Book excerpt: What happens when machines become smarter than us? Forget images of Terminators and Cylons: artificial intelligences (AIs) will achieve power through their intelligence, not brute strength. Just as humans shape the world in ways beyond the understanding of chimpanzees, AIs will shape our world, transforming it--whether slowly or blindingly fast--into whatever they are programmed to prefer. The future could be filled with joy, art, compassion, and beings living worthwhile and wonderful lives--but only if we're able to precisely define what a "good" world is, and skilled enough to describe it perfectly to a computer program. Philosophers have tried for thousands of years to define the ideal world, with little to show for it. The prospect of artificial intelligence gives this project a new urgency. Our values are fragile: miss a single piece of the puzzle, and the whole system collapses into a world empty of worth. And then comes the daunting task of encoding the entire system of human values for an AI: explaining them to a mind that is alien to us, defining every ambiguous term, clarifying every edge case. AIs, like computers, will do what we say--which is not necessarily what we mean. Though an understanding of the problem is only beginning to spread, researchers from fields ranging from philosophy to computer science to economics are working together to conceive and test new approaches. The problem of AI safety isn't easy, but it is solvable. Are we up to the challenge?