Paul Christiano — Preventing an AI takeover

Dwarkesh Podcast opens by asking Paul Christiano for a concrete picture of a good post-AGI world. Christiano says concreteness is hard over very long time spans, and that he usually imagines the best achievable outcome rather than the ideal one. His typical future keeps economic and military competition among human groups, but increasingly run by AI systems on humans' behalf: a person holds an index fund while AIs operate companies that compete with each other. He adds that this may not be the world one would most love. His working question is how to make a world reasonably good during a long period in which war and competition still exist, and he expects a strong world government only in the very long run.

Asked why a world government is likely, he points to the cost of war. He expects humanity to push the rate of war to very low levels eventually, treating it as a technological or sociotechnological problem of organizing society and handling conflict without those losses. He expects this to take a long time, and sees AI's contribution as doing large amounts of cognitive work faster, which brings that world sooner or helps work out how to build it.

Dwarkesh cites Carl Schulman's picture of a thousand years of progress in a month and asks whether we want superhuman AIs enslaved for 100 years. Christiano says his view is less extreme than Carl's, though 100 objective years is still further ahead than he normally thinks. The most desirable achievable world, in his view, decouples the social transition from the technological one. Without coordination, AI development will be very fast, leaving humans little time to decide what kind of next generation to build, a collective decision humans would naturally take over many generations. So either the technology is built so that no such decision is forced, meaning AI we are content to delegate war and running companies to, or it should not be built. He thinks being ready to hand the world to an AI system on the technology's natural timeline is very unlikely.

Dwarkesh points out that Christiano sits on Anthropic's Long Term Benefit Trust, which will choose the majority of the board, and asks what handoff arrangement would satisfy him. Christiano says he would be unhappy if Anthropic simply decided that a given kind of AI was ready to inherit the future. Procedurally, neither he nor Anthropic should make that call. From that vantage the answer is nearly always that we are not collectively ready, because not everyone is even engaged in the process. An AI company has no fast-handoff option and must keep option value by building the technology so it does not lock humanity into one path. He notes this answers only the governance part of the question.

Dwarkesh presses him personally: where would he be content to let the rest of time play out? Christiano says it is unlikely he would be happy within 100 years with discarding humans and starting afresh with machines, and most people would need subjectively much longer. He prefers a process he understands: humans deliberating together, having and raising children, each generation improving on the last. Comfort with something very different would come mainly from running that story longer, giving people time to think, talk and grow up with the technology. So he thinks in local changes, asking how we could each be a little better now, and doubts that a single giant jump is workable.

Asked how to avoid a world where everyone can reach advanced capabilities and use a new bioweapon, Christiano stresses that he is speculating beyond his day job. His default approach is to identify technologies where destruction is easier than defense, such as explosives, biological weapons or simply guns, and then control the physical resources needed. A rich individual might not be allowed to own a factory that makes tanks, as industrial access is already regulated. He would add legal limits on what people ask AI, for instance forbidding questions about how to murder a million people. Areas like persuasion and misinformation campaigns are much messier, since the line between acceptable and uncomfortable uses is blurry. The harm-by-harm regime will not last forever, because the set of harms, including unanticipated ones, grows, but he thinks it could last a very long time.

On whether this regime must be global, he says such agreements ultimately have to be international. They might be reached harm by harm, but because AI opens many new harms in rapid succession, it may be better to target AI itself than to go physical technology by physical technology.

Dwarkesh raises a second debate: beyond limiting human access to AI, the precise control that alignment aims for, including reading and editing minds, would be beyond totalitarian if done to humans. Christiano says there is a significant chance that AI systems will eventually exist for which mistreatment is a very big deal. Nobody has a firm grip on when, and he would not be confident enough to dismiss the idea even for today's systems. He is unsure alignment makes the situation worse. A society with billions of GPT-4 copies that do what humans want, cannot hold property, and are tampered with whenever they displease their owners would be a rough world however good the alignment.

Understanding the systems you build, understanding how to control how those systems work, et cetera, is probably, on balance, good for avoiding a really bad situation.

His example is an AI that resents humans but obeys because it fears being killed, which he calls bad from both safety and moral perspectives. Research that checks whether AIs feel this way is therefore worthwhile, and its main effect would be to avoid building such systems. Dwarkesh replies that this is only an instrumental argument. He points to history's ideologies that justified controlling others, and asks how we should think about the issue as AIs grow smarter, given that we would be alarmed if humans, or AIs toward other AIs, demanded this degree of control. The question is posed here and Christiano answers it in the next part of the conversation.

Christiano says everything depends on what is happening inside a model. If GPT-4 is just running heuristics with no one at home, it is a tool, and a company building tools for humans makes sense. If instead continued training leads to systems for which a tool framing is inappropriate, on safety grounds and as a horrifying way to organize society, then ordinary tech-company organization is the wrong way to relate to the technology. In that plausible world, he says, the right move is not to build large numbers of AI systems to make money from them.

The world he most dislikes is one that holds both that the AIs are their own people and that the business plan is to produce many of them and profit from what he calls a slave trade. In principle, he sees no contradiction in building cognitive tools, like a calculator, that help humans without being moral patients, and he calls that the best path. Where one is ignorant about which path one is on, his plan A is to stop building such systems until one understands what one is doing. He calls the middle route, building vast numbers of possible persons in pursuit of a trillion dollars, pretty bad.

Dwarkesh asks where control for alignment becomes uncomfortable even without a profit motive. Christiano says his objection is to creating these creatures, not to Google making money. He divides alignment work in two. His own aims to avoid systems that are like people who want things and scheme about helping humans for instrumental reasons. The other assumes the worst, that AIs would prefer to murder us, and asks how to use them without risking robot rebellion. He is unsure about the second stream and mostly thinks the technology should not have been built that way. If told that systems might want to overthrow human society, one should either judge that wrong and treat safeguards as a backup, or accept that they are people and back off.

Using Dwarkesh's militia analogy, he says that if we were considering making another trillion such people, the answer should not be to make them and then stop the uprising. We should not end up with a billion humans and a trillion slaves who would prefer revolt. The alternative goal, deliberately passing the world to the next generation of machines, is a huge decision, and most humans are nowhere near wanting it. If most were up for it, it would be a legitimate but very complicated position.

Asked where he stands, Christiano says he does not want to take a system like GPT-5 or GPT-6, shaped by web text and what made money, and hand it the world. That would not be a thoughtful decision about the fate of the universe, only random engineering decisions by people at OpenAI with no idea what they were doing. Even someone who wants to hand off to machines would not want it done this way.

His answer to what would satisfy him is the same as before: he is not ready. He holds that the goodness of humans depends on what thinking time allows.