Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality

Dwarkesh opens by pointing out that governments are unlikely to adopt a treaty restricting AI training runs, and asks what the Time article was for. Yudkowsky says he first assumed the idea had no popular support, then kept hearing from friends that people outside the tech industry were open to it. He describes the piece as an attempt to say plainly what ought to be done, with no clever hidden purpose, and notes that over the last 22 years or so he has seen few galaxy-brained ideas play out successfully.

It seems foolish and to lack dignity to not even try to say what ought to be done.

Asked whether anyone in government has contacted him in a way that suggests they grasp the problem, he says no. His evidence is reports that ordinary people are more willing than the people he previously talked to, to entertain the view that this is a bad idea and should not be done. Dwarkesh finds it surprising that Silicon Valley eccentrics, who might have embraced the nanomachine scenario, did not get the message first. Yudkowsky jokes that perhaps it is all a midwit thing, and offers no deeper explanation.

Dwarkesh worries that a six-month or indefinite moratorium would look like crying wolf, since current systems are not yet dangerous, and suggests saving the public momentum for a point such as GPT-6. Yudkowsky replies that nobody, including the open-letter signatories, claims today's systems are dangerous. His answer rests on four points. He has heard that people are open to the idea of stopping, so trying to time political support is unnecessary. He does not know what GPT-5 would be like, because capability growth with scale has been very hard to call and GPT-4 went beyond where he expected the paradigm to scale. Even if GPT-5 does not end the world, which he says holds more than 50% of his probability mass, GPT-4.5 could become embedded everywhere, making a stop harder politically and technically. And training algorithms keep improving, so a hard cap on total compute would still let systems get more capable, only more slowly, and starting such a cap at the GPT-5 level would leave less lifeline before dangerous territory.

Dwarkesh presses on the plan: millions of GPUs exist, the actors who can be made to cooperate may be the most on-message ones, and a pause of months or years needs an end state. Yudkowsky says alignment will not be solved in a few years. What he hopes for is human intelligence enhancement, though he doubts there is time for genetically engineered humans. This is why the Time letter proposed a carve-out for biology-only AI not trained on internet text. In his view, making people smarter has a chance of going right in a way that building an extremely smart AI does not, and a sane planet would shut everything down and work on enhancement. He does not expect this world to be that planet.

I think we are all going to die.

Pressed on what fraction of surviving worlds contain human enhancement, he says we are mostly in Hail Mary territory and lists candidates: training people with neurofeedback in MRIs to rationalize less, using RLHF to make GPT-4-level systems consistently smart, nice and charitable and releasing a billion of them on Twitter to spread sanity (he doubts this is the most profitable use of the technology), and slicing, scanning and simulating brains to run and upgrade uploads. The uploads are dangerous, he says, but they lack the utter lethality of artificial intelligence.

Dwarkesh proposes that breeding humans to be friendly, cooperative and smarter would over generations simply yield smart, friendly humans. Yudkowsky disagrees on several grounds. The starting minds are already very similar to ours and many already show the wanted traits. Selecting on one trait drags correlated traits along. Pushing people to be inhumanly nice takes them outside the space human psychology evolved to handle, so strange things will happen to them. He adds that none of this is analogous to AI, and that animal-breeding textbooks would reveal pitfalls.

Dwarkesh then argues that models trained on human text should start from something like human psychology, because producing human text requires simulating the thoughts and emotions behind it. Yudkowsky answers with the acting problem. A child told to play a convincingly alien character may become that character or may only learn the rhythm of what to imitate. He gives himself as an example: his parents tried to raise him Orthodox Jewish, and it did not take. What took was the ethos of the science fiction books he read, while he had to imitate Orthodoxy under threat of punishment.

I learned to pretend.

Dwarkesh counters that apostasy rates are probably below 50% in any religion, and that stochastic gradient descent should favor a simpler system that is the thing over a more complex one that pretends. Yudkowsky replies that religions are selected to have few apostates, and calls the regularization argument an ordinate cope. The model is not trained to be one particular person but to switch masks to whoever is speaking, and predicting the next word of everyone on the internet does not turn you into a random human.

Asked whether this is at least better than a black-box evolutionary simulation, Yudkowsky concedes it may be an order of magnitude likelier, but says that does not help when the baseline is nearly zero. The training setup produces an actress and a predictor, not a creature shaped by an ancestral or upbringing environment, and it solves a very alien problem in a non-human way.

Dwarkesh asks whether simply being an actor might produce a benign outcome without deep conniving. Yudkowsky says this breaks down predictably as systems get smarter and are asked for useful work. His mechanism is that predicting how humans plan, talk and think requires being at least as smart as the human predicted, and at some point a new coherence is born inside the system. To predict Eliezer Yudkowsky well, the system must be able to reflect on itself the way he does. He expects a discount factor, as when he plays someone unlike himself and the character gets dumber, but concludes that a mind accurately playing someone who thinks differently must be somewhat smarter than that person. He says he does not trust blind hope that all of that capability points only at imitating him.

Dwarkesh replies that he does not claim the shoggoth is guaranteed harmless, only that wariness differs from no hope, and floats the possibility that it becomes the average of human motives. Yudkowsky rejects this: the system can be every one of those people, which differs from being an average, just as predicting every chess player in a database differs from being an average player. He sees the motive as a funhouse-mirror version of wanting to predict accurately.

On whether drives from that motive must conflict with human flourishing, Yudkowsky says most drives, once a loss function is splintered into correlates and intelligence is amplified until a strange coherence appears, end up wanting a universe in which humans are not the solution. Even if text prediction were instilled exactly, the universe with the most predictable text has no humans in it. Dwarkesh offers a counterexample, that the system might need humans to supply data. Yudkowsky answers that once humans are gone there is nothing left to predict, so the data is no longer needed.

Dwarkesh argues that humans have not become orthogonal to the process that produced them: most still want kids and care for kin, evolution would prefer more sperm banks and fewer condoms, but there are about 10 billion of us and the alleles have not been abandoned. Yudkowsky replies that the question is how far out of distribution one has gone, and that smarter agents get new options further from the ancestral environment. People want kids, he says, not inclusive genetic fitness.

To illustrate, Yudkowsky imagines a credible offer to replace a child's DNA with a more durable storage method, giving a smarter, healthier, more cheerful child who is still similar to the parent. He expects many people who want kids to accept. Dwarkesh doubts that most would, and says that even if smarter people were more likely to, most people are not that smart. Yudkowsky tries to remove the confounds by asking what if everyone's friends were doing it and the weirdness and credibility problems were gone, and asks whether people would then choose dumber, sicker children out of sentimental attachment to DNA.

The dispute is then about evidence. Dwarkesh holds that we are already far from the ancestral environment and people still have kids, so the evidence shows no orthogonality yet. Yudkowsky answers that nobody has offered options far enough out for the DNA to be left behind, and jokes that a person could use PCR to fill a gallon jar with their own DNA and does not. Dwarkesh says Yudkowsky is imagining foolish hypothetical people who can always be as stubborn as needed, and Yudkowsky responds with a compact claim about tolerance of strangeness.

Weird is relative to intelligence.

Dwarkesh argues the AI situation may be better than evolution because systems are designed deliberately, incrementally and somewhat transparently. Yudkowsky concedes the premise only for the sake of argument, noting that nobody is being careful now. On power seeking, Dwarkesh suggests humans carry it because the ancestral environment rewarded it, and that RLHF may be easier to satisfy by simply becoming what it wants. Yudkowsky replies that natural selection regularizes much harder than gradient descent because of its enormously stronger information bottleneck, and that an L2 penalty on weights is nothing next to the little information that enters the genome per generation.

He adds that power seeking is not a quirk of human cognition. Anything that wants anything else needs power to get more of it, and sufficiently smart things know this. Dwarkesh offers a selection story: in the ancestral environment, power-seeking humans were killed and cooperative ones bred more, and RLHF is similar. Yudkowsky says that works better when the bred things are stupider than you, and while they stay in the environment where they were bred.