Richard Sutton – Father of RL thinks LLMs are a dead end

Dwarkesh Patel opens by doing something unusual for an interviewer: he states his guest's position in its strongest form before the conversation begins. Richard Sutton, he notes, is a founder of reinforcement learning, a 2024 Turing Award winner, and the author of “The Bitter Lesson,” and he believes large language models are a dead end. Patel's reconstruction is deliberately charitable: models cannot learn on the job, so no amount of scaling will substitute for a new architecture built around continual learning, and once such an agent exists there will be no separate training phase at all — it will simply learn the way animals do. The framing matters because it removes the cheapest objection. Patel is not accusing Sutton of underestimating compute; he is conceding that compute is not the disputed variable.

The disagreement is therefore structural rather than quantitative. Patel says outright that he tried to argue the opposite case — that LLMs might serve as the substrate on which experiential learning happens — and that “Some sparks flew.” That single sentence sets the tone for the whole episode: this is a disagreement between two people who agree on the destination (agents that learn continually from experience) and disagree about whether the current paradigm is a stepping stone or a detour.

Sutton's first move is to deny that the two traditions are commensurable at all. They can “easily get separated and lose the ability to talk to each other,” he says, because the field is prone to bandwagons and has lost track of fundamentals. His self-description is pointed: “I consider reinforcement learning to be basic AI.” The claim is not merely that RL is older or more principled, but that LLMs are a peripheral application rather than the core subject matter.

Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do.

That sentence is the thesis of the first hour. On Sutton's account, intelligence is defined by the problem it solves — understanding a world well enough to act in it — and imitation of human text solves a different problem, namely predicting what a human would have written. The two activities can produce similar outputs while resting on entirely different machinery, which is why Sutton thinks benchmark success is weak evidence about what a system has actually learned.

Patel presses the most natural counterargument: emulating trillions of tokens of internet text should require a world model, and these models appear to possess unusually robust ones — the best AI has produced. Sutton's reply is a flat rejection: “I would disagree with most of the things you just said.” His reasoning turns on a distinction between modeling a world and modeling a modeler. A system trained to predict human text is, in his words, mimicking things that already contain a model of the world. The competence is inherited rather than constructed.

They have the ability to predict what a person would say. They don’t have the ability to predict what will happen.

This is the load-bearing claim, and it is testable in principle. If a model genuinely represented consequences, it should be surprised when the world deviates from its expectation, and that surprise should alter its future behavior. Sutton's Turing-derived formulation of the alternative is that we want a machine that learns from what actually happens in its life — you act, you observe, you update. The LLM, by contrast, learns from paired situations and human responses, with the implicit instruction to reproduce the response. He summarizes the training signal as: “here’s a situation, and here’s what a person did.”

Patel then offers the strongest version of the opposing case, and it is worth stating precisely because Sutton's rejection of it is the hinge of the interview. The argument is that imitation provides a good prior over reasonable approaches, and that this prior is exactly what makes experiential learning feasible: it lets the agent be right some of the time, and being right some of the time is what generates the reward signal you need to improve. On this view imitation is not an alternative to experience; it is the initialization that makes experience tractable.

Sutton's answer is that a prior presupposes a truth to be prior about, and that the LLM framework contains no such truth. “What is actual knowledge?” he asks, and then answers his own question by pointing at the absence of a goal. Without a goal there is no better or worse, and without better or worse there is no fact of the matter about what should be said. His illustration is concrete: you say something, and nothing tells you whether it was the right thing to say, because the framework never defined rightness. In RL the definition exists — the right action is the one that yields reward — which is precisely what allows human-supplied hints to be checked rather than merely absorbed.

Patel's strongest empirical objection follows immediately, and it is the one Sutton answers least convincingly. If you ask a model what a user will say next, it will produce an answer. Sutton's response is that producing an answer to that question is not the same as holding a prediction, because the model will not be surprised when the actual response differs, and will not change as a result. Patel then raises chain-of-thought: a model solving a math problem will announce an approach, notice mid-stream that the approach is conceptually wrong, and restart. That is real in-context flexibility. Sutton concedes nothing, replying that the model is not surprised by what happens next and makes no changes based on it.

The exchange exposes a genuine ambiguity rather than a simple error on either side. Patel asks whether next-token prediction is not literally prediction-plus-updating-on-surprise, and Sutton's reply draws the line sharply: the next token is what the model should say, not what the world will return. That distinction is defensible, but it also reveals that Sutton is using “prediction” in a specific sense — prediction of exogenous consequences — while Patel is using it to cover prediction of one's own conversational continuation. Much of the apparent deadlock in this section comes from that unstated equivocation, and neither participant names it.

Sutton then escalates from mechanism to definition. Intelligence, he argues, is the capacity to achieve goals, citing John McCarthy's formulation that intelligence is the computational part of the ability to achieve goals. A system without goals is merely a behaving system. He asks Patel directly whether LLMs have goals, and Patel says yes: next-token prediction. Sutton's rejection is memorable and, on its own terms, correct — predicting tokens does not change the world, so it cannot be a goal in the sense that matters. Patel immediately concedes the point: it is not a goal about the external world.

You can’t look at a system and say it has a goal if it’s just sitting there predicting and being happy with itself that it’s predicting accurately.

Patel then raises what he considers the decisive counterexample: models that win gold at the International Mathematical Olympiad appear to have the goal of getting math problems right, and that goal was installed through RL. Why should it not extend to other domains? Sutton's answer is that mathematics is not a domain in the relevant sense. Deriving consequences of assumptions is computational and resembles planning, whereas modeling the empirical world requires learning consequences that cannot be derived. In math, the goal of finding a proof can be given; in the physical world, the consequences have to be discovered.

This is the most contestable step in Sutton's argument, and it is worth flagging what he does not address. Patel's example is not that models understand physics, but that a general-purpose optimization procedure was able to install a goal-directed competence in a domain where verification is cheap. Sutton's response narrows the claim to mathematics without explaining why the same optimization procedure should fail in domains where verification is expensive but still available — a gap Patel does not exploit, but which a reader should notice.

Patel then raises the essay that made Sutton famous, and turns it into an objection. “The Bitter Lesson” is routinely cited as justification for scaling LLMs, on the grounds that this is the one method we have found for pouring enormous compute into learning about the world. On that reading, Sutton's own essay argues for the LLM program. Sutton's response is more concessive than his earlier positions: whether LLMs are an instance of the bitter lesson is “an interesting question,” partly sociological, partly industrial.

Will they reach the limits of the data and be superseded by things that can get more data just from experience rather than from people?