John Schulman, Beren Millidge, Charlie O’Neill: AI researchers debate how close we are to recursive self-improvement

Dwarkesh opens with a diagnostic rather than a prediction. Suppose it is 2036 and there are no billions of superintelligences running around, no alien world. Setting aside war, politics and outright bans, what is the most likely technical reason? The question is designed to make each guest name the specific thing they think could break — not the general shape of progress, but the joint that fails.

Beren Millidge answers with a version of Moravec's paradox that never resolves. Each capability, once achieved, turns out to matter less than expected: the model solves the hard maths problems, wins at chess, and the world does not change. If that pattern held forever — if there were never the true spark of generalisation — the result would be a system that is superb at everything anyone puts into a benchmark or an environment, with a persistent sim-to-real gap blocking the rest.

He is careful to say he thinks this unlikely. Generalisation out of RL is already visible in practice. But if meta-learning turned out to be ridiculously hard to generalise, and continual learning were never solved, this is the scenario he would default to.

John Schulman agrees and gives the mechanism in operational terms. Humans still hold many advantages, and each new model closes some of them; what remains is the bottleneck — the places where judgment is worse, where the model cannot check itself well enough. There is a cycle that keeps repeating: a new model lands, people are blown away, they declare AGI, they use it for a month, and it starts to feel dumb. How many times that cycle repeats is genuinely hard to forecast.

The reason capability growth is not explosive right now is that research and engineering are themselves bottlenecked. A model that writes far more code than a person does not make the person a hundred times more productive. So the cycle may simply have more iterations left in it than anyone expects.

Charlie O'Neill reframes the question as a distance problem: how far is the current recipe — transformer plus RL — from the global optimum of a learner you could put on a chip? The popular picture is that once you have an agent even 0.1% better than every human at AI research, the ability to run hundreds of thousands of them in parallel, faster as chips improve, outweighs every other bottleneck. Fast takeoff follows.

Charlie's counter is the history of scaling laws. Moore's law looked like a clean straight line for decades, but that line was sustained by a long series of discrete discontinuities and innovations, each of which had to be discovered. LLMs repeated the pattern: pre-training scaling hit diminishing returns, RL was invented to get around it, and the resulting curve again looked like a straight line going up.

The worry follows directly. If another such discontinuity is required, it is not obvious that the current method — training LLMs with RL environments, even RL environments explicitly aimed at recursive self-improvement — can discover it. Absent that, the curve asymptotes.

Dwarkesh presses the obvious objection: would that discontinuity really be harder than anything since 2012?

Charlie concedes the framing but sharpens the distinction. There is a difference between a discontinuity that adds to the current paradigm — something beyond RL that still sits on the same line, which the models might connect — and one that requires going back and throwing out gradient descent and neural nets altogether. Scaling the current paradigm, no matter how many instances you run, does not necessarily buy you a discovery that is too far away.

Dwarkesh's own read is that the only hope for the bear case is deep learning being unable to reach a system that dominates human research and development, including the human capacity to invent new paradigms. It is conceivable that humans would never have found the next architecture either. But given the slope since 2012, even granting that most of it was bought with compute, it would be strange if the curve simply stopped short of dominating R&D.

Dwarkesh brings in Ryan Greenblatt's argument and his own intuition pump. Chess engines have gained Elo almost linearly since the 1980s, yet there is a sharp discontinuity where they cross the human range: before it, human experts always win; after it, they never do. Nothing in the linear Elo curve announces the moment when the meaning of the game changes.

That, he suggests, reconciles two facts that otherwise look contradictory: AI has had little macroeconomic impact so far, and AI is improving steadily. The impact is waiting on the crossing, not on the slope.

Beren agrees, and turns it into a constraint on the 2036 scenario. If we are already close to crossing the human Elo equivalent, then for the world of 2036 to look normal the asymptote has to arrive just before the crossing — a very specific and unlikely place to stop. He adds that the more probable route to that world is not technical at all: dramatic regulation.

Charlie separates two kinds of research. In the autoresearch style, the objective is specified cleanly and the work is optimisation against it: drive pre-training loss down, drive environment reward up, and improvement follows. That is what most people picture when they imagine AI doing research.

What a paradigm shift requires is the other kind — the open-ended science where nobody, human or AI, can specify the objective in advance. And that raises the specification problem in its hardest form: if you cannot say what you want, you have to be extremely careful about how you tell the system to want anything.

Dwarkesh asks Schulman for the wisdom of the trenches. In 2012 nobody would have guessed that a nanoGPT speed run was the thing to optimise; now that the paradigm exists, of course you would train AIs to be excellent at exactly that. The worry is the next inner loop — the one the AIs would not anticipate — while the outer loop of revenue, which should eventually be decisive, turns far too slowly to guide them.

Schulman's answer is a confession. In the early OpenAI days he had the intuition that minimising log loss would never produce intelligence, because the important bits account for such a small fraction of the loss that they would be swamped by noise; better objectives were needed to weight the interesting things. There were respectable arguments for this — humans cannot reproduce a photorealistic scene from memory, so surely a better objective was required. Then the naive objective simply worked.

That is the deeper point about the field: it runs on generalisation, and generalisation is nearly impossible to predict. Training on the task you care about reliably helps. The advances that matter most are usually forms of generalisation nobody had a right to expect — from a naive next-token objective to tasks that require deep understanding of the input, from skills so rare in the data that they are barely represented, from verifiable tasks to unverifiable ones.

Dwarkesh offers the strongest version of the fast-takeoff intuition that does not depend on new inputs. Before every seven-figure experiment, you could spend an equivalent amount of compute on AI labour: automated versions of these three spending a century thinking about the optimal experiment, running small ablations, building a century of theory reaching back before deep learning — and then another century afterwards analysing what happened and choosing the next run.

Schulman's response is that we are nowhere near the ceiling of research quality. If you think hard enough you can often predict the result beforehand; there is usually some clever small-scale experiment that builds the theory which then generalises to the large one. He expects a future where AI spends compute on analysis and theory-building comparable to what is spent on the experiments themselves.

Charlie accepts this precisely where the objective is well specified, and offers the information-theoretic version: thinking can only update a posterior on the bits already received; it cannot manufacture new bits. But given a clean objective and data sitting around, the speed-up should be large.

His worked example is the Kaplan scaling laws. An AI looking at them would have noticed that the intermediate checkpoints were not annealed, and that the conclusion was therefore wrong — a catch that would have saved a year or two. The same holds for muP and the scaling of learning rate with model size, including the role of width. Back out those observations and a great deal of low-hanging fruit is picked before it is spent. His estimate is a tenfold speed-up on the current objective.