Ryan Greenblatt – What happens once AI can automate AI research?

Ryan Greenblatt is chief scientist at Redwood Research, where he works on technical AI safety and security.

Dwarkesh frames the subject as the single most important open question in the world: recursive self-improvement. Once we build human-level intelligences, do they slingshot within a short period toward tens of billions of superintelligences, each individually more competent than the top human expert in every field? He has historically been sceptical. Greenblatt thinks it is plausible, and Dwarkesh wants the case for it.

Greenblatt starts with a structural observation: AI R&D is a task AIs are especially good at, because companies are trying very hard to make them good at it. It also happens to have unusually convenient properties for how AI development works right now. It is fairly verifiable, and you can iterate on it in a way that hill-climbs on various metrics.

Once you have AIs roughly matching top human experts at AI R&D, that can kick off a feedback loop: AIs do AI research, which produces smarter AIs, which feeds back in. The loop could be strong enough to produce a great deal of progress in a short window. His median expectation is something like four or five years of ordinary AI progress compressed into a single year. That requires overcoming an enormous amount of diminishing returns in research and effectively reproducing what a very large compute scale-out would have bought — which is a big claim.

It is worth keeping in mind how much progress even three years represents. A little over three years ago GPT-4 had just come out; now there is Mythos 5, and perhaps something better inside Anthropic. That is already an enormous jump. Five years of progress might mean something closer to the distance from GPT-3 to Mythos 5.

Dwarkesh breaks the argument into three parts he wants to evaluate separately. First: AI R&D is very verifiable. Second: automating AI R&D yields four or five years of progress in one year. Third: whatever comes out the far end of that at current pace is an AI you can drop into essentially any job imaginable.

Drop it into 1940s Texas politics and it outmanoeuvres Lyndon Johnson. Drop it into TSMC and it learns to do better process engineering. It is certainly a better video editor — his own editors are excellent, but the model is simply better than humans at whatever job it finds itself attempting.

So he wants to test each of these sub-arguments, the ones that together imply ASI arrives shortly after a benchmark Greenblatt expects around 2030.

Greenblatt's numbers: full automation of AI R&D around 2030 or 2031. The milestone of beating all humans on the job, a median around 2033. But if he sees AIs fully automating AI R&D, he expects the second milestone within about a year. As forecasting goes, the gap between medians is larger than the median gap between milestones.

Dwarkesh notes a running joke: whenever he asks anyone about timelines — Dario included — he asks how long until his video editors are automated. There is now a meme about his video editor editing the podcast while he listens back.

The reason is deliberate. It is easy to lose yourself in abstractions when discussing jobs you do not understand. A job he does understand, and understands why it is currently hard for LLMs, is a concrete test of what automation actually requires.

Greenblatt agrees the video editor milestone comes earlier than automating all human jobs including Texas politics from a cold start. He puts video editing closer to full automation of AI R&D — though it is very sensitive to how much effort anyone puts into genuinely understanding video.

On verifiability, Greenblatt points to a family of environments that directly train the model on AI R&D or something very close to it. For example, an environment where the model is training some AI on just eight H100s — a model the size of GPT-2 medium, running something like a NanoGPT medium run — and in RL it is tweaking and iterating on that.

That can be done across many tasks: training image classifiers, video generation models, image generation models, all kinds of ML training problems. You can RL on the task of training increasingly good models, and also on narrower things — here is a particular algorithmic direction, go implement it.

So there is a whole class of containerisable, verifiable, small-scale AI R&D tasks to train on aggressively. Companies are presumably already doing some of this, and it can simply be scaled up: more tasks, more environments, better models. The implicit claim is that this transfers to the extremely load-bearing parts of AI R&D — a claim he wants to set aside for a moment and return to.

Dwarkesh makes it concrete. Suppose there is a GPT-7.5, and the instruction is: we want you so good at AI R&D that you help us train GPT-9. So you build a set of environments.

One already exists in spirit: the descendant of Andrej Karpathy's nanoGPT speedrun, where the goal is to change everything about the model — optimiser, hyperparameters, architecture — to reach a fixed training loss as fast as possible. Others can be invented. Train a really good video-game-playing model, one that improves as it plays the same game repeatedly, and in doing so learn to help a model get better at online learning. It does not matter how: bizarre neuralese, a vector memory, better long-context handling. The point is to figure out how to do online learning research.

GPT-7.5 will already be a strong model, and it will keep getting better at coding the way models generally are. You can imagine a hundred more environments like this, all incentivising AI R&D capability — containerised versions of getting GPT-7.5 to develop GPT-2-sized models, and so on. Run GPT-7.5 through all of it, and you build GPT-8, which is now an excellent ML researcher with a great deal of intuition from the training.

Dwarkesh says his own intuition pump is the progress AI has made in mathematics. He does not know the object-level details of maths research, but the lesson seems clear: if a domain is verifiable enough to be put into a full verification loop, capability can arrive like a flood, and it can produce genuine new breakthroughs.

He wonders whether ML research shares a property of mathematical research: an overhang that comes from connecting disciplines. Nobody single-handedly knows enough about algebraic geometry and — he gropes for the word —

— topology and algebraic whatever, in order to construct a counterexample to a major conjecture.

Greenblatt's view is that ML is a shallower domain than maths, so there is less of the pattern where individual experts combine deep expertise from separate areas. There will be some of it, but not much.

ML also has properties that make it more favourable to AI training than mathematics. You get a better sense of whether you are succeeding, and you can see intermediate progress. In maths there is often no easy way to tell whether you are close. If the goal is to reach some training loss twice as fast, you can see when you are halfway there. And ML innovations tend to be additive, or multiplicative depending on how you count — you can keep stacking them, and usually they do not interfere with each other.

So in many ways AI R&D should behave like maths: you can train on chunks that are structurally similar to the problem you actually care about, in a very verifiable way, and that transfers. How well it transfers is an open question, but transfer for maths currently looks good. He expects transfer for AI R&D to look pretty good — not amazing.

Dwarkesh's concern is that even in mathematics we have not seen impressive new theory. We have seen many impressive, verifiable, specific results — find a counterexample to this conjecture — but not the invention of topology, or of group theory.

ML research contains both kinds of work, and the less verifiable kind — inventing new ways of thinking about a problem — is harder to induce. Take scaling laws. There is an end verification loop: you can train GPT-4 better if you have the 2020 idea of scaling laws. But there is a longer and far more compute-laden road to getting an AI to the point of thinking carefully about how to scale parameters and data, of asking what investigations would reveal this, of inventing a visualisation and an isoFLOP analysis. That is a much longer verification loop than getting nanoGPT's loss to go down.

Greenblatt's answer: AIs can already do the equivalent of a baby's first new theory. They can prove interesting conjectures by making connections and producing new understanding — here is a construction the AI found that is genuinely interesting, or a slightly different way of thinking about the problem. We do see that. The examples are simply not as impressive as founding group theory.

Founding group theory is among the greatest mathematical accomplishments of all time, and the AIs are not that good at maths yet. He sees a continuum between that and what exists now, and the AIs are still marching up it.

Second, ML is a very shallow domain relative to maths. In maths, much more of the work is finding a true deep abstraction, and if you really understand the thing — which is hard — you get somewhere. The ML equivalents of that, in his view, are pretty dumb. Scaling laws? Come on, they can be explained quickly. The deepest and most important concepts in maths do not have the property that you can understand the underlying thing and why it matters in a very short time.

Dwarkesh worries that by 2030 all the low-hanging fruit will be gone. Scaling laws will occupy the place in the history of ML that Descartes's coordinate grid and very basic mathematics occupy in the history of maths. To keep making progress in the 2030s, the work will have to be whatever is happening at the frontier of mathematics right now.

Greenblatt allows that could be right. His sense is that domains differ structurally in how much they depend on deep abstractions. Physics and maths sit far on the side of hard-to-come-up-with ideas; ML and most other domains are far more amenable to hill climbing. That is how he expects this to go.

Even in the regime where the AIs have to plough — it is 2030, the low-hanging research fruit is gone, and progress has to be made anyway — Greenblatt suspects much of the work will live on the side of building increasingly complicated infrastructure and having good intuition about roughly what the experiments look like. So he is less sympathetic to the idea that what the AIs will lack is some deep insight. He is more sympathetic to the idea that they need taste about in-the-weeds experiments, and intuition for which training approaches will work — the kind current researchers have.

Even where AI has produced a breakthrough, in retrospect the bottleneck often looks like getting the micro-details and the mungy intuition right. Take training AIs to be good at reasoning by doing RL on chain of thought. It looks like you could plausibly have done RL on chain of thought with GPT-3 and got interesting maths results, had you scaled it up and executed well.

But at the time there was low-hanging fruit elsewhere, and doing that training well is in the weeds on implementation, scaling, and hyperparameters. You might demonstrate the whole thing on Qwen 1B and get a sense that it will work — but people did not demonstrate it as early as they could have, because of all those mungy details and the intuition about exactly how to tune and set things up.

Dwarkesh's remaining scepticism is exactly this. If research breakthroughs are so amenable to intelligence, why has AI progress not historically been faster than it was? By the time RLVR actually worked, even though it could have been done with less compute, the field had to wait for oceans of compute — gigawatts of it — to become available before anyone ran that training.

There were a lot of AI researchers in 2022 trying to crack reasoning. Were they simply bottlenecked on the ability to write infrastructure code, or was something else going on?

Greenblatt's answer is a complicated mix. They would have gone faster if, as soon as they thought of an experiment, they could have run it without bugs — and without bugs mattering very much. Separately, being able to run many experiments at high compute papers over the ways your implementation is not quite right or your hyperparameters are off. Compute is genuinely helpful for AI research and covers over a lot. But that does not mean a massive increase in labour would not also be helpful, especially labour carrying some of the best intuitions in the field.

He also expects somewhat more transfer than Dwarkesh seems to imagine. He pictures these AIs as genuinely good scientists in general, reasonable at all of it. Interacting with them does not feel like interacting with a hyper-specialised savant. They are pretty good at everything in R&D, and then extremely good at some subdomains: superhuman at writing kernels, superhuman at anything with a short feedback loop, and competent enough elsewhere to match other people.

That is already visible. AIs today can competently match humans who are mediocre at ML research. The problem is that being mediocre at ML research is not very useful; what you want are people who are good at it. His sense is that AIs are improving at all of these things at once — their taste and intuition are getting better, and their taste and intuition are no longer complete garbage.

Dwarkesh wants this made concrete. Suppose we were back at the moment GPT-3 was developed. The claim would be that, with the compute available in 2022, automating AI R&D then would have put Mythos on the table by the end of that year.

That is the idea, yes.

Mythos took far more compute than existed then. But under this story, with only 2022-level compute, not only do all the breakthroughs happen — you also train Mythos at that compute level.

What that requires is discovering all the algorithmic progress since then, and actually more than all of it, because you have to make up the gap in raw compute. GPT-3 was trained on something like 1e23 FLOPs. Is it plausible that Mythos is four orders of magnitude beyond that?

Greenblatt checks: GPT-3 training compute is about 3e23. His sense is that Mythos is a little over three orders of magnitude higher. So the question becomes whether you can overcome a thousandfold compute gap while also being the model.

He offers a concrete calibration. GPT-3 was released in 2020, trained roughly six and a half to seven years ago — perhaps too far back to be the cleanest example, but useful anyway.

If you trained a model today with GPT-3-level compute, how good would it be? Based on how algorithmic progress works, he thinks you would get something as good as the best model of roughly three years ago. So today you could train a version of GPT-3 that is somewhat, moderately better than GPT-4. That lines up with the historical rate of algorithmic progress.

Which means that to get five years of AI progress, you probably need something like eight years of algorithmic progress — a great deal. But most AI progress, in his view, has come from some mix of algorithms and data, and both of those can keep improving enormously, letting you train strong models with less compute.

Dwarkesh is glad he raised it, because something large has happened since GPT-3. Compute has scaled and algorithms have improved, but the field has also built a decabillion-dollar data industry that systematically collected and codified expert human judgment across disciplines — as RL environments, as SFT traces — built by experts so the model could understand how coding is done, how complex infrastructure projects are built, how law is practised. How do AIs replicate the effect that expert human judgment currently plays?