The data black hole at the center of AI

Patel opens with a definition that reorganizes the usual story about AI progress. Intelligence, on this definition, is sample efficiency: how much data you need to see in a domain before you can operate fluently in it. The claim that follows is uncomfortable for the standard narrative — it is not clear that training sample efficiency has improved much in recent years; what has happened instead is that the data distribution has been dramatically widened and improved.

The mechanism he offers is that AIs have improved mainly by adding more and better data, and by scaling the compute needed to produce that data in the first place. Reinforcement learning is the clearest instance, and his framing of it is deliberately deflationary: RL is a kind of synthetic data generation. You spend compute against a verifier to find the good data, then train the model to predict those correct rollouts, much as you would train it to predict the next word in internet text.

There is a precondition in that account that carries most of the essay's weight: for the process to work, the model must already assign some probability to the correct solution. That is why, in Patel's phrase, you need mind-stretching amounts of human expert trajectories in every field and skill you want competence in. The model cannot be taught to find an answer it cannot yet imagine.