Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment

Dwarkesh Patel introduces Carl Shulman as the source of many ideas about the intelligence explosion that his earlier guests credited to him. Shulman is an advisor to the Open Philanthropy project and a research associate at the Future of Humanity Institute at Oxford, keeps a low public profile, and this is his second podcast ever. Asked for the big picture of the feedback loops that start once AI approaches human-level intelligence, Shulman says the method he most wants to highlight, and thinks is underappreciated, is the input-output curve.

Today humans develop new chips and software and run larger training runs, and keeping Moore’s law going takes a great deal of work. Each doubling of computing performance is harder than the last, and eventually physical limits make further doubling impossible. The question Shulman poses is how much harder. Using the paper “Are Ideas Getting Harder to Find?”, he describes a period in which computing productivity rose a million fold, meaning a million times the operations per second per dollar. Over that period the investment and labor needed to keep advancing rose 18 fold. He says he did an early version of this analysis ten years earlier at MIRI, using mainly Intel and large semiconductor fabricator data.

Some read that as diminishing returns that will eventually end progress. Shulman’s reply is that if AI does the work, a doubling of computing performance translates directly into a doubling or better of the effective labor supply. If that million-fold increase had been used to run AIs replacing human scientists and engineers, the 18-fold rise in labor demand would be trivial. The data set gives more than four doublings of compute per doubling of labor requirement. So one doubling is used up meeting the rising difficulty, and the rest can accelerate the process: the first doubling takes some number of months, the next a smaller fraction of that, and so on. He adds one condition: this holds only so far as the outputs (compute for AI) can serve as the necessary inputs, because any other required input eventually becomes a bottleneck. Patel recalls the “bloom paper” (as transcribed) as showing a 35% rise in transistor density alongside a 7% yearly rise in researchers required, and Shulman confirms it is roughly right: four to five doublings of compute per doubling of labor inputs.

Asked why compute is a good proxy for the number of AI researchers, Shulman explains that hardware was his first example only because good historical data exists. For hardware the idea is simple: if an AI worker can substitute for a human, twice the computers lets you run two instances doing two jobs, managing two machines or working on two design problems. Gains can exceed that, because extra compute can also train larger AIs rather than just run more copies. Software differs from hardware because it can be copied freely, so spending on hardware at a given price is what matters. He identifies the recent off-trend change driving AI as the tremendously larger money spent on hardware for training big models.

The work of designing hardware and software is currently done by people. Computer time is not the primary cost for NVIDIA designing chips, TSMC producing them, or ASML making lithography equipment. He gives rough headcounts: low tens of thousands at NVIDIA designing AI-specialized GPUs, more than 70,000 at TSMC, and a thousand people at DeepMind according to its public filings, with OpenAI at a few hundred and Anthropic fewer. Adding Facebook AI research, Google Brain and other R&D gives thousands or tens of thousands working on AI research. Shulman says even in AI software research the salaries at DeepMind used to exceed experiment compute, though recent spending has shifted heavily toward compute.

He says the relevant people are those developing new methods rather than narrow applications; inventing the transformer counts, while cleaning one business’s data set probably does not. Across hardware and software the pattern is consistent: capabilities double on a shorter time scale than the people required. For hardware the historical ratio was four or five doublings of compute efficiency per doubling of human inputs. He thinks it is somewhat lower now near the end of Moore’s law, though not as much lower as one might expect because input growth has slowed too.

On software, Shulman cites work by Tamay Besiroglu and collaborators, “Are models getting harder to find?”, which applies the same analysis. From papers, citations and company employment, the doubling time of the workers driving software advances is several years, while the doubling of effective compute from algorithmic progress is faster. He then gives Epoch’s headline figures, noting Epoch has received Open Philanthropy grants: hardware efficiency doubling in about two years, possibly a bit better with AI-specific specialization; budget growth with a doubling time of roughly six months in recent years, which is tremendous relative to historical rates; and algorithmic progress, measured mainly on ImageNet-type data sets, with a doubling time under one year. Combined, he says the growth of effective compute for training big AIs is drastic.

Patel reports an estimate of around 50 million dollars to train GPT-4, and asks whether effective compute rises through three routes: more spending, better models, and cheaper chips. Shulman answers that software progress, defined by how much less compute achieves a previous benchmark, currently looks like two or three doublings of effective compute. He cautions against identifying this purely with software, since the two interact: abundant compute allows more experiments to find better algorithms, and running a bigger model on more GPUs can look like a software improvement even though it would not have been possible without a large training run.

Asked how the loop closes, Shulman says AI can help with hardware design. NVIDIA is a fab-less chip design company that sends instruction files to TSMC. Automating the work of its more than 10,000 people, with the equivalent of a million people doing it, would quickly capture the gains available on TSMC’s existing nodes. He nevertheless regards this as less important for the intelligence explosion, because a chip design improvement only applies to chips made afterward, whereas an AI software improvement can immediately apply to every GPU already owned. He therefore places the leading edge of AI automating its own inputs on the software side.

Patel presses on timing: when would AI contribute enough to count as additional researchers on AI progress and software? He notes that some claim programmers at OpenAI already use Copilot, so some feedback loop exists, but says he is somewhat skeptical of that as a mechanism.

Shulman says the quantitative magnitude of AI’s help is central. Many products raise productivity very slightly; he suggests Xerox’s fax machines might add about 0.1% to office productivity, which cannot produce explosive growth. The thing to watch is when AI’s contributions become as large as human contributions, boosting effective productivity by 50 or 100%. If that moves the doubling time of effective compute from software innovations (he names the transformer, Chinchilla scaling and flash attention) from eight months to four, the next application of the AI boosts productivity further, perhaps to 200%. AI need not automate everything in AI research. It can automate a bunch of things and do them in extreme profusion because they are cheap, with the effect of having twice as many people of the same skill distribution as the roughly 10,000 now finding these innovations.

He adds that this is demanding but far below a system with no weakness relative to any human. A system equal to humans in every respect and also holding AI advantages is well past the threshold. Existing fabs make tens of millions of advanced GPUs a year. If those ran software as sample-efficient as humans, they could work the 168-hour week, four times as long, with far more education than a PhD’s roughly 20 years, since models are trained on the internet, every published book and all of GitHub. He argues that tens of millions of GPUs each doing the work of 40 or more workers resembles going from a workforce of tens of thousands to hundreds of millions, and that human-level AI is therefore deep into an intelligence explosion, which must begin with something weaker.

Patel objects that being an OpenAI researcher is more than completing the hello world prompt that Copilot handles: it means choosing a new idea, working out how to approach it, and managing collaborators. He asks at what point the loop starts, when AI is the equivalent of a researcher or close to it rather than a 0.5% productivity tool.

Shulman answers with illustrations of how cheap AIs offset their weaknesses. Dumber AIs can be deployed thousands of times to equal one human worker, for example in voting schemes where an LLM generates many responses and takes a majority vote. They can use AlphaGo-style search that spends more compute to compensate for a model’s weaknesses. And they can do things impractical for humans because of the number of steps, such as designing synthetic training data. He compares this with schools, which teach in a sensible order and test the skill being taught, whereas models currently just take more data from the internet, a source he says is nearing its end. AlphaGo started from human Go data and then used reinforcement learning and Monte-Carlo tree search, while AlphaZero started from scratch, generating its own data through self-play, at a quality beyond any human in the data set and with a built-in curriculum of equally skilled opponents.