Holden Karnofsky — History's most important century

Holden Karnofsky co-founded GiveWell and later worked with Cari Tuna and Dustin Moskovitz on the project that became Open Philanthropy. He describes his specialty as hunting for ideas that are underappreciated and tremendously important, where he expects an “outsized return on investment.” He says the most important century idea is a conclusion many people in the Effective Altruist community helped him reach, not his own invention.

The argument starts from economic history. Growth has accelerated, and a simple loop of people producing ideas, ideas producing resources, and resources supporting more people predicts ever faster growth. Extrapolated naively, it gives an infinite growth rate this century. Karnofsky does not expect that by default, because one link broke a couple of hundred years ago: people got richer rather than more numerous. AI systems that could do everything humans do to advance science and technology would fill the “more ideas” link and restart the loop.

He judges it more than 50-50 that this century sees AI able to do all the key tasks humans perform to advance science and technology. The result would be thousands of years of change in a much shorter period. If progress that fast reaches the limits of science and technology, the outcome could be a very stable civilization that expands well beyond Earth and looks post-human in many respects, which is why he calls this our last chance to shape it.

The most important century hypothesis in a nutshell is that if we develop AI that can do all the things humans do to advance science and technology, we could very quickly reach a very futuristic world that’s very different from today’s.

Dwarkesh Patel presses on a 2014 conversation in which Karnofsky said he could see a path to doing good in Africa but none he felt good about on far-future questions. Karnofsky ties the two concerns together through impact per dollar. If advanced AI is near, the future could be very good or very bad: sloppily designed systems with goals of their own or powerful technology in ill-meaning hands, versus a world that eliminates many forms of material scarcity.

He gives two reasons for the shift. The first, which he calls more than half the answer, is persistence: he has spent far longer on the problem and knows more about research and actions that could reduce the risk of poorly designed AI running the world. The second is the deep learning revolution from about 2014, in which computationally intensive but fundamentally simple systems advanced on many unrelated tasks. He says it is not crazy to think this path could reach AI that automates roughly everything humans do to advance science and technology, though he does not call that the leading possibility.

On the worry that no one has a record of predicting decades ahead, he concedes that forecasting is hard and that his 2014 self leaned too far into that view. A recent post on futurists’ track record found it merely “fine,” and nobody has shown real precision. He still thinks the attempt is worth making if one stays aware that everything is less reliable than it feels. His job, he says, is to find ten things, expect nine to fail embarrassingly, and hope one succeeds enough to cover the rest.

Asked whether the thesis is that a weird era makes transformation unsurprising or that transformation could happen this century, Karnofsky says the central claim is the latter. The long section on how strange our era is answers an objection. In 2014 his main obstacle was not any specific claim about AI, economic models or alignment research, but that the whole idea sounded crazy and suspicious.