Joseph Carlsmith - Utopia, AI, & Infinite Ethics

Joseph Carlsmith introduces himself as a senior research analyst at Open Philanthropy who works on existential risk from artificial intelligence: what will happen with AI, how to make sure it goes well, and in particular how to make advanced AI systems safe. Host Dwarkesh Patel invited him because of his blog, which the transcript renders as Hands in Cities (as transcribed). The blog covers philosophy, morality, ethics and the future. Carlsmith calls it a side project, and says it grows out of his earlier background in academic philosophy, before he moved into AI and Open Philanthropy.

His current work is on AI timelines and takeoff speeds, meaning how quickly the world moves from pretty impressive AI systems to radically transformative ones. He uses that work to sharpen his estimate of the probability that everything goes terribly wrong. Asked what a change in that number would imply, he gives two effects. The sooner something is coming, the more you must plan for it arriving soon, cut corners, and count less on having more time. And the higher the probability of catastrophe, the easier it is for this to become the most important priority. He thinks there is a range of probabilities where the exact figure may not matter much, but the gap between 1% and 10% is substantive, as is the gap between 10% and 90%, and he knows people at every one of those levels.

The host opens the philosophical discussion by asking about long-termism. If someone dedicates a career to building a future, that future might strike people alive now as strange, disturbing or even disgusting. He gives an analogy: from a utilitarian view the present is much better than the past, yet a person from the past might find the present disturbing, for example the isolation of a modern city or the free, cheap information available on a phone.

Carlsmith answers in two steps. First, he guesses that most people from history, brought to the present, would fairly quickly come to prefer it, even with some future shock and some alienation. Second, the gap between historical humans and us is much smaller, in time and in other respects, than the gap he envisions between us and future humans living in a radically better situation. So he expects more distance and possibly more alienation on arrival. His own standard is not first impressions. The best futures, he says, are ones you would judge really good if you fully understood and experienced them, which may require extensive engagement and even changes to your capacity to understand and experience. For him the question is how he would feel once he truly understood what was going on.

The host objects that insider experience can mislead. An Islamist might say that feeling the bliss of fighting for the caliphate in Iraq would teach you more than the outside view from a couch, yet in that case the outside view seems more useful. Carlsmith separates two questions: what the experience is like from the inside, and what you would conclude if you fully understood the situation. Full understanding, in his sense, includes what the situation is causing, what beliefs structure the ideology, and whether those beliefs are true. He means the second question, not merely that the future would feel good to those living there. He still expects the residents to approve of it.

If the people who are there aren't thumbs up, that's a strange utopia.

Beyond the residents' own view, he adds a more holistic perspective, which he calls full understanding, and that is the perspective from which one would endorse the situation.

The host then asks whether future people will regret how we handled the 21st century, citing an analogy, attributed in the transcript to McCaskill (as transcribed) and his new book, that treats us as the teenagers of civilization. Adults often feel nostalgic about the risky things they did as teenagers because those things were formative. Carlsmith doubts that future people, assuming there is a future, will look back on the risks and horrors of the 21st century with pleasure. He says he has no such nostalgia about the Cuban Missile Crisis or World War II.

He also questions the teenager analogy itself. If you survived racing friends or playing chicken at a quarry, you can look back on it fondly, but the real reason not to do it is the part of the probability where you died.

chunk of probability where you just died

Looking back from after the fact, he concludes, is not the right perspective for calibrating how to feel about risks, especially risks of death.

Asked why utopia matters and why people struggle to imagine a compelling one, Carlsmith defines utopia simply as a profoundly better future. It matters because he thinks it is actually possible: if we play our cards right, in non-crazy ways, we could build a world radically better than today's. He says we often underestimate how large the difference in value could be between our current situation and what is available. Utopias are often anchored too hard on the status quo, changed in small ways while our fundamental situation stays basically unaltered. His example is the step from a crappy job to a beach vacation, with utopia imagined as everyone having a beach vacation. He thinks the real difference is more like being asleep versus awake, or living in a cave versus under the open sky, and that this matters a lot.

The host points to an apparent contradiction with Carlsmith's essay, which says utopia should be recognizable to a person alive now. Carlsmith agrees there is at least a tension and resolves it through the idea of understanding. Ideally, we reach utopia through a long, patient process of becoming wiser, better and more capable as a species, and it is because that process culminates that we can build a civilization that is profoundly good and radically different. If someone were transported straight there, skipping the process, they might not like it, and it would be quite alien. A person who had gone through the process of understanding and becoming wiser would endorse it.

The host reads this as a philosophical process rather than, for instance, working out how to convert everything into hedonium and enjoying eternal bliss. Carlsmith agrees that he does not think we now know that utopia is hedonium. He is not especially into the notion, and says that the brand is bad: people talk about pleasure dismissively, and the picture of tiling the universe with hedonium suggests sterile uniformity. In his view the relevant perspective on something like hedonium is the internal one, the subject's own experience, and that point is where this part of the conversation stops.