Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Emmanuel Ameisen opens with a definition that sets the tone for the entire conversation. He works with a team of interpretability researchers at Anthropic whose goal is to understand how large language models such as Claude work by looking inside them. The premise is that behavior is visible while the mechanism is not: the architecture is only a scaffold, data is passed through it, and the computation that results takes shape in ways nobody specified. That gap is why studying the model is a job at all rather than a matter of reading its design.

One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them.

Lenny Rachitsky picks up the distinction between models that are built and models that are grown, and Ameisen walks through the pipeline. Pre-training takes a large corpus, feeds the model the first half of a document, and asks it to predict what comes next, repeated across an amount of data he describes as ungodly. A fine-tuning phase then substitutes examples of desired behavior, such as dialogue in which a helpful assistant gives a recipe or solves a coding problem. What is being trained through all of this is not a set of rules but a chain of mathematical operations on numbers converted from words, and those computations are the part that grows. Parameters move in one direction or another, the model learns formulas, and the team has no idea what those formulas are.

The reason for this, Ameisen argues, is necessity rather than preference. Ordinary programming handles self-contained behavior well: if someone clicks a button, then something happens. Nobody knows how to write that kind of rule for classifying images, or for a machine that can take in arbitrary text and produce summaries, emails or code. A simple training objective is therefore the only route anyone has found to a system this general, which is exactly why the result has to be investigated rather than read. Lenny sharpens the oddity: the two are describing code the team wrote that must now be watched the way a neuroscientist watches a brain. Ameisen's reply is that an objective that sounds simple is probably deceptively complicated, and that these models are doing more than the phrase about predicting the next thing suggests.

Asked what has surprised him after watching these systems closely, Ameisen goes after the standard intuition first. People picture iPhone autocomplete, correct perhaps sixty per cent of the time and funny and weird the rest of the time, and treat that as the whole story. He calls it the easiest version of predicting the next word and says it is unfortunately the mental model most people carry.

His argument is that getting better at the task forces the task to become harder. He dates the point at which language models surpassed humans at predicting the next word to around 2019, and notes that continued improvement runs out of easy words. First the model only has to capture sentence structure. Then it has to predict simple facts about the world, such as the capital of France being Paris. Once those are mastered, what remains is the answer to a complex physics equation or the final paragraph of a PhD thesis. His test case is the capital of the state that contains Dallas: producing Austin requires, at least for a human, a working model of geography, with Dallas in Texas, states having capitals, and Texas having Austin. Pursued far enough, the seemingly trivial objective demands an understanding of the structure of the world.

Lenny connects this to a line he attributes, with admitted uncertainty, to Ilya Sutskever: that predicting the next token well requires understanding the entire universe. Ameisen adds a related example of his own, the final line of a mystery novel naming the murderer, which cannot be predicted without having followed the whole plot. Neither of them resolves the puzzle. Ameisen frames it instead as the question that drew him into the field: how something so simple can give rise to something so powerful.

How can something so simple give rise to something so powerful?

Lenny then extends the analogy to people, citing an unattributed observation that humans are next-token predictors, and adding that meditation makes the impulse to say the next thing noticeable. Ameisen agrees only in a qualified form. While answering, he says, at least two processes are running: a destination, the broad point the sentence is meant to land on, and the moment-to-moment production of one word after another, with pauses when the sentence seems to be working. He presents the comparison as a way of understanding what language models do, not as a claim that human speech and model inference are the same mechanism. The two-track description is the more careful version of the idea, and it previews the planning evidence he turns to later.