Sholto Douglas & Trenton Bricken — How LLMs actually think

Sholto Douglas argues that long context is more than a larger storage buffer: it can let a model absorb a codebase or an unfamiliar language quickly enough to behave as though it had acquired new expertise. In one paper evaluation, a model learned an esoteric human language from examples in context and outperformed a human expert's learning over a couple of months. Douglas also proposes testing whether models can infer how to play Atari games from labeled frames. These demonstrations suggest strong in-context adaptation, though the game example is still a prediction rather than a reported result, and the language evaluation is only a small demonstration.

One proposed explanation treats in-context learning as a form of gradient descent performed through attention. Douglas and Trenton Bricken point to a linear-regression experiment: as examples accumulate in the prompt, ordinary least-squares loss falls in a way that matches the number of gradient-descent steps. Bricken notes a possible safety consequence: a model that adapts during its forward pass may effectively become a different model, so harmful or jailbreak examples in context could change its behavior even if its original training emphasized harmlessness. The analogy is suggestive, but the conversation does not establish that attention literally updates model weights or that all in-context learning follows this mechanism.

The speakers disagree about what currently prevents AI agents from taking off. Dwarkesh Patel emphasizes long-horizon assignments, where an assistant must work for hours or days. Douglas instead argues that reliability is the binding constraint: when a task requires many steps, the probability of success compounds across them. A 90% success rate on each step can make a long chain unreliable, while additional nines could abruptly make agents useful. Bricken connects this to the appearance of emergent capabilities in benchmarks, arguing that skills may have existed gradually but become visible only after a threshold of repeated success. They call for evaluations that measure completion across minutes, hours and days, because short single-question tests such as MMLU cannot directly predict how much of a job family can be automated.

The discussion also complicates the claim that long context is prohibitively expensive. Douglas says that in dense Transformers, the MLP's model-dimension cost can dominate attention's quadratic term, and that autoregressive decoding attends from each new query to prior key-value vectors with cost linear in context length. Long context may therefore be practical before attention becomes the dominant expense. Patel then frames context as additional computation during inference and asks whether forward-pass learning could make models more sample-efficient. Douglas is more cautious: context provides working memory and enables adaptation, but it may not be the core of reasoning; the GPT-2-to-GPT-3 transition suggests meta-learning depends on both context and scale. The conversation presents these as competing interpretations rather than a settled account of reasoning.

The conversation turns from context length to the residual stream as a possible form of working memory. Patel compares it to a boat moving down a river: attention heads and MLPs act like side channels that add information to the current state. Douglas says many features are encoded in superposition within a high-dimensional vector, so each layer can read, combine and rewrite information before the model predicts the next token. This gives a concrete picture of how a model can carry compressed information across layers, but it does not by itself explain how the model reasons.

Bricken connects this architecture to neuroscience through the cerebellum. He describes a direct and an indirect information path, then points to associative-memory algorithms whose circuits resemble cerebellar circuitry. He also cites reports that the cerebellum is active in many fMRI tasks and in a study of next-token prediction, while noting that it contains a large share of the brain's neurons. The analogy is striking, but the guests acknowledge that they are drawing from a limited research base and that the brain's circuitry is not yet fully understood.

Bricken's broader claim is that much of intelligence may be hierarchical pattern matching. Associative memory can retrieve or denoise a memory, and associations can be chained into more abstract concepts. He applies this idea to Sherlock Holmes: a long working memory could preserve clues, while attention repeatedly retrieves them and higher layers refine a hypothesis. Patel finds the account plausible, but notes that deduction seems to involve more than simple association. Their proposed answer is composition: multiple learned circuits and features can combine to support increasingly complex reasoning.