Reiner Pope – The math behind how LLMs are trained and served

Reiner Pope reduces inference on a GPU rack to two competing lower bounds: time spent computing and time spent moving data. The compute term scales with batch size and active parameters, while the memory term must fetch all model weights and the KV cache. For each decode step, the cache grows with the batch, context length, and bytes stored per token. Attention arithmetic is omitted from the first approximation because Pope expects it to be small beside the weight operations; he also stresses that the resulting equations are lower bounds, not exact runtime predictions.

The distinction between active and total parameters matters. Active parameters determine the work performed for each generated token, whereas the full weight set still has to be fetched from memory. The KV cache adds a separate traffic stream: each sequence in the batch needs its history read during attention. Pope therefore models total step time as whichever is larger, the combined memory-fetch time or the compute time. This simple roofline view cannot capture every architectural detail, but it makes the main constraints visible and gives a way to reason about how batch size and context length shift them.

Batching improves the economics because the cost of loading the model's weights can be shared across more sequences. At batch size one, that fixed weight-read cost is concentrated on a single token stream; as the batch grows, its per-token share falls. Compute and KV traffic, by contrast, grow roughly with batch size, so after dividing runtime by batch size they become approximately flat per-token costs. The resulting cost curve falls sharply at first and then approaches a floor set by compute and KV access. A larger batch is not free: it can increase step latency, and longer contexts raise the KV term enough to move the system from compute-limited to memory-limited operation.

This explains why a low-cost mode can benefit from waiting for more requests, but only up to the point where the batch amortizes the weights. KV data belongs to individual sequences, and each sequence still needs its own computation, so neither cost disappears as the batch expands. Pope's answer is a lower bound on cost per token rather than a promise that arbitrarily slow service can make inference arbitrarily cheap. The model also points to context length as an important qualification: dense attention makes KV reads increase with context, while sparse attention can improve that scaling, though the conversation does not assume every provider uses it.