Mark Zuckerberg — Llama 3, $10B models, Caesar Augustus, & 1 GW datacenters

Mark Zuckerberg describes Llama 3 as the central update behind a new version of Meta AI. Meta plans to release the models openly to developers while also using them in its assistant, which will draw on Google and Bing for real-time information. The company is adding the assistant to Facebook and Messenger search and expanding creation features, including image animation and real-time image generation as users type. The rollout will begin in a limited number of countries and widen over time.

At the time of the interview, Meta is training 8-billion- and 70-billion-parameter versions, plus a still-training 405-billion-parameter dense model. Zuckerberg says the smaller models are leading for their size, with the 70B model around 82 on MMLU and strong math and reasoning results; the 405B model is already around 85 during training. He also previews multimodality, more languages, and longer context windows. The 8B model is nearly as capable as Meta's largest Llama 2, illustrating how quickly capability per parameter has improved.

Zuckerberg says Meta began ordering H100 GPUs while scaling recommendation systems for Reels. The shift from ranking followed accounts to recommending 'unconnected' content expanded the candidate pool from thousands to hundreds of millions and required a different infrastructure. Meta struggled to match TikTok's pace, so he decided to buy enough GPUs for Reels and core feeds, then double that capacity to prepare for needs the company could not yet see. He describes this as a recurring operating principle: infrastructure should anticipate future products, not only current demand.

Asked whether he expected the next use to be AI, Zuckerberg says he expected large-model training but initially thought it would be for content. He connects major business bets to personal conviction more than precise valuation models. In 2006, he did not have the financial sophistication to analyze an offer for Facebook as a billion-dollar sale; he stayed because he liked building products that help people communicate and believed he would start a similar company if he sold. His retrospective point is that many consequential decisions are difficult to reduce to a forecast spreadsheet.

Meta has funded FAIR for about a decade as a research group whose advances would improve products, rather than as a product team in itself. ChatGPT and image-generation models changed the stakes: Meta created a generative-AI group to bring these capabilities into products and build foundation models. Early applications seemed mostly social—connecting users with creators and businesses, supporting sales and customer service, and providing assistant features across apps, glasses, and VR. In practice, Zuckerberg says the company came to see that more general intelligence was needed.