Francois Chollet — Why the biggest AI models can't solve simple puzzles

Chollet presents ARC as an intelligence benchmark designed to resist memorization. Unlike tests that reward accumulated facts or familiar patterns, its puzzles are intended to require only core knowledge—basic notions of objects, counting, and elementary physics—while presenting a novel task each time. A solver must infer a transformation from a few input-output grid examples and apply it to a new grid. Chollet says current language models can parse such small grids, but struggle because they need to synthesize a new solution program rather than retrieve one from memory.

The benchmark's interpretation is contested. Dwarkesh asks whether 80% performance by a future LLM would indicate a path to AGI, while Chollet says the method matters: training on millions of ARC-like puzzles could create overlap with test tasks and preserve the very memorization ARC aims to test. He acknowledges ARC is imperfect and could be hacked. His preferred evidence would be a critical mass of genuinely novel tasks that a model adapts to on the fly, a criterion he says would matter for any AI technique, not only LLMs.

The disagreement turns on what counts as being 'in distribution.' Dwarkesh argues that large models may have seen enough of the world to handle tasks that would be new to an individual human. Chollet replies that the world changes, so no fixed training set can cover every future task. He contrasts human intelligence with organisms whose behavior is more genetically hard-coded: if environments were static and predictable, evolution might have produced a fixed program instead of a species that begins with little knowledge and learns efficiently. This is Chollet's account of why adaptation to novelty matters, not simply a claim that current models have no useful knowledge.

Chollet says multimodality alone may not remove the ARC difficulty. The benchmark uses small two-dimensional grids with only ten possible symbols, which are easy to flatten into sequences; transformers can process them. Fine-tuning on familiar tasks and testing on small variations shows that LLMs can parse the input and encode solution programs. The failure, in his view, is unfamiliarity: each new puzzle requires a fresh program. Whether multimodal models improve performance remains an empirical question the speakers expect to test soon.

The hosts and guest discuss human performance as a comparison point. Chollet says smart adults should score 90–95% and even five-year-olds should exceed 50%; he reports that tests with average adults yielded about 85%, while Dwarkesh questions whether Mechanical Turk participants represent the global median. Dwarkesh notes that humans also vary on IQ-style tests and asks whether a small model's 35% score, reported by Jack Cole with 240 million parameters, signals an analogous capability spectrum. Chollet says this result depends on millions of generated ARC tasks and test-time fine-tuning, unlike a child's first exposure; he also notes that the approach is a genuine example of active inference, which he finds promising.