Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)

Brendan Foody argues that AI labs are entering an era in which evaluation, rather than raw model availability, determines how quickly capabilities can improve. Mercor works with leading labs because they need exceptional professionals who can measure what a model can do. Lenny Rachitsky frames the opportunity as unusually large, while Foody connects it to the willingness of the world’s wealthiest companies to spend heavily on better models.

If the model is the product, then the eval is the product requirement document.

The analogy is practical: researchers can run many experiments and make incremental improvements against a defined evaluation set. Foody points to Olympiad Math and SWE-bench as examples of benchmarks that can be rapidly saturated once researchers focus on them. In his view, the missing prerequisite for using agents across the economy is a way to define and measure success for each workflow, effectively writing a product requirement for every task an agent should perform.

For an enterprise, the first step is to identify its core value chain and build a systematic test for how well AI automates it. Foody uses an architecture firm as an example: the company would need a reliable way to assess the architecture diagrams it delivers to customers. A multi-product company may have several value chains, but the underlying requirement is the same. Measurement comes before broad deployment because without it the company cannot tell whether automation is improving the work it actually sells.

Foody also describes evals as commercial evidence. They show researchers what capabilities to build, but they can also demonstrate those capabilities to customers. The center of gravity is shifting from academic tests of advanced reasoning, such as GPQA, Humanity’s Last Exam, and Olympiad Math, toward evaluations tied to work people care about: building a software platform or completing an investment-banking analysis. Labs and application companies can therefore use the same kind of measurement to explain what their products can do.

Rachitsky places Mercor alongside three fast-growing categories: foundational-model companies, AI-assisted coding products, and data or labeling companies. Foody’s account explains why the third category is changing. Mercor began in January 2023 by recruiting people internationally, matching them with the founders’ network, and automating resume review, interviews, and hiring decisions with language models. The founders say this approach reached a one-million-dollar revenue run rate before they left college.

The company then encountered what Foody describes as a transition in the human-data market. Earlier systems needed large numbers of low- and medium-skilled contributors who could produce basic text. The newer problem is sourcing and vetting experienced professionals, including software engineers, investment bankers, doctors, and lawyers, who can evaluate and interpret the capabilities users want models to have. Foody says Mercor began working with the leading AI labs after this shift and grew from a one-million-dollar to a four-hundred-million-dollar revenue run rate in sixteen months.

In the interview’s simplest description of Mercor, the company helps labs hire experts rather than generalist labor to address specific gaps in model knowledge. Foody says this demand follows from the evaluation problem: labs need a definition of success both for testing progress and for acting as a verifier in reinforcement-learning environments that reward better model behavior. They need such definitions across domains and capabilities that models do not yet handle well.