Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

Hamel Husain defines evals as a systematic way to measure and improve an AI application. In his account, the work is fundamentally data analytics applied to an LLM product: examine what the application is doing, create metrics where they are needed, then use those measurements to run experiments and iterate. The definition makes evals a feedback practice rather than a mysterious specialist discipline.

His real-estate-assistant example gives that definition an operational meaning. An assistant may draft unsuitable customer emails, call the wrong tools, or fail in other ways. Without an evaluation process, a team might change a prompt and hope it has not broken something else. Informal vibe checks can be useful at the beginning, he says, but become hard to manage as the application grows. Metrics supply a feedback signal against which the team can improve with more confidence.

The conversation treats evals as a broad spectrum of quality measures, not simply a bank of tests. Shreya Shankar agrees that unit tests can check non-negotiable capabilities, but she argues that they cover only a small part of the problem. Open-ended AI work also raises questions that a fixed test may miss: how an assistant handles unfamiliar requests, whether a new user population is appearing, and whether the product should accommodate that group differently.

For Shankar, the evidence base can include regular inspection of product data to identify new user cohorts and metrics tracked over time, such as users giving a message a thumbs-up. Those signals need not be specifically AI-related to be useful in a product-improvement loop. This is why the unit-test analogy is helpful but incomplete: evaluation includes both explicit checks and ongoing observation of how the application behaves in changing conditions.

Husain places responsibility for open coding with a trusted individual rather than automatically with a committee. He warns that teams can get bogged down in committee work and make the process too expensive to sustain. His proposed decision-maker is someone whose judgment the team trusts and who has domain expertise, often the product manager. The point is not that collective review is never appropriate, but that it is unnecessary in many situations he describes.

The speakers frame the aim modestly: evals do not have to be perfect; they should lead to actionable product improvement. That emphasis matters because the subject already attracts strong opinions, and Shankar says poor past evaluation efforts have left some people distrustful enough to identify as anti-eval. A process that cannot produce useful action, or that repeats earlier bad experiences, can therefore undermine the confidence it is meant to create.

Another misconception appears at the outset: the idea that, because this is the age of AI, AI can simply evaluate the product itself. Husain says that shortcut does not work. The interview does not yet offer a technical substitute in this section; instead, it establishes that evaluation still requires deliberate measurement, data review, and accountable judgment rather than delegating the problem wholesale to an AI system.

In the host's introduction, Lenny Rachitsky says Husain and Shankar have taught more than 2,000 product managers and engineers across 500 companies and calls their Maven course the number-one course on that platform. These are the host's introductory claims, not evidence independently established in the discussion. The substantive case made in the opening exchange rests instead on the examples and reasoning about measurement, changing data, and product feedback.

Hamel Husain returns to the earlier warning against equating evals with tests. For an LLM application, he argues, teams should first analyze data to learn what is worth testing. Compared with conventional software engineering, the behavior has more surface area and is stochastic, so the assumptions behind a test plan are less settled. Error analysis is the opening move: inspect what is going wrong before deciding how to measure it.

The Nurture Boss example makes the analysis concrete. The anonymized property-management assistant works across chat, text, and voice, uses many tools for tasks such as appointments and availability, and retrieves customer and property information. Husain uses an observability view only as a way to see the application's data. He explicitly says there is no favored tool, naming Braintrust, Phoenix Arize, and LangSmith as possible places to load the logs.