Aishwarya Naresh Reganti, Kiriti Badam: Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon

Reganti opens with a qualification before the diagnosis: AI systems and software systems share most of their DNA, and only a few properties genuinely change how you build. The first is non-determinism. In ordinary software, intent travels down an explicit path. She uses Booking.com as the example — wanting two nights in San Francisco is funnelled through buttons, options and forms until the intention becomes an action. AI products replace that layer with natural language, so the same intent can arrive in an unlimited number of phrasings.

Both ends of that pipeline are opaque. The LLM is described as sensitive to prompt wording and effectively a black box, so even the shape of the output surface is unknown before runtime. The practical consequence is that input, output and process are all poorly understood at once, and the team is left anticipating behaviour rather than specifying it.

you don't know how the user might behave with your product, and you also don't know how the LLM might respond to that

Lenny's restatement narrows this to two consequences: the software may behave differently on each run, and the split of work between the AI and the person is a dial rather than a fixed setting. Reganti resists letting the point stand as a complaint. Natural language is also the reason the bar to adoption is low — people would rather talk than click through a form. The same openness is the defect: many ways to say a thing, aimed at deterministic systems that must produce a deterministic outcome. Her framing turns the design problem into intent fidelity.

That reframing is what carries the rest of the conversation. If the risk sits in interpretation rather than in code, the useful engineering work is deciding where interpretation is allowed to happen and how much of the result a person still signs off on — not choosing a larger model.

The second difference is what they call the agency-control trade-off. Reganti says she is surprised how rarely it is discussed, given how fixated teams are on autonomous agents that will do the work for them. The mechanism is simple and unforgiving: decision rights move to the system, and control moves away from the operator in the same transaction. Delegation is therefore a loan against trust, and it is only worth taking once reliability has been demonstrated rather than asserted.

more agency, which is the ability to make decisions, you're also losing some control

Badam describes what happens when a team skips the intermediate steps and goes straight for the most autonomous version. The agent fails in a hundred different ways, and nobody can tabulate that many failures, let alone fix them. Evaluation practice does not compensate for a bad starting point: once the error surface is that wide, the team cannot tell which corrections matter, so the project stalls no matter how much measurement discipline it has.

Lenny adds a blunter reason to hold autonomy back, one that has nothing to do with model quality: an agent with too much reach can corrupt a database or send mail nobody intended. Blast radius, not accuracy, is the case for staying on the bottom rung.

Badam's analogy is training for Half Dome rather than hiking it on day one: short sections first, then the summit. The operational version is to begin where impact is minimal and human control is high, so the team can learn the real capability boundary — what the system does well, and where it stops — and only then trade control for agency. He frames this as good news as much as caution: nobody has to look at the full apparatus of production agents and conclude it is out of reach, because everyone starts from a minimal structure and evolves.

Customer support is his worked example, including inside OpenAI, where launches such as Image and GPT-5 produced spikes in ticket volume and a different mix of problems. The first version does not answer customers at all. Human agents keep the conversation; the system proposes what it believes the right action is. Those accept and reject decisions become the product's first real training signal, and they show where the blind spots are before any customer is exposed to them.