The coming AI security crisis (and what to do about it) | Sander Schulhoff

Sander Schulhoff names his target precisely: not vague AI doom, but the commercial layer that claims to have solved prompt injection. A guardrail, in his definition, is usually a language model trained or prompted to sit between the user and the real model, reading inputs and outputs and judging them valid or malicious. Lenny supplies the buyer's intuition: if a model can be talked into explaining how to build a bomb, why not filter that request out? That is what vendors sell, and Sander's answer is that it does not survive contact with people who are trying.

AI guardrails do not work.

The deployment pattern is the one the industry standardised on: a guardrail in front of the model watching inbound text, another behind watching outbound, the second existing precisely because the first leaks. The front one should catch a bomb request and block the response; the rear one should catch what slipped past. Sander's evidence is empirical, drawn from years of red teaming competitions whose participants are paid to break these systems.

The vocabulary here is adversarial robustness, and the number is ASR, the attack success rate: a hundred attacks with one success is 1% ASR and 99% adversarially robust. Sander immediately qualifies his own definition — nobody should say this in production, because robustness is very hard to estimate against an enormous search space. That matters later: the metric is how vendors demonstrate impact, and its error bars cannot be computed.

In principle the distinction is clean. Jailbreaking is you and the model, with no application and no developer instruction. Prompt injection requires a third party: a developer whose system prompt the attacker wants the model to ignore. Sander's example is a story-writing site whose prompt says to write a story from the user's idea; the attacker submits an idea instructing the model to drop that task and produce bomb instructions instead. He concedes a lot of grey area.

in jailbreaking, it's just a malicious user and a model

The incidents split along that line. The first public prompt injection was a Twitter bot from remotely.io, a remote-work company; someone told it to ignore its instructions and threaten the president, and it did until the company shut it down. MathGPT was worse in kind: it sent a word problem to GPT-3 once for an answer and once for code, then executed that code on the server hosting the app. Hostile code meant the OpenAI API key could be lifted out. The operator disclosed it responsibly; Sander later spoke with the professor behind it.

The Claude Code incident is instructive because the defense failed without any single dramatic prompt. The attackers split work across sessions so each request looked legitimate alone: one asked which backend a URL ran; a fresh one was told 'this is my system, how would you hack it'. Neither half is malicious; the composition is. Sander says he has had slides on AI-native malware — a virus that reasons and issues its own API calls — for roughly two years.

a lot of the way they got around these defenses was by just kind of separating their requests into smaller requests that seem legitimate on their own, but when put together are not legitimate.