Home

Why AI code breaks in production

AI-built apps usually break in production for one structural reason: they were verified by looking finished, not by being proven finished. The demo passes because it runs the happy path the builder already knows. Real users don't take that path. And the gaps the demo never exercised are exactly where the app fails. The fix isn't more testing of the demo; it's changing what "done" is allowed to mean.

The collapse after the demo

The demoable part of an AI build arrives fast and looks complete. What comes after - the part that has to survive real users - is where it falls apart. That remainder is not "more of the same work". It is a different kind of correctness: the messy real paths, the cold start, the empty states, the inputs no one demoed. AI generation is strong at the part a demo can show and silently weak at the part it can't, and the gap is invisible until users find it.

"Works if you jump straight to the screen" is not working

The single most common production break: the feature works only when you load it in isolation, straight to the right screen, with the right state already in place. A real user arrives cold, from the front, and the path that should carry them there was never actually wired. So I refuse a shortcut-to-the-screen as proof of anything. Two checks decide "done":

the main action completing through the real, rendered interface, and a brand-new user reaching the feature by the route real people take

Both are things a demo skips and production demands.

You can't ask the build whether it's finished

The deeper reason these gaps ship is that the report is trusted. A build (or the AI that made it) reporting "done" tells you almost nothing. Again and again, that confidence came apart when checked against reality. The only reliable signal is evidence that can't be talked into existence:

the commit, the passing test, the action completing through the real UI

When "done" is computed from that instead of claimed, the demo-vs-production gap becomes visible before users find it, because the thing that looked finished now has to prove it.

What I do about it

I treat what comes after the demo as the actual job, not the cleanup. I break the product the way its users will - cold starts, real navigation, the inputs no one demoed - before they arrive to break it for me. And I separate the thing that grades the work from the thing that built it, so "finished" is a verdict from evidence, not a self-report. When a build is perfect in the demo and dying on real users, that gap is the first thing to read.