AI products that can't lie about being done
Building with AI creates a particular problem: the work can look finished long before there is any evidence that it is. The method on this page grew from building software, where its rules are easiest to see. The discipline underneath reaches further - separate the report from the evidence, test the real path rather than the demonstration, change nothing on the strength of a convincing story about what went wrong - and those habits still shape how I research, build and revise, whatever the finished result turns out to be.
An AI-built product says it works. The demo runs, the checklist is green. Under that sits a quieter question: do I actually know it works, or do I just know that something told me it does? Most AI code is trusted because it looks finished. I build on the opposite assumption.
That assumption isn't a pose. It's the lens I was trained to see through. Public administration, where I took my master's, taught me to look at how institutions turn decisions into outcomes. Auditing, which I did in consulting, added another reflex: trust the evidence, never the assurance. Both habits carried straight into building with AI: direct the work, then verify the result independently of whatever did it. Directing is the easier half. The other is proving that "done" is true - the half most easily skipped.
It says it's done. Is it?
I found something uncomfortable while building: you can't ask an AI whether it finished. Again and again, its confident "done" came apart the moment I checked it against the evidence - often enough that I stopped treating the report as information at all, the AI's and my own alike. In my work, "done" is not a claim anyone makes. It is computed from things that can't be talked into existence:
the real commit, passing tests, the actual action completing through the actual screen
And the thing that grades the work is never the thing that did the work. "Finished" becomes a fact rather than a feeling - which is the only thing that ever let me stop re-checking it.
Perfect in the demo. Dead when real users arrive
The demoable part of an AI build can come together in an evening and look flawless. Then real people arrive, and what the demo skipped begins to collapse. In the builds I have taken apart, the cause was almost always the same: it works if you jump straight to the right screen, and a real user never does that. So two things are the real finish line, both of which a demo skips:
the main action working through the real, rendered interface, not in isolation, and a brand-new user reaching the feature by the path people actually take
A shortcut straight to the screen does not count as proof. I break the product the way its users will, before they arrive to do it for me - because the collapse worth fearing is the one that happens in front of them.
Fix it, rebuild it, or leave it alone. The truth first
When a codebase is in trouble, the most expensive sentence available is "rewrite everything" from someone who never read it. My rule: read the real code first, then tell the truth - what to fix, what to rebuild, what to leave alone, including "this is fine" when that is the honest answer. The verdict has to be written so it stands without me: a plan any developer could execute. A diagnosis that only its author can act on is indistinguishable from a sales pitch, and the fear it feeds - of trusting someone who profits from the disease - is a reasonable fear. Under the mess, the job is never to patch the symptom. It is to find the clean architecture the broken version was blindly reaching for, and name it.
The invoice that quietly kills the project
AI tooling can burn a budget silently and end a project in a single month. I measured one version of this in my own work: a "cheap temporary version, just to test it" cost three to four times more than building the real thing once, and left nothing to keep. The lesson is not that small tests are a waste - a small test aimed at a real unknown is the cheapest thing you can buy. The waste was the disposable stand-in: a temporary copy of something already understood, built to be thrown away. So what is understood gets built properly once, what is uncertain gets the smallest test that can settle it, and spend stays capped and visible instead of being discovered at month's end, when the bill arrives too late to change anything.
Three months later the model updated and everything broke
Every time I have watched AI-heavy code age, it has aged fast: a new model or tool version ships, and yesterday's glue no longer fits. The most durable defence I have found is structural, so every product is split on purpose:
a stable core that holds the real logic and rules, and a thin outer layer that touches the fast-moving model
When the tooling changes, that limits what has to be rewritten - usually to the outer layer alone.
What if it's confidently wrong?
The most expensive mistakes don't look like mistakes. They look like a confident, reasonable answer that happens to be wrong. AI does this constantly; so do people. The instinct to "just fix the obvious cause" is exactly how a wrong guess becomes broken data and lost days. So there is a stop built into my own process:
nothing gets changed on the basis of a story about what's wrong, only on the actual evidence of what's wrong
That stop catches me too. More than once I've started on a fix that looked completely right and killed it within minutes, the moment I checked it against what had actually happened. The process worth trusting isn't the one that's never wrong. It's the one that catches the mistake fast, before a wrong guess breaks something real.
Why most AI builds come out shallow
Everyone knows you get more from an AI by asking better. That's why prompt engineering caught on. But the prompt is the small lever. Between the request and the answer sits work you never see: how much of the problem the model actually engages before it replies. What I found, working this way, is that putting the model into the right working state for the task ahead matters more than how you phrase the request. Run as an order-taker, it does what you said, at the shallowest reading that satisfies the words. That is where demo-deep, falls-apart-later output comes from. For real work I run it as an investigator: set up to go after the real problem instead of the stated one, and to argue with its own first answer before I ever see it.
The two halves fit together. I take everything I can from how the model works, and trust nothing of what it claims.
depth on the way in, evidence on the way out
That combination is the craft, and it is most of why what I make holds up where a quick build doesn't. It is also why the front page of this site talks about research the way it does.