Evidence-based development is my new hotness

When an agent tells me “done, it works”, I don’t believe it. Agents don’t lie on purpose, but “it works” is a claim, and I’ve started treating every claim like something that needs evidence.

So that’s the rule now. The agent has to prove it.

Show me

For every feature I want the agent to come back with evidence instead of a summary. Screenshots of the screen it changed, a video of the flow clicked through in a real browser, timings, database rows, logs. Whatever the product can cough up.

The more of that you can get out of your product, the less you have to trust anyone, human or agent. You can’t argue with a screenshot of a broken layout. “I tested it and it looks good” you can argue with all day.

And I’m not just being paranoid. Anthropic’s own Claude Code best practices say the same thing: when the agent says it’s done and it isn’t, that’s usually because it had no way to check its own work. So give it something to check with.

Every feature points at a test

In my chat workspace every feature is a user story with acceptance criteria and a Verify line that names the test that proves it. There are 113 stories, and 103 of them get checked automatically, most in a real browser.

There’s also a test for the tests. If a story claims a test that doesn’t exist, the test run fails, so the feature list and the proofs can’t drift apart without anyone noticing. When an agent adds a feature it has to add the story and the proof too, or it isn’t done.

That still left a loophole: a test that’s named after a story but checks nothing. So I also measure what each test actually runs. A test that names a story but runs no real code is hollow and gets flagged. Renaming a test takes five seconds. Faking what it runs is a lot more work.

Nothing is allowed to get worse

Proving the new thing works is only half of it. I also want proof that nothing else got worse, so I keep an eye on the stuff that rots slowly.

The chat has performance budgets for conversations of 10k, 100k and 1M tokens: how fast it loads, how smoothly it scrolls, how quickly a reply shows up. It also runs them on a CPU slowed down four times. At one point only 4 of those 12 budgets passed. Now it’s 12 of 12 and it has to stay there.

A script scores how hard every file is to read (decisions, nesting, the longest function), and the agent sees the numbers before and after its change. If a change makes the code harder to read, we both see it.

Another tool lists unused files, exports and dependencies, so leftovers from abandoned ideas don’t pile up. And because every feature has a story with a test, a feature can’t vanish without a test failing.

All of this is what lets me go fast with agents. They can change a lot, quickly, because I’m reading the evidence instead of every line.

Measure the real thing

Tests tell you the product can work. Usage tells you whether it does, and whether anyone cares.

So I log everything. For every reply, the chat workspace records what went to the model, what came back and where the time went: each tool call, time to the first token, cache hits. When something feels slow I just go and look.

The CMS keeps track of what people actually do, on my own server: which features get used, by how many people, who’s new and who comes back. Page reads are counted without cookies. Chat usage is tracked per model, project and person, including what it costs.

And that feeds the other half: if a feature isn’t used, I can delete it. Evidence tells you what to keep, and also what to throw away.

Why now

Your gut is a terrible judge here. In METR’s 2025 study, experienced developers felt about 20% faster with AI, while the stopwatch said they were 19% slower. The tools have improved since, and METR’s follow-up suggests developers probably are faster now. Still, how fast it feels and how fast it is turned out to be very different things. If you don’t measure, you don’t know.

Agents write code faster than I can read it. That’s the whole point, but it also means reading every line doesn’t work anymore. Checking evidence does. Screenshots, videos, budgets, metrics and usage take me minutes to check, and an agent seconds to produce.

I suspect that’s where the job is heading. Less writing code, less reading it even, and more deciding what proof you want and then asking for it.