Make it easy to undo your fuckups

This one’s about shipping fast. And I mean many agents at once, all changing things in parallel, more than one developer going fast.

That means you’re going to fuck up more often. More code, more changes, more experiments, faster than any human team ever worked. Google’s DORA research already sees it: more AI use came with about 7% lower delivery stability. The normal reaction is to slow down with more process, more sign-offs and longer PR reviews. I’d rather keep going and make every fuckup cheap to undo, with the tooling for that built right into the products.

Save everything, all the time

You can only undo what you kept, so I store stuff early and often.

In my chat workspace every agent reply takes a snapshot of the workspace before and after it runs. If the agent mangles a file there’s an Undo under that reply, and a Go back to before this that rewinds everything after it. Under the hood it’s a plain git repository per workspace, somewhere the agent can’t touch it.

My CMS does the same for content. Every publish writes a revision, you can compare any two versions side by side, and reverting doesn’t delete anything, it just adds a new revision on top. History only grows.

And it all gets backed up: accounts, chats, workspaces and their history, schedules. One script makes a backup, another restores it. For the databases I’m moving to streaming every write to S3 as it happens, so a crash costs me about a second of data instead of a day. A backup you’ve never restored is a hope, though, so a proper restore drill is on the list too.

Fork everything

What makes running lots of agents possible: nobody works on the real thing. You fork it.

We fork whole applications, databases included. In my table app every base is its own SQLite file, so copying a database is one call. An agent gets its own fork, does whatever it likes with it, and the original never notices. Good result? It goes live. Bad one? Throw the fork away. That’s the cheapest undo there is, since there’s nothing to undo.

The forks are also the preview environments. Every change can run as its own copy of the app with its own data, so you can click around in the real thing before anyone else sees it.

Same idea at a smaller scale: in the chat workspace you can continue any chat in a copy, and the assistant can start side chats that each work on their own copy of the files in the background. Ten agents, ten forks, and they never get in each other’s way.

The rest of the industry is moving the same way. Neon branches a whole Postgres database in about a second, so every pull request can have its own copy of the data.

Deploys you can take back

Every deploy goes into a new release folder, with a current link pointing at whichever one is live. The data lives outside the releases and survives every deploy. Rolling back means pointing the link at the previous folder. No rebuild needed.

If the code is fine but the data is broken, the restore script moves the existing data aside before it puts the backup back. So even the undo can be undone.

Roll out gradually

Nothing goes to everyone at once. A new app starts out visible only to me, then invite-only, then public. In the CMS you can limit features and content to a group, like one department, people on a paid plan or the people in one course. Plugins go on one site first. It’s the good old canary release: let a small group run into the problems before everyone else does.

When something breaks for five users you have a conversation. When it breaks for five thousand you have an incident.

Rip out what doesn’t work

Undo also works for whole features. Because every feature is a plugin, removing one means deleting a folder.

For a while I built the same chat app twice, on two different stacks, to compare them. When I had my answer I deleted one. It’s still in the git history and the measurements are written down, so all I lost was the maintenance.

Know what you did, and what’s used

You can’t undo what you can’t find, so everything gets tracked: which reply changed which files, which revision fixed which review note, who uses which feature. That makes it easy to see what to roll back, and what nobody would miss.

Let tools tell you where to look

Reading every line of every PR doesn’t work when agents write the code. I’d rather put that effort into tools that point at what needs my attention.

There’s a debt score, one number for how hard the code is to read, built from function complexity, long functions, huge files and copy-pasted code. There’s a baseline in the repo and the check fails when the score goes up. Performance budgets do the same for speed, and the user stories do it for features.

Most changes don’t need me to look at them at all. The ones that make something worse show up by themselves.

Write the playbook before you need it

The worst moment to figure out how to roll back is in the middle of the fire. So every risky thing comes with a playbook: how to deploy, how to back up, how to restore locally (with scheduled tasks turned off, so a restored copy doesn’t start running production’s jobs from my laptop) and how to restore production. There’s even a script that shows what the snapshots hold, for when undo doesn’t do what you expected.

The scripts are the playbooks, and the comments at the top of each one say when and how to use it.

Speed comes from safety

In my experience teams aren’t slow because they’re careful. They’re slow because mistakes are expensive. The DORA research has said this for years: the teams that deploy most often also recover the fastest. Make mistakes cheap and you can go as fast as your agents can type.

So don’t bother trying to prevent every fuckup. Just make them cheap.