A coding agent wrote a database migration for us that was, by every measure we normally check, correct: it applied cleanly, the schema came out right, and the tests passed. What it hadn't been asked to do, and what nobody had checked, was reverse cleanly, and we only found out because we made a habit of testing that separately.
Why forward migrations are the easy half
Generating a migration that adds a column, backfills it and enforces a constraint is a well-worn pattern with a lot of correct examples to draw on. Writing the migration that undoes it — dropping the constraint, the column, and any data that depended on it, without losing something you can't get back — is a less common pattern in the training data and a less common habit in human-written migrations too.
What we found
The generated migration's "down" step dropped the new column outright, discarding data that had, in the three weeks since deployment, become the source of truth for a downstream report. Rolling back to fix an unrelated issue would have silently destroyed real data, discovered only because we test rollbacks as a matter of course rather than trusting that they mirror the forward step correctly.
What we do now for every migration, generated or not
- Apply it, then reverse it, on a copy of production data, as a standing step in review rather than an occasional spot check.
- Treat data loss on rollback as a design decision, not an accident — sometimes it's genuinely unavoidable, and in that case the migration says so explicitly, with a note on what data would be lost and why that's acceptable.
- Never let "the tests passed" stand in for "the rollback was checked", because a test suite written for the forward state has no reason to exercise the reverse one.
A migration is a promise in two directions. Most reviews only read the promise going one way.
Why this isn't really an AI story
Human-written migrations have the same failure mode, and we found examples of it in older codebases we've audited that predate any of this tooling. What changed is the volume: more migrations are being written faster, so a check that was previously "usually fine to skip because migrations were rare" now needs to be a standing part of the process rather than an occasional discipline.
What it cost to add
A rollback test roughly doubles the time to review a migration. It has, on this project alone, caught two cases that would otherwise have been discovered during an actual incident, which is a considerably worse time to discover them.