The client could not accept downtime, so a schema migration that would ordinarily be an evening's maintenance window became a month of dual writes, careful comparison, and a cutover that most users never noticed.
Why the obvious approach was ruled out
Locking the table, migrating, and releasing was the fast option and the one nobody would approve, because the product runs continuously across time zones and there is no quiet hour to use. The migration had to happen while the system stayed live and correct.
Dual writes, then dual reads
For the first two weeks, every write went to both the old and new schema. Nothing read from the new schema yet — it existed purely to accumulate data and let us compare it against the old one continuously. Discrepancies were logged, not corrected automatically, because an automatic correction can hide a bug in the migration logic itself.
What the comparison found
- A default value that the old schema applied implicitly and the new one required explicitly, silently null in a small fraction of new rows.
- A timezone assumption baked into one old column that the new schema had, correctly, made explicit — meaning old and new disagreed by design, and the comparison needed an exception written for it.
- Ordering, in one case, mattered to a downstream report in a way nobody had documented, and the new schema's default ordering broke it.
None of these were migration bugs in the conventional sense. They were undocumented behaviour of the old system, discovered only by insisting on parity before switching anything over.
A migration with no downtime is not a faster version of a migration with downtime. It is a different, slower project that trades an evening of pain for a month of vigilance.
The cutover itself was uneventful
Once the discrepancy rate held at zero for a full week, we flipped reads to the new schema behind a flag, watched for a day, then removed the old writes. The most careful month of the project ended in an anticlimax, which was the entire point.
What we would do again
Running the comparison as a continuous, boring background job rather than a batch check run occasionally. The bugs we found were all low-frequency, and a batch check run once a day would have taken far longer to surface the same issues than a comparison running on every write did.