Every project we run gets a quarterly restore test: take the most recent backup, restore it to a clean environment, and verify the data is actually there and usable. Roughly a quarter of these tests, over two years, have found something wrong. That number should worry more teams than it seems to.
Why "we take backups" is not the claim that matters
A backup job that runs successfully every night tells you the job ran. It does not tell you the resulting file can be restored, restored within an acceptable time, or restored into a state the application can actually use. Those are three separate claims, and only a real restore tests all three at once.
What has actually gone wrong, across two years of these tests
- A backup that had been silently truncated for six weeks because a disk quota was hit partway through the job, and the job reported success regardless — it had written a partial file and exited cleanly.
- A restore that technically worked but took eleven hours, against a client expectation of "back up within the hour" that had never been tested against reality before this exercise.
- Foreign key constraints that made a partial restore fail in an order nobody had anticipated, because the backup and the restore process had been written at different times by different people with different assumptions.
None of these would have been visible from a dashboard showing "backup: successful" every night. Every one of them was visible within the first ten minutes of an actual restore.
A backup you have not restored is a theory about a backup. The only way to know if it's true is to act on it.
How we run it without disrupting anything live
Restores go to an isolated environment, never towards production, and the test includes a specific, checkable assertion beyond "the database exists" — a known record, present in the source, confirmed present and correct after restore. "The restore command exited zero" is not the bar; the actual data being right is.
What this costs
A few hours per project per quarter, mostly unattended once the process is set up. Weighed against the two incidents in two years where this test caught something that would otherwise have been discovered during an actual emergency, the arithmetic is not close.
The client conversation this enables
Being able to say "we restored your data successfully, from this date, in this environment" three months ago is a materially different claim from "we have backups", and clients who ask the right questions during procurement notice the difference immediately.