The standard response to a flaky test is to mark it as flaky, skip it, and move on. We have started treating that instinct as a warning sign in itself.
Flakiness is rarely random
A test that fails one time in twenty is not unlucky. It is deterministic, given the exact state of the system at that moment, and we just have not identified the state that triggers it. "Random" is the label we reach for when we have not looked hard enough.
Common causes, in order of how often we find them
- Shared state between tests. One test leaves a record behind that a later test did not expect.
- Time. Anything comparing timestamps, expiring tokens, or crossing midnight in the test run's timezone.
- Concurrency. Two requests racing for the same resource, passing when the timing happens to favour one order.
- External calls that are not actually mocked, just usually fast enough not to matter.
Every one of these is also a way the production system can misbehave. The test found it first, cheaply, in a place with logs and a rerun button.
Quarantine is a decision, not a default
We still quarantine tests, but it is now a logged decision with an owner and a date, not a silent skip annotation that nobody revisits. A quarantined test that has sat unexamined for a month is treated the same as a known bug with no ticket.
A flaky test is a bug report the system is filing on its own behalf. Skipping it does not close the ticket, it just stops the phone ringing.
The case that changed our minds
A payment reconciliation test failed intermittently for weeks before someone traced it to a race between two async jobs writing to the same ledger row. It had been quarantined twice. The same race existed in production, at low volume, showing up as a support ticket about a duplicate charge roughly once a quarter — rare enough to look like user error, until the test's failure rate lined up with it.
What we ask before skipping anything now
Has anyone reproduced it deliberately, rather than waiting for it to happen again. Has anyone checked whether the same condition can occur outside the test. If the answer to either is no, it stays in the pipeline, failing, until someone can answer both.