The first time a failing test turns out to be unrelated to your change, you investigate. The tenth time, you re-run the pipeline and move on. By the fiftieth time, the suite has stopped meaning anything.
What "flaky" actually costs
We tracked one project's pipeline for a month before intervening. Around one in eight test runs had at least one failure unrelated to the change under review. Each of those cost a re-run, a wait, and — the part that does not show up in any dashboard — a small erosion of trust in every other red result that month.
Where flakiness actually comes from
- Shared state between tests — a database row, a file, a mock server that one test leaves in a state another assumes is clean.
- Time-dependent assertions — anything comparing against "now" without controlling the clock in the test.
- Network calls that should have been stubbed and were not, so a slow afternoon on a third-party API becomes a slow afternoon in your pipeline.
- Concurrency in the test runner exposing a race that also exists in production but was never going to surface outside a parallel run.
The rule we adopted
A test that fails without a code change behind it gets fixed or deleted within one working day. Not quarantined indefinitely in a "known flaky" list — that list is where tests go to stop being looked at.
A flaky test does not fail randomly. It fails for a reason nobody has found yet. Quarantining it does not remove the reason, it just removes the reminder.
Where agents help and where they do not
A model is genuinely useful at spotting the shared-state pattern across a large suite — it can scan for tests that mutate global fixtures faster than a person reading file by file. It is not good at deciding whether a genuinely intermittent third-party dependency should be mocked, retried, or accepted as a real risk; that is a judgement about the system, not the test code.
What changed once we enforced the rule
Within six weeks the flake rate on that pipeline dropped from one in eight to under one in fifty, and — the part we actually cared about — the team started treating red pipelines as something to investigate immediately again, rather than something to shrug at.