Two releases in a row shipped with a fully green pipeline and a real bug in production. Neither was a gap in coverage — both had tests directly covering the broken behaviour. The tests were simply asserting something that had quietly become wrong.
How a passing test goes stale
A test asserted that a discount could not exceed 50%. The business rule changed to allow up to 75% for a promotional period, someone updated the test to match rather than questioning it, and the update was correct for that promotion and silently wrong once the promotion ended and the cap should have reverted. The test stayed green the entire time because it was checking the number it had been told to check, not the actual current rule.
The second case, similar shape
A test mocked an external shipping API's response format. The real API changed its format months later; the mock didn't, because nobody updates a mock in step with a vendor's changelog unless something forces them to. The test kept passing against a shape of data production no longer received.
Why coverage numbers didn't catch either
Coverage measures whether a line executed during a test, not whether the test's assertion still reflects reality. Both bugs sat in heavily covered code. The metric was answering a different question from the one we actually needed answered.
- Tests against external contracts need a live check, not just a mock, run on a schedule separate from the main suite, specifically to catch drift.
- Business-rule tests should cite the rule's source, a ticket or a policy document, so a reviewer changing the test can see whether the rule itself changed or just the number.
- Green CI answers "does this match what we told it to expect", not "is what we told it to expect still true".
A test suite can be entirely honest about what it checks and still be checking the wrong thing. Green is not the same as correct.
What we added
A quarterly pass, separate from normal review, where someone reads through the highest-risk tests and asks whether the assertion still matches current policy, not whether it passes. It's a small, deliberately manual step, because the entire failure mode here is one automation was silent about.
The uncomfortable conclusion
More tests would not have caught either bug. A different kind of attention would have, and no amount of pipeline tooling substitutes for someone occasionally asking whether the tests are still asking the right question.