Nothing paged anyone. The error rate stayed flat. Response times were normal. A background job had been silently returning empty results for four days, and every dashboard we had told us everything was fine.
Why the existing monitoring missed it
We were watching for errors and latency, which is the standard pair, and the job in question was not erroring — it was completing successfully and doing nothing, because an upstream data source had changed shape in a way that caused a filter to match zero records instead of throwing.
The gap this exposed
Monitoring for failure catches the case where something breaks loudly. It does not catch the case where something keeps running and quietly stops doing its job, which is arguably worse, because nothing about the system's own signals suggests a problem exists.
What we added
- Output volume checks — a job that normally processes hundreds of records and starts processing zero, for no scheduled reason, now raises an alert even though it technically succeeded.
- A dead man's switch on scheduled work — not "did it error" but "did it produce output within an expected range", checked independently of the job's own reported status.
- A weekly review of anything running at zero or near-zero volume, cheap to do and specifically designed to catch the slow, undramatic failures that alerts are bad at surfacing.
A system that fails successfully will never trigger an alert built to catch failure. It needs a different question asked of it: is this doing anything at all.
Why we did not just add more error handling
The instinct after an incident like this is to make the code throw in more cases. That helps, but it does not solve the general problem, which is that any given piece of logic can have a case nobody anticipated where it completes without doing meaningful work. Watching outcomes, not just exceptions, catches classes of failure that better exception handling never will.
What this cost us and what it would have cost otherwise
Four days of a feature quietly not working translated into a modest, recoverable amount of missed processing once we reconciled it. The client noticed before we did, which was the uncomfortable part of the postmortem and the reason the output-volume checks went in the same week rather than the same quarter.