An incident review last month ended, as they sometimes do, with "we can't actually tell what happened, because we didn't log the one field that would have told us." Nobody had done anything wrong at the time — the field simply had not seemed worth logging, until the one week it was the entire question.
Logging is a bet made in advance
Every log line is a small bet that some future question will need this information. Log too little and incidents become archaeology. Log everything and the signal drowns, storage costs climb, and nobody reads any of it because there is too much to read.
What we now log by default, everywhere
- Every state transition, not just the final state — what changed, what it changed from, and what triggered it. The trigger is the field most often missing and most often needed.
- Every external call's latency and outcome, including the ones that succeed, because "how long does this usually take" is unanswerable without a baseline of normal calls to compare against.
- Identifiers that let you trace one user's journey across services, consistently named, so a support ticket about one person's bad experience can be reconstructed rather than guessed at.
What we deliberately do not log
Full request and response bodies for anything touching personal data, by default — the cost of a leak outweighs the convenience of having it, and in the rare case we genuinely need that level of detail we add it deliberately and temporarily, not permanently.
The incident that prompted this
A refund was issued for the wrong amount, once, and by the time anyone noticed, the only record was the final state — refund issued, amount X — with no trace of what inputs produced that number. We could not tell if it was a calculation bug, a race condition, or a one-off manual override, because none of the intermediate steps had been recorded. It was never reproduced, and it remains unexplained.
The incident you can't diagnose is more expensive than several incidents you can. Insufficient logging doesn't cause fewer problems — it just makes each one impossible to close.
What changed afterwards
The refund calculation now logs every input to the formula, not just the output, at negligible additional cost. We have since used those intermediate values twice to resolve disputes in minutes that would previously have taken a day of guessing, which is roughly the return we hoped for and did not have until we needed it once and didn't have it.