Three unrelated incidents this quarter, on three unrelated projects, had the same root cause once we wrote them up side by side: a boundary between two systems where the data's shape was assumed rather than checked.
The three incidents
A webhook payload from a payment provider added an optional field, and a downstream job that assumed its presence threw on a small fraction of requests for two days before anyone noticed the pattern. A CSV export from a warehouse system quietly changed column order after a vendor update, and stock levels went wrong for a subset of products. A mobile app sent a timestamp in a slightly different format after an OS update, and a reconciliation job silently dropped rows rather than erroring.
What they have in common
None of these boundaries had validation. Each side trusted the other to keep sending the same shape indefinitely, which is a reasonable assumption right up until it isn't, and the failure was silent or delayed in every case rather than immediate.
The fix is boring and that's the point
- Validate at the boundary, not deep inside the code. A schema check on arrival turns a subtle downstream bug into an immediate, loud rejection.
- Log and alert on validation failures, don't just swallow them, because a validation failure is exactly the early warning that these three incidents lacked.
- Version the contract where you control both sides. If you own the producer and the consumer, an explicit version field costs almost nothing and removes the guesswork entirely.
An untyped boundary doesn't fail when the data changes. It fails later, for someone else, in a way that looks like a different bug.
Where we've added this systematically
Every external integration on our current projects now has a schema check on the way in, even for systems we don't control, precisely because we don't control them. The check doesn't fix the upstream change, but it turns three days of confused debugging into an alert with the exact field that changed.
The trade-off worth naming
This adds a small amount of upfront work to every integration, and it has, without exception, been cheaper than the incidents it would have prevented on every project where we've since added it.