A client's fulfilment app started marking orders as shipped that hadn't been, and the cause was a webhook from a courier integration that arrived twice, a day apart, for reasons the courier's support line couldn't explain and probably didn't know either.
The assumption the app was making
The handler treated each incoming webhook as a fresh, true event: courier says shipped, mark it shipped. That's a reasonable first version and a dangerous permanent one, because every webhook provider we've dealt with eventually resends something — after a timeout, after a deploy on their end, after nothing identifiable at all.
What we check now on every webhook
- Signature verification, unconditionally, even for providers we trust, because a compromised endpoint is worse than a slightly slower handler.
- An idempotency key or equivalent, derived from the payload if the provider doesn't supply one, checked before any write happens.
- A plausibility check against current state, not just the payload's own claim — a "delivered" event for an order that's still showing "awaiting pickup" is worth a beat of hesitation before it's accepted at face value.
- Out-of-order arrival handling, because a webhook can and does arrive after a later event from the same source, and applying it naively can roll a status backwards.
Why plausibility checking matters more than it sounds like it should
Couriers, payment providers and marketing platforms all occasionally send an event for the wrong order, a duplicate, or a stale retry of something already superseded. A handler that trusts the payload's timestamp over its own record of what already happened will occasionally undo a correct state with a stale one.
A webhook is a claim from another system, not a fact. Treat it the way you'd treat a hardware sensor: useful evidence, not automatic truth.
What this looks like in the code
Every webhook handler in the app now runs through the same small pipeline — verify, deduplicate, check plausibility, then apply — regardless of which integration it's for, so a new integration inherits the protection rather than needing it rebuilt.
The specific fix
The doubled shipping event now gets deduplicated by a key derived from the order id and status, with a short retention window, and the plausibility check would have caught it even without that, since the order was already marked shipped from the first event. Belt and braces, deliberately, because the two checks fail independently.