Nine customers were charged twice during a launch-week spike, and the cause was not fraud, a bug in the payment provider, or bad luck. It was a retry, working exactly as designed, hitting an endpoint that was not designed for one.
How it happened
Under load, the checkout call was timing out at the client before the server had finished processing it. The mobile app, quite reasonably, retried. The server, quite unreasonably, had no way to tell that the retry was the same order as the one it was still charging.
The fix is one idea, applied everywhere
Every request that changes money, stock, or an order state now carries an idempotency key generated once, on the client, at the moment the user presses the button — not regenerated on retry. The server checks the key before doing anything and returns the original result if it has seen it before.
- The key lives with the intent, not the request. Retrying with a new key defeats the whole point.
- The check happens before the side effect, inside the same transaction, not as a separate step that can itself race.
- Idempotency keys expire, deliberately, after a window generous enough to cover retries but short enough not to become a second database of orders.
The part that took longer than the fix
Auditing every endpoint that could plausibly be retried. Checkout was the obvious one; less obvious were the webhook handlers receiving events from the payment provider itself, which also retries on its own schedule and had the identical problem in the opposite direction.
If a request can be sent twice, it will be. Networks retry, users double-tap, and providers resend. Design for the second copy, not just the first.
What we test for now
Every write endpoint that touches money has a test that fires the same request twice in quick succession and asserts the side effect happened once. It is a cheap test to write and it has caught two regressions since, both introduced by engineers who did not know the rule existed until the test failed and told them.
The nine refunds
They went out within the hour, with an explanation rather than a form letter. None of the nine customers left a bad review about it, which we take as a small, specific argument for telling people what actually went wrong.