A dispatch platform we work on passes every synthetic load test we throw at it and still occasionally struggles on a Friday night. The gap between those two facts is the interesting part.
Synthetic load is evenly spread
Generated traffic tends to arrive at a steady rate because that's the easiest thing to script. Real demand on a taxi platform arrives in bursts correlated with weather, football matches and last orders, and the burst is exactly the condition the system needs to survive, not the average.
What we changed about our tests
- Replay real traffic shapes, taken from anonymised logs of the worst nights on record, rather than a flat ramp.
- Test the retry storm, not just the load. When a downstream service slows down, clients retry, which multiplies load exactly when capacity is lowest. We simulate the retry behaviour, not just the request volume.
- Include the human in the loop. A dispatcher clicking the same button twice because the first click seemed to do nothing is part of the load, not a separate concern.
The failure mode we found
Under a realistic burst, the queue that assigns drivers to rides backed up for around ninety seconds before recovering on its own. Nobody had reported it, because it recovered — but ninety seconds is a long time to a rider watching a spinner.
A system that recovers on its own still cost someone ninety seconds they noticed.
Fixing the number, not the symptom
We added backpressure earlier in the pipeline so the queue sheds load predictably rather than degrading uniformly, and we now alert on queue depth, not just error rate, because the error rate never moved during the incident.
What this means for the test plan
We keep a small library of real traffic shapes from past peak nights and run new releases against all of them before a Friday deploy, not after a Friday incident. It's slower than a synthetic ramp and it has caught two regressions the ramp would have missed entirely.