We assumed the deployment pipeline was the bottleneck in getting a change to production. It is not. Across four projects, the pipeline itself was the fastest part of the whole journey, and it was not close.
What we actually measured
From the moment a pull request merges to the moment someone with authority confirms the change behaves correctly in production — every stage, timestamped, across roughly sixty releases in a month.
Where the time went
- Build and deploy: minutes. Consistently the fastest stage, and the one everyone had spent the most effort optimising.
- Waiting for a deploy slot: hours, sometimes. Several teams deploy in batches at fixed times rather than continuously, which trades a little safety for a lot of latency.
- Waiting for someone to verify: often the longest stage. A change would sit "deployed" but unconfirmed for the better part of a day because verification depended on one person's attention and that person was in meetings.
The fix was not technical
We did not need a faster pipeline. We needed verification to be somebody's explicit, named job for a defined window after each deploy, rather than an implicit expectation that whoever happened to be free would notice. Once that was assigned, the longest stage shrank from most of a day to under an hour, with no change to any tooling.
We optimised the part of the process that was already fast, because it was the part with a dashboard. The slow part had no dashboard, so nobody had looked at it.
A smaller finding worth mentioning
Releases that included a database migration took disproportionately longer at the verification stage, not the deploy stage — reviewers were, reasonably, more cautious about confirming a migration had gone cleanly, and that caution took time nobody had budgeted. We now flag migration-bearing releases for a slightly longer verification window up front, rather than discovering the need for one in the moment.
What we would measure next
The gap between a bug being reported and someone starting to look at it, which we suspect has the same shape — a fast technical path (alerting, logging, tracing) surrounding a slow human one (whose job is it to look, right now). We will report back once we have a month of numbers on that too.