Our incident runbooks were written for a human on call at 3am — terse, assuming context, full of "use your judgement here". We wanted a model to run the first diagnostic pass on common alerts, and the existing runbooks were unusable for that purpose without a rewrite.
What "use your judgement" actually meant
It meant: an engineer with two years on this system would know what to check next. A model, like a new hire, has none of that accumulated context, and "use your judgement" resolved to either a guess or a refusal, neither of which is what you want at 3am.
What we rewrote
- Explicit conditions, not vague ones. "If latency is high, investigate the database" became "if p95 latency exceeds 400ms for more than two minutes, check the slow query log for the previous ten minutes".
- Named commands, not described steps. A description of what to check is not the same as the exact query or dashboard link to check it with.
- A stated stopping point. The runbook now says explicitly where automated diagnosis ends and a human is paged, rather than trailing off into "escalate if needed".
What we let the model do and not do
It gathers information — recent deploys, error rates, the specific log lines matching a pattern — and produces a summary attached to the alert before a human is even paged for lower-severity issues. It does not restart anything, roll anything back, or close an incident. Those actions stay behind a human decision, deliberately.
A runbook vague enough to need judgement was already a runbook that depended entirely on who was on call that night.
The unexpected benefit
Rewriting the runbooks to be explicit enough for a model also made them faster for humans to follow, particularly for the engineers newest to on-call rotation. The rewrite was aimed at one audience and helped both.
Where we're being cautious
We're not extending this to anything that touches customer data or billing without a much longer track record. The diagnostic summaries have been reliably useful for a few months now; that's a good sign and not yet a reason to widen the blast radius of what runs unattended.