Runbooks
The steps, written while nobody was panicking.
Incidents & On-Call → Runbooks holds the written procedure for a situation.
Why they are in here
A runbook in a wiki is a runbook nobody opens at 4 a.m. Attached to the alert, it is a link in the page that woke you.
Writing one that works
The test: could somebody who has never seen this system follow it?
| Do | Do not |
|---|---|
| Exact commands, copyable | "Check the logs" |
| What a healthy result looks like | Leave the reader guessing whether it worked |
| What to do when a step fails | Assume the happy path |
| Who to escalate to, by role | Name an individual who may have left |
| When to stop and escalate | Imply the reader must solve it alone |
Structure
## Symptom
What you are seeing that brought you here.
## Impact
Who is affected, and how badly. This decides how fast to move.
## Check
1. `curl -s https://… | jq .status` → expect "ok"
2. Open <dashboard> → expect requests > 0
## Fix
1. …
## If that did not work
Escalate to the payments on-call. Include: the output of step 2, the
deploy id from the change feed, and what you have already tried.
## Afterwards
Link the runbook from the post-mortem and note anything that was wrong.Attaching to alerts
A rule carries a runbook URL. It appears in the alert, in the notification, and on the alert detail page — which means the link is in the message that woke somebody rather than two searches away.
Keeping them true
A runbook that has drifted is worse than none, because it is trusted under pressure and its instructions are followed literally.
Fix the runbook during the incident, while you know what was wrong with it. "We should update that runbook" said in the post-mortem is a task nobody ever does.
Every post-mortem should answer: did the runbook exist, and was it right?
Related
- Post-mortems
- Copilot — runbooks are part of its knowledge base, so it can answer procedural questions from them
Where this behaviour lives: frontend/src/app/runbooks/, backend/internal/handler/runbook_handler.go. If the code and this page disagree, the code is right — please fix the page.
Part of Incidents and on-call — Turning a reading into a page, and a page into a resolved incident.