Incidents
The human response — who is handling it, what has been tried, what happened.
An alert is a machine noticing. An incident is people responding.
They are separate objects because most alerts never become incidents, and some incidents start with a phone call rather than an alert.
Opening one
Three ways:
| From an alert | Carries the alert, its context and its timestamps across |
| By hand | For something a customer or a colleague reported |
| Automatically | A rule can open an incident directly for conditions that always warrant one |
Severity
Incident severity is about impact, not about the alert that started it.
| Severity | Convention |
|---|---|
| SEV1 | Major outage. Revenue or safety impact |
| SEV2 | Significant degradation, or a subset of users entirely blocked |
| SEV3 | Degraded, with a workaround |
| SEV4 | Minor, or internal only |
Agree the severity ladder as a team and write it down here. An organisation where SEV1 means different things to different people cannot compare two incidents, and its "mean time to resolve" is an average over things that were not the same.
Roles
| Role | Does |
|---|---|
| Commander | Decides. One person, always named |
| Scribe | Keeps the timeline so the commander does not have to |
| Comms | Talks to everyone who is not in the response |
| Responder | Does the work |
For a small incident one person holds all four, and that is fine. For anything longer than half an hour, separating commander from responder is the single highest-return thing you can do — the person typing cannot also be the person deciding.
The timeline
Every action, observation and decision, timestamped.
Populated automatically from:
- alerts firing and resolving
- deployments in the window
- configuration changes
- feature-flag flips
- status-page updates
…and by hand for everything else. The automatic entries are what make the timeline usable afterwards: nobody writes down what they did not do, and "a deploy went out at 14:02" is exactly the kind of thing everyone forgets.
Communicating
- A status page for people outside the company
- Notification channels for people inside it
- The incident's own updates feed, which both read from
Resolving
Resolving records when, and by whom. The alerts that led to it are linked — they resolve on their own measurements, not because the incident was closed.
Closing an incident does not resolve its alerts, and it should not. An incident is a human judgement that the response is over; an alert is a measurement. If the condition is still true, something is still wrong.
Afterwards
Within a working day, write the post-mortem. The timeline is already most of it.
Related
Where this behaviour lives: backend/internal/domain/incident.go, frontend/src/app/incidents/. If the code and this page disagree, the code is right — please fix the page.
Part of Incidents and on-call — Turning a reading into a page, and a page into a resolved incident.