How alerting works

Alerting sources differ — a service check, an agent rule, a Cloudflare error rate, a warehouse query — but they all produce alerts through the same model. Learn it once.

The lifecycle

   reading ──► condition true? ──no──► (state stays ok / recovery clock runs)
                    │ yes
                    ▼
            firing clock starts
                    │
            sustained for `For` minutes?
                    │ yes
                    ▼
                FIRE  ──► notification routed to channels
                    │
            reading says condition false
                    │
            sustained for `Recovery` minutes?
                    │ yes
                    ▼
               RESOLVE ──► recovery notification
                    │
            `Cooldown` minutes before the same condition may fire again

The four clocks

For: 0 and Recovery: 0 mean fire and resolve immediately. That is the right setting for a binary condition ("the host is unreachable") and the wrong one for almost any rate.

No reading is not evidence of recovery

This is the rule that governs everything else, and it is worth stating on its own because the alternative is so tempting to implement.

An alert never resolves because of an absence:

• an empty query result
• a check that could not run
• a host that stopped answering
• a run window that closed
• a rule that was paused

All of those leave the alert exactly where it was. Only a real measurement that finds the condition false will close it.

Fingerprints: one condition, one alert

Every alert has a fingerprint derived from the rule and the subject. The same condition firing again updates the existing alert; it does not create a second one. This is what stops a five-minute rule producing 288 alerts a day about one problem.

For a rule that produces one row per subject (a per-country error rate, a per-queue depth), each subject gets its own fingerprint — so each has its own clocks, its own recovery and its own history. See per-row alerts.

Severity

Severity drives routing and escalation, not just colour. An escalation policy can page different people for critical than for warning.

Many rule types support a warn threshold alongside the main one, so a single rule fires at warning first and escalates itself to critical when the value keeps going — rather than needing two rules that can drift apart.

Routing

An alert reaches people through notification channels. What goes where is decided by:

1. Channel kind — alerts, incidents, or both.
2. Routing rules — match on severity, service, team, environment.
3. Escalation policy — page a person, then the next one if nobody acknowledges.
4. Silences and maintenance windows — see Maintenance and silences.

Every attempt is logged, and failures are retried — see Notification delivery.

Where alerts come from

Two failure modes this model is built to avoid

Alert storms. One transition that affects fifty subjects should be one conversation, not fifty. Grouping resolves the individual alerts and raises one that closes when the last subject recovers — not the first.

Silent degradation. A rule that cannot evaluate, a data source that stopped answering, a fallback to a weaker source — all of these are surfaced as faults rather than being treated as "nothing to report". A rule that has never produced a reading is marked never run, not healthy.