Data warehouse/SQL alerts/Per-row alerts

Per-row alerts

One alert per subject, each with its own clocks, recovery and history.

Turn on Group by and name a key column, and every row of your result becomes its own alert.

sql

SELECT
    subject,                        -- the key
    count()                  AS volume,
    countIf(status='bounced') AS bounced
FROM warehouse.mail_message_logs
WHERE created_at > now() - INTERVAL 1 HOUR
GROUP BY subject

What each subject gets

its own fingerprint, so it is one alert that updates rather than many
its own For and Recovery clocks
its own state and history
its own recovery — one subject recovering does not close another's alert

This is what makes "queue payments is backed up" and "queue email is fine" two independent facts rather than one averaged one.

Choosing the key

The key must be text and must identify the subject. If the query has several text columns, name the one you mean — the rule will not guess, because guessing produces alerts keyed on the wrong thing and nobody notices until they try to silence one.

If the rule reports "the query returns 7 rows but this rule is not set to one-row-per-subject", you have a grouped result with grouping switched off.

Subjects that come and go

Keys rotate. A reconciliation query keyed on recon/<date> emits a different set every day; a per-subject query loses a subject the moment it goes quiet.

AccelerUp tracks which keys were present in each run. A key that was firing when it stopped appearing is resolved explicitly — otherwise its alert would stay open forever, pointing at a row that no longer exists, with no possible recovery.

Note

This is the one place where an absence does close an alert, and it is not an exception to the rule: the run did produce a reading, and that reading says the subject is gone. That is different from no reading at all.

Row limits

A run evaluates at most 2000 rows. Past that the query is describing a shape rather than a set of subjects, and 2000 alerts is not monitoring.

If you are near the limit, aggregate further — group by queue rather than by message, by country rather than by city.

One message or many

A grouped rule with twelve subjects breaching produces twelve alerts, and by default twelve messages. Set notify mode to summary and it produces one message listing every subject that breached, while keeping the state per subject.

See Notifications.

Per-condition, per-subject

Grouping and per-column conditions compose. A query with three conditions and forty subjects tracks 120 independent states — each with its own clocks. The subject table on the rule page shows all of them.

Where this behaviour lives: backend/internal/worker/dwh_alert_worker.go. If the code and this page disagree, the code is right — please fix the page.

Part of Data warehouse — Querying the warehouse, and alerting on what the query returns.