Anomaly detection
Statistical detection for the things a threshold cannot catch — including the slow rise that never trips one.
Logs & Errors → Anomalies watches metrics for behaviour that is unusual for that metric, rather than above a number you picked.
When a threshold is the wrong instrument
- The normal value varies by time of day and day of week
- You do not know what normal is yet
- The problem is a change in shape, not a level
- The metric rises slowly enough that no threshold is ever crossed in one step
The last one is the important case, and it is worth being specific about.
The level shift that nobody caught
A week-long rise in 401s never alerted, for two related reasons:
- Every detector in use was a point detector. They ask "is this reading unusual?" — and no individual reading in a gradual rise is.
- The two detectors that were trend-aware voted it down: the mean move (0.74) was small against the median absolute deviation (3.85), so it looked like noise.
A DetectLevelShift function existed with zero callers, and would not have
worked on that data anyway.
The fix was three things together, and all three were necessary:
| Change | Why |
|---|---|
| A rank-based test | Insensitive to the variance that hid the shift |
| Its own decision path | Not a vote among point detectors that will always outnumber it |
| A dense hourly series | A sparse series cannot show a gradual slope |
4xx and 5xx are now always tracked, whether or not anybody configured a rule for them. The class of problem above is invisible until it is large, and by then it has usually been happening for a week.
Sources
Anomaly detection is source-agnostic. It works on service metrics, Cloudflare metrics, log volumes, database metrics, warehouse queries and agent metrics.
AccelerUp already has a full multivariate anomaly engine. To add a source you
write a MetricSourceAdapter — you do not build a second detector. A parallel
implementation is how two systems come to disagree about whether something was
anomalous.
Tuning
| Setting | Effect |
|---|---|
| Sensitivity | How far from normal counts. Start low and tighten |
| Minimum volume | Below this, do not evaluate — small numbers are always anomalous |
| Seasonality | Compare against the same hour last week rather than the last hour |
| Exclusions | Do not learn from windows that were already incidents |
The last one matters more than it sounds: without it every outage raises the baseline, and the next one is harder to detect.
Alerting
Anomalies raise alerts through the standard model, with severity from the magnitude of the deviation.
Run anomaly detection in a non-paging channel for a couple of weeks first. Look at what it flags and how often. Then decide what deserves to wake somebody.
Where this behaviour lives: backend/internal/worker/anomaly_workers.go, backend/internal/service/anomaly_svc.go. If the code and this page disagree, the code is right — please fix the page.
Part of Logs and errors — What the software said about itself, and what was unusual about it.