Rules and silences
Alerting on agent data, and acting on it — with the three guardrails that stop a restart loop.
Fleet → Alerts & silences defines rules over agent data.
Rule sources
| Source | Examples |
|---|---|
| Metric | CPU, memory, disk, load, clock drift |
| Prediction | Disk time-to-full |
| Unit | A systemd unit not running |
| Probe | Reachability, latency, loss |
| Log watch | A rate of matching lines |
| Health document | A dependency the application itself reports unhealthy |
Rule groups
Rules can be grouped and applied across the fleet by tag or environment, so "disk above 85% on every production box" is one object rather than sixty.
Alert on the prediction, not the level
A disk at 71% is not interesting. A disk at 71% that was at 40% this morning is.
Rules on time-to-full give you the one thing a percentage cannot: warning while there is still time to act.
Log watches alert on rate
A log-watch rule fires on a rate, not on presence. "Error X appeared" is noise on a busy box; "error X went from 2/hour to 400/hour" is a signal.
Actions
A rule can run a command in response — restart a unit, clear a cache, roll a log.
Three guardrails, and the order matters:
- Give up after N attempts. Checked first, so a loop cannot outrun its own limit.
- Cooldown between attempts.
- Capped by profile. An
observeagent runs nothing regardless.
Auto-remediation without a give-up counter turns one failing unit into a restart loop. It looks like flapping, it hides the real cause, and it can keep a broken service in a half-started state indefinitely. The counter is not optional.
Every action is audited alongside the rule that triggered it.
Silences
Silence a rule, a server, or a rule-on-a-server. For planned work, prefer a maintenance window — it covers everything in scope and is recorded as planned.
Grouping
One transition that affects fifty servers is one conversation.
Grouping resolves the individual alerts and raises one that stands for the transition. It closes on the last recovery, not the first — a group that closed when the first server recovered would mark an ongoing outage as resolved.
Staged rollout of rules
Like agent rollout, a rule change can go out in waves rather than to the whole fleet at once. A threshold that is wrong is wrong everywhere, and finding that out on one canary is cheaper.
Where this behaviour lives: backend/internal/domain/agent_rule.go, backend/internal/service/agent_rule_svc.go. If the code and this page disagree, the code is right — please fix the page.
Part of Fleet — The agent on each server: what it collects and what it may be asked to do.