Background workers
What runs in the background, how often, and what happens when one fails.
Around fifty background workers run inside the backend process. They are registered through a helper that leader-gates them: with two replicas, each worker runs on exactly one.
Without that gate, every scheduled rule would evaluate twice and every notification would be sent twice. If you add a worker, register it the same way as the others.
The main ones
| Worker | Interval | Does |
|---|---|---|
monitor_worker | 30 s | HTTP/TCP service checks |
dwh_alert_worker | 1 min | Evaluates SQL alerts |
notification_retry_worker | 30 s | Drains the delivery retry queue |
cf_hourly_worker | hourly | Cloudflare metric rollups |
cf_alert_worker | minutes | Cloudflare error-rate rules |
cf_datasource_alert_worker | minutes | Per-endpoint guard |
elk_sync / elk_alert_worker | minutes | Log analytics + rules |
anomaly_workers | minutes | Anomaly detection across sources |
mysqlnode_sync / _alert_worker | minutes | Database node health |
proxysql_sync / _alert_worker | minutes | Connection-pool health |
synthetic_worker | per check | Synthetic checks |
agent_watchdog | minutes | Agents that stopped reporting |
slo_burn_worker | minutes | Error-budget burn rate |
jenkins_poll | minutes | Build status |
release_sync_worker | minutes | Release + changeset sync |
cost_sync_worker | daily | Cloud spend snapshots |
copilot_kb_worker | scheduled | Knowledge-base embedding |
report_worker / saas_report_worker | scheduled | Scheduled reports |
cleanup | daily | Retention pruning |
internal/worker/ holds the full list.
Failure behaviour
A worker that fails logs and continues on its next tick. It does not take the process down, and it does not stop its siblings.
Workers that talk to slow external systems carry their own timeouts — an individual SQL alert rule, for example, gets two minutes, so one hanging warehouse query cannot hold up every rule behind it. The symptom of missing that is unrelated alerts going quiet, which looks like nothing being wrong.
Rules that cannot evaluate
A worker that cannot do its job raises a fault rather than being silent — and reports it on transition, not on every run. A misconfigured rule evaluated every minute should produce one message, not one a minute forever.
Adding a worker
- Put it in
internal/worker/. - Extract anything that decides into a pure function with its own tests — the worker should call a decision, not contain one.
- Register it through the leader-gating helper in
main.go. - If it writes a high-volume table, add that table to the cleanup worker's retention list.
Observing them
Settings → System Health shows worker liveness and last-run times. Start-up
lines in the log name each worker as it registers — notification retry worker started, and so on — which is the quickest way to confirm a worker is running
on the replica you are looking at.
Where this behaviour lives: backend/internal/worker/, backend/cmd/server/main.go. If the code and this page disagree, the code is right — please fix the page.
Part of Running AccelerUp — Operating the platform itself: architecture, deploys, backups.