Troubleshooting
Failures this team has actually hit, with the cause and the fix.
Nobody was paged
Work down this list in order.
- Did the alert fire? Look at Alerts. If not, the rule is the problem — see below.
- Did it route anywhere? The alert detail page shows delivery. "No channels" means no routing rule matched, or the channel's kind does not include alerts.
- Did delivery fail? Notifications shows the error, the attempt count and a retry button.
- Was it silenced? Check for an active silence or maintenance window.
- Was the rule inside its run window? Outside it, the query is not run at all.
A Slack message never arrived
Check Notifications. The common errors:
| Error | Fix |
|---|---|
400 invalid_attachments | The message exceeded Slack's 3000-character section limit. Now trimmed at the transport — if you still see this, the deployed version predates that fix |
404 | The incoming webhook was revoked. Re-create it, update the channel, retry |
channel "X" is switched off | Re-enable it in Channels, then retry |
| Nothing at all — no row | No channel matched. Check the channel kind and the routing rules |
A rule fires constantly
See Alert intelligence — it grades rules by how often anybody acts on them.
- Fires often, auto-resolves quickly → raise
For - Fires and resolves repeatedly on the same value → set a
Cooldownor widen the gap between firing and recovery - Fires on a percentage computed from tiny numbers → add a volume guard
A rule has never fired
The most dangerous state, because it is indistinguishable from working monitoring. Work through the checklist in Testing a SQL alert — Preview run answers most of it in one click.
The usual cause: the query returns no rows when everything is healthy, so
there is nothing to evaluate. Return 0 instead.
A panel shows less data than expected over a long range
Several external sources have retention shorter than the range you can select. The panel says so rather than silently showing a subset — look for a note naming the source and the window it actually covered.
The known one: Cloudflare's load-balancing dataset is kept about three days. A seven-day routing matrix is measured only as far back as AccelerUp's own capture reaches. See Load balancing.
An agent is offline
| Symptom | Cause |
|---|---|
| "Has stopped reporting" | The agent process is down, or the machine is |
| "Is unreachable" | The network path to it is broken |
These are deliberately different alerts. On the machine:
systemctl status accelerated
journalctl -u accelerated -n 50
accelerated statusIf the websocket will not establish, the proxy chain is the usual culprit —
Connection and Upgrade are hop-by-hop headers and any hop may drop them.
/api/v1/agent/ws-check says whether the upgrade survived. The terminal also
has an SSE + POST fallback that does not need the upgrade at all.
The terminal button does nothing
The agent's profile caps what it will accept, and observe does not include
a shell. The profile is on the machine, not in AccelerUp, and cannot be widened
remotely. It is logged at every agent start. See
Access and grants.
Alerts arrived twice
Check that the worker is leader-gated. With two replicas and an ungated worker, everything runs twice. See Background workers.
Timestamps are an hour out
Check the rule's timezone. An unknown IANA name falls back to UTC — never to the server's local time — so an unrecognised zone shifts every window predictably rather than mysteriously. See Schedules.
Everything is rate limited
TRUSTED_PROXY_CIDRS is not set, so every request appears to come from the
proxy and the whole deployment shares one bucket. See
Environment variables.
A migration will not apply
SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 10;Migrations are sorted by version, not by position in the file, and run under an advisory lock. A stuck one usually means a previous run died holding the lock — check for an idle Postgres session before assuming the SQL is wrong. See Migrations.
Where to look next
- Settings → System Health — worker liveness, integration status
- Settings → Platform → Change Feed — what changed just before it broke
- Audit log — who did what
Part of Running AccelerUp — Operating the platform itself: architecture, deploys, backups.