Notification delivery
Every dispatch attempt, whether it arrived, how many retries are left, and how to send it again.
Incidents & On-Call → Notifications answers three questions, in this order:
- Did the page go out?
- If not, is it coming back?
- Can I make it go out now?
Every dispatch attempt is written to notification_delivery_log with the full
message payload, so all three have real answers rather than inferences.
Statuses
| Status | Means |
|---|---|
| Pending | Handed to the channel; no outcome recorded yet |
| Delivered | The channel accepted it |
| Retrying | A retry is in flight right now |
| Failed | It failed, and a further attempt is scheduled — the page shows when |
| Gave up | Every automatic attempt failed. Nothing more happens unless you retry it |
The distinction between Failed and Gave up is the one that matters. The first is a notification still on its way; the second is one that is never arriving.
The retry budget
A failed delivery is re-tried three times on top of the original dispatch — four attempts in total.
| Attempt | When |
|---|---|
| 1 | Immediately, as part of the dispatch |
| 2 | 1 minute after the first failure |
| 3 | 5 minutes later |
| 4 | 30 minutes later |
| — | DLQ ("Gave up") |
The budget and the schedule are one tested decision (domain.DecideRetry), and
the page reads the numbers from the worker rather than hardcoding a copy — so
"attempt 2 of 4" on screen is always the worker's own arithmetic.
retry_count counts failed attempts, and the first of those is the
original dispatch rather than a retry. This is why the total is four attempts
for three retries.
Failures that are not retried
Some failures cannot be fixed by asking again, and burning three attempts on them delays the honest answer by thirty-six minutes:
- the channel was deleted
- the channel is switched off — the error says so, and tells you to re-enable it and retry
Both go straight to Gave up with a message naming the cause.
Retrying by hand
The Retry button appears on any failed row — not only ones that have given up. It resets the retry budget and schedules immediate re-delivery.
Retry all failed at the top does the same for everything currently failing.
Retry is the right move after you have fixed the cause — re-enabled a channel, rotated a webhook, unblocked an egress rule. The retry re-sends the original message content, not a summary of it.
Reading a failure
Click a row to expand it. The full error text is there, along with the channel, the event kind, the attempt count, the latency and the timestamps.
Errors worth recognising:
| Error | Cause | Fix |
|---|---|---|
slack returned status 400: invalid_attachments | The message exceeded Slack's 3000-character limit for a section block | Now prevented at the transport — every outgoing message is trimmed to Slack's limits before sending |
slack returned status 404 | The incoming webhook was revoked | Re-create the webhook, update the channel, retry |
channel "X" is switched off | Somebody disabled it | Re-enable it, then retry |
the channel this was sent to no longer exists | It was deleted after the alert fired | Nothing to retry — re-route the rule |
context deadline exceeded | The channel did not answer in time | Usually transient; the automatic retries handle it |
Limits are enforced at the transport
Every Slack message is trimmed to Slack's documented limits immediately before it is sent: 3000 characters per section block, 2000 per field, ten fields, fifty blocks, and empty blocks removed. Telegram is capped at its own 4096-character limit the same way.
This is done once, at the client, rather than in each of the dozen places that build messages — because the message that overruns is always the one about the busiest incident, and that is exactly the one that must not be dropped.
Message builders still bound their own output so the cut lands somewhere sensible. A SQL alert rolling up many subjects trims at a line boundary and appends "…and N more", counted against the real total.
When a delivery fails, AccelerUp tells you
A failed delivery raises a system alert naming the channel, the event, the message title and the body size — so the next one can be diagnosed from the alert rather than from a database query.
Stuck in "Retrying"
A retry is claimed with a five-minute lease. If the worker dies mid-attempt, the row is picked up again once the lease expires — once, rather than being re-claimed every tick forever without ever counting an attempt.
If a row has been Retrying for much longer than that, the retry worker is not
running. Check the backend logs for notification retry worker started.
Where this behaviour lives: backend/internal/notification/, backend/internal/worker/notification_retry_worker.go, backend/internal/domain/notification_retry.go. If the code and this page disagree, the code is right — please fix the page.
Part of Incidents and on-call — Turning a reading into a page, and a page into a resolved incident.