Databases (MySQL)
Node health, replication lag, GTID gaps and configuration drift.
Monitor → Databases tracks MySQL nodes: their role, their health, and whether the replicas are keeping up.
Node roles
| Role | Meaning |
|---|---|
writer | Takes writes |
replica | Follows the writer |
either | Can take either role |
Roles are recorded because the same reading means different things in each. A replica that is not replicating is an incident; a writer that is not replicating is normal.
Replication
| Measured | Notes |
|---|---|
| Seconds behind | The classic measure, and the one that lies during a stall |
| GTID gap | Which transactions the replica has not got |
| IO / SQL thread state | Whether replication is even running |
| Last error | errno and message |
"Seconds behind" reads 0 when replication has stopped, because there is
nothing in flight to be behind on. The GTID gap does not have that failure
mode, which is why both are tracked.
A one-directional gap is ordinary lag
A replica missing transactions the writer has is lag. The writer missing transactions the replica has is something else entirely — a split brain, or a replica that was written to directly. These are distinguished rather than collapsed into one number.
Configuration drift
Node configuration is compared across the set. A differing binlog_format is
reported as critical: it is the kind of difference that works perfectly
until a failover, at which point it does not.
Drift is reported only where there is something to compare — a single node has no drift, and one node missing a variable is not drift, it is one-sided data.
Runtime-only differences are called out specifically. A value changed at runtime but not in the config file reverts on the next restart, which means the box is one reboot away from behaving differently.
Silence and staleness
A node that stops reporting produces a different alert from one that is unreachable:
| Alert | Cause |
|---|---|
| "has stopped reporting" | The collector is not sending. The node may be fine |
| "is unreachable" | The connection failed. The node is probably not fine |
A sample older than ten minutes is treated as stale rather than current — judging a ten-minute-old reading as current is how a node that died at 03:00 looks healthy until somebody opens the page.
Alerting
Rules on lag, GTID gap, thread state, connection count, slow queries and drift. The usual alerting model applies.
Alert on the GTID gap and on thread state. Lag alone misses a stopped replica; thread state alone misses one that is running and falling behind.
Related
- Database pool (ProxySQL) — the layer in front
- SQL alerts — alerting on the contents rather than the health
Where this behaviour lives: backend/internal/integration/mysqlnode.go, backend/internal/worker/mysqlnode_alert_worker.go. If the code and this page disagree, the code is right — please fix the page.
Part of Monitor — Everything that watches something and reports what it saw.