Probes
Checks run from a server, measuring the path your traffic actually takes.
A probe is a check the agent runs from a server.
Why this matters
A service check runs from the monitoring system. A probe runs from your application server.
They frequently disagree, and when they do the disagreement is the finding: the monitoring system reaching a service says nothing about whether the machine that actually depends on it can.
What a probe can check
| Kind | Checks |
|---|---|
| TCP | Can this server open a connection to that host and port |
| HTTP | Status, latency, body match |
| DNS | Does this name resolve, from this resolver |
| Ping | Reachability, latency, loss |
Defining one
Fleet → Probes. Give it a target, an interval and which servers run it.
Probes assigned to many servers become a reachability matrix — which server can reach which.
Probes keep running through a disconnect
A probe list already received keeps being checked while the agent is disconnected from the control plane. Results are spooled and delivered when it reconnects.
This is deliberate. A network problem is exactly when you want the probes running, and it is also when the agent is least likely to be connected. A probe schedule that stopped on disconnect would go blind at precisely the moment it was needed.
Alerting
Probe results feed rules: reachability lost, latency above a threshold, loss above a percentage.
Alert on change rather than on an absolute. A path that is normally 90 ms is fine; one that was 14 ms yesterday is not.
Related
- Network — the aggregated view
- Reachability — the agent-to-agent matrix
Where this behaviour lives: backend/cmd/accelerated/, backend/internal/integration/netprobe.go. If the code and this page disagree, the code is right — please fix the page.
Part of Fleet — The agent on each server: what it collects and what it may be asked to do.