Agent rollout
Upgrading the fleet in waves — canary first, and a wave passes only on reconnect at the new version.
Fleet → Agent rollout upgrades agents in controlled waves.
Why not all at once
An agent upgrade that fails takes monitoring with it. Upgrading sixty servers simultaneously with a bad build means sixty blind servers and no alerting to tell you.
Waves
canary → wave 1 → wave 2 → … → the restEach wave must pass before the next begins.
The canary is an unsupervised box
Deliberately: the first machine to take a new version should be one where a failure costs least. Not the busiest, not the most critical, and not one somebody is actively working on.
A wave passes only on reconnect
This is the important rule.
A wave is judged by agents reconnecting at the new version — not by the upgrade command returning successfully.
An agent that upgrades and then fails to start has run a successful command and is dead. The only evidence that an upgrade worked is the agent coming back and saying which version it is.
Halting
A wave that fails halts the rollout. Remaining waves do not start.
A halted rollout is not rolled back. Agents already upgraded stay upgraded.
Automatically downgrading a fleet on one failure is a second mass change made under exactly the conditions where you understand least. The rollout stops, tells you, and waits for a decision.
Resuming
After fixing the cause, resume from the failed wave. Already-upgraded agents are skipped — the rollout knows which version each agent reported.
Watching one
The rollout page shows, per wave: agents targeted, upgraded, reconnected, failed, and still pending. "Upgraded but not reconnected" is called out separately, because that is what a failed start looks like from here.
Before you start
- Check Coverage — a machine that is already not reporting will not reconnect, and will be counted as a failure
- Avoid a deploy freeze window
- Do not run one during an incident. Upgrading monitoring while relying on it is a bad trade
Where this behaviour lives: backend/internal/domain/agent_rollout.go, frontend/src/app/agents/rollout/. If the code and this page disagree, the code is right — please fix the page.
Part of Fleet — The agent on each server: what it collects and what it may be asked to do.