Chaos experiments
Breaking things deliberately, in a bounded way, while you are watching.
Delivery → Chaos runs controlled failure experiments.
Why
Every system has failure modes nobody has tested. The choice is whether you find them during business hours with the team watching, or at 3 a.m. with one person and no context.
An experiment
| Field | Notes |
|---|---|
| Hypothesis | What you expect to happen. Write it first |
| Blast radius | What is in scope. Keep it small |
| Method | What is broken, and how |
| Abort conditions | What makes you stop immediately |
| Duration | Bounded |
Write the hypothesis before you run it. Without one you are not experimenting, you are breaking something and seeing what happens — and you will rationalise whatever you observe as expected.
Running one
Experiments are recorded like deployments: what ran, when, what the observed effect was, and whether the hypothesis held.
They appear on the timeline, so an alert during an experiment is obviously related rather than mysterious.
Abort
Every experiment has abort conditions and an abort button. The conditions should be specific — "error rate above 5%", not "if it looks bad".
Before your first one
- Start in staging. Not production, whatever anybody says about realism.
- Tell people. A surprise chaos experiment is an incident with extra steps.
- Make sure monitoring covers what you are about to break. An experiment you cannot observe teaches you nothing.
- Have a rollback that does not depend on the thing you broke.
What to test first
The things you assume work and have never verified:
- a replica failover
- losing one availability zone
- a dependency timing out rather than erroring
- the disk filling
- the certificate expiring
The fourth and fifth are the ones that catch most teams, because both fail in a way that looks like something else.
Afterwards
Write it up like a post-mortem, even when it went well. An experiment that confirmed the hypothesis is evidence worth keeping — particularly the next time somebody asks whether failover works.
Where this behaviour lives: frontend/src/app/chaos/. If the code and this page disagree, the code is right — please fix the page.
Part of Delivery — Deployments, pipelines, releases and everything that ships code.