A/B tests
Experiments with confidence intervals, p-values, guardrails and automatic ramping.
Feature Flags → A/B Tests runs experiments on top of flag variants.
Setting one up
| Field | Notes |
|---|---|
| Flag | The variants being tested |
| Primary metric | The one thing you are optimising |
| Guardrail metrics | Things that must not get worse |
| Audience | Who is eligible |
| Duration | Minimum, before you are allowed to look |
Pick one primary metric
One. An experiment with five "primary" metrics will show a significant result on at least one by chance, and the temptation to declare that one the answer is very hard to resist after two weeks of waiting.
The others are guardrails: things that must not get worse, not things you are trying to improve.
Statistics
Results carry a confidence interval and a p-value, not just a percentage difference.
| Reading | Means |
|---|---|
| +3.2% [−0.4%, +6.8%] | Might be +7%, might be slightly negative. Not a result |
| +3.2% [+1.9%, +4.5%] | Confidently positive |
Do not stop an experiment the first time it looks significant. Peeking and stopping on a good result inflates the false-positive rate substantially — this is the single most common way experiments produce wrong answers. Set the duration in advance and honour it.
Guardrails and automatic ramping
Guardrail metrics are monitored throughout. A breach stops the ramp, and can roll the experiment back.
Automatic ramping increases exposure on a schedule while guardrails hold. It turns a manual ramp — the one people forget to continue — into something that happens on its own and stops itself when it should.
correlate_window
An experiment can be correlated with operational data over its window: error rates, latency, incidents. A variant that improves conversion by 2% and adds 300 ms to checkout is not the win the primary metric claims.
Reading results
Wait for the planned duration. Then:
- Did the primary metric move, with a confidence interval that excludes zero?
- Did any guardrail get worse?
- Does the effect hold across segments, or is it one country?
Related
Where this behaviour lives: frontend/src/app/ab-tests/, backend/internal/service/feature_flag_svc.go. If the code and this page disagree, the code is right — please fix the page.
Part of Feature flags — Shipping code dark, then turning it on for a percentage of people.