Monitor/APM and traces

APM and traces

Where the time actually goes inside a request.

Monitor → APM shows traces: a request broken into spans, with timing for each.

Getting data in

Source	How
Browser	The web-monitor SDK posts spans to `/ingest/spans`
Backend services	Emit spans in the same shape

Reading a trace

A trace is one request. Spans are the operations inside it, nested by causality:

text

GET /checkout                                    812 ms
├─ auth.verify                                    14 ms
├─ cart.load                                      41 ms
│  └─ db.query cart_items                         38 ms
├─ pricing.calculate                             602 ms   ←
│  ├─ db.query price_rules                        22 ms
│  └─ http POST pricing-service                  571 ms   ←
└─ render                                        141 ms

The two marked spans are the answer. Everything else is noise on this request.

The endpoint view

Per endpoint: request count, error rate, and the p50/p95/p99 latency distribution.

Important

Watch p95 and p99, not the mean. A mean of 200 ms with a p99 of 9 seconds means one request in a hundred is unusable — and the people hitting it are disproportionately your heaviest users, because heavy accounts produce the slow queries.

Finding the slow one

Sort by p99, open the slowest trace, and look for the span that owns most of the duration. Almost always one of:

a query with no index
a serial loop of calls that could be one call
an external call with no timeout
a lock

Traces and errors

A trace that ended in an error links to its error group, and an error group links back to example traces. The error tells you what broke; the trace tells you what it was doing at the time.

Sampling

High-volume services sample. Errors and slow requests are kept preferentially — those are the ones worth having, and a uniform sample of a million healthy requests tells you nothing you did not already know from the aggregate.

Retention

Traces are telemetry and fall under RETENTION_TELEMETRY_DAYS (default 90 days). See Data retention.

Where this behaviour lives: frontend/src/app/apm/, backend/internal/handler/web_ingest_handler.go. If the code and this page disagree, the code is right — please fix the page.

Part of Monitor — Everything that watches something and reports what it saw.