Traces

Each trace is one request or operation followed across every service it touched. Traces arrive over OTLP; see OpenTelemetry, Prometheus & Tracing for how to send them.

Finding the right tracelink

Filter by service, operation, duration or error status. Sorting by duration surfaces the slow tail, which is usually more informative than the average. Averages hide the requests users actually complain about.

From an error to a tracelink

Errors and logs that carry a trace id link straight to the trace. That path, issue then trace then the exact span, is normally the fastest route to a root cause.

Traces only appear if your services propagate trace context between each other. A service that does not forward the incoming headers shows up as a separate trace instead of part of the request.

Reading one tracelink

Each bar is a span. Nesting shows causation: a child span was started by its parent. Width is duration, so the widest bar with no children of its own is usually the real cost.

Common shapes worth recognising:

• One wide leaf span - a slow query or slow downstream call.
• Many identical narrow spans in sequence - an N+1 pattern.
• A gap with no spans - time spent somewhere that is not instrumented.