When a user request flows through a dozen microservices, each adding its own latency and potential failure points, understanding what actually happened becomes a forensic exercise. Distributed tracing solves this by stitching together a coherent narrative from fragments of execution scattered across processes, hosts, and availability zones.

The core challenge is deceptively simple: maintain a shared context as execution hops between services. In a monolith, a stack trace gives you the full picture. In distributed systems, you need to explicitly thread context through every network call, message queue, and async boundary.

Trace Context and Propagation

Every traced operation gets assigned a trace ID when it enters the system, typically at an edge service or API gateway. This identifier stays constant for the entire request lifetime. Individual operations within that request become spans, each with its own span ID and a pointer to its parent span.

The trick is propagation. When Service A calls Service B, it must inject the trace context into the outgoing request. This happens through HTTP headers, message metadata, or RPC context fields. Service B extracts this context, creates a child span under the same trace ID, and continues the chain. Without careful instrumentation at every service boundary, the trace breaks and you lose continuity.

Standards like W3C Trace Context define how to serialize this information. A typical implementation includes a trace ID, parent span ID, and sampling decision packed into headers that traverse with the request. The sampling flag is particularly important because it determines whether downstream services should continue recording detailed span data or just pass context along without overhead.

Instrumentation and Auto-Discovery

Manual instrumentation requires developers to wrap every significant operation in tracing code. This is brittle and incomplete. Modern tracing systems lean heavily on automatic instrumentation through library hooks, bytecode manipulation, or eBPF probes.

An HTTP framework can be instrumented once to automatically create spans for all incoming requests and propagate context on all outgoing calls. Database drivers can create spans for queries. Message queue clients can extract context from message headers and inject it when publishing. This coverage reduces the chance that a critical hop goes untraced.

The challenge is knowing what to trace. Not every function call deserves a span. Too much detail creates overwhelming data volume and obscures the meaningful structure. Good instrumentation focuses on service boundaries, external calls, and semantically significant operations like cache lookups or authorization checks.

Sampling Strategies

Recording every span for every request doesn’t scale. At meaningful traffic volumes, the data and processing costs become prohibitive. Sampling is unavoidable, but naive approaches like “keep 1% of traces” create gaps in coverage.

Head-based sampling makes the decision at trace creation. The edge service decides whether to trace this request and communicates that decision downstream. This is simple and avoids wasted work, but it can miss rare, important events. If only 1% of traces are kept, an error that occurs once in 500 requests likely won’t be captured.

Tail-based sampling defers the decision until after execution completes. Services provisionally record all spans but hold them in a buffer. A central collector examines completed traces and decides what to keep based on error status, latency, or other signals. This captures the interesting traces but requires infrastructure to buffer and coordinate sampling decisions across distributed agents.

Adaptive sampling adjusts rates based on traffic patterns. High-volume endpoints might be sampled at 0.1% while rare code paths get 100% coverage. This balances data volume with visibility where it matters.

Storage and Query Challenges

Traces arrive as streams of individual spans that must be reassembled. A trace with 50 spans might arrive out of order from 20 different services. The backend needs to index spans by trace ID for retrieval and by timestamp for time-range queries.

Columnar storage works well here because queries typically filter on a few fields like service name, error status, or duration. Compressed columnar formats provide good density for the write-heavy workload tracing generates.

The real query challenge is aggregation. Finding slow traces is straightforward, but questions like “which service added the most latency to checkout flows last hour” require joining span data, grouping by service, and aggregating durations. This pushes tracing backends toward specialized time-series databases or analytics engines rather than general-purpose stores.

The Observability Stack

Distributed tracing sits between metrics and logs. Metrics tell you that latency increased, logs tell you what a specific service logged, but traces show you the end-to-end flow. The integration matters: jumping from a high error rate metric to example traces to specific log entries creates the full debugging path.

The overhead tradeoff is real. Even sampled tracing adds latency to service calls and background CPU cost for span creation and export. The value proposition depends on system complexity—in architectures with dozens of interacting services, the visibility is indispensable. In simpler systems, structured logging might suffice.