Distributed Tracing for Microservices with OpenTelemetry

What distributed tracing is and when it answers questions logs and metrics cannot

A trace records the path of one request through a system. It is a tree of spans that share a single trace ID. Each span covers one unit of work, such as an inbound HTTP handler, a database query or an outbound call. A span carries its own span ID, its parent's span ID (empty for the root), start and end timestamps, attributes (key-value pairs such as the route or database system), timestamped events and a status. Since every span points to its parent, a backend can rebuild the whole call tree. That shows where the time went and which hop produced the first error.

Tracing is most useful for backend and platform engineers who run request paths that cross several services. In that setup, a p99 latency spike or a 500 at the edge could start in any of a dozen dependencies.

In a monolith, or with only two or three services, tracing often costs more than it gives back. The same is true if you can't instrument every hop, because one uninstrumented service splits the tree in two. In those cases, start with these cheaper options:

• Structured logs with a correlation ID that is passed between services
• RED dashboards that track rate, errors and duration per service
• Profiling for CPU or memory hot spots inside one process

How trace context propagates across service boundaries

A correlation ID only works if every service agrees on its format. W3C Trace Context makes that agreement explicit, so a Go service and a Java service can share a trace without coordinating with each other.

The traceparent header

A typical value looks like 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. It has four dash-separated fields:

• version (00): the format version.
• trace-id (32 hex chars): identifies the whole request across all services. It never changes along the path.
• parent-id (16 hex chars): the span ID of the caller's span. Each hop replaces it, which is how the backend rebuilds parent-child edges.
• trace-flags (01): bit 0 is the sampled flag. Downstream services read it so they keep or drop the same traces as upstream.

The optional tracestate header carries vendor-specific key-value pairs alongside traceparent without changing it.

Inject and extract

On the caller side, the configured propagator injects the active span context into outgoing headers. On the callee side, the propagator extracts that context, and the new server span becomes its child. HTTP and messaging instrumentation libraries usually do both steps for you.

Baggage is a separate header for application key-value pairs, such as tenant=acme. It is forwarded to every downstream hop, including third-party APIs, and it can end up in logs. Never put tokens, emails or other personal data in baggage.

Worked example

checkout calls payments over HTTP. payments then publishes to a queue that ledger consumes. If the producer writes traceparent into the message headers, the trace tree looks like this:

• checkout POST /order (root)
  • checkout HTTP client span
    • payments HTTP server span
      • payments publish payment.completed
        • ledger process payment.completed

For batch consumers, the processing span typically uses span links to each message's context instead of a single parent.

Copy# Producer (payments) injects trace context into message headers;
# consumer (ledger) extracts it so its span joins the same trace.
from opentelemetry import trace, propagate, baggage, context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import SpanKind

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("demo")
queue = []  # stand-in for a real broker

def publish(payload):
    # Only non-sensitive keys in baggage: it travels to every hop.
    token = context.attach(baggage.set_baggage("tenant", "acme"))
    try:
        with tracer.start_as_current_span("orders publish", kind=SpanKind.PRODUCER):
            headers = {}
            # Default propagators write 'traceparent' and 'baggage'.
            propagate.inject(headers)
            queue.append({"headers": headers, "body": payload})
            print("sent headers:", headers)
    finally:
        context.detach(token)

def consume():
    msg = queue.pop(0)
    # Rebuild the remote parent context from the message headers.
    ctx = propagate.extract(msg["headers"])
    with tracer.start_as_current_span(
        "orders process", context=ctx, kind=SpanKind.CONSUMER
    ) as span:
        # Same trace-id as the producer; parent is the producer span.
        print("trace_id:", format(span.get_span_context().trace_id, "032x"))
        print("tenant:", baggage.get_baggage("tenant", ctx))

publish({"order_id": "ord_123"})
consume()

A queue hop where the producer injects traceparent and baggage into message headers with propagate.inject and the consumer uses propagate.extract, so its CONSUMER span becomes a child of the PRODUCER span in the same trace.

Where propagation breaks: queues, async runtimes and header-stripping hops

Message queues and event buses. A broker does not carry your in-process context, so the producer must write it into the message itself. On publish, call the propagator's inject with the message's headers or attributes as the carrier, which writes traceparent and, if set, tracestate. On consume, call extract on the same carrier before starting the processing span. Use a child span when one message drives one unit of work and the consumer is logically part of the producer's request. Use span links when one span handles many messages, or when processing happens much later and a parent-child relationship would stretch a trace across hours.

Thread pools and async runtimes. Context lives in thread-local or task-local storage. When work moves to a pooled thread, a callback, or a detached task, that storage does not follow unless you capture the current context and reattach it on the other side. Most SDKs provide context-wrapping helpers for executors. Without them, child spans silently become new root spans.

Proxies, gateways, and legacy services. Any hop that rebuilds requests from an allowlist of headers will drop traceparent. The trace does not error. It splits into disconnected fragments, and nothing alerts you. Fixing it usually means changing configuration owned by another team, so treat propagation headers as part of your inter-service contract.

The edge. Do not trust incoming trace context from external clients. A forged traceparent can force sampling decisions or pollute your traces. Discard it at the boundary and start a new trace. If you need the client's ID for correlation, record it as a span link or attribute.

Copyfrom opentelemetry import trace, propagate
from opentelemetry.trace import SpanKind, Link

tracer = trace.get_tracer("orders.worker")

# `broker` and `process` are placeholders. Adapt the headers dict to your
# client's header format, for example a list of (key, bytes) tuples.

def publish(broker, body):
    with tracer.start_as_current_span("orders publish", kind=SpanKind.PRODUCER):
        headers = {}
        # Write the current context into the message: traceparent, plus tracestate if set
        propagate.inject(headers)
        broker.send(topic="orders", value=body, headers=headers)

def handle_one(msg):
    # One message drives one unit of work, so continue the producer's trace as a child
    ctx = propagate.extract(msg.headers)
    with tracer.start_as_current_span("orders process", context=ctx, kind=SpanKind.CONSUMER):
        process(msg.value)

def handle_batch(msgs):
    # One span handles many messages and no single producer is the parent.
    # Start a new trace and link it to every producer's span.
    links = []
    for m in msgs:
        producer_ctx = propagate.extract(m.headers)
        sc = trace.get_current_span(producer_ctx).get_span_context()
        if sc.is_valid:  # skip messages that arrived without trace headers
            links.append(Link(sc))
    with tracer.start_as_current_span("orders process batch", kind=SpanKind.CONSUMER, links=links):
        for m in msgs:
            process(m.value)

A producer injects trace context into message headers, and the consumer extracts it either as the parent of a single-message processing span or as span links on a batch span.

Implementing instrumentation: API vs SDK, auto vs manual

OpenTelemetry splits instrumentation into two layers, and keeping them separate prevents dependency conflicts. The API defines how code creates spans. It does nothing until something configures it. The SDK is that configuration: the tracer provider, span processors (batching, filtering), exporters, and resource attributes such as service.name.

Libraries should depend only on the API. If an HTTP client or shared internal package bundles the SDK, it forces exporter and sampling choices on every application that imports it. Applications own the SDK and configure it once at startup.

Start with zero-code auto-instrumentation to get request flow across services quickly. Then add manual spans at business-significant boundaries, such as "authorise payment" or "reserve inventory". Auto-instrumentation can't infer these, and they are what on-call engineers search for. Name attributes using semantic conventions so that dashboards and queries work across services. Conventions have been renamed over time, so check attribute names against the current specification at https://opentelemetry.io/docs/ and pin instrumentation package versions. Otherwise an upgrade can silently break your queries.

Never record PII, auth tokens or raw query parameters as attributes. Traces are often retained longer and shared more widely than application logs.

Copy# --- Application startup: the app owns the SDK and configures it once ---
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Resource attributes identify this service on every span it emits.
resource = Resource.create({
    "service.name": "checkout-service",
    "service.version": "1.4.2",
})

provider = TracerProvider(resource=resource)
# Batch spans in memory and send them to a local collector over OTLP/gRPC.
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="", insecure=True))
)
trace.set_tracer_provider(provider)

# --- Library or business code: depends only on the API (opentelemetry.trace) ---
tracer = trace.get_tracer("checkout.payments")

def authorise_payment(order_id: str, amount_minor: int, currency: str) -> None:
    # Manual span at a business boundary that auto-instrumentation cannot infer.
    with tracer.start_as_current_span("authorise payment") as span:
        # Record identifiers and low-risk context only. Do not record card numbers,
        # emails, auth tokens or raw query strings.
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.amount_minor", amount_minor)
        span.set_attribute("payment.currency", currency)
        # The outbound HTTP call to the payment provider is captured as a child span
        # by auto-instrumentation, if it is enabled for the HTTP client.
        ...

A Python application configures the SDK once with a tracer provider, batch span processor, OTLP exporter and service.name resource, while business code uses only the API to add a manual span with non-sensitive attributes.

Exporting through the OpenTelemetry Collector with OTLP

Once spans are created, they still have to reach a backend. Sending them straight from each service works, but it couples every deployment to one vendor's endpoint, credentials and retry behaviour. The OpenTelemetry Collector sits in between and receives, processes and forwards telemetry (docs).

A Collector pipeline has three stages:

• Receivers accept data. The OTLP receiver listens on gRPC (port 4317) and HTTP (port 4318) by default.
• Processors transform data in order. Put memory_limiter first so the Collector refuses data under pressure instead of crashing. Then add batch, which groups spans into fewer, larger export requests and reduces per-request overhead.
• Exporters send data onward, usually over OTLP to a backend or to another Collector.

Applications only ever export OTLP to a local or nearby address. Switching backends, adding a second destination, or redacting attributes centrally becomes a Collector config change. No services need to be rebuilt.

Many teams run both: agents forward to a gateway.

The cost is a new tier you must deploy, scale and monitor. An overloaded or misconfigured Collector drops spans without any error in your application. To catch this, scrape the Collector's own metrics and alert on:

• exporter queue size approaching capacity
• failed sends
• refused spans

Also expose a health check endpoint to your orchestrator.

Copy# Minimal Collector config: OTLP in, memory_limiter then batch, OTLP out.
extensions:
  health_check:
    endpoint: 0.0.0.0:13133      # liveness/readiness probe target for the orchestrator

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317   # apps export OTLP/gRPC here
      http:
        endpoint: 0.0.0.0:4318   # or OTLP/HTTP

processors:
  memory_limiter:                # first: refuse data under pressure instead of crashing
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  batch:                         # fewer, larger export requests
    send_batch_size: 512
    timeout: 5s

exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317  # backend or gateway; apps never see this
    headers:
      authorization: "Bearer ${env:BACKEND_API_KEY}"  # credential lives here, not in services
    sending_queue:
      enabled: true
      queue_size: 1000           # alert as queue size nears this capacity
    retry_on_failure:
      enabled: true

service:
  extensions: [health_check]
  telemetry:
    metrics:
      level: normal              # Collector's own metrics: queue size, failed sends, refused spans
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

A minimal OpenTelemetry Collector traces pipeline that receives OTLP, applies memory_limiter before batch, exports over OTLP with a bounded retry queue, and exposes a health check and internal metrics so dropped spans can be detected.

Sampling strategy and its impact on cost

Sampling decides which traces you keep. Where that decision happens determines what you can keep and what it costs to run.

Head-based sampling decides at the root span. A ParentBased sampler wrapping TraceIdRatioBased makes the root keep a fixed fraction of trace IDs. Every downstream service then follows the sampled flag in traceparent instead of deciding again, so traces stay complete. The decision is cheap, but it is made before anything has failed. A 1% rate drops 99% of your errors.

Tail-based sampling runs in a Collector gateway after a trace finishes. This means you can keep every error and every slow trace. For that to work, all spans of a trace must arrive at the same Collector instance. That requires a trace-ID-aware load-balancing tier in front of the gateway, plus buffering of in-flight traces until a decision window closes.

Illustrative estimate (assumed numbers, not a benchmark). Assume 500 requests/s, 20 spans per trace and 1 KB stored per span.

Daily storage = requests/s × spans per trace × 86,400 × span size × sample rate

• 100%: about 864 GB/day
• 10%: about 86 GB/day
• 1%: about 8.6 GB/day

Sampling reduces stored spans, not their size. High-cardinality attributes such as user IDs or full URLs make each span larger and indexes more expensive. A span created inside a tight loop multiplies spans per trace, and every one of them costs CPU and memory in the service before any sampler discards it.

Verifying the setup and linking traces to logs to save engineering time

…the data ever leaves the process. That cost is why you should prove the pipeline works on a small scale before rolling it out.

Verification checklist

• Export locally to the console. Point the SDK at a console exporter, or add the Collector's debug exporter to your pipeline. You can then see spans, attributes and parent IDs without depending on a backend.
• Confirm traceparent on outbound calls. Inspect a request at the receiving service, or in a local proxy. If the header is missing, the client library is not instrumented or the propagator is not configured.
• Hunt for orphaned root spans. A root span on a service that should never start a trace, such as a queue consumer or an internal API, means context was lost one hop upstream. Search for these regularly, not just at launch.

Linking traces to logs

Have your logging layer read the active span context and write trace_id and span_id as fields in every structured log record. During an incident, an engineer can start from a latency spike on a dashboard, open an exemplar trace, and filter logs by that trace_id. No one has to match timestamps across services by hand.

Weighing the payoff

Tracing pays off most when a failure crosses team boundaries. The trace shows which service added latency, so teams spend less time arguing about where the problem is. The cost is ongoing. Instrumentation drifts as code changes, new transports need propagation work, and attribute hygiene needs review. Budget for that maintenance explicitly.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed tracing, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.