How Micrometer, Micrometer Tracing, and the Observation API fit together
Three layers, each with a narrow job.
Micrometer is a metrics facade. Your code records to the MeterRegistry interface, and an implementation turns those measurements into a backend's format. For example, PrometheusMeterRegistry holds them in memory and renders them in the Prometheus exposition format when scraped. Swapping backends means swapping the registry, not rewriting instrumentation.
Micrometer Tracing does the same for spans. It defines a Tracer abstraction and relies on a bridge to a real tracer, such as OpenTelemetry or Brave, to create, propagate, and export spans. Application code depends on the facade, so the choice of tracer and exporter stays a dependency and configuration decision.
The Observation API sits above both. You wrap an operation in an Observation registered with an ObservationRegistry. Handlers attached to that registry react to its lifecycle (start, stop, error). A metrics handler records a timer, and a tracing handler opens and closes a span. One instrumented call yields both signals with the same name, the same timing boundaries, and the same error status. This removes a common source of drift: metrics and traces that describe the same operation with different names or tags.
The API also separates low-cardinality key values from high-cardinality ones. Low-cardinality values become metric tags and span attributes. High-cardinality values, such as an order ID, go only to the span, which keeps your Prometheus series count bounded.
This stack suits backend and platform engineers running Spring Boot services that need Prometheus metrics and distributed traces from a single, vendor-neutral instrumentation point.
+---------------------------------------------------------------+
| Application code wraps an operation in an Observation |
| key values: low-cardinality + high-cardinality |
+-------------------------------+-------------------------------+
|
v
+---------------------------------------------------------------+
| ObservationRegistry |
| lifecycle: start / stop / error -> notifies handlers |
+--------------+---------------------------------+--------------+
| |
v v
+-----------------------------+ +-----------------------------+
| Metrics handler | | Tracing handler |
| records a timer | | opens and closes a span |
| tags: low-cardinality only | | low + high card. key values |
+--------------+--------------+ +--------------+--------------+
| |
v v
+-----------------------------+ +-----------------------------+
| Micrometer: MeterRegistry | | Micrometer Tracing: Tracer |
| metrics facade | | tracing facade |
+--------------+--------------+ +--------------+--------------+
| |
v v
+-----------------------------+ +-----------------------------+
| PrometheusMeterRegistry | | Bridge: OpenTelemetry/Brave |
| Prometheus format on scrape | | creates/propagates/exports |
+-----------------------------+ +-----------------------------+
One Observation feeds two handlers: a metrics handler records a timer through Micrometer's MeterRegistry, and a tracing handler manages a span through Micrometer Tracing's bridge. High-cardinality values go only to the span.
Dependencies, Actuator exposure, and securing the scrape endpoint
Four pieces go on the classpath. Spring Boot Actuator provides the management endpoints and wires Micrometer into the application context. A Prometheus registry for Micrometer turns meters into the Prometheus text format and adds a scrape endpoint. A Micrometer Tracing bridge connects the Observation API to a concrete tracer; the OpenTelemetry bridge is the usual choice. An exporter, typically OTLP, ships finished spans to a collector. Artifact names, starter packaging and dependency management have changed across Boot releases, so check the Spring Boot reference documentation for your exact version rather than copying coordinates from older posts.
Expose only what you scrape
Actuator can expose far more than metrics. Endpoints such as env, configprops, heapdump, threaddump and loggers can reveal credentials, internal hostnames and memory contents. Some, like loggers, also accept writes that change runtime behavior. A blanket wildcard in management.endpoints.web.exposure.include publishes all of it over HTTP. List the endpoints you need, usually health and prometheus, and nothing else.
Keep the management surface off the public path
Set management.server.port to run Actuator on its own port. Bind that port to an internal interface or restrict it with network policy, so only your Prometheus scrapers and health probes can reach it. If a separate port isn't possible, put the endpoints behind Spring Security and give the scraper its own credential. Either way, a misrouted ingress rule won't hand your heap dump to the internet.
# application.yml: management settings only
management:
server:
# Run Actuator on its own port, separate from application traffic
port: 8081
# Bind to an internal interface so only in-cluster scrapers and probes reach it
address: 10.0.0.12
endpoints:
web:
exposure:
# Explicit allow-list; never use "*" here
include: health,prometheus
endpoint:
health:
# Hide component details (datasource, disk, etc.) from unauthenticated callers
show-details: never
tracing:
sampling:
# Sample 10% of requests; tune per environment
probability: 0.1
otlp:
tracing:
# Collector endpoint for the OTLP span exporter (placeholder host)
endpoint:
An application.yml that moves Actuator to an internal management port, exposes only the health and prometheus endpoints, and points the tracing exporter at an OTLP collector.
What auto-instrumentation gives you before you write code
Spring Boot instruments inbound HTTP traffic through the Observation API as soon as Actuator and a meter registry are on the classpath. Every Spring MVC or WebFlux request produces a timer named http.server.requests and, with a tracer bridge present, a server span. The timer is tagged with method, status, outcome, exception, and uri. The uri tag uses the route template (/orders/{id}), not the raw path, so a million order IDs collapse into one series instead of a million.
Outbound calls get the same treatment, but only when the client comes from a Spring-managed builder. If you inject RestTemplateBuilder, RestClient.Builder, or WebClient.Builder, the resulting client records http.client.requests and propagates trace headers. If you call new RestTemplate() or RestClient.create() directly, the client bypasses that auto-configuration and records no metrics or spans. This gap is common and easy to miss. Search the codebase for direct construction before assuming downstream calls are covered.
Before writing any custom instrumentation, inventory what already exists. /actuator/metrics lists every registered meter name. You can drill into one with tag filters, for example /actuator/metrics/http.server.requests?tag=uri:/orders/{id}. Then read /actuator/prometheus to see the exact series names and labels your queries will use. JVM memory, GC, thread pools, and connection-pool meters are usually already there.
Starting from the defaults saves engineering time. You avoid duplicating meters under different names, and any custom observations you add later fill real gaps rather than re-measuring request latency.
@Configuration
class ClientConfig {
// Instrumented: Boot's auto-configured RestClient.Builder already carries
// the ObservationRegistry, so every call records http.client.requests
// and propagates trace headers to the downstream service.
@Bean
RestClient inventoryClient(RestClient.Builder builder) {
return builder.baseUrl("").build();
}
}
@Service
class OrderService {
private final RestClient inventoryClient;
OrderService(RestClient inventoryClient) {
this.inventoryClient = inventoryClient;
}
StockLevel stockFor(String sku) {
// Pass a URI template plus variables, not a concatenated string,
// so the client "uri" tag stays low-cardinality: /stock/{sku}
return inventoryClient.get()
.uri("/stock/{sku}", sku)
.retrieve()
.body(StockLevel.class);
}
// Not instrumented: created outside the auto-configured builder, so it
// records no metrics, no spans, and sends no trace context downstream.
// RestClient untraced = RestClient.create("");
}
record StockLevel(String sku, int available) {}
A RestClient built from the injected RestClient.Builder gets http.client.requests timers and trace propagation automatically. A client from RestClient.create() silently gets neither.
Instrumenting business logic with the Observation API
Custom observations belong where auto-instrumentation can't see the business meaning: pricing a cart, reserving inventory, settling a payment. Wrap the work with Observation.createNotStarted(name, observationRegistry), attach key values, and call observe(...) with a lambda. The observation starts, runs the lambda, records any exception as an error, and stops. The registered handlers then produce a timer and a span from that single observation.
@Observed does the same job declaratively on a Spring bean method. It depends on AOP: an ObservedAspect bean must be registered and AspectJ must be on the classpath. Because it uses proxies, a call from one method to another within the same class bypasses it.
Each key value is either low or high cardinality, and the choice decides where it goes:
| Key value type | Becomes metric tag | Becomes span attribute | Safe values |
|---|---|---|---|
| Low cardinality | Yes | Yes | Bounded sets: payment.method, region, outcome |
| High cardinality | No | Yes | Unbounded: order ID, user ID, raw URI |
Every distinct combination of low-cardinality values creates a new time series in Prometheus. If you put an order ID on a low-cardinality key, each order gets its own series. Memory and storage then grow without limit. The API accepts any string and will not warn you. Review these keys the way you would review a database index.
Don't observe small hot-path methods such as validators, mappers, or cache lookups. Each observation allocates context, calls its handlers, and may create a span. That overhead can exceed the cost of the work being measured, and the extra spans clutter traces. Observe operations that a person would actually investigate.
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;
import io.micrometer.observation.annotation.Observed;
import org.springframework.stereotype.Service;
@Service
class PaymentService {
private final ObservationRegistry registry;
private final PaymentGateway gateway;
PaymentService(ObservationRegistry registry, PaymentGateway gateway) {
this.registry = registry;
this.gateway = gateway;
}
Receipt settle(String orderId, String paymentMethod) {
return Observation.createNotStarted("payment.settle", registry)
// Bounded set of values: becomes a metric tag AND a span attribute
.lowCardinalityKeyValue("payment.method", paymentMethod)
// Unbounded value: span attribute only, so no new time series per order
.highCardinalityKeyValue("order.id", orderId)
// Starts, runs the lambda, records any exception as an error, stops
.observe(() -> gateway.charge(orderId, paymentMethod));
}
}
@Service
class InventoryService {
// Declarative form. Requires an ObservedAspect bean and AspectJ on the
// classpath; calls from another method in this same class bypass the proxy.
@Observed(name = "inventory.reserve",
lowCardinalityKeyValues = {"warehouse.region", "eu-west"})
Reservation reserve(String sku, int quantity) {
return doReserve(sku, quantity);
}
// Small hot-path helper: intentionally NOT observed
private Reservation doReserve(String sku, int quantity) { /* ... */ return null; }
}
Wrapping a payment settlement with Observation.observe and an inventory reservation with @Observed, keeping the bounded payment method as a low-cardinality key and the unbounded order ID as a high-cardinality key.
Context propagation, log correlation, and sampling
Trace context travels in request headers, by default the W3C traceparent header. Spring injects it only into clients it instruments. That means RestTemplate, RestClient, or WebClient instances built from the auto-configured builders. A client created with new sends no header, and the downstream service starts a new, unrelated trace.
Async boundaries break in a similar way. Context lives in thread-locals, so a task submitted to a plain ExecutorService runs without it. Wrap custom executors with the context-propagation library's wrappers so context is captured at submit time and restored on the worker thread. Reactor pipelines hop threads freely and need automatic context propagation enabled in Reactor's hooks. Messaging needs observation turned on for both the producer template and the listener container, so the header is written into the message and read back out.
To confirm propagation, send one request to service A, which calls service B. Then check that both services log the same trace ID. If B logs a different ID, the hop is broken.
Make that check routine by adding %X{traceId} and %X{spanId} to the logging pattern. With the IDs in every line, a stack trace leads directly to its trace.
Set management.tracing.sampling.probability per environment. Use 1.0 in development and something lower in production. Sampling discards spans only. Metrics still count every request, so dashboards and alerts stay accurate. The cost of a low probability is that the one failing request you need may never have been traced. At 1.0, every span is exported, sent over the network, and stored by the backend, and that cost grows with traffic.
Prometheus queries and choosing histograms deliberately
Three queries cover most dashboards for the default http.server.requests timer. Request rate per endpoint is sum by (uri) (rate(http_server_requests_seconds_count[5m])). Error ratio divides the 5xx rate by the total rate: sum(rate(http_server_requests_seconds_count{outcome="SERVER_ERROR"}[5m])) / sum(rate(http_server_requests_seconds_count[5m])). Latency at p99 comes from buckets: histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket[5m]))). The _bucket series exist only if you enable a percentile histogram for that meter, for example management.metrics.distribution.percentiles-histogram.http.server.requests=true. Enable it per meter rather than globally, because each one multiplies your series count.
| Approach | Property | Aggregates across instances | Precision |
|---|---|---|---|
| Client-side percentiles | percentiles |
No. Averaging p99s is meaningless. | Exact per instance |
| Server-side buckets | percentiles-histogram |
Yes, via sum by (le) |
Approximate within bucket bounds |
Client-side percentiles are computed in the JVM, so you cannot combine one pod's p99 with another's. Buckets can be summed, but histogram_quantile interpolates within a bucket, so the answer is only as precise as the bucket boundaries.
Illustrative estimate. Assume 50 uri/method/status combinations, 60 buckets, and 10 instances. Without a histogram, each combination exports three series (count, sum, max), which gives 1,500 series. With buckets, each combination exports about 63 series, which gives roughly 31,500. That is about a 20× increase in scrape payload and in active series. You can reduce it by setting minimum-expected-value and maximum-expected-value to narrow the bucket range, or by defining explicit slo boundaries.
When to avoid it, alternatives, and impact on cost and engineering time
This stack pays off when several Spring services call each other and you already run a metrics store and a trace backend. It fits poorly in three cases:
- A single small service, where Actuator health checks and a few counters answer every question you ask.
- A polyglot estate where most services are not on the JVM.
- A team with nowhere to send spans. Instrumentation without a backend costs effort and returns nothing.
| Approach | Strength | Trade-off |
|---|---|---|
| Observation API + Micrometer Tracing | One instrumentation point produces both metrics and spans | You debug a facade plus a bridge |
| OpenTelemetry API or Java agent directly | One model across languages; the agent needs no code changes | Weaker Spring-native integration; agent behaviour is harder to inspect |
| Metrics only | Cheap and simple to operate | Shows that latency rose, not which downstream call caused it |
| Logs only | No extra infrastructure | You reconstruct cross-service requests by hand |
Most of the development cost is configuration, not code. You need to:
- Keep Micrometer, Micrometer Tracing and the bridge on aligned versions. Let Spring Boot's dependency management handle this instead of pinning each one.
- Put exactly one bridge on the classpath.
- Point the exporter at the right endpoint.
- Confirm the backend actually receives data.
When spans go missing, the cause can be in any of those four layers.
The return comes from moving through one connected path. You start at a slow-request metric, follow an exemplar to its trace, then read logs that carry the same trace ID. Without that path, you grep timestamps across services.
Illustrative estimate (assumed numbers, not a benchmark): hours saved per month = incidents × (minutes uncorrelated − minutes correlated) ÷ 60. Assuming 6 incidents per month, 90 minutes each without correlation and 30 minutes with it: 6 × (90 − 30) ÷ 60 = 6 engineer-hours per month. Weigh that against your one-off setup effort and the ongoing backend cost.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to spring boot micrometer tracing, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.