How to Reduce Observability Costs: Logs, Cardinality and Sampling

What drives observability cost: a three-multiplier model

Observability spend is driven by three separate multipliers. Each grows for different reasons, so each needs its own control.

• Logs: bytes ingested per day, multiplied by days retained. Ingest and storage are often priced separately, but both scale with volume.
• Metrics: active time series. One metric's series count is the product of its label value counts, so a single high-cardinality label multiplies everything else.
• Traces: spans ingested and stored. Span volume equals request rate times spans per request, unless you sample.

Illustrative estimate (assumed numbers, not a benchmark). The unit prices below are placeholders. Substitute your own.

With these inputs, unsampled traces dominate, and log ingest costs far more than log storage. Change one input, such as adding a user_id label, and the ranking flips. Cutting every signal by the same percentage ignores that structure. It removes data you need from cheap terms while leaving the expensive term barely changed.

This approach suits backend and platform engineers who own both the telemetry pipeline and the bill that comes with it.

Find the top contributors before cutting anything

Measure before cutting. A short ranking exercise usually shows that a few sources produce most of the bill, and it tells you which multiplier (volume, cardinality or retention) to reduce first.

Logs: group by service, logger and level

Aggregate ingested bytes, not line counts, over a representative window such as the last seven days. Group by service, then by logger and level within the top services. Bytes matter because one verbose logger that dumps request bodies can outweigh thousands of short lines. If the top rows are DEBUG or INFO from a single logger, the problem is volume. If the volume is spread evenly but stored for a long time, retention is the bigger lever.

Metrics: series per name, then per label

In Prometheus, topk(20, count by (__name__) ({__name__=~".+"})) ranks metric names by active series. For each of the top names, find which label is responsible by counting its distinct values, for example count(count by (path) (http_requests_total)). A label with thousands of values, such as a raw URL path, user ID or pod name on a long-lived metric, is a cardinality problem. Dropping the label usually saves far more than dropping the whole metric.

Traces: spans per service and operation

Count spans grouped by service name and span name. High-throughput operations such as health checks or cache reads often dominate, and these are good candidates for sampling.

Look at the three rankings together before changing anything.

Copy#!/usr/bin/env bash
# Rank the biggest log and metric contributors before changing anything.
# Requires: curl, jq (1.6+), awk. Point PROM at your Prometheus server.
set -euo pipefail
PROM="${PROM:-"
LOG_DIR="${LOG_DIR:-./logs-last-7d}"   # NDJSON log export, one JSON object per line

echo "== Logs: ingested bytes by service / logger / level (top 20) =="
# Sum bytes, not lines: one logger dumping request bodies outweighs many short lines.
cat "$LOG_DIR"/*.ndjson \
  | jq -r '[.service // "unknown", .logger // "unknown", .level // "unknown",
            (tojson | utf8bytelength)] | @tsv' \
  | awk -F'\t' '{ b[$1"\t"$2"\t"$3] += $4 }
      END { for (k in b) printf "%12d\t%s\n", b[k], k }' \
  | sort -rn | head -20

echo "== Metrics: active series per metric name (top 20) =="
curl -sG "$PROM/api/v1/query" \
  --data-urlencode 'query=topk(20, count by (__name__) ({__name__=~".+"}))' \
  | jq -r '.data.result[] | "\(.value[1])\t\(.metric.__name__)"' \
  | sort -rn

echo "== Metrics: distinct values of one suspect label on one metric =="
# Swap in a top metric name and a label you suspect (path, user_id, pod...).
curl -sG "$PROM/api/v1/query" \
  --data-urlencode 'query=count(count by (path) (http_requests_total))' \
  | jq -r '.data.result[0].value[1] // "0"'

echo "== Metrics: labels with the most distinct values (TSDB head stats) =="
curl -s "$PROM/api/v1/status/tsdb" \
  | jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"'

A script that ranks log bytes by service, logger and level from an NDJSON export, then uses the Prometheus HTTP API to rank metric names by series count and find the labels with the most distinct values.

Reducing log volume and setting tiered log retention

Start with the cheapest and safest control: stop emitting logs nobody reads. Debug output left on in production and per-request "200 OK" lines are often a large share of volume, and they duplicate information your metrics and traces already carry. Lower their level or remove them at the source, before they incur ingestion and indexing charges.

Next, deduplicate high-frequency identical events. A retry loop that logs the same message thousands of times per minute can be collapsed into one line with a count and a time window. You lose no information, only repetition.

Sample only low-value categories, such as verbose informational logs from stable services. Never sample errors, warnings, or audit and security events. Those are the records you need for rare incidents and investigations, and a sampled-away line cannot be recovered.

If you only ever count or chart a log line, replace it with a counter or histogram. A metric costs roughly the same whether traffic is ten requests or ten million.

Then set retention per data class:

Keep the hot window short and move older data to cold storage, which can be queried or rehydrated on demand. Two limits apply. Legal and compliance requirements set minimum retention periods, and those periods override cost targets. Rehydration also adds delay, so do not tier data that on-call engineers routinely query during incidents.

Copyimport logging, random, time

class CostFilter(logging.Filter):
    """Drop debug, collapse repeats, sample low-value INFO. Never touches WARNING+ or audit."""

    def __init__(self, info_sample_rate=0.1, dedupe_window_s=60):
        super().__init__()
        self.rate = info_sample_rate
        self.window = dedupe_window_s
        self.seen = {}  # (logger, message) -> (window_start, suppressed_count); evict old keys in production

    def filter(self, record):
        # Errors, warnings and audit/security events always pass, unsampled.
        if record.levelno >= logging.WARNING or getattr(record, "audit", False):
            return True
        # Debug never leaves the process in production.
        if record.levelno < logging.INFO:
            return False
        key = (record.name, record.getMessage())
        now = time.monotonic()
        start, count = self.seen.get(key, (None, 0))
        if start is not None and now - start < self.window:
            self.seen[key] = (start, count + 1)  # identical event inside window: count it, don't emit
            return False
        self.seen[key] = (now, 0)
        if count:
            # Emit one summary line so the repetition count is not lost.
            record.msg = f"{record.getMessage()} (repeated {count}x in previous {self.window}s)"
            record.args = ()
            return True
        # Verbose INFO from stable code paths is sampled.
        return random.random() < self.rate

handler = logging.StreamHandler()
handler.addFilter(CostFilter(info_sample_rate=0.1, dedupe_window_s=60))
log = logging.getLogger("checkout")
log.addHandler(handler)
log.setLevel(logging.DEBUG)

log.info("cache refreshed")                         # sampled at 10%
log.error("payment declined")                        # always kept
log.info("role changed", extra={"audit": True})      # always kept

A Python logging filter that drops debug output, collapses repeated identical messages into a counted summary, and samples routine INFO logs. Warnings, errors and audit-tagged events always pass through.

Controlling metric cardinality

A metric's cost is driven by its number of time series, not its number of samples. Every unique combination of label values creates a separate series with its own index entry and storage. Labels such as user ID, request ID, full URL or raw error message have no upper bound, so each new value adds new series. Because label counts multiply, one unbounded label can outweigh everything else on the metric.

Illustrative estimate: suppose a request counter has 5 methods, 10 status codes and raw paths that include IDs, producing about 20,000 distinct paths. That allows up to 5 × 10 × 20,000 = 1,000,000 series. If you normalize the paths to 40 route templates such as /orders/{id}, the ceiling drops to 5 × 10 × 40 = 2,000.

There are four fixes:

• Normalize at the source. Emit the route template, not the raw path.
• Bucket values. Store latency in histogram buckets and status codes as 2xx or 5xx classes.
• Drop labels at ingest. Use relabeling, for example Prometheus metric_relabel_configs with a labeldrop action, to remove known offenders before they are stored.
• Pre-aggregate. Use recording rules to keep dashboard-level rollups. You can then retain raw series for a shorter period.

Each fix has a cost:

• Once a dimension is dropped, you can no longer slice by it.
• Per-tenant or per-user debugging moves to logs or traces. That shifts cost to another system instead of removing it.
• A recording rule only answers the question it was written for. An unanticipated question during an incident still needs raw data.

Copy# prometheus.yml (scrape config excerpt)
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api-1:9100"]
    metric_relabel_configs:
      # Drop unbounded labels before the series is stored.
      - action: labeldrop
        regex: "user_id|request_id"
      # Collapse raw paths into a route template (Prometheus anchors the regex).
      - source_labels: [path]
        regex: "/orders/[0-9]+"
        target_label: path
        replacement: "/orders/{id}"
        action: replace
      # Bucket status codes into classes such as 2xx or 5xx.
      - source_labels: [status]
        regex: "([1-5]).."
        target_label: status_class
        replacement: "${1}xx"
        action: replace
      - action: labeldrop
        regex: "status"
---
# rules/http_rollups.yml (loaded via rule_files)
groups:
  - name: http_rollups
    rules:
      # Dashboard-level rollup. It answers this one question only, so keep
      # the raw series around long enough to handle incident queries.
      - record: job_path:http_requests:rate5m
        expr: sum by (job, method, path, status_class) (rate(http_requests_total[5m]))

A Prometheus scrape config drops user and request ID labels, rewrites order paths to a route template and buckets status codes at ingest, and a separate recording-rule file pre-aggregates request rates for dashboards.

Trace sampling strategies and how they work

Sampling decides which traces you pay to store. The decision can happen at trace creation or after the trace completes, and that timing sets both what the strategy can see and what it costs to run.

Head-based probabilistic sampling

The root span decides at creation, for example keep 5%, and the decision travels downstream in the trace context so every service agrees. It costs almost nothing: no buffering and no extra infrastructure. The weakness is that it decides before anything has happened. A trace for a rare 0.1% failure is dropped at the same rate as a healthy one, so at 5% you lose most of those failures.

Tail-based sampling

Collectors buffer every span of a trace for a decision window, often several seconds, and then apply rules such as "keep errors, keep anything over 2 s, keep 2% of the rest." This only works if every span with the same trace ID reaches the same collector, so you need a routing tier in front:

Copy[SDKs, 100% export] ──► [load-balancing collectors] ──hash(trace ID)──► [tail-sampling collector A]
                                                                      ├► [tail-sampling collector B]
                                                                      └► [tail-sampling collector C] ──► backend

Buffer memory grows with span throughput multiplied by window length. When the tail tier scales up or down, the hash ring changes, and traces in flight get split across collectors. Spans that arrive after their trace's decision has been made are evaluated without the rest of the trace.

Rate-limited and per-operation sampling

These cap traces per second, either globally or per endpoint. A health check or a chatty polling route then can't take over the budget, and low-traffic operations still get enough samples to be useful.

Copy  ┌────────────┐   ┌────────────┐   ┌────────────┐
  │ Service A  │   │ Service B  │   │ Service C  │
  │    SDK     │   │    SDK     │   │    SDK     │
  └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
        │ 100% of spans  │                │
        └────────────────┼────────────────┘
                         ▼
  ┌──────────────────────────────────────────────┐
  │        Load-balancing collector tier         │   scaling tail tier
  │        route spans by hash(trace ID)         │   reshuffles the ring;
  │        same trace ID → same tail node        │   in-flight traces split
  └─────┬────────────────┬────────────────┬─────┘
        │                │                │
        ▼                ▼                ▼
  ┌────────────┐   ┌────────────┐   ┌────────────┐
  │Tail-sampler│   │Tail-sampler│   │Tail-sampler│   decision per trace:
  │collector A │   │collector B │   │collector C │   - keep errors
  │buffer spans│   │buffer spans│   │buffer spans│   - keep latency > 2 s
  │(N s window)│   │(N s window)│   │(N s window)│   - keep 2% of the rest
  └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
        │ kept traces    │                │
        └────────────────┼────────────────┘
                         ▼
                  ┌────────────┐
                  │  Backend   │
                  │ kept traces│
                  └────────────┘

Tail-based sampling: SDKs export every span, a load-balancing tier hashes on trace ID so each trace lands on one tail-sampling collector, and that collector buffers the trace for a decision window before forwarding only the kept traces to the backend.

Keeping sampled and filtered data trustworthy

Sampling only saves money if the remaining data can still answer questions accurately. Three habits make that possible.

Record the sample rate on every kept item. Attach the effective rate as an attribute on each span or log line. At 1-in-20 sampling, each kept span represents 20 requests, so queries can multiply counts by the inverse of the rate. Without this field, a sampling change looks like a traffic drop, and nobody can tell the two apart afterward.

Propagate the decision, not just the trace ID. The sampled flag in the W3C traceparent header lets downstream services follow the upstream choice. If each service decides on its own, you get fragmented traces: the checkout span is kept but the payment span it called is gone.

Compute RED metrics before sampling. Rate, errors and duration should come from every request, either in the application or at the first collector hop. Dashboards and SLO alerts then reflect real traffic, and sampled traces are used only for investigation.

Watch the pipeline itself

Filtering rules spread across agents and collectors drift apart over time. A badly written drop rule fails silently: no error appears, and the data simply never shows up. Treat this configuration like code. Keep it in version control, require review, and roll it out from one place.

Then monitor the pipeline as a production system:

• Export counters for items received, dropped, filtered and failed at each stage.
• Alert when ingest volume for a service moves sharply away from its recent baseline, in either direction.

A sudden 40% drop in volume is as likely to be a broken rule as a successful cost cut.

Alternatives, governance, and the impact on cost and engineering time

Targeted controls versus the alternatives

Global cuts are easy to apply but do not distinguish a chatty debug loop from an audit trail. Paying for growth defers the work until the bill forces it, and that usually leads to a rushed global cut.

Governance that keeps savings

Savings erode when nobody owns them. Attribute ingest and storage cost to each team, and give each service a cardinality budget and a volume budget, with alerts on growth. In code review, check new metric labels and log statements, because a single unbounded label can cancel months of reductions. Audit dashboards and alerts every quarter and delete the ones nobody has viewed or acted on.

Effect on engineering time

The main return is less noise during incidents, so responders reach the relevant data faster. The cost is pipeline maintenance: someone has to keep the filter and sampling rules correct as services change.

Over-aggressive cuts backfire. Teams that lose data they need will run their own logging, which ends up costing more and is harder to see. Set each cut where teams still trust the data.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce observability costs, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.