Alerting Best Practices: SLO Burn-Rate Alerts That Cut Alert Fatigue

What deserves a page: pages, tickets and dashboards

An alert exists to get a human to do something. The person it serves is the on-call backend or platform engineer, and every notification spends some of their attention and sleep. Good alerting starts by sorting each signal by how urgently a human must act on it.

Alert on symptoms, not causes

Pages should fire on symptoms at the service boundary: error rate, latency and availability as users experience them. A symptom alert covers every failure mode, including ones you never predicted, because it measures the outcome directly.

Cause-based signals such as high CPU, memory pressure or one pod restarting are poor pages. A pod restart behind a load balancer with healthy replicas affects no one. A CPU spike during a batch job may be expected. Paging on these teaches engineers to ignore pages, which is how alert fatigue starts. Keep causes on dashboards, where they help you diagnose why a symptom alert fired.

The rule

A page that requires no action is a defect. Fix it by deleting it, raising its threshold, or demoting it to a ticket. Treat it like a bug, not like background noise you learn to live with.

How SLO burn-rate alerting works

An SLI (service level indicator) is a measured ratio of good events to total events, such as the fraction of HTTP requests that return a non-5xx response within 300 ms. An SLO (service level objective) is the target for that ratio over a window. Paging on the SLO ties alerts to user impact instead of guesses about causes.

Error budget and burn rate

Illustrative estimate: a 99.9% success SLO over 30 days leaves a 0.1% error budget. In time terms, that is 30 × 24 × 60 × 0.001 = 43.2 minutes of full outage.

Burn rate is how fast you spend the budget compared with the planned pace. At burn rate 1, the budget runs out exactly at the end of the 30-day window, which is acceptable. At burn rate 14.4 sustained for one hour, you spend 14.4 / 720 hours = 2% of the monthly budget. That pace would exhaust the budget in about two days, so it justifies waking someone.

Multiwindow, multi-burn-rate

Page only when both a long window (for example, 1h) and a short window (for example, 5m) exceed the fast-burn threshold. The long window shows the problem is significant. The short window shows it is still happening, so the alert resolves within minutes of recovery and does not stay red for the next hour.

Copyerror rate   ┌───────────┐
             │   burn    │ recovery
─────────────┘           └──────────────
5m window    [=fires=====][clears]
1h window       [===over threshold========...]
page             ^ both over   ^ 5m drops: page resolves

Slow burns, measured over 6h or 3d windows at lower thresholds, should go to tickets. Treat every window and threshold here as a starting point. Tune them against your own traffic and incident history.

Copy            ┌────────────────────────────────────┐
            │ Request events (good / total)      │
            └─────────────────┬──────────────────┘
                              │ SLI = good / total
                              ▼
            ┌────────────────────────────────────┐
            │ Burn rate per window               │
            │ = (1 - SLI) / 0.001 error budget   │
            └────────┬──────────────────┬────────┘
                     │                  │
          fast burn  │                  │  slow burn
                     ▼                  ▼
  ┌──────────────────────────┐  ┌──────────────────────────┐
  │ 1h burn rate > 14.4      │  │ 6h or 3d burn rate       │
  │ AND                      │  │ > lower threshold        │
  │ 5m burn rate > 14.4      │  │                          │
  └──────────────────┬───────┘  └───────┬──────────────────┘
                     │ both true        │ true
                     ▼                  ▼
  ┌──────────────────────────┐  ┌──────────────────────────┐
  │ PAGE on-call             │  │ TICKET (no page)         │
  │ clears when 5m drops     │  │                          │
  └──────────────────────────┘  └──────────────────────────┘

Request events become an SLI and per-window burn rates against a 99.9% SLO. A page fires only when both the 1h and 5m windows exceed the fast-burn threshold, and slow burns over 6h or 3d go to tickets; every window and threshold shown is a starting point to tune.

Implementing burn-rate alerts step by step

Pick the SLI at the boundary

Measure request success where users meet the service, such as the load balancer, API gateway or ingress. Internal component metrics can look healthy while users see failures. Define "bad" explicitly: 5xx responses and timeouts usually count, while 4xx responses usually don't, because a client sending malformed requests is not your outage.

Record ratios once, per window

Write one recording rule per window: the rate() of failed requests divided by the rate() of all requests, each wrapped in sum by (service). Use the same windows the alerts need, for example 5m, 1h, 30m and 6h. Precomputing these ratios keeps alert expressions short and makes dashboards match exactly what fired.

Express page and ticket rules

The fast-burn page requires both the long and short windows to exceed the threshold, joined with and. The long window shows the burn is significant. The short window shows it is still happening, so the alert clears soon after recovery. Add a brief for duration to absorb scrape jitter. Set a severity label (page or ticket) and a team or service label so the routing layer, not the rule author, decides who gets notified.

Make every page actionable

Each page should carry annotations covering four things:

• What is broken: which service, and which SLO is burning.
• User impact: current error ratio and burn rate.
• Links: {{ runbook_url }} and {{ dashboard_url }} placeholders.
• First diagnostic step: for example, "check the most recent deploy."

If you can't write that first step, ask whether the alert should page anyone.

Copy# Example Prometheus rules for a 99.9% availability SLO (error budget = 0.001).
# Metric and label names are placeholders; adapt to what your ingress exports.
groups:
  - name: slo-recording
    rules:
      # Bad = 5xx at the boundary (gateway timeouts surface as 504); 4xx excluded.
      - record: service:request_error_ratio:rate5m
        expr: |
          sum by (service) (rate(http_requests_total{job="ingress", code=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total{job="ingress"}[5m]))
      # Repeat the rule above with [1h], [30m] and [6h], named ...:rate1h etc.
  - name: slo-alerts
    rules:
      # Fast burn (14.4x budget): long window = significant, short = still happening.
      - alert: ErrorBudgetFastBurn
        expr: |
          service:request_error_ratio:rate1h > (14.4 * 0.001)
          and
          service:request_error_ratio:rate5m > (14.4 * 0.001)
        for: 2m  # absorb scrape jitter
        labels:
          severity: page   # routing layer decides who is notified
          team: payments
        annotations:
          summary: "{{ $labels.service }}: 99.9% availability SLO burning fast"
          impact: "1h error ratio {{ $value | humanizePercentage }} (budget 0.1%)"
          runbook_url: "<RUNBOOK_URL>"
          dashboard_url: "<DASHBOARD_URL>"
          first_step: "Check the most recent deploy of {{ $labels.service }}."
      # Slow burn (6x budget): real but not urgent, so it opens a ticket.
      - alert: ErrorBudgetSlowBurn
        expr: |
          service:request_error_ratio:rate6h > (6 * 0.001)
          and
          service:request_error_ratio:rate30m > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          team: payments

Prometheus recording and alerting rules that precompute per-window error ratios at the ingress and use multi-window conditions to page on fast burn and open a ticket on slow burn, with annotations that make the page actionable.

Cutting noise mechanically

Even a well-chosen alert can generate noise. These controls work at evaluation and delivery time rather than in the alert's query, so they apply across the whole rule set.

• Require a duration. Fire only when the condition has held for a set window, such as five minutes. A single slow scrape or a brief GC pause then never reaches a human. The cost is detection delay, so keep the window short relative to how fast the budget burns.
• Add hysteresis. Use a lower threshold to resolve than to fire. For example, fire at 2% errors and resolve below 1%. A metric hovering at the boundary then produces one page instead of a fire-resolve cycle every few minutes.
• Group and deduplicate. Batch alerts that share labels like service and cluster into one notification. Fifty failing pods are one problem, not fifty.
• Route by ownership. Send each alert to the team named in its team or service label. A page that reaches someone who can't act on it is noise to them.
• Inhibit downstream symptoms. While an upstream dependency such as the primary database is alerting, suppress the alerts it predictably causes in its consumers.

The config example below shows grouping keys, an ownership-based route, and one inhibition rule.

Grouping and inhibition share a risk: they can hide an unrelated second failure during an incident. Limit that risk in three ways:

• Make inhibition rules require matching labels, such as the same cluster, so they suppress only alerts that depend on the failing component.
• Never inhibit SLO burn-rate alerts.
• Show suppressed alerts on the incident dashboard so responders can see what was silenced.

Copy# Alertmanager-style routing, grouping and inhibition (illustrative)
route:
  receiver: default-oncall
  # Batch alerts sharing these labels into one notification
  group_by: ['alertname', 'service', 'cluster']
  group_wait: 30s        # wait to collect related alerts before the first send
  group_interval: 5m     # minimum gap before notifying about new alerts in a group
  repeat_interval: 4h    # re-notify an unresolved group at most this often
  routes:
    # Route by ownership: the team label decides who gets paged
    - matchers: ['team="payments"']
      receiver: payments-oncall
    - matchers: ['team="storage"']
      receiver: storage-oncall

inhibit_rules:
  # While the primary DB is down, mute the symptoms it causes in its consumers
  - source_matchers: ['alertname="PrimaryDatabaseDown"']
    target_matchers:
      - 'depends_on="primary-db"'
      # Never suppress SLO burn-rate alerts; they must always page
      - 'alert_kind!="slo_burn_rate"'
    # Only inhibit targets in the same cluster as the failing DB,
    # so an independent failure elsewhere still pages
    equal: ['cluster']

receivers:
  # Delivery integrations (pager, chat) omitted; add per team
  - name: default-oncall
  - name: payments-oncall
  - name: storage-oncall

An Alertmanager-style config that groups alerts by service and cluster, routes them by the team label, and inhibits downstream symptoms of a database outage only within the same cluster, without ever muting SLO burn-rate alerts.

When burn-rate alerts fit and when to avoid them

Burn-rate alerts work best for user-facing, request-driven services with steady traffic: APIs, web frontends, and RPC backends where every request is a small vote on whether users are being served. Outside that shape, they need adapting.

Low traffic. With few requests, a single failure moves the error percentage sharply. If an endpoint sees 20 requests an hour, one 500 is a 5% error rate and can trip a fast-burn threshold on its own (illustrative estimate). You can fix this in four ways: require a minimum request count before the alert can fire, add synthetic probes to create a steady baseline, lengthen the windows, or merge related endpoints into one SLI so the sample is larger.

Batch jobs. A nightly job has no request stream to burn against. Alert on whether it finished, how late it finished, and whether its output is correct.

SLI blind spots. An SLI only measures what it counts. A 200 response carrying stale or empty data passes. A broken endpoint that few people use barely moves the aggregate, so it stays silent.

Overambitious targets. Choosing 99.99% because it sounds rigorous leaves a tiny error budget. Minor blips then page someone, and you can end up with more fatigue than the cause-based alerts you replaced.

Window tension. Long windows suppress noise, but they detect problems later and keep firing after recovery because old errors are still inside the window. Multiwindow alerting reduces this effect but does not remove it.

Alternatives and the cause-based pages worth keeping

Burn-rate alerts are one option among several, and most mature setups combine them.

Cause-based pages worth keeping

Some failures arrive with warning but are invisible to the SLO until they cause an outage. The SLO sees nothing while a disk fills. By the time error rates rise, the database has stopped accepting writes. Keep a small set of cause-based pages for these cases:

• Disk filling with a predictable time-to-full. Fit a linear trend to recent usage, for example over the last few hours, and page only if the projection reaches zero within a window shorter than your response time. A disk at 85% that is flat is not a page.
• Certificate expiry. Page when expiry falls inside a window too short for a ticket to reach someone, and file tickets well before that.
• A backup job that hasn't succeeded within its expected interval. Alert on the absence of success. Alerting only on failure misses a job that never ran.

Each of these must still be urgent and actionable. If no one needs to act tonight, make it a ticket.

Removing cause-based alerts has a cost. Symptom alerts tell you something is wrong, not why. If your dashboards and runbooks are weak, the on-call engineer starts each incident with less context. Before you delete an alert, move its signal onto a dashboard that the runbook links to.

Copygroups:
  - name: cause-based-pages
    rules:
      # Page only if the last 6h trend says the disk hits zero within 4h.
      # The second clause skips disks that still have plenty of room,
      # so a short write burst on a mostly empty disk does not page.
      - alert: DiskWillFillWithin4h
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
          and
          node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.20
        for: 30m  # the trend must hold, not just one noisy sample
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} projected full within 4h"
          runbook: "runbooks/disk-full.md"

      # Certificate expiry, from a blackbox_exporter TLS probe. Page at 3 days.
      # A ticket rule with a 21-day threshold should catch this long before it pages.
      - alert: TLSCertExpiresWithin3d
        expr: probe_ssl_earliest_cert_expiry - time() < 3 * 86400
        labels:
          severity: page

      # Alert on missing success, not on failure. This also catches a job that never ran.
      # backup_last_success_timestamp_seconds is a custom metric that the backup job
      # pushes after each successful run. The daily job gets a 2h grace period.
      - alert: BackupNotSucceededIn26h
        expr: time() - backup_last_success_timestamp_seconds{job="db-backup"} > 26 * 3600
        labels:
          severity: page

Prometheus alerting rules for the three cause-based pages worth keeping: a predict_linear disk-fill projection, a TLS certificate expiry check, and a backup rule that fires when no recent success has been recorded rather than when a failure is reported.

Impact on cost and engineering time

The upfront work is real. Someone has to choose SLIs that reflect what users experience, agree on SLO targets with the people who own the product, write a runbook for every page, and build the dashboards those runbooks point to. You then tune thresholds against your own traffic and incident history. During that period, expect some alerts to fire on nothing and some incidents to slip past. Plan it as a project rather than a side task.

The ongoing cost is smaller but cannot be skipped. Alert hygiene needs a named owner and a fixed review cadence. Without both, new cause-based alerts pile up and the pager drifts back to noise. In each review:

• For every page, record whether it was actionable and what the responder actually did.
• Delete, demote to a ticket, or retune any alert that keeps producing no-action pages.
• Track pages per shift and off-hours pages over time. They are on-call health signals. A rising trend means the rotation is absorbing a cost that no service dashboard shows.

The payoff is fewer interruptions and faster triage. A burn-rate page already tells the responder which user journey is affected and how fast the error budget is being spent.

Illustrative estimate (assumed numbers, not a benchmark)

Weekly time saved = no-action pages removed × (handling minutes + refocus minutes)

Assume 10 pages removed per week, 10 minutes to acknowledge and check each one, and 20 minutes to get back into the previous task:

10 × (10 + 20) = 300 minutes, or about five hours per week.

Off-hours pages cost more than this formula captures, because lost sleep reduces the next day's work too.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to alerting best practices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.