Circuit Breaker Pattern in Microservices: Timeouts, Retries, Limits

How cascading failures propagate between services

A cascading failure starts when one dependency degrades and its callers degrade with it, until services that were never broken stop answering. The trigger is rarely a hard crash. A dead dependency refuses connections or returns errors quickly, and callers can fail fast and move on. A slow dependency is worse because every request to it holds resources in the caller while it waits: a worker thread, a pooled connection, a socket, and the memory for the request and its buffered response.

Consider an illustrative estimate with assumed inputs. A service has 200 worker threads and normally spends 50 ms on a downstream call. If that dependency starts taking 5 seconds, each thread is held 100 times longer. Throughput through those threads drops by the same factor. Requests that have nothing to do with the slow dependency now queue behind blocked threads, upstream load balancers see timeouts, and their own pools begin to fill. Retries from each layer add more load to the component that is already struggling.

The circuit breaker pattern interrupts this chain. A breaker tracks failures on calls to a dependency. When failures cross a threshold, it stops sending traffic for a while and fails those calls immediately, which releases the caller's resources. A breaker can only count failures it can see, though. Without timeouts, a slow call never counts as a failure. Without retry budgets, the breaker counts amplified load. Without concurrency limits, pools are exhausted before the breaker trips. This article covers those layers for backend and platform engineers who run synchronous service-to-service calls.

Copy  Clients / upstream LB   (timeouts -> retries -> even more load)
            |
            v
  +--------------------------------------------+
  | Service A            threads: 200/200 busy |
  |   /checkout  --> calls B      (blocked)    |
  |   /profile   --> no B call    (queued too) |
  +--------------------------------------------+
            |  one worker thread held per call
            v
  +--------------------------------------------+
  | Service B            conn pool: exhausted  |
  |   sockets + buffered responses in memory   |
  +--------------------------------------------+
            |  normally 50 ms, now 5 s (100x)
            v
  +--------------------------------------------+
  | Service C            SLOW, not dead        |
  +--------------------------------------------+

  Same call path from B, with the protective layers in place:

  B --> [conc. limit] --> [retry budget] --> [timeout] --> [breaker] -x-> C
             |                  |                |             |
             v                  v                v             v
        caps held         caps retry       slow call =    OPEN: fail
        threads           amplification    failure        fast, free
                                                          thread/conn

A slow Service C holds connections in B and threads in A until unrelated endpoints such as /profile stall, and a circuit breaker only stops this when concurrency limits, retry budgets and timeouts beneath it turn slowness into countable, bounded failures.

Who should use the circuit breaker pattern?

These controls are for teams whose services make synchronous network calls to other services, databases or third-party APIs, and whose own availability depends on those calls. In practice that means:

• Backend engineers who own a service with outbound dependencies. They set per-call timeouts, decide which operations are idempotent and safe to retry, and write the fallback behaviour, because only the application knows what a sensible degraded response looks like.
• Platform and SRE teams who run shared infrastructure such as gateways, sidecar proxies or client libraries. They set fleet-wide defaults for timeouts and connection limits and decide which layer owns which policy.
• On-call engineers who need breaker state, rejection counts and retry counts to diagnose incidents quickly.

Teams with a single service, or with purely asynchronous, queue-based communication, get less from a breaker. A queue already decouples producer from consumer, and backpressure there is usually handled with queue depth limits and consumer concurrency rather than a breaker on each call.

When should you use a circuit breaker?

Use a circuit breaker when all of the following hold:

• The call is synchronous and on a request path. A caller is holding threads, connections or memory while it waits for the answer.
• The dependency can degrade independently. It is a separate process, service or external API that can slow down or fail while your service is still healthy.
• Failing fast is better than waiting. You can return an explicit error, serve cached or degraded data, or skip an optional feature instead of blocking.
• Traffic is high enough to judge health. The breaker needs enough calls in its window to tell an outage from a few unlucky requests.

Add it alongside timeouts, bounded retries and concurrency limits, not instead of them. Timeouts come first, because a breaker cannot count a slow call that never ends.

Reconsider or skip a breaker in these cases:

• Low-traffic, non-critical calls that are already bounded by a timeout. The breaker rarely collects enough samples to trip usefully.
• Critical writes such as payments. Failing fast may be the wrong outcome. Durable queuing or a longer bounded wait is often more correct.
• Policies already enforced by a service mesh. Adding the same policy in the application stacks controls and multiplies retries. The library-versus-mesh section below covers how to split responsibility.

How does the circuit breaker pattern work?

The breaker wraps calls to a dependency and keeps a short history of their outcomes. While results are healthy, it stays closed and passes calls through. When the share of failed or slow calls in its recent window crosses a threshold, and the window holds a minimum number of calls, it trips to open. While open, it rejects calls immediately without touching the network. That frees the caller's threads and connections and removes load from the struggling dependency. After a set open duration, it moves to half-open and lets a few probe calls through. If they succeed, it closes. If they fail, it opens again.

The breaker depends on the layers around it:

1. Timeouts turn slowness into a failure the breaker can count.
2. Retry limits and budgets keep retries from inflating the failure count and the load on the dependency.
3. Bulkheads and concurrency limits cap how many resources a single dependency can hold while the breaker is still collecting evidence.
4. Fallbacks decide what the caller returns once the breaker rejects a call.

The sections that follow cover each layer in that order, then the state machine itself, with code sketches for each.

Timeouts and deadline propagation come first

Every network call needs an upper bound. Without one, a slow peer decides how long your threads, connections and memory stay occupied, and that is how the pool exhaustion described above begins. Three timeouts bound different things:

Library defaults are often infinite, or chosen without your traffic in mind. Instead, measure the dependency's latency under normal load and set each attempt's timeout from that distribution.

The overall deadline should travel with the request. Each hop:

• reads the remaining budget,
• subtracts the time it has already spent and a margin for its own work,
• passes a smaller deadline downstream.

When the budget runs out, downstream services stop working on a response nobody will read.

Illustrative estimate (assumed numbers, not a benchmark)

Per-attempt timeout. Assume a dependency p99 of 120 ms and a 1.5× headroom factor. Per-attempt timeout = p99 × headroom = 180 ms.

Downstream budget. Assume a 500 ms caller budget, 60 ms already spent and a 40 ms local margin. Downstream budget = 500 − 60 − 40 = 400 ms. Two attempts fit within it; a third does not.

Cost of loose timeouts. In-flight requests ≈ arrival rate × timeout (Little's law). With 50 req/s into a hung dependency:

• a 5 s timeout holds about 250 requests in flight;
• a 180 ms timeout holds about 9.

Cost of tight timeouts. A timeout that is too tight cuts off healthy tail requests. That produces retries, which add load to the dependency at the moment it is already slowest.

Copyimport time
import requests

CONNECT_TIMEOUT_S = 0.05   # bounded by network RTT, not service latency
ATTEMPT_TIMEOUT_S = 0.18   # dependency p99 (120 ms) x 1.5 headroom
LOCAL_MARGIN_S = 0.04      # reserved for our own work after the call returns
BUDGET_HEADER = "X-Request-Budget-Ms"  # placeholder name; a relative budget avoids clock skew

def call_dependency(url, incoming_budget_ms, received_at):
    # Overall deadline: the caller's budget, measured from when we received the
    # request, minus a margin for our own work.
    deadline = received_at + incoming_budget_ms / 1000 - LOCAL_MARGIN_S
    last_error = None
    for _ in range(3):
        remaining = deadline - time.monotonic()
        # Do not start an attempt that cannot finish inside the deadline.
        if remaining < ATTEMPT_TIMEOUT_S:
            break
        try:
            # Pass the smaller, remaining budget downstream so the next hop
            # can stop work once nobody is waiting for the answer.
            # Note: requests' read timeout bounds the gap between received bytes,
            # not the total response time.
            return requests.get(
                url,
                timeout=(CONNECT_TIMEOUT_S, ATTEMPT_TIMEOUT_S),
                headers={BUDGET_HEADER: str(int(remaining * 1000))},
            )
        except (requests.ConnectionError, requests.Timeout) as exc:
            last_error = exc  # retry only transient failures, within the budget
    raise TimeoutError("deadline exhausted") from last_error

def handle(request_headers):
    received_at = time.monotonic()
    budget_ms = int(request_headers.get(BUDGET_HEADER, "500"))
    return call_dependency("", budget_ms, received_at)

A Python client sets separate connect and per-attempt timeouts. It derives an overall deadline from the caller's budget, skips any attempt that cannot finish in the time left, and passes the shrinking budget to the next hop in a header.

Retries without amplification: backoff, jitter and retry budgets

Retries are the right response to a transient fault and the wrong response to overload. The problem is that a caller cannot tell the two apart from a single failed request, and every layer that retries independently multiplies the load that reaches the bottom of the stack.

How per-layer retries compound

Illustrative estimate (assumed numbers, not a benchmark). Assume a chain of four hops (gateway → API → orders → inventory database), and assume each layer makes up to 3 attempts (1 original plus 2 retries). If the bottom dependency is failing every call, the worst-case number of attempts it receives for one user request is:

attempts at depth = attempts_per_layer ^ layers = 3^4 = 81

That load multiplier hits a dependency that is already struggling. Substitute your own hop count and retry settings.

Rules that keep retries bounded

• Retry at one layer only. This is usually the layer closest to the failing dependency. The others should fail fast and pass the error up.
• Cap attempts. Use a small fixed limit, and stop early once the propagated deadline cannot fit another attempt.
• Use exponential backoff with jitter. Doubling the delay gives the dependency time to recover. Randomising each delay, for example by drawing it uniformly between zero and the current cap, spreads clients out so they do not retry in synchronised waves.
• Retry only idempotent or safe operations. A timed-out POST may have succeeded. Without an idempotency key, retrying it can charge a customer twice.
• Enforce a retry budget. Allow retries only while they stay below a fixed fraction of normal request volume, for example 10%, tracked over a sliding window. During a broad outage the budget runs out and retries stop, which turns a 3× multiplier into roughly 1.1×.

Copyimport random, time
from collections import deque

class TransientError(Exception):
    """Raised by the operation for faults worth retrying (timeouts, 503s)."""

class RetryBudget:
    """Allow retries only while they stay under `ratio` of recent requests.
    Single-threaded sketch: add a lock before sharing it across threads."""
    def __init__(self, ratio=0.1, window_s=10.0):
        self.ratio, self.window_s = ratio, window_s
        self.requests, self.retries = deque(), deque()

    def _trim(self, q, now):
        while q and now - q[0] > self.window_s:  # drop events outside the window
            q.popleft()

    def record_request(self):
        self.requests.append(time.monotonic())

    def try_spend(self):
        now = time.monotonic()
        self._trim(self.requests, now); self._trim(self.retries, now)
        if len(self.retries) >= self.ratio * len(self.requests):
            return False  # budget exhausted: fail fast instead of piling on
        self.retries.append(now)
        return True

def call_with_retries(op, budget, deadline, max_attempts=3, base_s=0.1, cap_s=2.0):
    """Only pass idempotent ops. `deadline` is a time.monotonic() timestamp."""
    budget.record_request()  # count original attempts, not retries
    for attempt in range(max_attempts):
        try:
            return op()
        except TransientError:
            if attempt == max_attempts - 1:
                raise  # attempt cap reached
            # Full jitter: uniform in [0, exponential cap] desynchronises clients.
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))
            if time.monotonic() + delay >= deadline or not budget.try_spend():
                raise  # no time left, or retries already over 10% of traffic
            time.sleep(delay)

A Python sketch of capped retries using exponential backoff with full jitter. Each retry is first checked against the propagated deadline and then against a sliding-window budget that keeps retries under 10% of recent requests.

How the circuit breaker state machine works

A breaker sits in front of a dependency and stops calls that are likely to fail. It has three states.

• Closed: calls pass through. The breaker records outcomes over a sliding window, either the last N calls or the last N seconds. It trips when the error rate or slow-call rate crosses a threshold. It only evaluates the threshold once the window holds a minimum number of calls, so two failures out of three requests do not count as an outage.
• Open: calls fail immediately without touching the network for a fixed open duration. This gives the dependency room to recover and frees the caller's threads.
• Half-open: a small, fixed number of probe calls go through. If they succeed, the breaker closes. If they fail, it reopens.

Granularity decides what the breaker can see:

Finer scopes have their own costs. Each breaker needs a window, thresholds and an open duration, and every one of those values must be tuned. Low-traffic endpoints may never reach the minimum volume, or they trip on samples too small to mean anything. Consider a fleet whose breakers opened together during an outage. With identical open durations, those breakers all enter half-open at the same moment, and the burst of probes can knock a recovering service back over. Adding jitter to the open duration spreads the probes out.

Copyimport random
import time
from collections import deque, defaultdict

class CircuitBreaker:
    # Not thread-safe; a real breaker needs a lock around state changes.
    def __init__(self, window=50, min_calls=20, failure_rate=0.5, slow_rate=0.5,
                 slow_s=2.0, open_s=30.0, jitter=0.2, probes=3):
        self.window = deque(maxlen=window)  # count-based: last N (failed, slow) outcomes
        self.min_calls, self.failure_rate, self.slow_rate = min_calls, failure_rate, slow_rate
        self.slow_s, self.open_s, self.jitter, self.probes = slow_s, open_s, jitter, probes
        self.state, self.opened_until, self.in_flight, self.passed = "closed", 0.0, 0, 0

    def allow(self):
        if self.state == "open":
            if time.monotonic() < self.opened_until:
                return False  # fail fast, no network call
            self.state, self.in_flight, self.passed = "half_open", 0, 0
        if self.state == "half_open":
            if self.in_flight >= self.probes:
                return False  # only a fixed number of probes get through
            self.in_flight += 1
        return True

    def record(self, ok, elapsed_s):
        good = ok and elapsed_s <= self.slow_s
        if self.state == "half_open":
            if not good:
                return self._trip()  # any failed probe reopens
            self.passed += 1
            if self.passed == self.probes:
                self.state = "closed"
                self.window.clear()
            return
        self.window.append((not ok, elapsed_s > self.slow_s))
        n = len(self.window)
        if n < self.min_calls:
            return  # too few samples to judge
        if (sum(f for f, _ in self.window) / n >= self.failure_rate
                or sum(s for _, s in self.window) / n >= self.slow_rate):
            self._trip()

    def _trip(self):
        # Jitter the open duration so a fleet does not probe in lockstep.
        self.state = "open"
        self.opened_until = time.monotonic() + self.open_s * random.uniform(
            1 - self.jitter, 1 + self.jitter)

breakers = defaultdict(CircuitBreaker)
breaker = breakers[("payments", "POST /charges")]

A minimal count-based circuit breaker that trips on error or slow-call rate only after reaching a minimum call volume. It lets a fixed number of half-open probes through, jitters the open duration to avoid synchronized probing, and keys breakers per dependency and endpoint.

Bulkheads, concurrency limits and what to do when the breaker is open

A breaker reacts only after failures accumulate, so it needs a partner that limits damage while the breaker is still deciding. A bulkhead gives each dependency its own connection pool or semaphore. If the recommendations service hangs, it can hold at most its own twenty slots, and checkout traffic keeps flowing through a separate pool. A concurrency limit applies the same idea to inbound work. When the queue in front of a worker pool fills, reject new requests immediately. A request that waits thirty seconds in a queue has usually been abandoned by its caller already, so serving it wastes capacity.

When the breaker is open

The default response is to fail fast with an explicit error, such as a 503 with a reason, so callers can tell this from a bug. If the data can tolerate staleness, serve a cached or degraded response instead. For optional features like reviews or personalisation, skip the feature and render the page without it. A fallback must never call the failing dependency through a different path, because that recreates the load the breaker was meant to remove.

Failing fast is wrong for critical writes such as payments. For those, durably queuing the request, or waiting longer behind a bounded timeout, is more correct. Untested fallbacks carry their own risk. A fallback that returns zero for an account balance looks like success and silently corrupts downstream decisions. Exercise every fallback path in tests, and alert on how often it runs.

Application libraries versus service mesh: where the controls should live

A sidecar proxy can enforce timeouts, retries, outlier ejection and connection limits the same way for every service, with no code changes. In a fleet written in several languages, that consistency is worth a lot. The limitation is that the proxy sees requests, not intent. It cannot tell whether a POST is safe to repeat, what the caller should return when a dependency is unavailable, or how much of the end user's deadline is left. Those decisions need application context. Even when the mesh does the breaking, the client code still owns idempotency, fallbacks and deadline propagation.

Stacking controls across layers is where trouble starts. Here is an illustrative estimate: if the app makes up to 3 attempts and its sidecar makes up to 3 per app attempt, one slow dependency receives up to 9 calls. Add another hop configured the same way and the worst case reaches 81. Stacking also hides the real failure. Proxy retries that eventually succeed never appear in application metrics, and a breaker that opens in the mesh shows up to the caller as an unexplained 503.

Assign each policy to exactly one layer and write that assignment down. A good default is mesh-level timeouts and connection limits, with retries and fallbacks in the app. Skip the breaker entirely for a single non-critical, low-traffic call that is already bounded by a timeout. At low volume it rarely collects enough samples to trip in a useful way, and it adds state you will have to tune.

Observability, fault injection and the cost of ownership

A breaker you cannot see is a breaker you cannot trust. Each one should emit:

• State transitions (closed → open → half-open → closed), tagged by dependency, so a spike in opens can be matched to a deploy or a downstream incident.
• Rejection counts: calls refused while open. These are failures your users experienced, even though no request reached the network.
• Timeout counts, kept separate from error responses. Rising timeouts with flat error rates usually mean latency, not breakage.
• Retry counts per attempt number. Growth in second and third attempts is the early signal of a retry storm.

Alert on a breaker that stays open past several open-duration cycles. This means half-open probes keep failing. Either the dependency is still down or the probe criteria are wrong, and someone needs to decide which.

Verify before you depend on it

Configuration that has never fired is a guess. In staging, inject added latency and error responses at the dependency boundary. Confirm that timeouts fire at the expected deadline, retries back off and stop within budget, and the breaker opens and recovers. Also confirm that the fallback path works.

The ownership cost

This work takes real engineering time. You have to design deadlines and budgets across call chains, tune thresholds against real traffic, and review window sizes, thresholds and open durations as traffic grows or shifts. A threshold tuned for last year's volume can trip on noise or never trip at all.

Designing timeouts, retries, breakers and limits together pays that cost once. The alternative is bolting a breaker onto each call after an outage. That produces overlapping, inconsistent settings, and those make incidents longer, retry storms harder to diagnose and on-call rotations more draining.

How can circuit breakers and these controls save engineering time?

The time savings come from fewer and shorter incidents, and from less per-service, per-incident decision-making.

• Smaller blast radius means fewer incidents to work. When a dependency degrades, timeouts, bulkheads and an open breaker keep the failure contained to the features that use it. One team handles a degraded dependency instead of several teams chasing stalled services that were never broken.
• Faster diagnosis. Breaker state transitions, rejection counts and per-attempt retry counts point straight at the failing dependency. Without them, on-call engineers often start from symptoms such as thread-pool exhaustion or upstream timeouts in a service that is only a victim, and work backwards through the call chain.
• Self-recovery without manual intervention. Half-open probing closes the breaker on its own once the dependency recovers. Retry budgets stop retry storms without someone scaling down clients or restarting pods. Engineers do not have to be paged to undo load that the system generated against itself.
• Decisions made once, not per incident. A documented split of which layer owns timeouts, retries and breakers, together with shared defaults in a client library or mesh, saves each service team from re-deriving these values. It also avoids the debugging time lost to stacked retries and unexplained 503s.
• Fault injection replaces guesswork. Testing timeouts, breakers and fallbacks in staging finds misconfiguration during a planned exercise, when it is cheap to fix, rather than during an outage.

Illustrative estimate (assumed numbers, not a measured result). Assume a team has 6 dependency-related incidents a year. Assume each one, without these controls, pulls in 3 engineers for 4 hours to find the slow dependency and stop the cascade. That is 6 × 3 × 4 = 72 engineer-hours. Now assume containment and clear breaker metrics reduce each incident to 1 engineer for 2 hours. That is 6 × 1 × 2 = 12 engineer-hours. The difference has to be weighed against the up-front design and ongoing tuning described above. Substitute your own incident history; the point is that the cost of designing the controls once is compared with recurring incident time, not that these numbers apply to your system.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to circuit breaker pattern microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.