Microservices Partial Failure Handling in Fan-Out Requests

Why partial failure is the normal case in fan-out requests

In the aggregator pattern, one inbound request fans out to several downstream services, and their responses are combined into one payload. A partial failure is when some of those calls succeed and others fail, time out, or return unusable data. The aggregator has to decide what to do with an incomplete set of results.

This happens often because the probabilities multiply. If each of N independent calls succeeds with probability p, all of them succeed with probability p^N. Illustrative estimate, assumed inputs: 10 downstream calls, each 99.9% successful, gives 0.999^10 ≈ 99.0% of requests where everything succeeds. Roughly 1 in 100 requests sees at least one failure, even though every dependency looks healthy on its own dashboard. Real failures are rarely fully independent, but the direction holds: wider fan-out means more partial failures.

Latency compounds in the same way. The response cannot be sent until the slowest required call returns, so the aggregator's latency follows the maximum of its dependencies' latencies, not the average. Illustrative estimate, assumed inputs: if each of 10 calls has a 1% chance of exceeding its own p99, then 1 − 0.99^10 ≈ 9.6% of requests hit at least one of those slow calls. A downstream p99 becomes roughly your p90.

This applies to backend and platform engineers who own aggregators, backends-for-frontends, or API gateways that call several services. It matters most when fan-out is wide and the request sits on a user-facing latency budget.

Copy                       +-------------+
                       |   Client    |
                       +------+------+
                              |  1 inbound request
                              v
          +-------------------+-------------------+
          |  Aggregator / BFF / API gateway       |
          |  response = slowest required call     |
          |  combines whatever came back          |
          +-------------------+-------------------+
                              |  fan-out: N = 10 calls
  +------+------+------+------+------+------+------+------+------+
  |      |      |      |      |      |      |      |      |      |
  v      v      v      v      v      v      v      v      v      v
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+
| S1 | | S2 | | S3 | | S4 | | S5 | | S6 | | S7 | | S8 | | S9 | |S10 |
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+
  ok     ok   FAIL     ok     ok   SLOW     ok     ok     ok     ok

Illustrative estimate, assumed inputs (N = 10 independent calls):
  all succeed at 0.999 each:   0.999^10     ~ 99.0%  -> ~1 in 100 partial
  >=1 call beyond its p99:     1 - 0.99^10  ~ 9.6%   -> p99 becomes ~p90

One inbound request fans out to ten downstream services, so a single failed call (S3) makes the result partial and a single slow call (S6) sets the aggregator's response time; the figures are illustrative estimates from assumed inputs, not measurements.

The per-dependency contract: required, optional, or best-effort

Before any timeout or retry is tuned, each downstream call in a fan-out needs a written classification. It is assigned per endpoint, not per service. The same inventory service can be required for a checkout endpoint and merely optional for a product-listing page. The classification decides one thing: what the response looks like when that dependency does not answer.

This decision has to come first because it sets the failure budget for everything else. A required dependency justifies a larger share of the request deadline and a retry. An optional one should get a short timeout and usually no retry, since waiting on it only delays a response that is already valid without it. A best-effort call should not sit on the request path at all. If you tune timeouts before classifying, you end up giving every call the same budget, and the slowest optional dependency sets your tail latency.

The correctness trap

Some data looks like decoration but is actually a contract. Prices, inventory availability, and permission or entitlement checks must never be classified as optional, and they must never be served from a stale fallback. A cached price can sell goods at the wrong amount. A defaulted permission can expose data to someone who should not see it. When one of these is missing, fail the request.

How it works: deadlines, structured concurrency, and cancellation

Enforcing that contract at runtime depends on three mechanisms: one deadline, concurrent calls whose lifetimes are tied to the request, and cancellation that reaches every downstream hop.

One budget, derived timeouts

The edge sets a single deadline for the whole request. Each outbound call gets a timeout computed from the remaining budget, minus a margin for the aggregator's own merge and serialization work. Fixed per-call timeouts don't account for time already spent, so they can add up to more than the caller is willing to wait.

Propagating the deadline

With gRPC, the deadline travels on the call, and servers can read it from their incoming context. Over HTTP, send the remaining time in a header rather than an absolute timestamp, so that clock skew between hosts does not distort the budget. Each hop then derives its own context from the value it receives.

Structured concurrency in Go

In Go, wrap the request context with context.WithDeadline and pass it to errgroup.WithContext. Run each required call in the group and return its error. The first failure cancels the shared context, which stops the sibling calls, and Wait returns that error. Optional calls also run under the same context, but they record failures in their result slot and return nil. That way they never cancel the request, and they still stop at the deadline.

Cancellation only works if every hop honors it: database queries, HTTP clients, and loops must all observe ctx.Done(). A single component that ignores the context keeps consuming resources after the caller has given up.

Copy// Go. Page, Profile, Rec, fetchProfile and fetchRecs are placeholders.
import (
	"context"
	"time"

	"golang.org/x/sync/errgroup"
)

func buildPage(parent context.Context, budget time.Duration) (*Page, error) {
	// One deadline governs the whole request.
	ctx, cancel := context.WithDeadline(parent, time.Now().Add(budget))
	defer cancel()
	g, gctx := errgroup.WithContext(ctx)

	var profile *Profile
	var recs []Rec
	var recsErr error

	g.Go(func() error { // required: a returned error cancels gctx for all siblings
		cctx, done := callCtx(gctx)
		defer done()
		var err error
		profile, err = fetchProfile(cctx)
		return err
	})
	g.Go(func() error { // optional: record the failure, never cancel the request
		cctx, done := callCtx(gctx)
		defer done()
		recs, recsErr = fetchRecs(cctx)
		return nil
	})
	if err := g.Wait(); err != nil {
		return nil, err // first required failure
	}
	return &Page{Profile: profile, Recs: recs, RecsDegraded: recsErr != nil}, nil
}

// callCtx derives a per-call deadline from the remaining budget, reserving time to merge.
func callCtx(ctx context.Context) (context.Context, context.CancelFunc) {
	const mergeMargin = 20 * time.Millisecond // placeholder value
	dl, _ := ctx.Deadline()
	return context.WithDeadline(ctx, dl.Add(-mergeMargin))
}

A Go fan-out under one request deadline: errgroup cancels siblings when the required call fails, the optional call records its error and returns nil, and each call's deadline is the remaining budget minus a merge margin.

Designing response shapes and fallbacks for graceful degradation

Once the aggregator knows which calls finished, it has to report that honestly. Give each section its own status, such as "recommendations": {"status": "unavailable", "data": null}, or add a top-level errors array that names the section and the reason. Either way, the client can see exactly which part is missing.

Choose between null and omitted fields on purpose, and document the rule. One workable convention: "items": [] means the cart is confirmed empty, null with a status means the cart service didn't answer, and an omitted field means the client never asked for it. Without this distinction, a client renders an empty cart or a $0.00 balance during an outage. Users then take action on false data, such as abandoning a purchase or calling support about missing money.

Return 200 with degraded markers when the required dependencies succeeded and only optional ones failed. Return an error status (503, 504) when a required dependency failed. A response missing its core data is a failure, even if most fields are filled in.

Prefer stale data that reports its own age over a default that hides the failure.

Copy{
  "status": "degraded",
  "cart": {
    "status": "ok",
    "data": { "items": [] }
  },
  "balance": {
    "status": "stale",
    "as_of": "2026-10-22T09:14:05Z",
    "max_age_seconds": 300,
    "data": { "amount": "42.17", "currency": "USD" }
  },
  "recommendations": {
    "status": "unavailable",
    "data": null
  },
  "shipping_estimate": {
    "status": "fallback",
    "source": "static_default",
    "data": { "days": "3-7" }
  },
  "errors": [
    {
      "section": "recommendations",
      "reason": "timeout",
      "required": false
    },
    {
      "section": "balance",
      "reason": "upstream_unavailable",
      "required": false,
      "fallback": "stale_cache"
    }
  ]
}

A 200 degraded response where an empty items array means a confirmed empty cart, null with status unavailable means no answer, and stale or default data carries its age and source.

Retries, hedging, bulkheads, and circuit breakers inside a fan-out

Fallbacks cover a dependency that has already failed. The mechanisms below keep one struggling dependency from dragging down the whole fan-out.

Retries are only safe on idempotent calls. Each retry must fit inside the remaining deadline: if 40 ms of a 200 ms budget are left and the dependency's p50 is 60 ms, a retry cannot succeed and only adds load. Cap retries with a budget, such as a token bucket that allows retries up to a fixed fraction of recent successful calls. Add jittered backoff so clients do not retry in sync. Without a budget, every caller retries at the same moment, and a brief slowdown becomes a retry storm.

Hedging sends a duplicate request after a delay near the dependency's high-percentile latency and uses whichever response arrives first. It cuts tail latency on read-only calls. The cost is that every hedge doubles the load for that request, so limit it to a small share of traffic.

Bulkheads and circuit breakers isolate dependencies from each other. Give each dependency its own concurrency limit so a slow optional service cannot hold all of the aggregator's threads or connections.

Timeouts pull in two directions. Tight timeouts degrade the response early and protect latency, but they also cut off calls that would have succeeded. Loose timeouts recover those calls, but they hold resources longer and make bulkheads fill up faster.

Making degradation observable and testing it with fault injection

Fan-out handlers fail quietly. The handler returns 200, the load balancer counts a success, and the dashboard stays green while a dependency has been timing out for an hour. To catch this, record what happened to each dependency, not only what happened to the request.

Per-dependency outcomes

For every downstream call, emit a counter labeled with the dependency name and one outcome from a fixed set: success, timeout, error, cancelled, fallback_served. Put the same outcome on the child span as an attribute. On the parent span, add a list of the dependencies that degraded, so a single trace shows which sections of the response are stale or missing. Keep these label values bounded. A free-form error string will blow up metric cardinality.

A separate degradation SLI

Define a degraded-response rate: the share of successful responses where at least one optional dependency was served from a fallback. Alert on it separately from the error rate, with its own threshold per endpoint. A required dependency that fails should still show up in the error rate. An optional one that fails should never be invisible.

Fault injection in staging

Inject faults at the client or proxy layer for one dependency at a time. Typical faults are a delay just over its timeout, a connection reset, or a 5xx. For each fault, a test should:

• send a representative request
• assert the status code and that the affected field carries its fallback marker
• assert that the unaffected fields are still complete
• assert that total latency stays under the request deadline plus a small margin
• assert that the outcome metric recorded the expected label

Copyimport time
import pytest
import requests

BASE = ""
DEADLINE_S = 0.8   # request deadline enforced by the fan-out handler
MARGIN_S = 0.1     # allowance for serialization and network jitter

def set_fault(dependency, fault):
    """Placeholder: call your proxy or mesh fault-injection admin API here.
    fault=None clears any fault for that dependency."""

def outcome_count(dependency, outcome):
    # Read one counter series from the service's Prometheus text endpoint.
    # Assumes labels are exported in declared order (dependency, outcome).
    key = f'dependency_calls_total{{dependency="{dependency}",outcome="{outcome}"}}'
    for line in requests.get(f"{BASE}/metrics", timeout=2).text.splitlines():
        if line.startswith(key + " "):
            return float(line.split()[1])
    return 0.0

@pytest.mark.parametrize("fault,expected_outcome", [
    ({"delay_ms": 350}, "timeout"),   # just over the 300 ms reviews timeout
    ({"http_status": 503}, "error"),
])
def test_reviews_degrades_gracefully(fault, expected_outcome):
    set_fault("reviews", fault)
    try:
        before = outcome_count("reviews", expected_outcome)
        start = time.monotonic()
        resp = requests.get(f"{BASE}/products/sku-1234", timeout=5)
        elapsed = time.monotonic() - start

        assert resp.status_code == 200
        body = resp.json()
        assert body["reviews"]["degraded"] is True       # fallback marker present
        assert body["price"] and body["inventory"]       # unaffected sections intact
        assert elapsed < DEADLINE_S + MARGIN_S            # latency bound holds
        assert outcome_count("reviews", expected_outcome) == before + 1
    finally:
        set_fault("reviews", None)  # never leak a fault into the next test

A pytest sketch that injects one fault at a time into the reviews dependency in staging and checks the status code, the fallback marker, the unaffected fields, the latency bound and the per-dependency outcome counter.

When to avoid it, alternatives, and impact on cost and engineering time

Partial responses are the wrong fit for transactional or correctness-critical endpoints. A checkout total, a permissions check, or a balance read that silently omits one input is worse than an error, because the client acts on an answer that looks complete. They also add little value when fan-out is narrow (two or three calls), or when clients render every field unconditionally and cannot interpret a fallback marker.

Keep the optional set small. Each optional dependency can independently succeed or degrade, so the number of response states grows quickly.

Illustrative estimate (assumed numbers, not a benchmark): states = 2^k, where k is the number of optional dependencies. With k = 3 there are 8 states. With k = 5 there are 32, and each one is a combination the client must render and the test suite should cover.

The development cost is real. You get more states to test, client changes to handle degraded fields, and ongoing tuning of deadlines and fallbacks. That cost buys back engineering time in other places. A single dependency stops causing page-level incidents. The written contract makes it clear who owns each degraded field. Per-dependency telemetry also points on-call engineers to the failing service, so they don't have to search the whole call graph.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservices partial failure handling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.