What is load shedding in microservices and why do queues hide overload?
Load shedding is overload protection for a service's own capacity. When more work arrives than the service can finish, it rejects the excess early and cheaply, usually with a fast error the caller can act on, so the requests it does accept still complete in time.
Queues make this necessary because they hide overload. Little's Law says that in a stable system, concurrency equals throughput times latency. A service has a fixed number of requests it can process at once, so its throughput has a ceiling. When the arrival rate exceeds that ceiling, the system has no steady state. The surplus waits in queues, and queue length and latency keep growing for as long as the overload lasts.
Illustrative estimate. Assume 100 workers, 50 ms per request and a 1-second client timeout:
- Capacity is 100 / 0.05 s = 2,000 requests per second.
- Arrivals rise to 2,500 rps, so the queue grows by 500 requests every second.
- After 4 seconds, 2,000 requests are queued. A new arrival now waits a full second before any work starts on it.
- From that point on, nearly every response the server produces goes to a client that has already timed out.
The server still completes 2,000 requests per second, so CPU and throughput dashboards look healthy. Goodput, meaning responses delivered to callers who are still waiting, drops toward zero. All of that capacity is spent on wasted work, and client retries add more arrivals on top.
Any service needs this protection if it has bounded capacity, requests whose cost varies widely, or dependencies that can slow down. A slow database lowers capacity in the same way that extra traffic raises demand.
How does load shedding work?
Load shedding is a feedback loop around a single admission decision made for each request as it arrives:
- Measure a capacity signal. The most useful signal is the number of requests currently in flight. Queueing delay and recent latency compared with a no-load baseline are also used. Each of these reflects work the service is actually doing. Arrival counts do not.
- Compare the signal against a limit. The limit can be fixed, derived from a load test and Little's Law, or adaptive, recalculated continuously from observed latency.
- Admit or reject immediately. If the request fits under the limit, it takes a permit and runs. If it does not, the service returns a fast, explicit error such as HTTP 503 with
Retry-After, or gRPCRESOURCE_EXHAUSTEDorUNAVAILABLE. The service does not wait for capacity to free up, because waiting recreates the unbounded queue. - Choose what to reject. When requests carry a priority or criticality tag, the least valuable work is rejected first. Requests whose deadline has already passed, or will pass before the work could finish, are dropped regardless of tier.
- Release and feed back. When an admitted request finishes, by success, error or cancellation, its permit is returned. If the limit is adaptive, the request's latency feeds into the next limit calculation.
- Callers back off. Clients treat the rejection as a signal. They retry within a budget, with jittered backoff, or throttle themselves locally. Load balancers route to instances with free capacity. Autoscaling adds capacity, and while it does, shedding keeps latency bounded.
The rest of this article covers each step: where the decision is made, how to build the limit, how to make it adaptive, which queue and priority tactics to combine with it, and how the surrounding system has to cooperate.
When should you use load shedding?
Use load shedding when a service handles online, request/response traffic and its callers give up after a timeout. In that setting, a response that arrives late is worth nothing, so rejecting quickly is better than finishing slowly. The following conditions make shedding especially valuable:
- Many callers share one service. A fan-in service such as an auth, profile or pricing service gets overloaded by its combined callers, and no single caller's rate limit stops that.
- Traffic is bursty or hard to predict. Launches, batch jobs that start on the hour, and reconnect storms after an outage all push arrivals past capacity faster than any human or autoscaler can react.
- Request cost varies widely. When a few expensive requests can tie up workers, a concurrency limit protects capacity that an RPS limit cannot see.
- Dependencies can slow down. If a database or downstream service can go from fast to slow, in-flight work piles up even though arrivals have not changed. Shedding caps that pile-up.
- Callers retry. Retries turn a brief overload into a sustained one. Shedding, together with client retry budgets, breaks the loop.
- Autoscaling lags behind load. Shedding holds latency for admitted requests during the time it takes new instances to start and warm up.
The warning signs are usually visible in existing metrics. Latency rises while throughput stays flat. Timeouts increase across callers at the same moment. One slow dependency causes failures in services that do not call it directly.
How is load shedding different from rate limiting and other alternatives?
Rate limiting answers "is this caller using more than its share?" Load shedding answers "can this server take one more request right now?" The first enforces fairness and quotas per tenant or API key. The second protects the server's capacity no matter who is calling. You usually need both, and neither replaces the other.
A fixed requests-per-second ceiling is a poor capacity signal. A cache hit and a cold report query count as one request each, yet their costs can differ by orders of magnitude. Real capacity also moves. A deploy can change CPU per request, a noisy neighbor can steal cycles, and a slow dependency holds threads longer. A static RPS number set last quarter is either too loose to prevent collapse or too tight to use the hardware you pay for. In-flight concurrency tracks actual work in progress, which makes it a better signal.
| Approach | What it protects | Fits when | Weakness under overload |
|---|---|---|---|
| Static rate limit | Fairness between callers | Quotas, abuse control | Blind to request cost and capacity drift |
| Unbounded queue | Nothing | Never, for online traffic | Latency grows until every request times out |
| Bounded queue | Memory | Absorbing short bursts | Still adds wait time once full |
| Circuit breaker | The caller, from a failing dependency | Fast failure on downstream errors | Acts on the client side and does not protect the server |
| Autoscaling alone | Long-term capacity | Gradual load growth | Takes minutes to react, while overload builds in seconds |
| Overprovisioning | Headroom | Predictable peaks | Costly and still has a ceiling |
When should you avoid load shedding?
Skip load shedding, or put something else in place first, in these cases:
- Low-traffic internal tools, where overload is unlikely.
- Workloads where every request must eventually complete, such as payments ingestion. A durable queue is the right model there. The producer writes to the queue and consumers process at their own pace, so a backlog becomes delay, not lost work.
- Teams that cannot tolerate any visible error rate. Rejecting requests is the whole mechanism, so these teams should fix that constraint before adding shedding.
- Pull-based consumers. A worker that reads from a queue or stream controls its own intake. It applies backpressure by pulling less, and it has nothing to reject.
- Callers that cannot handle rejections. If clients retry immediately and without limit, every rejection turns into more requests. Add retry budgets and backoff on the client side first, or shedding can make the overload worse.
- A permanent capacity shortfall. If the service is short of capacity at normal peak every day, shedding just turns that shortfall into a steady error rate. Shedding handles transient overload. A persistent gap needs more capacity or cheaper requests.
Where should you shed load in a request path?
Every request you reject costs something. The goal is to make that cost as small as possible. A request rejected before the body is read costs a few microseconds and one connection slot. A request rejected after JSON decoding, a token validation call, and two cache lookups has already consumed most of the resources you were trying to protect. Each layer in the path makes a different trade between how cheap the rejection is and how much it knows.
Edge proxy
The proxy can count in-flight requests per upstream and return 503 immediately, without touching the backend. This is the cheapest place to reject. The proxy only sees latency and error codes, though, so it cannot tell a slow request from an overloaded pool. Use it as a coarse ceiling, not the primary limit.
Service middleware
The first middleware, placed before deserialization and auth, should acquire a concurrency permit or reject. At this point the service knows its own in-flight count and recent latency, which makes it the best-informed cheap decision. It should also check the propagated deadline here. If the remaining budget is shorter than the typical handler latency, reject now rather than doing work nobody will wait for.
Dependency client
Each outbound client holds its own limit per dependency. When the database pool is saturated, fail fast instead of queuing on the pool. Check the deadline again before every call, because the time left has shrunk since the request entered.
reject cost: cheap ----------------------------------------> expensive
backend info: none -----------------------------------------> full
Client
|
v
+--------------------------+
| EDGE PROXY | in-flight per upstream at ceiling?
| coarse ceiling |---- yes ---> 503, backend untouched
+--------------------------+
| no
v
+--------------------------+
| SERVICE MIDDLEWARE | no permit, or remaining deadline
| 1. acquire permit | < typical handler latency?
| 2. check deadline |---- yes ---> 503, body never read
+--------------------------+
| admitted
v
+--------------------------+
| decode body, auth, | cost paid only by admitted work
| cache lookups, handler |
+--------------------------+
|
v
+--------------------------+
| DEPENDENCY CLIENT | per-dep limit full, or deadline gone?
| limit per dependency |---- yes ---> fail fast, no pool queue
| re-check deadline / call |
+--------------------------+
| admitted ---> DB pool, cache
Rejection points from top to bottom of a request path: each layer down knows more about backend state, but a rejection there costs more because more work has already been done.
How do you implement concurrency limits as the core admission control primitive?
A rate limit counts arrivals. A concurrency limit counts work in progress, and work in progress is what actually exhausts threads, connections and memory. When a downstream call slows from 50 ms to 500 ms, the same request rate produces ten times as many in-flight requests. A concurrency limit trips immediately. A rate limit sees nothing wrong.
Little's Law gives a starting value: in-flight requests = throughput × mean latency. As an illustrative estimate, suppose a load test shows one instance sustaining 400 requests per second at a mean latency of 50 ms. That gives 400 × 0.05 = 20 concurrent requests. Add headroom for variance, say 1.5×, and you get a limit of 30. Then confirm that latency stays flat up to that point in the same load test.
The middleware is a counter with a try-acquire operation. On arrival, it attempts to take a permit. If none is free, it responds at once:
- HTTP: 503 with
Retry-After, or 429 when the rejection is attributable to the caller. - gRPC:
RESOURCE_EXHAUSTEDorUNAVAILABLE.
It never waits for a permit, because waiting is just an unbounded queue under another name. Release the permit on every exit path, including errors and client cancellation, or the limit leaks down to zero.
Requests rarely cost the same. A bulk export can take a weight of ten permits while a lookup takes one. Weights drift as code and data change, though, so re-derive them periodically from measured per-route latency. An alternative is to use separate pools instead of weights.
Put all of this in shared middleware. You get one implementation, consistent rejection semantics and uniform metrics, instead of hand-tuning every endpoint and letting each one go stale independently.
import threading
class ConcurrencyLimiter:
"""Bounds in-flight work with try-acquire. It rejects and never queues."""
def __init__(self, limit):
self.limit = limit # e.g. 30 = 400 rps x 0.05 s x 1.5 headroom
self.in_flight = 0
self._lock = threading.Lock()
def try_acquire(self, weight=1):
weight = min(weight, self.limit) # a heavy request can still run on an idle instance
with self._lock:
if self.in_flight + weight > self.limit:
return 0 # no permits free: caller rejects immediately
self.in_flight += weight
return weight # permits actually taken, for an exact release
def release(self, taken):
with self._lock:
self.in_flight -= taken
ROUTE_WEIGHTS = {"/export": 10, "/lookup": 1}
limiter = ConcurrencyLimiter(limit=30)
def admission_middleware(handler):
# `request` and `Response` stand in for your framework's types.
def wrapped(request):
taken = limiter.try_acquire(ROUTE_WEIGHTS.get(request.path, 1))
if not taken:
# Fast, explicit shed. A gRPC server would return RESOURCE_EXHAUSTED or UNAVAILABLE.
return Response(status=503, headers={"Retry-After": "1"})
try:
return handler(request)
finally:
# Runs on success, exceptions and cancellation, so permits never leak.
limiter.release(taken)
return wrapped
A framework-agnostic weighted concurrency limiter wraps every handler, rejects with 503 and Retry-After when permits run out, and releases permits on every exit path.
How does adaptive concurrency adjust the limit automatically?
A fixed limit is only correct for the traffic mix and dependency health you tuned it against. Adaptive concurrency recalculates the limit continuously. It compares observed latency with the lowest latency recently seen, which approximates service time when nothing is queued.
AIMD raises the limit by one after each window in which latency stays below the baseline times a tolerance, such as 2×. When latency exceeds that threshold or requests time out, it multiplies the limit by a factor such as 0.9. Gradient algorithms scale the limit proportionally, using limit × (minRTT / observedRTT) plus a small headroom term so the limit can still probe upward.
Illustrative estimate: take a limit of 100, a baseline of 20 ms and a smoothed latency of 40 ms. The gradient is 0.5, so the new limit is 50 plus headroom. Queueing falls and latency eases to 25 ms. The next gradient is 0.8, which gives about 40. Latency then settles near 21 ms and the limit holds in the low 40s. If the algorithm used raw samples instead, one 200 ms garbage-collection pause would cut the limit to 10. Use an exponential moving average or a percentile over a window.
Failure modes:
- Oscillation: aggressive gains overshoot in both directions.
- Limits that collapse too low: noisy latency keeps the gradient below 1. A stale baseline from a quiet period, or from cheap cache hits, has the same effect. Reset the baseline periodically.
- Slow dependencies: when a downstream service is slow, a local limit only sheds load in front of it. It cannot make that service faster.
Each instance limits itself and has no view of fleet state. A global limit adds coordination traffic, staleness and a new failure dependency.
import math, time
class GradientLimiter:
"""Per-instance gradient limiter. It is not thread-safe and has no view of fleet state."""
def __init__(self, initial=100, min_limit=10, max_limit=1000, alpha=0.2, baseline_reset_s=30.0):
self.limit = float(initial)
self.min_limit, self.max_limit = min_limit, max_limit
self.alpha = alpha # weight of the newest sample in the EMA
self.smoothed_rtt = None # EMA of observed latency (ms)
self.min_rtt = None # baseline: approximates service time with no queue
self.reset_s, self.last_reset = baseline_reset_s, time.monotonic()
self.in_flight = 0
def try_acquire(self):
# Admission control: reject early instead of queueing past the limit.
if self.in_flight >= int(self.limit):
return False
self.in_flight += 1
return True
def release(self, rtt_ms):
self.in_flight -= 1
# Smooth the signal so a single 200 ms GC pause cannot slash the limit.
s = self.smoothed_rtt
self.smoothed_rtt = rtt_ms if s is None else self.alpha * rtt_ms + (1 - self.alpha) * s
# Periodically re-seed the baseline. Otherwise a minimum from a quiet
# period or from cheap cache hits keeps the gradient below 1 indefinitely.
now = time.monotonic()
if self.min_rtt is None or now - self.last_reset > self.reset_s:
self.min_rtt, self.last_reset = self.smoothed_rtt, now
self.min_rtt = min(self.min_rtt, self.smoothed_rtt)
# Cap the gradient at 1.0 so the limit grows only through headroom.
# Floor it at 0.5 so a single update can cut the limit by half at most.
gradient = max(0.5, min(1.0, self.min_rtt / self.smoothed_rtt))
new_limit = self.limit * gradient + math.sqrt(self.limit) # sqrt term keeps probing upward
# Move only part of the way toward new_limit to damp oscillation.
blended = 0.8 * self.limit + 0.2 * new_limit
self.limit = max(self.min_limit, min(self.max_limit, blended))
A minimal gradient-style concurrency limiter that smooths latency with an EMA, periodically resets a stale baseline, clamps and damps each adjustment to limit oscillation, and enforces the resulting in-flight limit on a single instance.
Which queue management and priority tactics complement shedding?
A concurrency limit decides how many requests run at once. The queue in front of it decides which waiting requests are still worth running. A request that waits past its caller's timeout still consumes a slot and produces a response nobody reads.
| Tactic | Purpose | Trade-off |
|---|---|---|
| Bounded queue | Caps memory and worst-case wait | Rejects short bursts that would have drained |
| Drop expired requests | Skips work the caller already abandoned | Every request needs a deadline |
| Deadline propagation | Downstream sees remaining budget, not a fresh timeout | Every hop must forward it; absolute timestamps suffer clock skew |
| LIFO under load | Serves the newest requests, whose callers are most likely still waiting | Oldest requests starve, widening the tail |
| CoDel-style delay target | Sheds only when queueing delay stays above a target, so brief spikes pass | Target and interval need tuning per service |
Criticality-aware shedding tags traffic at the edge so the least valuable work goes first. A typical order sheds batch jobs first, then retries, then non-critical user calls, and user-facing requests last.
Retries need their own tier because a retry during overload adds load without adding demand. Shedding retries before first attempts damps that amplification.
Health checks need the opposite treatment. If a user-traffic limit rejects them, the load balancer ejects instances that are busy but alive. Their load then shifts to peers and overloads those too. Serve health checks from a small reserved pool and keep them cheap.
Classification can fail. Untagged traffic falls into some default tier, and any default will be wrong for some callers. Alert on the untagged rate and audit callers that claim top priority. Strict priority also causes priority inversion. Batch can starve indefinitely, and a critical request that depends on a low-tier downstream call gets shed at that hop. To prevent this, reserve a minimum share for each tier and propagate criticality alongside the deadline.
How do clients, load balancers and autoscaling need to cooperate with load shedding?
A rejection only reduces load if the caller treats it as a signal to back off. If every client retries a shed request three times right away, one rejection becomes four requests, and the overloaded service sees more traffic than before it started shedding.
Clients: retry budgets, backoff and throttling
Give each client a retry budget, for example retries capped at 10% of recent successful requests. When the budget runs out, the client fails fast and does not queue more retries. Space the retries that remain with exponential backoff plus full jitter: the delay is a random value between zero and base * 2^attempt, capped at a maximum. The randomness matters because without it, clients rejected at the same moment retry at the same moment. Clients can also throttle themselves. If the recent rejection ratio climbs, drop a matching fraction of requests locally before they go out, so the server stops paying even the cost of saying no.
Load balancers: rejections are not failures
A fast rejection from a shedding instance means the instance is working as designed. If health checks or outlier detection count those rejections as errors, the balancer ejects the instance. Its traffic then moves to the remaining instances, which overload and get ejected in turn. Exclude shed responses from health signals.
Use least-outstanding-requests routing instead of round-robin. Round-robin keeps sending traffic to an instance that is already at its concurrency limit. Least-outstanding-requests sends new work to instances with free slots.
Autoscaling: shedding buys time
Autoscaling takes time to add capacity. New instances have to be scheduled, started and warmed up, so the extra capacity arrives late. Shedding keeps latency bounded until it does. Include the shed rate in your scaling signals. Otherwise a fleet that holds CPU steady by rejecting traffic can look healthy to the autoscaler.
import random
import time
class Shed(Exception):
"""Raised when the server rejects a request because it is overloaded (e.g. 503/429)."""
class RetryBudget:
"""Token bucket: each success earns `ratio` tokens and each retry costs 1.
Retries therefore stay at about 10% of successful traffic. Not thread-safe as written."""
def __init__(self, ratio=0.1, max_tokens=10.0):
self.ratio = ratio
self.max_tokens = max_tokens # caps the burst of retries allowed after a quiet period
self.tokens = max_tokens
def on_success(self):
self.tokens = min(self.max_tokens, self.tokens + self.ratio)
def try_spend(self):
if self.tokens >= 1.0:
self.tokens -= 1.0
return True
return False # budget exhausted: caller should fail fast
def backoff_delay(attempt, base=0.1, cap=5.0):
# Full jitter: pick uniformly from [0, min(cap, base * 2^attempt)] so that
# clients rejected at the same moment do not all retry at the same moment.
return random.uniform(0, min(cap, base * 2 ** attempt))
def call_with_retries(send, budget, max_attempts=3):
for attempt in range(max_attempts):
try:
result = send()
budget.on_success()
return result
except Shed:
out_of_attempts = attempt == max_attempts - 1
if out_of_attempts or not budget.try_spend():
raise # stop adding load to an overloaded service
time.sleep(backoff_delay(attempt))
A minimal client-side sketch that limits retries of shed requests with a token-bucket retry budget and spaces the remaining retries with exponential backoff and full jitter.
How do you observe, test and budget for load shedding in production?
Each shedding node should emit a small, fixed set of signals. Without them, an overloaded fleet that is shedding traffic can look the same as a healthy one.
| Metric | What it tells you |
|---|---|
| In-flight count | Actual concurrency against the limit |
| Current limit | Whether adaptive control is shrinking capacity |
| Rejections by priority | Whether critical traffic is being protected |
| Queueing delay | Overload building before rejections start |
| Goodput vs. throughput | Useful completions vs. raw work performed |
Keep shed responses out of the error budget for admitted traffic, and track them against their own budget. Shedding accepts a measured rejection rate so that admitted requests keep their latency. A single combined error SLO would page on-call engineers for the protection working as designed, while a slow collapse in goodput went unnoticed. Alert on shedding of high-priority traffic, on sustained shedding of any priority, and on goodput falling while throughput holds.
Load tests must go past capacity, to two or three times the expected peak. Tests that stop at the expected peak never exercise the limiter. The pass criteria are that p99 latency for admitted requests stays flat, critical traffic is shed last, and the limit recovers once the load drops.
The upfront work covers limiter middleware, priority tagging, client retry handling, and overload tests. Weigh it against engineering time spent on incidents.
Illustrative estimate (assumed numbers, not a benchmark). Annual savings = incidents avoided × engineers × hours per incident + manual tuning hours.
Assume:
- 6 cascading-failure incidents avoided per year
- 5 engineers per incident
- 8 hours per incident
- 100 hours per year of manual capacity tuning
Annual savings = 6 × 5 × 8 + 100 = 340 engineer-hours per year. Compare that figure with your build estimate.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to load shedding microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.