What are liveness, readiness and deep health checks, and who consumes each?
A service usually exposes three health signals. Each one is read by a different consumer, and each consumer takes a different action. Liveness tells the orchestrator whether the process is stuck and should be restarted. Readiness tells the load balancer or orchestrator whether this instance should receive traffic right now. Deep, or diagnostic, health shows humans and dashboards the state of the service and its dependencies.
| Signal | Consumer | Action taken | What it should check | Failure consequence |
|---|---|---|---|---|
| Liveness | Orchestrator | Restart the container or process | The process can still make progress: no deadlock, event loop responsive | Process is killed and restarted |
| Readiness | Load balancer, orchestrator | Add or remove the instance from rotation | This instance can usefully serve requests now: started, warmed up, not overloaded | Instance stops receiving traffic |
| Deep health | Humans, dashboards, alerting | Investigate and diagnose | Dependency reachability, latency, queue depth, config state | None automatic; it informs people |
The core rule is that readiness should answer only one question: can this instance usefully serve traffic right now? Most cascades start when the signals are mixed. Suppose a shared database check is placed in readiness or liveness. When that database slows down, every instance fails the check at the same moment. Then either the whole fleet is pulled from rotation or the whole fleet restarts, and the database outage becomes a total outage.
This split concerns two groups of engineers. Backend engineers write the endpoints and decide what each one checks. Platform engineers configure the probes and load balancers that act on those checks. Both groups need to agree on what each signal means.
How does a readiness cascade spread from one dependency to the whole fleet?
Consider a shared Postgres primary that starts timing out under lock contention. It isn't down, just slow. Each step below follows from the one before it:
- Every caller checks the same thing at the same time. If the readiness endpoint of the
ordersservice runs a database query, all replicas hit the same timeout within one probe interval. Their failures are correlated, so having many replicas does not help. - The orchestrator removes every endpoint. After the failure threshold is reached, each replica is marked not-ready. No healthy subset remains to absorb traffic.
- The failure moves upstream.
checkouthas a readiness check that callsorders. That call now fails, socheckoutgoes not-ready, and the API gateway in front ofcheckoutfollows. - Healthy capability is lost.
checkoutcould have served cached carts, andcatalognever touched the database at all. Both are now unreachable.
gateway [x][x][x] not-ready: checkout unready
|
checkout [x][x][x] not-ready: orders unready
|
orders [x][x][x] not-ready: DB query timed out
|
postgres (slow, still up)
Platforms differ in what happens when every endpoint reports not-ready. Some load balancers fail open and spread traffic across all targets, treating "everything unhealthy" as more likely a bad check than a real outage. Others, including typical orchestrator service routing, send traffic to no endpoints, and clients get connection errors or 503s. Find out which behavior your platform has, and test it, before you count on a fail-open safety net.
clients
|
v
+-------------------------------+
| routing: 0 ready endpoints |--> 503s / connection errors
+-------------------------------+
|
+-------------------------------+
| gateway [x] [x] [x] | not-ready: checkout unready
+-------------------------------+
| |
+--------------+ +--------------+
| checkout | | catalog |
| [x] [x] [x] | | [ok][ok][ok] | healthy, never touches DB,
+--------------+ +--------------+ but unreachable via gateway
| not-ready: orders unready
+--------------+
| orders |
| [x] [x] [x] | not-ready: DB query timed out
+--------------+
|
+--------------+
| postgres | slow under lock contention, still up
+--------------+
[x] replica failed readiness, removed from endpoints
[ok] replica ready, but no traffic can reach it
readiness failures propagate bottom-up, layer by layer
A slow Postgres primary fails the readiness checks of every orders replica, then checkout and gateway in turn, until routing has no ready endpoints and even the healthy catalog service is unreachable.
Why do recursive deep health checks amplify load on a struggling dependency?
A deep health check turns every probe into a distributed call. When an orchestrator probes a caller instance, that instance checks its downstream services, and those services check theirs. Nothing coordinates or caches these calls, so each layer multiplies the traffic from the layer above.
Illustrative estimate (assumed inputs, not a measurement): 3 caller services with 20 replicas each, 2 probe sources per instance (orchestrator plus load balancer), a 5-second probe period, and recursion depth 2. Each caller checks the shared database directly and also calls an intermediate service's deep endpoint, which checks the same database. The intermediate service runs 10 replicas and receives its own probes.
| Source | Arithmetic | Requests/s to database |
|---|---|---|
| Callers' direct checks | 60 instances × 2 ÷ 5 s | 24 |
| Callers via intermediate | 60 × 2 ÷ 5 s × 1 hop | 24 |
| Intermediate's own probes | 10 × 2 ÷ 5 s | 4 |
| Total | 52 |
That load is constant and carries no user value. Its cost grows when the database slows down. By Little's law, in-flight checks equal rate × latency. If each check waits for a 5-second timeout, 52 req/s × 5 s means about 260 concurrent checks. Each one holds a pool connection on the database side and a thread or socket on the caller side. Those are the same resources that real requests need. Any retries on failed probes multiply the load again. The database gets the most health traffic at the moment it has the least capacity, and the callers exhaust their own pools while they wait for it.
What should a readiness check actually verify?
Readiness should answer one narrow question: can this instance serve a request right now, given what it controls? That means startup work is finished, configuration is loaded, required caches are warmed, connection pools are initialized, the listener is bound, and the instance is not draining.
Shared dependencies need a split. If the service cannot function at all without a dependency at boot, such as a schema it must read before handling traffic, gate startup on it. Once the instance is running, a temporarily unavailable shared database is not a readiness failure. Every replica sees the same outage, so failing readiness everywhere removes capacity without fixing anything.
The rule targets correlated failures. It is not a ban on external checks. Dependencies local to one instance are fair to check: a crashed sidecar, a full local disk, or a broken connection on this pod while its peers are healthy. These failures are uncorrelated, so pulling one instance out of rotation lets the others absorb its traffic.
Permissive startup readiness has a cost. If an instance reports ready before its caches are warm, the first requests hit cold paths and cause latency spikes. Warm the required caches before reporting ready, and accept this trade-off only for caches that are genuinely optional.
Liveness should be stricter still. It should check only unrecoverable local conditions, such as a deadlocked event loop. If liveness depends on downstream services, a single outage restarts the whole fleet. Every replica then cold-starts at once, which loads the struggling dependency again, and all in-memory state is lost.
The example below separates the three handlers and sets a draining flag on shutdown.
state = { started: false, draining: false, last_tick: now() }
on_boot():
load_config("/etc/orders/config.yaml")
read_schema_with_retry(db) # hard boot requirement: gates startup, not readiness
warm_required_caches() # avoid cold-path latency spikes on first requests
init_connection_pools()
bind_listener(port = 8080)
state.started = true
every 1s on the event loop:
state.last_tick = now() # stops advancing if the loop deadlocks
on_signal(SIGTERM):
state.draining = true # readiness fails, load balancer stops routing here
sleep(drain_grace = 10s) # let in-flight requests and routing updates settle
close_listener(); finish_inflight(); exit(0)
GET /startupz:
return state.started ? 200 : 503
GET /readyz:
if not state.started or state.draining: return 503
# Local, uncorrelated failures: fair to check
if not sidecar_ok(): return 503
if local_disk_free("/var/lib/orders") < 500MB: return 503
# Shared database outage: do NOT fail here. Every replica would see it.
# Let requests fail fast or degrade instead.
return 200
GET /livez:
# Only unrecoverable local conditions. Never call downstream services.
# Run this check off the main loop (watchdog thread) so a stall is observable.
if now() - state.last_tick > 30s: return 500
return 200
Separate startup, readiness and liveness handlers: startup gates on hard boot requirements, readiness checks only local state and a draining flag set on SIGTERM, and liveness detects only a stalled event loop.
How can services handle dependency failures without failing readiness?
Keep readiness answering one question: can this process serve requests? Everything that concerns a dependency's health belongs in the request path. There, a failure stays confined to the calls that need that dependency.
Bound every outbound call. Give each dependency call a timeout and a small, jittered retry budget. Without them, a slow database ties up worker threads until the instance looks dead, and the instance then fails readiness for reasons that never reach the probe.
Add a circuit breaker. After repeated failures, the breaker fails calls immediately instead of sending more traffic to a dependency that is already struggling. This prevents the load amplification described in section 3.
Serve degraded responses where you can. Cached data, partial results, or a feature switched off are often more useful to callers than an error.
Otherwise, fail per request. A fast 503 lets the caller's own retries and breakers react. Reserve readiness failures and load shedding for saturation of the instance itself.
The trade-off is deliberate. With shallow readiness, the load balancer keeps sending traffic to instances that will return errors while a hard dependency is down. You accept fast, scoped failures so that you never remove all capacity at once. For some clients, such as non-idempotent writers that retry badly, honest errors are worse than removal, so decide this per service.
Breakers and fallbacks need tuning and add code paths you have to test. They can also hide outages. Emit a metric whenever a breaker opens or a fallback is served, and alert on it.
In the example below, the dependency call is wrapped in a timeout and a breaker that returns cached data, and the readiness handler is unchanged.
// Stand-in for your metrics client; degraded mode must be visible.
declare const metrics: { increment(name: string, tags?: Record<string, string>): void };
// Base address of the pricing service, supplied by configuration.
declare const PRICING_BASE_URL: string;
type Price = { sku: string; cents: number };
const lastKnown = new Map<string, Price>(); // last good value per SKU
// Minimal breaker: opens after N consecutive failures, allows trial calls after a cooldown.
class CircuitBreaker {
private failures = 0;
private openedAt = 0;
constructor(private threshold = 5, private cooldownMs = 10_000) {}
async call<T>(fn: () => Promise<T>, fallback: () => T): Promise<T> {
if (this.failures >= this.threshold && Date.now() - this.openedAt < this.cooldownMs) {
metrics.increment("pricing.fallback", { reason: "breaker_open" });
return fallback(); // fail fast, add no load to a struggling dependency
}
try {
const result = await fn();
this.failures = 0; // a successful trial call closes the breaker
return result;
} catch {
if (++this.failures >= this.threshold) {
this.openedAt = Date.now();
metrics.increment("pricing.breaker_opened"); // alert on this
}
metrics.increment("pricing.fallback", { reason: "call_failed" });
return fallback();
}
}
}
const pricingBreaker = new CircuitBreaker();
export function getPrice(sku: string): Promise<Price> {
return pricingBreaker.call(
async () => {
// Hard 300 ms timeout so a slow dependency cannot pin request handlers.
const res = await fetch(`${PRICING_BASE_URL}/prices/${encodeURIComponent(sku)}`, {
signal: AbortSignal.timeout(300),
});
if (!res.ok) throw new Error(`pricing returned ${res.status}`);
const price = (await res.json()) as Price;
lastKnown.set(sku, price);
return price;
},
() => {
const cached = lastKnown.get(sku);
if (cached) return cached; // degraded: possibly stale, but useful
throw new Error("pricing unavailable"); // caller maps this to a fast 503 for this request only
},
);
}
// Readiness is untouched: it reports only whether this process can serve requests.
export function readiness(): { status: number } {
return { status: 200 };
}
A dependency call wrapped in a 300 ms timeout and a simple circuit breaker that serves last-known cached data or fails only the affected request. Each breaker open and each fallback emits a metric, and the readiness handler never consults the dependency.
How do you build a safe aggregated deep health endpoint?
A deep health endpoint is for humans and dashboards, not orchestrators. A request to it should never trigger a synchronous downstream call. If it did, every poll would add load to whichever dependency is already struggling.
Refresh in the background, serve from cache
A background refresher owns all dependency checks. On a fixed interval, with jitter so replicas don't probe in lockstep, it runs each check under a strict timeout and writes the result to an in-memory map. Checks touch only direct dependencies, such as a connection-pool ping or a lightweight query. They never call another service's deep endpoint, because that recursion is what turns one slow database into fleet-wide probe traffic. The handler only reads the map.
Each entry in the JSON response should carry:
nameandcriticality(criticaloroptional). Only critical failures change the top-levelstatus.status(healthy,degraded,unhealthy,unknown).last_checkedandlatency_ms.stale: true whenlast_checkedis older than a threshold, such as three refresh intervals.
The staleness flag matters because cached status lags reality in both directions. Early in an incident it can still show healthy, and after recovery it can still show unhealthy. Readers need to know how old each verdict is. A stale entry should report unknown rather than its last value.
Lock it down
The detailed response maps your topology: hostnames, dependency names, and failure modes. Keep it off public routes. Serve it on an internal port or path that requires authentication or network-level restriction. Rate-limit the handler, or cache the serialized response for a few seconds, so a flood of requests stays cheap.
import asyncio, random, time
INTERVAL = 10.0 # seconds between refresh rounds
TIMEOUT = 2.0 # hard cap on any single check
STALE_AFTER = 3 * INTERVAL # older than this, the verdict is not trusted
CHECKS = {}
cache = {}
async def run_check(name, crit, fn):
start = time.monotonic()
try:
await asyncio.wait_for(fn(), timeout=TIMEOUT)
status = "healthy"
except Exception: # timeouts and connection errors both count as failure
status = "unhealthy"
cache[name] = {"name": name, "criticality": crit, "status": status,
"last_checked": time.time(),
"latency_ms": round((time.monotonic() - start) * 1000)}
async def refresher():
while True:
await asyncio.gather(*(run_check(n, c, f) for n, (c, f) in CHECKS.items()))
# Jitter so replicas don't probe the same dependency in lockstep.
await asyncio.sleep(INTERVAL + random.uniform(0, INTERVAL * 0.2))
def snapshot():
"""Used by the HTTP handler: reads the cache only, never calls downstream."""
now = time.time()
deps = []
for e in cache.values():
stale = now - e["last_checked"] > STALE_AFTER
deps.append({**e, "stale": stale, "status": "unknown" if stale else e["status"]})
bad = [d for d in deps if d["status"] != "healthy"]
top = ("unhealthy" if any(d["criticality"] == "critical" for d in bad)
else "degraded" if bad else "healthy")
return {"status": top, "generated_at": now, "dependencies": deps}
A background refresher runs each direct-dependency check under a timeout with jitter, and the handler serves cached results with staleness marked as unknown and only critical failures making the top-level status unhealthy.
How does the dependency-isolated health model work end to end?
The pattern has four parts, and each one handles a different kind of failure:
- Startup gates on hard boot requirements: config, schema, required caches, pools, bound listener. Until these finish, the instance gets no traffic and is not restarted for being slow to boot.
- Readiness reports only local, uncorrelated state, plus the draining flag. A failure here removes one instance while its peers keep serving.
- Liveness detects only conditions a restart can fix, such as a stalled event loop.
- Dependency health lives in two places. The request path handles it with timeouts, breakers and fallbacks. The cached deep endpoint and per-dependency metrics report it to people.
Replay the earlier incident under this model. The Postgres primary slows down. Requests to orders that need the database hit their timeout. After repeated failures the breaker opens, so those requests get a cached response or a fast 503, and the database stops receiving extra load from orders. Readiness on every orders replica keeps returning 200 because nothing local is wrong. checkout does not check orders in its readiness, so it stays in rotation and serves cached carts. catalog never notices. The deep endpoint shows postgres as unhealthy with a fresh timestamp, and the breaker-opened and dependency-latency metrics fire alerts that name the database. The outage stays as large as the set of requests that actually need the slow database.
When should you use this approach?
Use dependency-isolated readiness and liveness when any of the following is true:
- Several replicas share a dependency. Examples include a database, a cache cluster, a message broker, or a third-party API. Any check against it fails on all replicas together, which is exactly the correlated failure that empties a pool.
- The service has useful work that doesn't need every dependency. Read paths with caches, endpoints backed by different stores, or features that can be switched off all benefit from staying in rotation.
- Services call each other in chains. Once one service's readiness depends on another's, a single failure can climb the whole call graph, as in the cascade above.
- Your routing layer sends traffic to zero endpoints when all are not-ready. Without a fail-open fallback, a fleet-wide readiness failure is a full outage.
- On-call engineers need to see which dependency is at fault. Per-dependency metrics and a cached deep endpoint name the cause. A fleet-wide not-ready state hides it.
When should you avoid it or adapt it?
The pattern is a default, not a law. Adjust it in these cases:
- The dependency really is per instance. If each replica has its own sidecar, local database, or dedicated connection, checking it in readiness is correct. Its failure isn't correlated, and removing the instance helps.
- There is no degraded mode, and clients handle errors badly. If every request needs the dependency and clients such as non-idempotent writers retry harmfully against 503s, removing instances may be the safer behaviour. Make this choice per service, with full knowledge that it can empty the pool, and confirm what your routing layer does in that state.
- Traffic is steered between independent failure domains. When each region or zone has its own copy of a dependency, a dependency-aware signal at the domain level can correctly drive failover between domains, because failures are not correlated across them. Keep that signal at the domain level and out of per-instance readiness inside the domain.
- The workload doesn't receive load-balanced traffic. Queue consumers and batch workers have little use for readiness. When a dependency is down they should pause consumption or back off, not report not-ready or restart.
- The service is small and simple. A single-dependency internal tool may not justify breakers and fallbacks. Shallow readiness and dependency-free liveness still cost almost nothing, so keep those even when you skip the rest.
What are the alternatives?
- Deep readiness that checks dependencies directly. This is the simplest to write and the most common source of cascades. It is acceptable only for the per-instance dependencies described above.
- Relying on fail-open load balancing. Some load balancers route to all targets when every target is unhealthy. This limits the damage but depends on platform behaviour you may not control. It also routes traffic blindly, including to instances that really are broken.
- Proxy-level outlier detection. A proxy or service mesh can eject individual endpoints based on observed request errors instead of probe results. It reacts to real traffic, but it has the same risk if all endpoints fail together. Check whether your proxy can cap the fraction of endpoints it ejects.
- Local load shedding and admission control. Instances reject work when their own queues, concurrency, or latency pass a limit. This complements shallow readiness rather than replacing it, since it handles saturation of the instance itself.
- External synthetic monitoring. Black-box checks from outside the system measure what users see. They are good for alerting and SLOs but should not drive per-instance traffic decisions.
- Push-based health reporting to a registry. Instances send heartbeats instead of being probed. The same rules apply: if the heartbeat content depends on a shared dependency, every instance deregisters at once.
How can this approach save engineering time?
The savings come mainly from incidents that stay small and are quicker to diagnose:
- Fewer fleet-wide incidents. A slow shared dependency degrades the requests that need it and leaves the rest of the system serving. Fewer engineers get pulled in, and fewer services need to be investigated.
- Faster triage. Per-dependency metrics, breaker-open alerts and a timestamped deep endpoint point at the failing dependency directly. Without them, on-call engineers first have to work out why every service went not-ready and which one went first.
- No restart storms to untangle. Liveness that ignores dependencies avoids mass restarts. Those lose in-memory state, cold-start every replica at once, and hide the original cause in a pile of crash-loop noise.
- Less probe tuning and flapping work. Local-only probes have predictable latency. Teams spend less time adjusting timeouts and thresholds to stop healthy instances from being removed during dependency slowdowns.
- Reusable building blocks. A shared library for the probe handlers, the breaker wrapper and the cached deep endpoint lets new services adopt the pattern without each team designing health checks from scratch. The checklist below turns reviews of existing services into a short, repeatable exercise.
The upfront cost is real: breakers, fallbacks and degraded paths need code, tuning and tests. That cost is paid once per service during normal work, not during an incident.
Probe tuning, observability, and a review checklist for existing services
Probes share the same parameters across platforms. The period sets how often the probe runs, and the timeout caps a single attempt. The failure threshold is how many consecutive failures trigger action, and the success threshold is how many consecutive passes restore the instance. The startup grace period suppresses probing while the process initializes. As an illustrative estimate, a 10-second period with a failure threshold of 3 removes an instance after about 30 seconds of sustained failure. A single slow response, such as a GC pause or a cold cache, should never be enough to remove an instance. Loosening thresholds reduces flapping but leaves genuinely broken instances serving traffic longer, so choose them based on how long you can tolerate a bad replica.
Dependency awareness belongs in observability. Emit latency and error-rate metrics for each dependency, and alert on those metrics. Use the deep endpoint for dashboards and incident triage, never for automated traffic decisions.
Shallow probes are cheap to write and easy to reason about. Breakers and fallbacks have an upfront cost, and their degraded paths need ongoing tests. That cost pays off in smaller incidents. A named failing dependency is quicker to debug than a fleet that has gone not-ready all at once.
| Question | Red flag | Fix |
|---|---|---|
| Does any readiness or liveness check make a network call? | Yes | Check local state only |
| Could all replicas fail at once from one external cause? | Yes | Remove the shared dependency from the probe |
| What happens if every instance reports not-ready? | Total outage | Know your platform's behaviour; keep probes shallow |
| Is there a degraded mode, visible in metrics? | No | Add a fallback and emit a degraded-state metric |
| Is the deep endpoint cached, time-bounded, non-recursive and protected? | Any "no" | Cache results, set deadlines, check direct dependencies only, restrict access |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice health check dependencies, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.