What do liveness, readiness, and startup probes actually decide?
Orchestrators and load balancers do not interpret health checks. They act on a pass or a fail, and each probe type maps that result to a different action. Kubernetes uses three probes, and each one answers a separate question:
- Liveness: Is this process stuck in a state that only a restart will fix?
- Readiness: Should this instance receive traffic right now?
- Startup: Has initialization finished, so the other probes can begin?
Because the actions differ, the same dependency failure needs a different answer on each probe. Suppose the database becomes unreachable. Restarting the process will not bring the database back, so the liveness probe should keep passing. Readiness is less obvious, and later sections cover it. The cost is plain, though. If every replica reports not-ready together, the load balancer has nowhere to send requests.
| Probe type | Question answered | Action on failure | External dependencies belong in it? |
|---|---|---|---|
| Liveness | Is the process deadlocked or wedged? | Container is killed and restarted | No. A restart cannot repair a remote system |
| Readiness | Can this instance usefully serve traffic now? | Removed from service endpoints; not restarted | Rarely. Only ones this instance cannot function without, and only when other replicas would not fail the same way |
| Startup | Has initialization completed? | Liveness and readiness are held off until it passes; repeated failure past the threshold restarts the container | Only what startup strictly requires, such as loading config or warming a local cache |
The core principle is that a health check reports whether this process can usefully serve traffic. It does not report whether the whole dependency graph is healthy. Each probe should only be able to fail for a reason that its triggered action can actually fix.
Who should use shallow, local-only health checks?
This approach is for engineers who own services that run as multiple replicas behind an orchestrator or load balancer, where probe results trigger automated actions. Three groups in particular need it:
- Service owners who write the
/livezand/readyzhandlers and decide what they check. - Platform and SRE teams who set probe defaults, review deployment manifests, and get paged when a shared dependency slows down and takes a whole fleet out of rotation.
- Teams with shared stateful dependencies. Any service where many replicas share one database, cache, message broker, or auth service is exposed to the correlated-failure pattern described below.
It matters less for a single-instance service with no automated restart or traffic shifting. In that case, nothing acts on the probe result, so a deep check only adds load. It also matters less for batch jobs that are not health-checked at all. If a pass or fail from your endpoint can cause a restart or remove an instance from rotation, the rules in this article apply.
How do deep dependency checks turn a slow database into a fleet-wide outage?
Consider a service with many replicas behind a load balancer, all sharing one database. The readiness endpoint runs a query against that database with a short timeout. Then the database slows down, perhaps from lock contention or a heavy batch job. It is still answering, just slowly.
┌───────────────┐
│ Load balancer │ → no ready backends → 503 for all traffic
└──────┬────────┘
┌───────────────┼───────────────┐
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│replica 1│ │replica 2│ │replica 3│
│ UNREADY │ │ UNREADY │ │ UNREADY │
└────┬────┘ └────┬────┘ └────┬────┘
└───────────────┼───────────────┘
┌──────▼──────┐
│ database │ slow, not down
└─────────────┘
The failure unfolds in three steps:
- All probes fail together. Every replica's check goes through the same slow dependency, so they all exceed their timeout within the same probe period.
- The orchestrator removes every replica at once. Each replica looks broken, so each is pulled from rotation. Nothing in the check can tell "I am broken" apart from "something we all share is slow."
- A partial outage becomes a total one. Before the probes failed, some requests still succeeded, and others, such as cached reads, didn't touch the database at all. After removal, every request fails.
If the deep check is wired to liveness instead, the outcome is worse. Failing replicas get restarted. Each restart opens a fresh connection pool, re-runs migrations or cache warm-up, and retries queries. All of that adds load to the database that is already struggling, which keeps the probes failing and keeps the restart loop going.
The probes also add load in steady state. Illustrative estimate: assume 60 replicas, each probed every 5 seconds, with each probe running one query. That adds 12 queries per second to the database before any user traffic. Under normal conditions this costs little. During an incident, it is extra load on the component that is already the bottleneck.
┌─────────────────┐
client traffic ──►│ Load balancer │──► 503: no ready backends
└────────┬────────┘
┌────────────────┼────────────────┐
▼ (3) removed ▼ (3) removed ▼ (3) removed
┌─────────┐ ┌─────────┐ ┌─────────┐
│replica 1│ │replica 2│ │replica 3│
│ UNREADY │ │ UNREADY │ │ UNREADY │
└────┬────┘ └────┬────┘ └────┬────┘
│ (2) readiness │ (2) readiness │ (2) readiness
│ query times out│ query times out│ query times out
└────────────────┼────────────────┘
▼
┌─────────────────┐
│ shared database │ (1) slow, not down
└─────────────────┘
▲
│ extra load on the database:
│ replicas × probe rate
│ + reconnects and warm-up if the
│ deep check drives liveness
A slow shared database makes every replica's deep readiness check time out in the same probe period, so the load balancer has no ready backends left and a partial slowdown becomes a full outage.
When should a health check include a dependency, and when should it be avoided?
For backend and platform engineers running services behind an orchestrator or load balancer, the deciding question is what the probe's consequence can actually fix.
Liveness triggers a restart, so it should only detect local states that a restart can fix: a deadlocked event loop, a worker pool with every thread stuck, or in-process state the code knows is corrupt. A restart cannot repair a remote database. Restarting every replica because the database is slow adds cold starts and connection storms to an existing incident.
Readiness removes an instance from rotation. It should reflect local capacity: initialization finished, not draining for shutdown, and not saturated beyond what it can serve.
| Dependency type | Example | Gate readiness? | Why |
|---|---|---|---|
| Hard local requirement | Sidecar proxy, local cache file | Yes | Failure is specific to this instance, and other replicas can serve. |
| Shared remote dependency | Primary database, auth service | No | Every replica fails together, which leaves no healthy targets. |
| Instance-specific connection | Broken connection pool on one replica | Yes | Only this replica is degraded, so shifting traffic helps. |
Readiness gating helps only when the failure is isolated to a subset of instances. One legitimate exception is a load balancer that spans zones with zonal dependencies, such as a per-zone cache. Failing readiness in the affected zone moves traffic to zones that are still healthy.
The isolation assumption is easy to get wrong. A "per-instance" pool error is often caused by something shared, such as DNS, expired credentials, or a database at its connection limit. All replicas then go unready at once. Depending on the load balancer, it either fails open or returns errors for everything.
How does the shallow-probe approach work end to end?
The design splits health information into two paths, each with a different consumer.
Path 1: probes, consumed by automation. The orchestrator calls local-only endpoints on a timer. Each endpoint reads state the process already holds in memory and returns a status code:
- At boot, the startup probe holds off the other probes until initialization finishes.
- The worker loop updates a heartbeat timestamp on every iteration. The liveness handler compares that timestamp with the current time and fails only if the loop has stopped making progress. The orchestrator restarts the container, which is the one action that can fix a wedged process.
- The readiness handler reads flags for initialization, draining, and saturation, plus any cached instance-local signals computed by a background task. When it fails, the instance leaves rotation, and the other replicas keep serving.
None of these handlers makes a network call, so a slow dependency cannot make them fail. This keeps the replicas from all failing in the same probe period.
Path 2: the request path, consumed by callers and humans. Dependency health is measured where it matters, on real outbound calls:
- Every outbound call has a timeout and goes through a circuit breaker.
- When a dependency slows down, timeouts bound the cost per request. When failures continue, the breaker opens and calls fail immediately with an explicit error or a degraded response.
- Per-instance metrics for error rate, latency, and breaker state feed dashboards and alerts. A separate diagnostic endpoint shows dependency status to operators, and no automation reads it.
When the shared database slows down under this design, replicas stay in rotation. Requests that need the database fail fast with a clear error. Requests that don't need it, such as cached reads, keep succeeding. Alerts fire on the dependency itself. The orchestrator takes no action because restarting or removing replicas would not help.
How do you implement shallow liveness and readiness checks?
The liveness handler should answer one question: is this process still making progress? Have the worker loop write a monotonic timestamp on each iteration, and have the handler compare it to the current time. If the heartbeat is older than a few loop intervals, return 503. The handler does no network I/O. A static 200 OK would keep answering from a wedged process whose HTTP thread is alive but whose workers are deadlocked. A deep check would restart healthy pods whenever the database is slow.
Readiness reports local state only:
- Initialization: whether caches are warm and config is loaded.
- Draining: set to failing as soon as SIGTERM arrives, so traffic moves away before the process exits.
- Saturation: for example, in-flight requests above a threshold.
If an internal check is expensive, such as scanning a queue, compute it on a background timer and have the probe read the cached result.
In Kubernetes, use startupProbe with failureThreshold × periodSeconds above your worst observed boot time, such as 30 × 5s. Liveness and readiness probes do not run until the startup probe succeeds, so slow boots are not killed. Set timeoutSeconds above your longest expected GC pause. Give liveness a higher failureThreshold than readiness, because a false restart costs more than a brief removal from rotation.
Illustrative estimate, assuming periodSeconds: 10, timeoutSeconds: 2, and failureThreshold: 3: a wedged instance is detected after about 22 to 32 seconds. That window is how long a bad instance keeps serving traffic. Halving the threshold shortens the window, but load spikes then cause probes to flap.
spec:
terminationGracePeriodSeconds: 30 # time to drain after SIGTERM flips readiness
containers:
- name: orders-api
image: registry.example.com/orders-api:1.0.0
ports:
- name: http
containerPort: 8080
# Startup: tolerate up to 30 x 5s = 150s of boot (above worst observed boot).
# Liveness and readiness do not run until this succeeds.
startupProbe:
httpGet:
path: /livez
port: http
periodSeconds: 5
failureThreshold: 30
# Liveness: heartbeat-age check only. Higher threshold than readiness,
# because a false restart costs more than brief removal from rotation.
# Detection after ~2x10s + 2s timeout + up to 10s phase = ~22-32s (illustrative).
livenessProbe:
httpGet:
path: /livez
port: http
periodSeconds: 10
timeoutSeconds: 2 # above the longest expected GC pause
failureThreshold: 3
# Readiness: init state, draining flag, in-flight saturation (cached).
# Fails faster so traffic shifts away during load spikes or shutdown.
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
successThreshold: 1
A Kubernetes probe configuration where a startup probe covers slow boots, liveness checks only the local heartbeat with a lenient threshold, and readiness fails faster to move traffic away during draining or saturation.
What are the alternatives to deep dependency checks in probes?
Probes are the wrong place to learn that a dependency is unhealthy. The request path is the right place, because each call can report its own outcome accurately and immediately.
Wrap every outbound call in a timeout shorter than your own caller's timeout, so a slow database costs a bounded amount of time per request. Add a circuit breaker. After a run of failures it opens and rejects calls immediately for a cooldown period, then lets a few trial requests through. Use bounded retries with exponential backoff and jitter only for idempotent operations. Unbounded or synchronized retries multiply load on a dependency that is already struggling. When the breaker is open, return a fast, explicit error such as a 503 with a clear reason, or a degraded response such as cached data. The caller then gets an accurate signal for that request, and the instance stays in rotation for the work it can still do.
Without timeouts and breakers, requests pile up waiting on the dead dependency until threads or connection pools run out. That is a self-inflicted outage. Degradation has its own risk. Serving stale data can hide a real outage, so emit metrics for breaker state, fallback rate, and data age.
Deep dependency status still belongs somewhere. Expose it on a separate diagnostic endpoint and in metrics and alerts. Humans and dashboards read these sources. Automation should never act on them. A diagnostic endpoint can reveal internal topology, so put it behind access control. Give it a name like /debug/dependencies so nobody later mistakes it for a probe target and wires it into one.
// Node 18+: global fetch and AbortSignal.timeout.
class CircuitBreaker {
private failures = 0;
private openedAt = 0;
constructor(private maxFailures = 5, private cooldownMs = 10_000) {}
async call<T>(fn: () => Promise<T>): Promise<T> {
const open = this.failures >= this.maxFailures;
if (open && Date.now() - this.openedAt < this.cooldownMs) {
// Fail fast. No socket or worker waits on a dependency we believe is down.
throw new Error("circuit_open");
}
// Breaker is closed, or the cooldown has elapsed and this is a trial call.
// Simplification: every request after the cooldown is a trial, not just a few.
try {
const result = await fn();
this.failures = 0; // A success closes the breaker.
return result;
} catch (err) {
this.failures++;
if (this.failures >= this.maxFailures) this.openedAt = Date.now(); // (Re)open.
throw err;
}
}
}
const inventory = new CircuitBreaker();
// Base address of the inventory service, supplied by configuration.
const INVENTORY_BASE_URL = process.env.INVENTORY_BASE_URL ?? "";
async function getStock(sku: string): Promise<{ status: number; body: unknown }> {
try {
const body = await inventory.call(async () => {
// 300 ms budget, kept shorter than our own caller's timeout.
const res = await fetch(`${INVENTORY_BASE_URL}/stock/${encodeURIComponent(sku)}`,
{ signal: AbortSignal.timeout(300) });
if (!res.ok) throw new Error(`inventory_http_${res.status}`);
return res.json();
});
return { status: 200, body };
} catch {
// Explicit per-request error. Liveness and readiness probes are unaffected.
return { status: 503, body: { error: "inventory_unavailable" } };
}
}
A dependency call with a 300 ms timeout inside a minimal circuit breaker that returns a fast, explicit 503 while the breaker is open, so the request path reports the failure and probes never have to.
What do shallow checks miss, and how do you cover the gap?
The trade-off is that a shallow check only proves the process can answer HTTP. It says nothing about whether the requests that process serves succeed. If one replica starts with a rotated-out database credential, or its connection pool fills with dead connections after a network blip, /healthz keeps returning 200 while every real request on that instance fails. The load balancer sees a healthy target and keeps sending it a full share of traffic, so a fraction of users get errors that no probe will ever catch.
Cover this gap with signals taken from real traffic:
- Per-instance error rate and latency. Label request metrics with the pod or instance identity, not only the service name. Aggregate dashboards hide one bad replica out of twenty.
- Outlier alerting. Compare each replica against its peers, not against a fixed threshold. One replica at 100% errors while the others sit near zero is an instance problem, and you can usually fix it by restarting or draining that replica.
- Request-path handling. Fail fast with clear errors, set timeouts on every outbound call, and let the connection pool evict and replace broken connections rather than reuse them indefinitely.
Some instance-local signals can go into readiness safely: "this replica's pool has had zero successful checkouts for 30 seconds," or "credentials failed to load at startup." This is safe only when the condition belongs to that replica. Before wiring a signal in, ask whether a shared outage would trip it on every replica at the same moment. If it would, keep it out of the probe.
Checklist for reviewing an existing health endpoint, and what it costs
If a check would fail on every replica at the same moment during a shared outage, it belongs in alerting, not in a probe. That test is the core of reviewing any existing health endpoint. Walk each one through these questions:
| Question | Red flag | Recommended fix |
|---|---|---|
| What does the endpoint call? | Database, cache, or downstream service | Check only in-process state |
| Which probe uses it? | Same endpoint for liveness and readiness | Separate endpoints with separate intent |
| What happens on failure? | Restart or traffic removal on every replica at once | Limit liveness to conditions a restart fixes |
| Can that action fix the condition? | Restarting cannot repair a remote outage | Report via metrics; let circuit breakers shed load |
| Does it do network I/O? | Yes, synchronously per probe | Read cached state updated in the background |
| What is its timeout? | Longer than the probe timeout, or unset | Bound internal calls below the probe timeout |
| What load does it generate? | Probe traffic scales with replica count | Calculate it, then remove or cache the call |
Illustrative estimate (assumed numbers, not a benchmark): probe load on a dependency = replicas × probes per replica ÷ period. With an assumed 200 replicas, 2 probes each, and a 10-second period, that is 200 × 2 ÷ 10 = 40 requests per second hitting the database from health checks alone. That load does not drop when the database slows down.
Shallow probes take less effort to write and maintain. That does not remove the work. It moves it into timeouts, circuit breakers, and dependency metrics. This is the better place for the effort, because each of those tools acts on the caller without restarting it. Deep checks cost more time over a service's life. When they fail, they produce fleet-wide restart storms. Logs from those storms point at every service except the one that failed, so diagnosis is slow.
How can this approach save engineering time?
Most of the savings come from incidents that never happen or that are faster to diagnose, plus less ongoing maintenance:
- Fewer amplified incidents. A slow dependency stays a dependency problem. It does not turn into a fleet-wide outage of every service that checks that dependency in its probes. One team handles one incident, instead of several teams joining a bridge for an outage that their own services did not cause.
- Faster diagnosis. When probes fail only for local reasons, a failing probe points at the instance that reports it. Dependency problems show up in breaker-state and per-dependency error metrics, labelled with the dependency's name. Engineers can skip the step of working out which shared component caused a wave of restarts.
- Less probe tuning. Deep checks need timeouts and thresholds tuned against the latency of every dependency they call, and they need retuning whenever that latency changes. Local checks depend only on the process's own behaviour, so their settings stay stable.
- Simpler reviews and onboarding. The rule "probes read in-memory state only" is easy to state and enforce in code review, and the checklist above can be applied the same way to every service.
- No restart-loop recovery work. With nothing in liveness that a remote outage can trip, nobody has to temporarily disable probes or scale deployments down by hand to break a restart loop during an incident.
Illustrative estimate (assumed inputs, not a measured result): assume a deep-check cascade causes 2 incidents a year, each pulling 5 engineers into a 3-hour response plus 1 engineer-day of follow-up. That is 2 × (5 × 3 + 8) = 46 engineer-hours a year. Assume moving the dependency checks out of the probes, and adding timeouts and breakers where they are missing, takes 2 engineer-days (16 hours) per service. Under these assumptions, the change pays for itself within the first year for that service. Use your own incident history in place of these inputs. The arithmetic is the same.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice health check dependencies, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.