What are deep health checks in microservices, and what question should each health signal answer?

A deep health check is an endpoint that doesn't just report on its own process. It calls outward: it pings the database, opens a connection to the cache, and hits the health endpoints of downstream services. It returns green only if all of them answer. A shallow check confirms that the process is running and can serve a trivial request, without touching anything it doesn't own.

Teams adopt deep checks for an understandable reason. One URL that returns 200 or 503 looks like a complete answer to "is this service OK?" Load balancers, orchestrators, uptime monitors and on-call engineers can all point at the same endpoint. The trouble is that "OK" hides three different questions, and each has a different consumer that takes a different action.

Signal Question it answers Who acts on it Action on failure
Liveness Is the process alive and not wedged? Orchestrator Kill and restart the instance
Readiness Should this instance receive traffic right now? Load balancer or orchestrator Remove the instance from rotation
Dependency health Are the things I rely on working? Humans, dashboards, alerting Investigate, page, correlate

The first two signals trigger automated, mechanical responses. A restart can fix a deadlocked thread pool. It can't fix a database that is down for every instance at once. That gives the rule the rest of this article builds on: an automated health signal should reflect only conditions the instance itself can fix or control. Everything else belongs in observability, where a person can decide what to do.

How does a health check work, from probe to action?

Every health check, deep or shallow, runs the same loop. The difference lies in what happens inside the handler.

  1. A consumer polls on a schedule. An orchestrator agent (in Kubernetes, the kubelet on each node), a load balancer, or an uptime monitor sends a request to the endpoint at a fixed interval. Each request has a timeout.
  2. The handler evaluates conditions. A shallow handler reads in-process state, such as a heartbeat timestamp or a startup flag, and answers in microseconds. A deep handler fans out. It runs a probe query against the database, connects to the cache, and calls the /health endpoints of downstream services. It waits for all of them, which may happen in parallel or one after another, and then combines the results into one status.
  3. The handler returns a status. Usually this is HTTP 200 for healthy and 503 for unhealthy. A timeout counts as a failure.
  4. The consumer counts consecutive results. It does not act on a single failure. It acts once the number of consecutive failures reaches a threshold, and it restores the instance once enough consecutive successes come back.
  5. The consumer acts. A readiness failure removes the instance from the set of load-balancing targets. A liveness failure kills and restarts the process.

The key property of a deep check is in step 2. The status of every instance now depends on the response time of systems outside that instance. Each replica runs the same fan-out against the same dependencies on the same schedule, so their results are correlated. The sections below show what that correlation does under load.

How does a dependency health check trigger a cascading failure?

Picture a shared database that starts answering in 800 ms instead of 20 ms. It is not down. Most queries still succeed, just slowly. But every replica's /health endpoint runs a probe query with a 500 ms timeout, and every replica hits the same slow database at roughly the same moment. Every check fails together.

The orchestrator does what it was told. Failed readiness pulls every replica out of the load balancer. Failed liveness restarts them. Now no instance is serving traffic, including those handling endpoints that never touch the database. A partial slowdown has become a full outage.

Restarts make it worse. Each new process opens a fresh connection pool, warms caches with queries, and may run startup migrations or config fetches. That connection storm lands on the database while it is trying to recover, which keeps it slow. The slow database then fails the next round of checks, and the cycle repeats.

Chained checks spread the failure upward. If A's check calls B's /health, and B's calls C's, then C's database slowdown marks C, B, and A unhealthy. Services with no direct relationship to the database go dark with it.

Illustrative estimate (assumed inputs, not a measurement): 30 replicas × 0.2 probes/s (one every 5 s) × 4 dependencies per probe × 3 levels of graph depth ≈ 72 check requests per second. That load does no useful work. During an incident it holds connections open until timeout, on the exact systems that are already struggling.

          +----------------------------------------------------+
          | Orchestrator: probe every 5 s, 500 ms timeout      |
          |   readiness fail -> remove replica from LB         |
          |   liveness fail  -> restart replica                |
          +----------------------------------------------------+
               |               |               |
             probe           probe           probe
               v               v               v
 Clients  +-----------+   +-----------+   +-----------+
 ---X---> | Service A |   | Service B |   | Service C |
 (no      | /health --+-->| /health --+-->| /health   |
 target)  |   FAIL    |<--+   FAIL    |<--+   FAIL    |
          +-----------+   +-----------+   +-----------+
          <------ failure propagates up the chain ----  |
                                                        | probe query
                                                        v (times out)
                                              +--------------------+
                                              | Shared database    |
                                              | 20 ms -> 800 ms    |
                                              | slow, not down     |
                                              +--------------------+
                                                        ^
 Restarted replicas: new conn pools, cache warmup       |
 --------------------------------------------------------+
 connection storm keeps DB slow -> next probes fail -> repeat

 Illustrative estimate (assumed inputs, not a measurement):
 30 replicas x 0.2 probes/s x 4 deps x 3 levels ~= 72 checks/s

A slow shared database fails C's deep check, the failure propagates through B to A, the orchestrator pulls or restarts every replica, and the restart connection storm keeps the database slow.

Why is correlated failure the real problem with deep checks?

The extra load is a cost, but it isn't what turns a dependency blip into an outage. The deeper problem is that a deep check makes every instance's health depend on the same shared signal. If forty replicas all ping the same database, they don't fail independently. They fail together, within one probe interval of each other.

Removing an instance from rotation only helps under a specific condition: the fault is local to some instances, and healthy instances remain to absorb their traffic. A replica with a full disk, a wedged thread pool, or a bad node fits that model. Taking it out moves its requests to replicas that can serve them.

A shared dependency outage breaks that condition. When the database is unreachable for everyone, every replica reports unhealthy, and the orchestrator or load balancer removes all of them. No healthy instance is left to take the traffic. Requests that could have been served from cache, served in degraded form, or failed fast with a clear error now fail at the routing layer, often as a generic 503 that hides the real cause.

Some load balancers and service meshes soften this with fail-open behaviour. When every backend is marked unhealthy, they ignore health status and route to all of them. That can turn a total outage back into a partial one. Don't treat it as a safety net, though. Whether fail-open exists, what threshold triggers it, and whether it is enabled by default all vary by implementation and configuration. Test what your own stack does in this case. Don't assume it will save you.

Who should use shallow probes, and who still benefits from deep checks?

Service owners running more than one replica behind an orchestrator or load balancer should use split, shallow liveness and readiness probes. They are the people most exposed to correlated failure, because their fleet can be pulled from rotation all at once. This includes most teams deploying stateless HTTP or gRPC services on Kubernetes or a similar scheduler.

Platform and infrastructure teams that own deployment templates, Helm charts, or service scaffolding should adopt the pattern as the default. A probe configuration copied into dozens of services spreads its flaws to every one of them. Fixing the template once is cheaper than fixing each service.

SREs and on-call engineers are the intended consumers of deep dependency information. They should keep it, but read it through dashboards, alerts, and a diagnostic endpoint, not through probes that drive automation.

Teams with a single instance or no shared dependencies gain little from the split. A deep check is tolerable in that setting, because there is no fleet to fail together. The same applies when the only consumer of the check is a human-facing monitor that pages someone and takes no automated action.

When should deep dependency checks be used?

A check that calls outward is reasonable when its result drives a decision that a person makes, or an automated decision where the alternative target does not share the failing dependency.

  • Alerting and dashboards. A diagnostic endpoint or synthetic monitor that exercises the database, cache, and downstream calls is useful for detection. Its consumer is a person or an alerting rule that pages. No instance is restarted or removed.
  • Startup and deploy gates. A one-time dependency check during boot, or a post-deploy smoke test, catches a build that can't reach its database or has a bad credential. This runs once per rollout, not on every probe interval, and it stops a bad version from replacing healthy pods.
  • Per-instance dependencies. A sidecar proxy, a local socket, or a node-local agent can fail on one instance while working on others. Checking it in readiness keeps the fault local, which is the condition under which removing an instance from rotation helps.
  • Failover between independent failure domains. If traffic can move to another region or cluster that does not share the failing dependency, a check that reflects dependency health can drive that move. This only holds if the domains really are independent. If both regions depend on the same backing store, the check will fail in both at once and the failover has nowhere to go. Verify the independence before relying on it.

When should deep dependency checks be avoided?

Avoid wiring dependency checks into any signal that triggers automated restarts or traffic removal across a fleet:

  • Liveness probes. Never. A restart can't repair a remote database, and the restart storm adds load to it.
  • Readiness probes for shared dependencies. A database, message broker, shared cache, or downstream service is shared by every replica. Checking it in readiness makes all replicas fail together.
  • Chained checks. If your /health calls another service's /health, any failure deep in the call graph will mark every service above it unhealthy. That includes services whose own requests never touch the failing component.
  • Services with partial functionality. If some endpoints work without a given dependency, a deep check that fails on that dependency throws away the requests you could still have served.
  • Tight timeouts against dependencies that slow down under load. A dependency that gets slow, rather than going down, will turn into probe timeouts across the fleet at the moment it is already struggling.
  • Autoscalers and other automation reading a diagnostic endpoint. Once anything automated acts on it, a diagnostic endpoint becomes a deep check with all the risks above.

What should liveness and readiness endpoints actually check?

Start from what each probe's failure does. A failed liveness probe restarts the process, so liveness should test only conditions a restart would fix: a deadlocked event loop, an internal thread pool that will never drain, or in-process state that has become corrupted. It should make no network calls and do almost no work. Serve it from a path that does not share the request-handling worker pool. If it does, a liveness request queued behind slow user traffic times out, and the orchestrator kills a process that was busy, not broken. That restart removes capacity at the moment you need it most.

A failed readiness probe removes the instance from load balancing, so readiness should report whether this instance can accept work. That means startup has finished, configuration is loaded, caches are warm, and the instance has spare capacity. It can also cover a strictly required dependency that is local and recoverable per instance, such as a sidecar proxy or a local socket.

For any candidate check, ask: is this dependency local to this instance, or shared by all of them? If it is shared, such as a database, a downstream service or a message broker, every replica fails readiness together, and you have built the correlated failure described earlier.

The trade-off is that shallow checks can keep a broken replica in rotation. An instance with a misconfigured credential will pass readiness and fail every real request. Catching that is a judgment call. One option is a one-time credential check at startup rather than a recurring probe.

A minimal handler shape uses two routes. /livez returns 200 if a heartbeat timestamp, updated by the main loop, is recent. /readyz returns 200 only after the startup flag is set and the local sidecar socket accepts connections.

import socket, threading, time
from http.server import BaseHTTPRequestHandler, HTTPServer

HEARTBEAT_MAX_AGE = 10.0  # seconds without a main-loop tick before we call it stuck
SIDECAR_SOCKET = "/var/run/sidecar/proxy.sock"  # placeholder: local, per-instance dependency
last_heartbeat = time.monotonic()
startup_complete = threading.Event()

def sidecar_accepts():
    # A short local connect only. Keep the timeout well under the probe timeout.
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    s.settimeout(0.2)
    try:
        s.connect(SIDECAR_SOCKET)
        return True
    except OSError:
        return False
    finally:
        s.close()

class ProbeHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/livez":
            # No network calls. Only asks whether the main loop is still ticking.
            ok = time.monotonic() - last_heartbeat < HEARTBEAT_MAX_AGE
        elif self.path == "/readyz":
            # Instance-local only. No database or downstream calls: those are shared by every replica.
            ok = startup_complete.is_set() and sidecar_accepts()
        else:
            ok = None
        self.send_response(404 if ok is None else 200 if ok else 503)
        self.end_headers()

threading.Thread(target=HTTPServer(("0.0.0.0", 8081), ProbeHandler).serve_forever, daemon=True).start()

load_config(); warm_caches()  # placeholders for real startup work
startup_complete.set()
while True:
    last_heartbeat = time.monotonic()  # stops updating if the loop deadlocks
    process_next_batch()  # placeholder for request handling

This minimal standard-library Python service serves probes on a dedicated thread and port: /livez checks only a main-loop heartbeat, and /readyz checks the startup flag and a local sidecar socket, with no checks on shared remote dependencies.

How should you configure probes so they don't cause false failures?

Point each probe at its own route: livenessProbe at /livez, readinessProbe at /readyz. Sharing one endpoint means any condition that should only pull a pod from rotation can also get it restarted, and a restart is far more disruptive than a readiness flip.

Set timeoutSeconds well below periodSeconds. A probe that is still waiting when the next one fires is measuring queueing, not health. Set failureThreshold high enough that a single GC pause or slow scheduling tick does not count as a failure. Liveness should usually tolerate more failures than readiness, because acting on a false liveness failure costs much more.

Add a startupProbe for anything with a variable boot time, such as cache warming, migrations or JIT warm-up. While the startup probe is running, liveness and readiness checks are held off, so a slow boot is not mistaken for a hang. initialDelaySeconds works when boot time is predictable, but it is a fixed guess, and the guess gets tuned to the best case.

Tight probes are dangerous during an incident because the probe handler runs on the same CPU, thread pool and event loop as real traffic. When the pod is saturated, the probe is the first request to time out. The kubelet then restarts the pod, and its load shifts onto peers that are already overloaded.

Shallow readiness won't catch a build that boots cleanly but can't reach its database. Catch that at rollout instead. Run dependency verification in the startup path or in a post-deploy smoke test. Use minReadySeconds and progressDeadlineSeconds so a bad rollout stalls rather than continuing to replace healthy pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 4
  minReadySeconds: 15           # a new pod must stay Ready this long before it counts as available
  progressDeadlineSeconds: 300  # rollout reports Progressing=False if stuck; it does NOT auto-roll back
  strategy:
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}  # never remove a healthy pod before a new one is available
  selector:
    matchLabels: {app: orders-api}
  template:
    metadata:
      labels: {app: orders-api}
    spec:
      containers:
        - name: app
          image: registry.example.com/orders-api:1.4.2
          ports: [{containerPort: 8080}]
          startupProbe:            # holds off liveness/readiness until boot completes
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 60   # allows up to ~300s for cache warming or migrations
          readinessProbe:          # shallow: can this pod accept traffic right now?
            httpGet: {path: /readyz, port: 8080}
            periodSeconds: 5
            timeoutSeconds: 2      # well below periodSeconds so probes never queue
            failureThreshold: 3
          livenessProbe:           # process-level only; never checks downstream dependencies
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 6    # ~60s of failures before a restart, so GC pauses don't trigger one

A Deployment with separate liveness and readiness endpoints, timeouts shorter than probe periods, lenient liveness thresholds, a startup probe for slow boots, and rollout settings that stall a bad deploy instead of replacing healthy pods.

What are the alternatives to dependency health checks?

Dependency health still matters. It just belongs in places where a failure changes one request's outcome, not the fleet's membership.

Metrics and alerts track error rate and latency per dependency, so humans learn that the payments database is slow without the scheduler pulling every instance at once. Request-path resilience handles the failure where it happens. Timeouts bound how long a call can hold a thread. Circuit breakers stop calls to a dependency that is clearly down. Retries with backoff and jitter recover from transient errors without synchronized retry storms. Load shedding rejects excess work early. Graceful degradation lets requests that never touch the broken dependency keep succeeding. A diagnostic endpoint reports dependency status for dashboards and on-call engineers, and no probe or autoscaler reads it.

Approach Protects Cost Failure modes
Dependency metrics and alerts Detection time Alert design, tuning Weak alerting delays detection
Timeouts, retries with jitter Threads, transient errors Per-client config, tests Retries amplify load if unbounded
Circuit breakers Struggling dependency, caller latency Threshold tuning Flapping when badly tuned
Load shedding, degradation Unaffected requests Feature-level design Users see errors or partial responses
Diagnostic endpoint Human debugging Low Becomes a deep check if automation consumes it

These patterns take more engineering and testing than one deep check. Caching or computing status asynchronously reduces probe load, but the status goes stale, and it carries the same correlated risk if anything automated acts on it. A service that stays in rotation during an outage will serve errors, so its error responses need to be explicit and its load shedding deliberate.

How do you migrate an existing deep /health endpoint, and what does it cost?

Start by inventorying every consumer of the current endpoint: orchestrator probes, load balancer target checks, uptime monitors, and other services that poll it before making calls. Group access logs by source address and user agent. This usually turns up callers nobody documented, and each one will react differently when the endpoint's semantics change.

Next, split the endpoint into three:

  • Shallow liveness. Is the process responsive?
  • Local readiness. Is the instance initialised, not draining, and not saturated?
  • Diagnostic dependency report. Humans and dashboards read it. Nothing automated acts on it.

Keep the old path temporarily. Then move consumers one at a time: orchestrator first, then load balancers, then monitors. Each move can be rolled back on its own.

Finish with a game day. Degrade a shared dependency and confirm three things: instances stay in rotation, nothing restarts, and dependency alerts fire.

Prioritise this work if you have shared databases, deep call graphs, or a history of restart storms. A simple deep check is tolerable with a single instance or no shared dependencies. In those cases there is no fleet that can fail together.

The up-front cost covers three things:

  • alerting to replace what the deep check signalled implicitly
  • resilience patterns such as timeouts and circuit breakers
  • testing

The return is fewer fleet-wide incidents, recoveries that restart storms don't slow down, and clearer signals during triage.

Illustrative estimate (assumed numbers, not a benchmark): Hours saved per year = incidents avoided × engineers per incident × hours per incident.

Assume 2 incidents avoided, 4 engineers and 6 hours each: 2 × 4 × 6 = 48 hours per year. Compare that with an assumed 40 hours of migration work. Substitute your own incident history.

How can splitting health checks save engineering time?

The savings come from removing specific kinds of work that deep checks create during and after incidents.

  • Smaller incidents. When a shared dependency slows down, shallow probes keep instances in rotation. Requests that don't need that dependency keep succeeding. The incident stays scoped to one dependency instead of becoming a fleet-wide outage that pulls in several teams.
  • Faster recovery. Without a restart storm, the struggling dependency is not hit by fresh connection pools and cache warm-up from every replica at once. Engineers don't have to scale replicas down, pause probes, or edit deployment manifests in the middle of an incident just to let the dependency recover.
  • Shorter triage. A generic 503 from the routing layer hides the cause. Explicit errors from the application, together with per-dependency latency and error metrics, point straight to the failing component. Engineers spend less time working out why every service went red at once.
  • Fewer false pages. Lenient liveness thresholds and timeouts kept below the probe period reduce restarts caused by GC pauses or brief saturation. That removes a class of pages and post-incident investigations that turn out to have no real fault behind them.
  • Clearer ownership. When a service's health reflects only its own state, its owners are not paged for another team's database. Alerts go to the team that can act on them.
  • Safer rollouts. Startup checks, minReadySeconds, and progressDeadlineSeconds stall a bad build before it replaces healthy pods. That avoids manual rollbacks and the clean-up that follows a broken deploy.

Use the illustrative formula in the previous section with your own incident records to judge whether these savings outweigh the migration cost for your services.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to deep health checks microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.