What is the difference between liveness vs readiness probes in Kubernetes?

Kubernetes probes do not report health in any general sense. Each probe type is tied to a specific action the kubelet takes when the check fails. Choosing the wrong one means the platform does the wrong thing to your workload.

Probe What happens on failure Restarts container? Typical purpose
Liveness Kubelet kills the container and restarts it according to the pod's restart policy Yes Recover a process that is stuck and cannot fix itself
Readiness Pod is marked not ready and removed from Service endpoints. Checks continue, and traffic returns once it passes No Stop routing requests to an instance that cannot serve them right now
Startup Liveness and readiness checks are held off until it succeeds. If it never succeeds, the container is killed Yes, if it exhausts its failure threshold Give slow initialization time without loosening liveness settings

The core rule follows from the table: pick the probe by the action you want, not by the word "health." Ask what should happen when the check fails:

  • If a restart would fix the problem, it belongs in liveness.
  • If the instance should only stop receiving traffic for a while, it belongs in readiness.
  • If neither action helps, the check probably belongs in neither probe.

Liveness is optional. A process that exits on a fatal error is restarted by the kubelet anyway. What you give up without it is detection of hangs, such as deadlocks or a wedged event loop, where the process stays up but does no work.

This matters to backend engineers, who write the endpoints the probes call. It also matters to platform engineers, who set default probe templates that every service inherits.

Who should use liveness, readiness and startup probes?

Not every workload needs all three probes, and some need none.

  • Long-running services behind a Service are the main audience for readiness probes. Without one, a pod is added to Service endpoints as soon as its containers start, before it can handle requests. Any HTTP or gRPC API that is still loading config, warming caches or opening connection pools at startup should have a readiness probe.
  • Processes that can hang without exiting are the audience for liveness probes. Examples include event-loop servers, worker pools that take locks, and runtimes with known deadlock risk. If your process reliably crashes when it breaks, the kubelet already restarts it, and a liveness probe adds risk with little benefit.
  • Applications with slow or variable startup should use a startup probe, so that liveness can stay strict once the app is running. JVM services, apps that load large models or datasets, and apps that run migrations at boot are typical cases.
  • Batch Jobs and one-shot tasks usually need no readiness probe, because nothing routes traffic to them. A liveness probe can still help if a job can hang indefinitely, but an activeDeadlineSeconds on the Job is often the simpler guard.
  • Platform teams that maintain shared Helm charts or base manifests should care most about the defaults. A probe template that checks the database, copied across many services, turns one database outage into a fleet-wide restart.

How do Kubernetes probes work?

The kubelet on each node runs the probes for the containers on that node. It does not go through the Service or any load balancer. For each configured probe it waits initialDelaySeconds, then runs the check every periodSeconds, and treats a check that takes longer than timeoutSeconds as a failure.

A probe uses one of four mechanisms:

  • httpGet: the kubelet sends an HTTP GET to the pod's IP on the given port and path. A status code from 200 up to 399 is success. Anything else, or a timeout, is failure.
  • tcpSocket: success if a TCP connection to the port can be opened. This only proves something is listening, not that it can serve.
  • exec: the kubelet runs a command inside the container. Exit code 0 is success. Each run starts a process in the container, which costs CPU and memory under load.
  • grpc: the kubelet calls the standard gRPC health checking service on the given port. The application must implement that service.

A single failed check does not trigger an action. The kubelet counts consecutive results. A probe moves to failed after failureThreshold consecutive failures and back to passing after successThreshold consecutive successes. Then the action depends on the probe type:

  • Liveness failure: the kubelet kills the container and restarts it according to the pod's restartPolicy. Repeated restarts are subject to exponential back-off, which shows up as CrashLoopBackOff.
  • Readiness failure: the kubelet sets the pod's Ready condition to false. The control plane then removes the pod's address from the Service's EndpointSlices, and node-level proxies and ingress controllers stop sending it new traffic once they see the update. That propagation is not instant, which is why the shutdown sequence later in this article includes a delay.
  • Startup probe: while it is running, liveness and readiness probes are disabled. Once it succeeds it never runs again for that container. If it reaches its failure threshold, the container is killed like a liveness failure.

Every pod evaluates its probes independently. The kubelet has no fleet-wide view and no notion that "all replicas are failing at once, so maybe the problem is elsewhere." That independence is exactly why a check on a shared dependency fails on every pod together, as the next section shows.

How do dependency health checks in probes cause cascading failures?

The trouble starts when a probe endpoint checks a dependency every replica shares. Suppose the handler pings the database before returning 200. While the database is healthy, the probe passes. When the database is down, every pod fails the probe at about the same moment, and the kubelet acts on each one separately.

                 database outage
                        │
      ┌─────────────────┼─────────────────┐
      ▼                 ▼                 ▼
   pod A probe       pod B probe       pod C probe
    fails             fails             fails
      │                 │                 │
      ▼                 ▼                 ▼
 liveness: restart all pods / readiness: remove all pods
      │                 │                 │
      └─────────────────┼─────────────────┘
                        ▼
          Service has zero ready endpoints
                        ▼
      callers get connection refused or timeouts

If the check is in liveness, the kubelet restarts every replica. Each restart throws away warm caches, open connection pools and JIT state. The new processes then open connections and reload caches all at once, which puts the most load on a dependency at the moment it is trying to recover. If the database is still down, the restarted pods fail again. The restarts turn into a storm, and back-off pushes pods into CrashLoopBackOff. None of this fixes the database.

If the check is in readiness, all pods go unready together and the Service loses every endpoint. Callers lose the option of a fast, explicit degraded response, such as a 503 with a clear reason or cached data. They get connection errors instead, and those errors trigger their own retries and timeouts further upstream.

Avoid dependency checks in both probes when the dependency is shared by all replicas. Restarting your pods or removing them from the Service cannot fix that dependency.

                    ┌──────────────────────┐
                    │ shared database DOWN │
                    └──────────┬───────────┘
             ┌─────────────────┼─────────────────┐
             ▼                 ▼                 ▼
      ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
      │ pod A probe │   │ pod B probe │   │ pod C probe │
      │ DB ping FAIL│   │ DB ping FAIL│   │ DB ping FAIL│
      └──────┬──────┘   └──────┬──────┘   └──────┬──────┘
             └─────────────────┼─────────────────┘
                               ▼
              kubelet acts on each pod separately
                               │
             ┌─────────────────┴─────────────────┐
             ▼                                   ▼
   ┌───────────────────┐               ┌───────────────────┐
   │ LIVENESS check    │               │ READINESS check   │
   │ restart all pods  │               │ mark all unready  │
   └─────────┬─────────┘               └─────────┬─────────┘
             ▼                                   ▼
   ┌───────────────────┐               ┌───────────────────┐
   │ caches, pools,    │               │ Service has zero  │
   │ JIT state lost    │               │ ready endpoints   │
   └─────────┬─────────┘               └─────────┬─────────┘
             ▼                                   ▼
   ┌───────────────────┐               ┌───────────────────┐
   │ reconnect storm   │               │ callers get       │
   │ hits recovering DB│               │ refused/timeouts  │
   └─────────┬─────────┘               └─────────┬─────────┘
             ▼                                   ▼
   ┌───────────────────┐               ┌───────────────────┐
   │ fails again ->    │               │ no degraded reply;│
   │ CrashLoopBackOff  │               │ upstream retries  │
   └───────────────────┘               └───────────────────┘

A single shared database outage fails the probe on every pod at once. In liveness it drives restart storms and CrashLoopBackOff, and in readiness it empties the Service so callers get connection errors instead of a degraded response.

What should a liveness check actually detect?

A liveness probe should answer one narrow question: is this process stuck in a way that only a restart will fix? Good examples are a deadlocked event loop, a worker pool whose threads are all blocked on a lock that will never be released, or a process that accepts connections but can no longer serve any request. A slow database or a full queue does not belong here, because restarting the pod changes neither.

A simple HTTP handler that returns 200 is not enough, because it often runs on a separate listener thread and keeps responding while the real work has stopped. A more useful approach is a heartbeat, also called a watchdog. The main processing loop writes the current monotonic timestamp to a shared variable on every iteration. The liveness endpoint reads that variable and fails if the timestamp is older than a threshold, for example several times the longest expected iteration.

This pattern has two common failure modes:

  • False positives. An idle loop that blocks waiting for work never updates the timestamp, so it looks stuck. To avoid this, wake the loop on a timer so it records a beat even when there is nothing to do, or have the blocking wait time out and record one.
  • False negatives. A background thread that updates the heartbeat on its own schedule will keep it fresh while request handling is wedged. The beat must come from the code path that does the real work. In a worker pool, that means each worker records its own beat, and the check fails if any worker, or a set share of them, has gone quiet.
import queue, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

NUM_WORKERS = 4
IDLE_WAKE = 5.0     # blocking wait times out so an idle worker still records a beat
STALE_AFTER = 60.0  # must exceed IDLE_WAKE + the longest expected job, with margin

jobs = queue.Queue()
beats = [time.monotonic()] * NUM_WORKERS

def handle(job):
    time.sleep(job)  # placeholder for real work

def worker(i):
    while True:
        beats[i] = time.monotonic()  # beat comes from the loop that does the work
        try:
            job = jobs.get(timeout=IDLE_WAKE)
        except queue.Empty:
            continue  # idle, not stuck: loop around and beat again
        handle(job)  # if this wedges, beats[i] stops advancing
        jobs.task_done()

class Liveness(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/livez":
            self.send_response(404); self.end_headers(); return
        now = time.monotonic()
        stale = [i for i, t in enumerate(beats) if now - t > STALE_AFTER]
        # Fail if any worker has gone quiet; a restart is the only fix for a wedged worker.
        self.send_response(503 if stale else 200)
        self.end_headers()
        self.wfile.write(f"stale_workers={stale}\n".encode())

for i in range(NUM_WORKERS):
    threading.Thread(target=worker, args=(i,), daemon=True).start()
ThreadingHTTPServer(("0.0.0.0", 8080), Liveness).serve_forever()

A per-worker heartbeat in which each worker stamps a monotonic timestamp from its own processing loop, uses a timed queue wait so idle workers do not look stuck, and the /livez endpoint fails when any worker's beat is older than the threshold.

When should readiness depend on something other than the instance itself?

Readiness answers a narrower question than "can this request succeed?" It asks whether this instance should receive traffic right now, compared with its peers. That keeps it to conditions local to the pod: still warming caches or loading models, saturated with work, draining after receiving SIGTERM, or unable to reach a dependency that other replicas can reach. In each case, routing elsewhere helps, because somewhere else is in a better state.

There are two legitimate exceptions. The first is a hard local dependency, such as a sidecar proxy that all traffic passes through. If the sidecar is down, the pod cannot serve, and peers are unaffected. The second is a multi-zone deployment where only replicas in one zone lose a zonal dependency. Failing readiness there shifts traffic to zones that still work. The test is the same in both cases: is the failure partial, and is there healthy capacity to absorb the traffic?

If every replica would fail the same check, leave it out. Pods that stay ready during a shared outage will receive requests they can only fail or degrade. That outcome is better than having zero endpoints only if the application fails fast, with a clear error such as a 503 that names the unavailable dependency. If the pod instead hangs until timeout, it ties up callers.

Readiness-based load shedding needs care. When an overloaded pod drops out, its traffic moves to the remaining pods, which can push them over the threshold too. Pods then flap in and out of the endpoint list. To prevent this, use hysteresis, with a lower threshold for returning than for leaving, and keep enough spare capacity that losing one pod does not overload the rest.

What are the alternatives to checking dependencies in probes?

Dependency failures are better handled in the request path, where each call fails on its own terms and the pod remains in rotation. Every outbound call needs its own timeout, shorter than the caller's deadline, so a slow database returns an error quickly and does not hold a worker thread. Around that timeout, add the mechanisms below, chosen by the failure each one handles.

Mechanism Failure it handles Why it beats a probe
Per-call timeout Slow or hung dependency Caps how long one request waits; the pod stays healthy
Circuit breaker Dependency down for a sustained period Stops calls after repeated failures and probes recovery with a few trial requests
Bulkhead One dependency exhausting shared threads or connections Gives each dependency a separate pool, so endpoints that don't use it keep working
Retry with backoff and jitter Brief, transient errors Jitter spreads retries out instead of synchronizing them into a spike
Graceful degradation Optional feature unavailable Serves cached or partial results rather than a 503

Dependency status still matters, but it belongs in metrics and a separate diagnostic endpoint that humans and alerting read. The orchestrator must never act on it. During an incident, this endpoint tells on-call engineers which dependency is failing directly, so they don't have to work backward from mass restarts that are hiding the original fault.

The endpoint has costs. It exposes internal topology, so restrict it to internal networks or authenticated callers. Each request to it can also fan out to every dependency, so cache results for a short interval to keep scrapes and repeated page refreshes from adding load to a backend that is already struggling.

How do you configure and tune Kubernetes probes?

Give each probe its own endpoint: /startupz, /livez and /readyz, each answered from in-memory state. For slow starts, prefer a startupProbe over a large initialDelaySeconds. A fixed delay has to be sized for the worst start you have ever seen, and it delays liveness checks on every start. A startup probe instead allows up to periodSeconds × failureThreshold for startup, then hands over to liveness as soon as it succeeds.

Set timeoutSeconds against p99 probe latency under peak load, not on an idle laptop. Use failureThreshold to ride out a GC pause or CPU throttling. Use readiness successThreshold to stop flapping back into rotation; liveness and startup require it to be 1. Aggressive settings detect failure faster but restart healthy pods during transient stalls. Lenient settings keep a broken pod in rotation longer.

Illustrative estimate. Assume readiness periodSeconds: 5, failureThreshold: 3, and roughly 5 seconds for endpoint changes to reach every proxy and load balancer. A pod that stops serving keeps receiving traffic for up to 5 × 3 + 5 = 20 seconds.

Serve probes from a thread or listener that does not queue behind request workers. If /livez waits in a saturated pool, it times out under overload. The kubelet then restarts pods exactly when capacity is scarcest, and the surviving pods take on their load.

On SIGTERM, follow this sequence:

  1. Fail readiness.
  2. Keep serving while a preStop sleep covers endpoint propagation.
  3. Drain in-flight requests.
  4. Exit.

Set terminationGracePeriodSeconds larger than the preStop delay plus the longest expected drain, or the kubelet will send SIGKILL mid-request.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  # Must exceed preStop (10s) + longest drain (30s); otherwise SIGKILL lands mid-request.
  terminationGracePeriodSeconds: 45
  containers:
    - name: orders-api
      image: registry.example.com/orders-api:1.4.2
      ports:
        - name: http
          containerPort: 8080
        # Separate listener so probes never queue behind saturated request workers.
        - name: health
          containerPort: 8081
      startupProbe:            # allows up to 5s x 24 = 120s to start; liveness waits until it passes
        httpGet: {path: /startupz, port: health}
        periodSeconds: 5
        failureThreshold: 24
      livenessProbe:           # lenient: ~30s of consecutive failures before a restart
        httpGet: {path: /livez, port: health}
        periodSeconds: 10
        timeoutSeconds: 2      # above p99 probe latency at peak load, not idle latency
        failureThreshold: 3    # rides out a GC pause or CPU throttling
      readinessProbe:          # stricter: leaves rotation within ~15s, plus propagation delay
        httpGet: {path: /readyz, port: health}
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 3
        successThreshold: 2    # two passes before rejoining, to stop flapping
      lifecycle:
        preStop:
          # Runs before SIGTERM. Keep serving while endpoint removal reaches proxies.
          # After SIGTERM, the app fails /readyz, drains in-flight requests, then exits.
          exec:
            command: ["sh", "-c", "sleep 10"]

A pod spec with startup, liveness and readiness probes on a dedicated health port, a preStop delay that covers endpoint propagation, and a termination grace period sized to cover that delay plus the drain.

How do you decide which probe a check belongs in?

Run every candidate check through three questions, in order:

  1. Would restarting this container fix the condition? If not, it cannot be a liveness check. A restart only adds a cold start to whatever is already wrong.
  2. Is the condition specific to this instance, or shared by every replica? Removing one pod from the Service only helps if the other pods don't have the same problem.
  3. What happens if every pod reports this result at the same moment? If the answer is "the Service has zero endpoints" or "the whole Deployment restarts," the check can take down the entire fleet at once. Expose it as a metric with an alert instead.
Candidate check Restart fixes it? Instance-local? Belongs in
Request-handling threads deadlocked Yes Yes Liveness
Loading config, warming local caches Not applicable (one-time) Yes Startup
Shutting down after SIGTERM No Yes Readiness
Local work queue saturated No Yes Readiness
Database unreachable No No Metrics and alerts, plus fail-fast handling in the request path
Downstream API slow No No Metrics, plus a circuit breaker

Instance-local probes are cheap to build. The handler reports in-process state and has no dependency timeouts to tune, so it is easy to reason about when it flips. The trade-off is that the effort moves into the application, and you pay for it up front. You need bounded timeouts on every outbound call, circuit breakers, and fast 503 responses when a dependency is down. That work has a fixed scope, and you do it once.

The return comes during incidents. A failing probe points at one pod, so you don't have to work out whether a dependency caused it. A database outage shows up as a single dependency alert, not hundreds of restarts and empty endpoint lists that someone has to sort out before finding the real cause.

How can well-designed probes save engineering time?

The time savings come from removing work that badly designed probes create, mostly during and after incidents.

  • Shorter diagnosis. When probes only report instance-local state, a probe failure means something is wrong with that pod. On-call engineers can go straight to that pod's logs and metrics. With dependency checks in probes, the first symptom of a database outage is often mass restarts or CrashLoopBackOff, and the first part of the incident goes to ruling out the application, the nodes and the rollout before anyone looks at the database.
  • No secondary outage to clean up. A restart storm leaves its own damage: cold caches, connection floods against a recovering backend, and pods stuck in back-off that recover more slowly than the dependency did. Keeping pods up and failing fast in the request path means that when the dependency returns, the service returns with it, and nobody has to scale deployments down and up or delete pods by hand to clear back-off.
  • Fewer false alarms. Liveness probes tuned against peak-load latency, with a startup probe for slow boots, stop restarting healthy pods during GC pauses, CPU throttling or slow starts. Each avoided false restart is a page, a restart-count alert or a "why did this pod restart?" investigation that nobody has to do.
  • Safer deploys. Accurate readiness means a rollout only sends traffic to pods that can serve it, and a correct SIGTERM sequence means scale-downs and rolling updates do not drop in-flight requests. Teams spend less time chasing error spikes that line up with every deploy.
  • One decision, reused. A platform team that builds the three-question test and the instance-local endpoint pattern into a shared chart or library makes the decision once. Individual service teams no longer design probes from scratch or copy a dependency check from another service.

Illustrative estimate. Assume a team has two incidents a quarter where a shared dependency fails, and that in each one, probe-driven restarts add an assumed 30 minutes of diagnosis and an assumed 15 minutes of manual recovery for two engineers. That is 2 × 45 × 2 = 180 engineer-minutes, or three engineer-hours, per quarter spent on effects of the probes rather than on the actual fault. Your own numbers depend on incident frequency and team size, and you can check them against past incident timelines by marking where the response time went to restarts and empty endpoints rather than to the failing dependency.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to liveness vs readiness probes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.