What is the difference between a liveness vs readiness probe?
A probe is defined by what happens when it fails. The kubelet runs all three probes against the container, but each probe triggers a different action.
| Probe | Question it answers | On failure | Runs when |
|---|---|---|---|
| Liveness | Is this process irrecoverably stuck? | Kubelet kills the container, which is restarted according to restartPolicy |
After startup succeeds, then for the container's lifetime |
| Readiness | Should this pod get traffic right now? | Pod is marked not Ready and removed from Service endpoints. No restart. | After startup succeeds, then for the container's lifetime |
| Startup | Has the app finished initializing? | Kubelet kills the container, as with liveness | Until it first succeeds. Liveness and readiness are held off until then. |
Kubelet → runs probes against → Container ├─ liveness fails → kubelet restarts the container ├─ readiness fails → pod marked not Ready → removed from Service endpoints (traffic stops, process keeps running) └─ startup not yet passed → liveness and readiness not run
These questions have different answers in common situations. During a slow downstream call, a cache warm-up, or a burst of load, the process is healthy but shouldn't receive new requests. That is a readiness concern. A restart won't fix it and usually makes it worse, because the replacement container starts cold.
A deadlocked worker or an event loop that never returns is different. Only a restart helps there, so it's a liveness concern.
Engineers who own health checks often point both probes at one /health endpoint. That ties a restart decision to conditions that only call for removing traffic. Keeping the probes separate is what allows a pod to step out of rotation temporarily without being killed.
+-------------+
| kubelet |
+------+------+
| runs probes against the container
v
+-------------------+
| startup probe |
+--+-------------+--+
fails | | first success
+-------------+ | (until then, liveness and
| | readiness are not run)
| +------------+------------+
| v v
| +-----------------+ +-----------------+
| | liveness probe | | readiness probe |
| +--------+--------+ +--------+--------+
| | fails | fails
| v v
| +-----------------+ +-----------------+
+---->| RESTART | | pod not Ready |
| container | +--------+--------+
| (restartPolicy) | |
+-----------------+ v
+-----------------+
| removed from |
| Service |
| endpoints |
+-----------------+
traffic stops; process keeps running
A failing startup or liveness probe leads the kubelet to restart the container, while a failing readiness probe only marks the pod not Ready and removes it from Service endpoints, leaving the process running.
Who should use liveness, readiness, and startup probes?
Probe configuration touches several roles, and each one needs a slightly different part of this article:
- Application developers implement the endpoints the probes call. They decide what
/livezand/readyzactually check, and they are the only ones who know whether a given failure is fixed by a restart. - Engineers writing deployment manifests set the timing fields. They need measured startup times and probe-endpoint latency to choose values that don't kill working containers.
- Platform teams who maintain Helm charts, base manifests, or admission policies set the defaults that every service inherits. A bad default, such as a liveness probe that checks a shared database, gets copied into every service at once.
- SREs and on-call engineers diagnose restart loops and outages. They need to tell a probe-induced kill from an application crash or an OOMKill quickly.
By workload type:
- Long-running services behind a Service should almost always have a readiness probe. Without one, a pod is added to endpoints as soon as its containers are running, before the application can serve requests.
- Services with slow or variable startup, such as JVM applications, services that run migrations, or anything that loads a large cache, should use a startup probe.
- Processes with a known hang risk, such as deadlock-prone worker pools or single-threaded event loops that can block, are the main case for a liveness probe.
- Batch Jobs and processes that exit on fatal errors often need neither liveness nor readiness. The kubelet already handles a process that exits.
How do Kubernetes probes work?
Probes are executed by the kubelet on the node where the pod runs, not by the control plane. For each container, the kubelet runs every configured probe on its own schedule, defined by periodSeconds, and records each attempt as a success or failure.
Each probe uses one mechanism:
httpGet: the kubelet sends an HTTP GET to the pod's IP on the given port and path. A status code from 200 to 399 counts as success. Any other code, a refused connection, or a response slower thantimeoutSecondscounts as failure.tcpSocket: the kubelet tries to open a TCP connection to the port. If the connection opens, the probe succeeds. It says nothing about whether the application behind the port works.exec: the kubelet runs a command inside the container. Exit code 0 is success, anything else is failure. Each attempt starts a new process in the container.grpc: the kubelet calls the standard gRPC health checking service on the given port, for applications that implement it.
The kubelet counts consecutive results. When failures reach failureThreshold, the probe is considered failed, and the action depends on the probe type:
- Startup runs first. Until it succeeds, the kubelet does not run liveness or readiness. If it fails past its threshold, the kubelet kills the container and
restartPolicydecides what happens next. After its first success, it does not run again for that container. - Liveness then runs for the life of the container. When it fails, the kubelet kills the container, sending SIGTERM and, after the grace period, SIGKILL, and the container is restarted. Repeated restarts are spaced out with exponential backoff, which is what shows up as
CrashLoopBackOff. - Readiness runs alongside liveness. Its result sets the container's ready state. A pod is Ready only when all of its containers are ready. The endpoints controller includes only Ready pods in the Service's endpoints, so a failing readiness probe stops new traffic through the Service without touching the process. When readiness passes again for
successThresholdconsecutive attempts, the pod is added back.
Because the kubelet runs the probe from the node, the probe request competes for the container's CPU and threads like any other request. That is why probe timing and probe-endpoint latency, covered below, matter as much as what the endpoint checks.
How do probes cause CrashLoopBackOff and cluster-wide outages?
Liveness that can't tell slow from dead
A liveness probe has one remedy: the kubelet kills the container and restarts it. If failureThreshold × periodSeconds is shorter than the time the process actually needs, the probe kills containers that were working. That happens during a cold start that loads a large cache. It happens during a stop-the-world GC pause that outlasts timeoutSeconds. It also happens when CPU throttling against a tight limit stalls the HTTP handler behind request threads. The restarted container hits the same condition, gets killed again, and the kubelet starts spacing out restarts with exponential backoff, capped at five minutes. The pod then shows CrashLoopBackOff. The application never crashed. The probe kept killing it, and each backoff interval is time the replica serves nothing.
Shared dependencies turn one failure into all of them
Suppose the liveness endpoint checks the database, Redis, or a downstream API. When that dependency degrades, every replica fails liveness within roughly the same probe window, and every replica restarts. Restarting does not fix the database. The new containers come up, fail the same check, and enter backoff together. Meanwhile each restart can cost connection-pool warm-up and reconnect storms against the dependency that is already struggling. An outage in one backend becomes zero healthy pods in your service. Recovery is also slower, because pods must climb out of backoff after the dependency returns.
The timing trade-off
Tight thresholds detect a hung process sooner. But probes run on the same CPU and event loop as real traffic, so they fail first under load. Aggressive liveness timing produces false positives at peak traffic, which is exactly when removing a replica pushes more load onto the remaining ones and makes them fail too.
How do probe timing fields work together?
Five fields decide how quickly the kubelet acts on a probe result, and they compound.
| Field | What it controls |
|---|---|
initialDelaySeconds |
Wait after container start before the first probe |
periodSeconds |
Interval between probe attempts |
timeoutSeconds |
How long one attempt may take before it counts as a failure |
failureThreshold |
Consecutive failures before the probe is considered failed |
successThreshold |
Consecutive successes needed to recover (must be 1 for liveness) |
For a container that never becomes healthy, time to the first restart is roughly initialDelaySeconds + periodSeconds × failureThreshold. Illustrative estimate, assuming a 15s initial delay, a 10s period and a threshold of 3: 15 + 10 × 3 ≈ 45 seconds before the kubelet kills the container. timeoutSeconds also matters. A slow response is a failure, so a timeout set below your tail latency turns ordinary slowness into failed probes.
Readiness flapping happens when an endpoint sits near its timeout and alternates between passing and failing. Each transition adds or removes the pod from Service endpoints, so traffic shifts back and forth between replicas. Raising failureThreshold makes the pod tolerate brief blips before it is removed. Raising successThreshold requires sustained health before it is re-added. Both changes stabilize routing, and both mean the pod reacts more slowly to real failures and real recoveries.
Base these values on measurements rather than defaults:
- Set the startup window from observed worst-case startup time, plus margin.
- Set
timeoutSecondswell above the probe endpoint's p99 latency under peak load. - Give liveness a longer failure window than readiness, so a pod is taken out of rotation well before it is restarted.
containers:
- name: api
image: example.com/api:1.0.0
ports:
- containerPort: 8080
# Covers slow startup: up to 30 x 5s = 150s before liveness takes over.
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 30
# Readiness: short failure window, so the pod leaves rotation quickly.
# successThreshold: 2 needs two passes in a row before re-adding, which damps flapping.
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2 # well above the endpoint's p99 under peak load
failureThreshold: 3 # ~15s of failures before removal from endpoints
successThreshold: 2
# Liveness: longer failure window, so the pod is removed from traffic
# well before it is restarted. successThreshold must stay 1 here.
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 6 # ~60s of consecutive failures before restart
successThreshold: 1
Probe configuration where a startup probe covers slow boot, readiness removes the pod from rotation after about 15 seconds of failures and needs two passes in a row to return, and liveness restarts the container only after about 60 seconds of failures.
When should you use a startup probe instead of initialDelaySeconds?
A large initialDelaySeconds on the liveness probe looks like a fix for slow boots, but it only postpones the first check. It doesn't respond to how long a given start actually takes. If you size the delay for the worst case you've seen, every pod waits that long before liveness begins. If a cold cache or a slow node pushes boot past the delay, the kubelet kills a container that was working correctly. Teams usually respond by loosening failureThreshold and periodSeconds as well. Those loose settings then apply for the pod's whole life, so a container that hangs hours after startup goes undetected for much longer.
A startup probe separates the two concerns. While it is running, the kubelet does not run liveness or readiness checks. Once it succeeds, it stops, and the normal probes take over. The startup budget is roughly failureThreshold × periodSeconds. Take a JVM service that runs schema migrations on boot (illustrative estimate). With periodSeconds: 10 and failureThreshold: 30, it gets about five minutes to come up. It is checked every ten seconds, so it moves on to liveness as soon as it is ready instead of waiting out a fixed delay. Liveness can then stay tight, for example a 10-second period with a threshold of 3, because it never sees the slow boot.
The cost is that a container that will never start also gets the full budget before it is restarted. That slows rollouts and delays the signal that a release is broken. A generous budget can also hide startup-time regressions. If boot time creeps from 40 seconds to four minutes, nothing fails. Size the budget from measured boot times plus a margin, and track startup duration as a metric.
containers:
- name: orders-api
image: registry.example.com/orders-api:1.4.2
ports:
- containerPort: 8080
# Startup budget ~= failureThreshold x periodSeconds = 30 x 10s = ~5 minutes.
# Liveness and readiness are not run until this probe succeeds,
# and it is checked every 10s, so a fast boot hands off early.
startupProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
failureThreshold: 30
# Tight liveness: never sees the slow boot, so a hang after startup
# is detected in ~30s (3 failures x 10s). No large initialDelaySeconds.
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
# Readiness gates traffic; failing it removes the pod from Service
# endpoints without restarting the container.
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5
failureThreshold: 3
A startup probe with a roughly five-minute failureThreshold × periodSeconds budget covers JVM warm-up and migrations, so the liveness probe can stay tight and needs no large initialDelaySeconds.
How should you implement liveness and readiness health check endpoints?
A liveness endpoint should answer one question: would restarting this process fix it? Keep it cheap, local, and free of dependencies. A good liveness check confirms that the event loop is still scheduling work, or that a worker thread has updated a heartbeat timestamp recently. It should not check the database. A database outage is not fixed by a restart, so failing liveness on it only adds restarts to an existing outage.
Readiness answers a different question: should this pod receive traffic right now? It should report false during warm-up, when the connection pool is exhausted, and as soon as the process receives SIGTERM, so endpoints are removed before in-flight requests drain.
Hard-dependency checks in readiness are a judgment call. If every replica checks the same shared cache, one cache outage marks every pod unready. The Service then has zero endpoints, and clients that could have received degraded responses get connection failures instead.
| Mechanism | Cost per period | Checks | Caveat |
|---|---|---|---|
| HTTP GET | One request | Application logic | Handler must stay fast |
| TCP socket | One connection | Port is accepting | A wedged app can still accept |
| exec | Spawns a process | Anything scriptable | CPU and PID overhead at scale |
Where you serve probes is also a trade-off. Serving them on the main request path reflects real health, but under saturation the probe times out along with user traffic. A separate listener stays responsive under load, but it can report healthy while the real request path is failing.
apiVersion: v1
kind: Pod
metadata:
name: orders-api
spec:
# Must exceed readiness failure time plus in-flight drain time
terminationGracePeriodSeconds: 30
containers:
- name: app
image: registry.example.com/orders-api:1.4.2
ports:
- containerPort: 8080
# Startup: allows up to 24 x 5s = 120s for warm-up before liveness starts counting
startupProbe:
httpGet:
path: /livez
port: 8080
periodSeconds: 5
failureThreshold: 24
# Liveness: /livez checks only a local worker heartbeat. No DB or cache calls,
# because a restart does not fix a dependency outage.
livenessProbe:
httpGet:
path: /livez
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
# Readiness: /readyz returns 503 while warming up, when the connection pool is
# exhausted, and immediately after SIGTERM, so the pod leaves endpoints before draining
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
A pod spec in which the startup probe covers warm-up, the liveness probe checks only a dependency-free local heartbeat, and the readiness probe reports warm-up, pool exhaustion, and shutdown drain.
How do you diagnose a probe-induced restart loop?
Start with kubectl get pods and read the RESTARTS column. A count that climbs steadily while STATUS alternates between Running and CrashLoopBackOff means the container keeps getting killed. Note when the restarts started: if many pods began restarting together, look for a shared dependency or a node-level problem before blaming one container.
Next, run kubectl describe pod on an affected pod and read two areas. The first is the Events list. A liveness kill leaves "Liveness probe failed" messages followed by a note that the container will be restarted. The failure text tells you whether the probe timed out or got a bad status code. Events are only kept for a limited time, so capture them early. The second area is the container's Last State. Its termination reason and exit code show how the previous instance died.
Then run kubectl logs --previous to read the output of the killed instance. The current container's logs only show the new start, which usually looks clean.
| Cause | Last State reason | Exit code | Probe events | Points to |
|---|---|---|---|---|
| Liveness kill | Usually Error or Completed | Often 137 or 143 | Liveness failures just before the restart | Probe timing, a blocked event loop, or a dependency inside the check |
| Application crash | Error | App-specific, often 1 | None | Bug or bad config, found in the previous logs |
| OOMKill | OOMKilled | 137 | None needed | Memory limit too low, or a leak |
Exit code 137 only means the process received SIGKILL, so it does not identify the cause by itself. Use the reason field and the events to tell an OOMKill from a liveness kill. A probe-only failure needs different fixes than a memory or code fault.
When should you avoid liveness probes, and what are the alternatives?
Before tuning a liveness probe, ask whether the container needs one. A liveness probe exists to restart a process that is stuck and cannot recover on its own. If your runtime already exits on fatal errors, the kubelet restarts it anyway, so liveness adds risk without adding much coverage. A missing liveness probe is often safer than a badly designed one.
| Situation | Readiness | Liveness | Startup |
|---|---|---|---|
| Process exits on fatal errors | Yes | Usually omit | Only if startup is slow |
| Known deadlock or hang risk | Yes | Yes, local check only | If startup is slow |
| Slow or variable initialization | Yes | Optional | Yes |
| Failures come from dependencies | Yes | No | No |
The trade-off: without liveness, a truly deadlocked process stays running but unready. Traffic stops reaching it and capacity drops quietly until alerting catches the gap. Cover that case with alerts on ready replica counts and request throughput, not with an aggressive probe.
A probe sees only one container's view of the system, so it should not carry resilience logic. Handle dependency failures in the application, with circuit breaking and fallback responses. For conditions you know are fatal, use an in-process watchdog that exits deliberately, which gives a clear restart reason in the logs.
Cost follows the same pattern. Simple local probes are cheap to build and maintain. Dependency-aware probes couple your restart behavior to other services, which leads to more incidents and more debugging time. Good probe design reduces on-call load and the time spent chasing false restarts.
How can correct probe design save engineering time?
Most of the time lost to probes goes into incidents that look like application failures but aren't. Separating liveness from readiness and sizing timing from measurements removes several of those incidents outright:
- Fewer false restart incidents. A liveness probe that checks only local state can't restart every replica during a database or cache outage. The dependency incident stays a dependency incident, and nobody has to work out why the service also went to zero pods.
- Shorter dependency outages. When readiness, not liveness, reacts to a degraded backend, pods leave and rejoin endpoints without going through restart backoff. Once the dependency recovers, the service recovers with it instead of waiting for pods to climb out of
CrashLoopBackOff. - Faster diagnosis. With distinct
/livezand/readyzendpoints, a "Liveness probe failed" event means one thing: the process itself stopped responding. Thekubectl describe podandkubectl logs --previousworkflow above then points to the cause in minutes rather than requiring a search through every dependency. - Less repeated tuning. A startup probe absorbs slow boots, so teams stop loosening liveness thresholds after each slow rollout. Setting values once from measured startup time and p99 probe latency avoids repeated adjust-and-redeploy cycles.
- Reusable defaults. Platform teams can encode the patterns in shared templates: a dependency-free liveness check, readiness that fails on SIGTERM, and a startup probe for slow services. Each new service inherits safe behavior instead of each team rediscovering the same failure modes.
Illustrative estimate, using assumed inputs only: suppose a team has an assumed four incidents per quarter where liveness probes that check a shared dependency restart every replica, and each one takes an assumed three engineers about two hours to triage and resolve. That is an assumed 24 engineer-hours per quarter. If moving those dependency checks from liveness into readiness, or into application-level fallbacks, eliminates the restart cascades, most of that time is recovered, plus the reduced customer impact. Replace these inputs with your own incident history before using the figure for anything.
The effort needed is small. In most services it means splitting one health endpoint into two, adding a startup probe, and measuring startup time and probe-endpoint latency once.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to liveness vs readiness probe, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.