Health Checks and Graceful Shutdown for Containers: A Practical Guide

Liveness, readiness, and startup probes: one lifecycle contract

Kubernetes runs three independent probes against each container, and each one controls a different decision. The kubelet runs them on the node and acts on the results. Mixing up their roles causes many self-inflicted outages, such as a liveness check that restarts healthy pods because a database is slow.

The Kubernetes overview describes this as self-healing: the platform "kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve" (source).

Readiness and shutdown are two halves of one contract. Readiness controls when traffic starts arriving. Graceful shutdown controls how it stops: stop accepting new work, finish in-flight work, then exit. If either half is wrong, requests fail at the boundaries.

This applies to any team running HTTP or gRPC services, or queue workers, on Kubernetes-style orchestrators. It matters most where pods change often: frequent deploys, autoscaling in both directions, and node drains during upgrades. Each of these replaces pods while live traffic is still flowing.

What each health endpoint should and should not check

A probe answers one question, and the action the orchestrator takes on failure tells you which question. A failed liveness probe restarts the container. A failed readiness probe removes the pod from Service endpoints but leaves it running. Design each endpoint around the action it triggers.

Liveness: can this process make progress?

Liveness should fail only when a restart would fix the problem. Examples are a deadlocked worker pool, a wedged event loop, or a main loop that has stopped advancing. A handler that confirms the HTTP server can respond, or that a heartbeat timestamp updated recently, usually covers it.

Database, cache, and downstream API checks do not belong here. If the database is unreachable, restarting every replica does not bring it back. It does add a wave of cold starts, connection storms, and lost in-flight work at the moment the dependency is trying to recover.

Readiness: can this instance usefully serve?

Readiness can reflect local state: warm caches, loaded config, or an exhausted connection pool. Be careful with shared dependencies. If every replica checks the same database, an outage of that database makes the whole fleet unready at once. The Service then has no endpoints, and clients get connection errors instead of a meaningful response. Often it is better to stay ready and return degraded responses: stale cached data, a partial result, or a fast 503 with a clear body that callers can handle.

Keep handlers cheap

Probes run on every pod, every few seconds. A handler that runs a query or calls another service multiplies that load across the fleet. It can also time out under normal latency spikes, producing false failures that cause the instability the probe was meant to detect. Read in-memory state, avoid blocking I/O, and return quickly.

Copyimport threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

# In-memory state only. Probe handlers read this and never do I/O.
STATE = {"heartbeat": time.monotonic(), "config_loaded": False, "pool_in_use": 0}
POOL_SIZE = 20
HEARTBEAT_MAX_AGE = 30  # seconds; well above one loop iteration

def worker_loop():
    while True:
        # ... do one unit of real work here ...
        STATE["heartbeat"] = time.monotonic()  # proves the loop is still advancing
        time.sleep(1)

class Probes(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/livez":
            # Liveness: is the process making progress? A restart fixes a stalled loop.
            # No DB, cache or downstream calls: restarts can't fix those.
            ok = time.monotonic() - STATE["heartbeat"] < HEARTBEAT_MAX_AGE
        elif self.path == "/readyz":
            # Readiness: local conditions only. A shared-DB outage should lead to
            # degraded responses, not every replica going unready at once.
            ok = STATE["config_loaded"] and STATE["pool_in_use"] < POOL_SIZE
        else:
            self.send_response(404); self.end_headers(); return
        body = b"ok\n" if ok else b"unavailable\n"
        self.send_response(200 if ok else 503)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass  # probes hit every few seconds; keep them out of the logs

threading.Thread(target=worker_loop, daemon=True).start()
STATE["config_loaded"] = True  # set once real startup work has finished
ThreadingHTTPServer(("0.0.0.0", 8080), Probes).serve_forever()

A standard-library Python server with cheap probe handlers that read only in-memory state. /livez checks that a heartbeat is still advancing, and /readyz checks local readiness without calling any shared dependency.

How pod termination actually works and where requests get dropped

When a pod is deleted, the API server records a deletion timestamp and the pod enters the Terminating state. Two independent processes then start in parallel, and neither waits for the other.

The two parallel paths

Kubelet path. The kubelet on the pod's node runs the container's preStop hook, if one is defined. Only after the hook returns does the kubelet send SIGTERM to the container's main process. The grace period countdown (terminationGracePeriodSeconds, 30 seconds by default) starts at deletion, not at SIGTERM, so time spent in preStop counts against it. When the countdown expires, any process still running receives SIGKILL.

Networking path. The control plane updates the EndpointSlices so the pod is no longer a ready endpoint. Each node's kube-proxy then rewrites its forwarding rules. Ingress controllers and external load balancers pick up the change through their own watch or sync loops.

The race

Endpoint removal is eventually consistent, and it spreads through several layers, each with its own delay. If the application closes its listener as soon as it receives SIGTERM, some nodes or load balancers may still route new connections to it. Those connections are refused or reset, and clients see errors during every rollout.

The gap closes in one of two ways. A short preStop sleep delays SIGTERM until routing has likely converged. Alternatively, the application can keep accepting requests for a few seconds after SIGTERM before it closes the listener. Both are heuristics. A fixed delay estimates propagation time and does not confirm it. External load balancers that rely on their own health checks or deregistration delays can take much longer than kube-proxy. Size the delay to the slowest layer in front of the pod.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      # Budget for the whole shutdown: preStop sleep + app drain time.
      # The countdown starts at deletion, so the sleep is included here.
      terminationGracePeriodSeconds: 45
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                # Delay SIGTERM so EndpointSlice changes can reach kube-proxy
                # and load balancers while the app still serves traffic.
                # A heuristic: raise it if an external LB deregisters slowly.
                # Requires a 'sleep' binary in the image.
                command: ["sleep", "10"]
          # After the sleep, SIGTERM arrives; the app should stop accepting,
          # finish in-flight requests, and exit well before SIGKILL at 45s.

A Deployment that uses a preStop sleep to hold off SIGTERM while endpoint removal propagates, with a grace period sized to cover both the sleep and the application's drain time.

Making sure SIGTERM reaches your application

None of that delay matters if the signal never arrives. When the grace period starts, the container runtime sends SIGTERM (or the image's STOPSIGNAL, if one is set) to exactly one process: PID 1 inside the container. The kernel treats PID 1 specially. Signals whose default action is to terminate are ignored unless the process has installed a handler. A process that would exit on SIGTERM anywhere else can sit untouched as PID 1 until the grace period expires and SIGKILL arrives, which drops every in-flight request.

Shell form vs exec form

The form of ENTRYPOINT or CMD decides what PID 1 is:

• Shell form, CMD node server.js, runs as /bin/sh -c "node server.js". The shell becomes PID 1, and depending on the shell and the command, it may not forward SIGTERM to its child.
• Exec form, CMD ["node", "server.js"], runs the binary directly as PID 1, so your handler receives the signal.

Wrapper scripts

Entrypoint scripts that render config or wait for dependencies have the same problem: the script stays PID 1 and the app runs as its child. End the script with exec "$@" or exec your-binary. This replaces the shell process with the application rather than forking it.

When to add an init

If the application installs no signal handlers or spawns child processes, run a minimal init such as tini or dumb-init as PID 1. It forwards signals to the child and reaps zombie processes. Without reaping, orphaned children accumulate as defunct entries in the process table.

CopyFROM node:20-alpine

# tini is a minimal init: it forwards signals to its child and reaps zombies.
# The Alpine package installs it at /sbin/tini.
RUN apk add --no-cache tini

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

# docker-entrypoint.sh renders config, then hands off to the app:
#   #!/bin/sh
#   set -e
#   envsubst < /app/config.tmpl > /app/config.json
#   exec "$@"   # replaces the shell, so the app takes over its PID
# Without `exec`, the shell stays the parent and may never pass SIGTERM on.
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

# WRONG: shell form. /bin/sh -c becomes PID 1, and node is its child.
# CMD node server.js

# WRONG: exec form, but the wrapper script runs the app without `exec`.
# ENTRYPOINT ["docker-entrypoint.sh"]   # the script stays PID 1

# RIGHT: exec form throughout. tini is PID 1, the script execs into node,
# and tini forwards SIGTERM to node.
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/docker-entrypoint.sh"]
CMD ["node", "server.js"]

A Node.js Dockerfile that contrasts shell-form CMD with exec form and shows why the entrypoint wrapper must end with exec "$@". It runs tini as PID 1 so SIGTERM reaches the app and zombie processes are reaped.

Implementing connection draining in the application

Once SIGTERM arrives, the application should run a fixed sequence. Each step closes a specific gap.

The drain sequence

1. Fail readiness immediately. Endpoint removal is asynchronous, so load balancers and kube-proxy may keep routing to the pod for a few seconds. Keep accepting traffic during a short delay, often a few seconds, so those late requests still succeed.
2. Stop accepting new connections. Close the listener so no new TCP handshakes complete against a process that is shutting down.
3. Finish in-flight requests. Track active requests and wait for the count to reach zero.
4. Close idle keep-alive connections. A pooled client connection with no active request will otherwise receive its next request after the socket is gone. On HTTP/1.1, send Connection: close on the final response of each active connection. On HTTP/2 and gRPC, send GOAWAY so clients open new streams elsewhere.
5. Bound everything with a deadline. Set it a few seconds shorter than terminationGracePeriodSeconds so the process can log, flush telemetry, and exit cleanly instead of being SIGKILLed mid-write.

Long-lived connections and background work

Do not wait for WebSockets or long gRPC streams to finish, because they may never end. Send a close frame or GOAWAY, and make clients reconnect with jittered backoff. Queue consumers should stop fetching, finish or checkpoint the current message, and nack or requeue anything incomplete so another replica picks it up.

None of this covers SIGKILL, OOM kills, or node failures, where no handler runs. Idempotent handlers and client retries are still required. Draining reduces how often you fall back on them, but it does not replace them.

Copyconst http = require('node:http');

let draining = false;
const server = http.createServer((req, res) => {
  if (req.url === '/readyz') {
    // Readiness probe target: fail as soon as draining starts.
    res.statusCode = draining ? 503 : 200;
    return res.end(draining ? 'draining' : 'ok');
  }
  // While draining, tell HTTP/1.1 clients not to reuse this connection.
  if (draining) res.setHeader('Connection', 'close');
  res.end('hello');
});
server.listen(8080);

const READINESS_DELAY_MS = 5000;  // time for endpoint removal to propagate
const DRAIN_DEADLINE_MS = 25000;  // total budget, below a 30s terminationGracePeriodSeconds

process.once('SIGTERM', () => {
  draining = true; // 1. fail readiness but keep serving late-routed requests

  // 5. Hard deadline measured from SIGTERM, so we exit before the kubelet SIGKILLs us.
  setTimeout(() => {
    console.error('drain deadline reached; closing remaining connections');
    server.closeAllConnections();
    process.exit(1);
  }, DRAIN_DEADLINE_MS).unref();

  setTimeout(() => {
    // 2 + 3. Stop accepting new connections; the callback runs once
    // every in-flight request has finished and its socket has closed.
    server.close(() => {
      console.log('drained cleanly'); // flush logs/telemetry here
      process.exit(0);
    });
    // 4. Close keep-alive sockets that have no active request right now.
    server.closeIdleConnections();
  }, READINESS_DELAY_MS);
});

A Node.js HTTP server that handles SIGTERM in order: it fails readiness, waits for endpoint removal to propagate, stops accepting connections, lets in-flight requests finish with Connection: close, and closes idle keep-alive sockets, all under a deadline shorter than the grace period.

Tuning probe parameters and grace periods

Probe parameters decide two things: how quickly a broken container is noticed, and how often a healthy one is falsely killed. Set them deliberately, because each one moves both outcomes.

For slow startup, prefer a startup probe over a large initialDelaySeconds. A fixed delay is either too short on a cold node or wasted on a warm one.

Illustrative estimate (assumed values: period 10s, timeout 2s, threshold 3, grace 30s, 5s container start). A process hangs just after a successful liveness probe. Probes fail at roughly 12s, 22s and 32s, so detection takes about 32s. If the hung process ignores SIGTERM, SIGKILL arrives about 30s later, at around 62s. The replacement container is serving at about 67s.

Avoid tightening these settings blindly. During a load spike or a long GC pause, every replica can miss probes at once. Liveness then restarts all of them together, and the remaining capacity takes more load. This is a restart storm. A one-second timeout on a CPU-saturated instance measures queueing delay, not health.

Long grace periods have a cost too. Size the grace period to your observed drain time plus a margin, not to the worst case you can imagine. Every terminating pod holds its CPU and memory until it exits, which slows rollouts, node drains and scale-down.

Verification, alternatives, and impact on cost and engineering time

Verify with a rolling restart under load

Point a synthetic load generator at the Service, then run kubectl rollout restart deployment/<name>. Have the application log three timestamped events: SIGTERM received, readiness flipped to failing, and drain complete. The gaps between them are your real timings. The first gap shows how long endpoint removal takes to propagate. The second shows how long in-flight work actually needs. If the load tool records resets or 5xx responses during the restart, compare their timestamps against these phases to find the one that is too short.

Alternatives

These approaches work best layered on top of in-app draining. None of them replaces it.

Cost versus time saved

The upfront work is a signal-aware entrypoint, a readiness flag, a drain routine, and probe configuration.

Illustrative estimate (assumed numbers, not a benchmark)

Formula: annual hours saved = deploy incidents per month × 12 × engineer-hours per incident.

Assumed inputs:

• 2 incidents per month
• 3 engineer-hours per incident
• 16 engineer-hours of upfront work

Result: 2 × 12 × 3 = 72 hours saved per year, against 16 hours spent once. Plug in your own incident history to check whether the trade holds for your team.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to health checks graceful shutdown, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.