Kubernetes Pod Disruption Budget Guide: Safe Node Drains and Upgrades

What a pod disruption budget is and what it actually protects

A PodDisruptionBudget (PDB) is a rule that the Eviction API enforces. It is not an availability guarantee. When something asks the API server to evict a pod, the server checks every PDB whose selector matches that pod. If the eviction would leave fewer healthy pods than the budget allows, the server refuses it and the caller has to retry later. A PDB has no effect on anything that never sends an eviction request.

The distinction that matters is not "planned versus unplanned." It is whether the operation goes through eviction.

The last row is where teams get caught out. A kubectl delete pod or a bad rollout can drop a service to zero even when a strict PDB exists.

Platform teams adopt PDBs so they can automate node maintenance without coordinating with every service owner. Upgrades, kernel patching and node replacement can drain nodes continuously. Each service team declares its minimum safe floor once, and the drain tooling waits whenever a service is at that floor. Without PDBs, the only safe way to drain is to do it slowly and by hand, or to accept the risk of an outage.

How kubectl drain and the PDB controller work together

Draining a node with kubectl drain <node> --ignore-daemonsets --delete-emptydir-data happens in two phases. First the node is cordoned, which marks it unschedulable so no new pods land there. Then drain sends an eviction request for each pod through the Eviction API instead of deleting pods directly. The API server checks every eviction against any PDB whose selector matches the pod. If the eviction would push the workload below its budget, the request is rejected with HTTP 429, and drain waits and retries until the budget allows it or the timeout expires.

The two flags handle cases where eviction does not make sense. DaemonSet pods tolerate the unschedulable taint, so their controller would recreate them on the same node right away. --ignore-daemonsets tells drain to skip them rather than abort. Pods using emptyDir keep scratch data that is lost when the pod leaves the node. --delete-emptydir-data is your explicit acknowledgment of that loss.

The disruption controller recomputes the budget continuously. It counts the pods that match the selector and are Ready, compares that count to minAvailable or maxUnavailable, and publishes the result. Readiness probes therefore set the budget directly. A pod that is already unready counts as disrupted, so it uses up allowance before drain touches anything. A probe that reports Ready too early inflates the budget instead.

kubectl get pdb shows ALLOWED DISRUPTIONS next to the configured minimum or maximum. kubectl describe pdb adds current healthy, desired healthy and total expected pods, which tells you why the allowance is what it is.

Copy# Inspect the budget before draining.
# Columns include MIN AVAILABLE, MAX UNAVAILABLE and ALLOWED DISRUPTIONS.
kubectl get pdb -n shop

# describe shows why the allowance has its value. Read these Status fields:
#   Allowed disruptions  - evictions the API server will accept right now
#   Current              - selector-matched pods that are currently Ready
#   Desired              - healthy pods the budget requires
#   Total                - pods the owning controller expects to exist
kubectl describe pdb web-pdb -n shop

# The same numbers in machine-readable form, for a pre-drain check in a script.
kubectl get pdb web-pdb -n shop \
  -o jsonpath='{.status.disruptionsAllowed} allowed, {.status.currentHealthy}/{.status.expectedPods} healthy (need {.status.desiredHealthy}){"\n"}'

# If Current is already below Total, an unready pod is using up the budget.
# Fix that pod before draining, or the drain will stall on 429 retries.

# Cordon, then evict each pod through the Eviction API.
#   --ignore-daemonsets     skip DaemonSet pods instead of aborting
#   --delete-emptydir-data  accept loss of emptyDir scratch data
#   --timeout               stop retrying rejected evictions after this long
kubectl drain worker-node-3 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

# After maintenance, let the scheduler place pods on the node again.
kubectl uncordon worker-node-3

Read a PDB's allowed disruptions and its current, desired and expected healthy counts before draining a node, then drain and uncordon it with the flags that handle DaemonSet and emptyDir pods.

Choosing minAvailable or maxUnavailable

…budget is or isn't leaving room for an eviction. That calculation depends on which field you set and whether you used an integer or a percentage.

A PDB sets exactly one of the two fields. minAvailable is the number of pods that must stay healthy. maxUnavailable is the number of pods that may be down at once. Both fields accept either an integer or a percentage of the expected pod count. Kubernetes rounds percentages up, and this affects the two fields in opposite directions:

With minAvailable, rounding up raises the floor. At low replica counts the floor can reach the total, and then no eviction is ever allowed. With maxUnavailable, rounding up grants at least one disruption whenever the percentage is above zero.

Recommendations

Scaling Deployments: use maxUnavailable. Set an integer like 1 or a percentage like 25%. Either way, the budget keeps tracking the workload as an autoscaler adds or removes replicas, and you avoid the zero-disruption trap described above.

Quorum systems such as etcd or ZooKeeper: use an integer minAvailable. Size it to the quorum. A three-member ensemble needs minAvailable: 2. The safety requirement here is a fixed majority, not a share of the current pod count, so the number should not move.

One constraint applies to both cases. maxUnavailable and percentage values only work when the pods are owned by a controller that exposes a scale subresource. For bare pods, only an integer minAvailable is accepted.

Copy# Scaling web Deployment: cap concurrent disruptions instead of fixing a floor.
# 25% rounds up, so even at 1-3 replicas at least one eviction is allowed,
# and the budget follows the replica count as an autoscaler changes it.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: shop
spec:
  maxUnavailable: 25%
  selector:
    matchLabels:
      app: web
---
# Three-member ZooKeeper ensemble: quorum is a fixed majority (2 of 3),
# so use an integer floor rather than a percentage of current pods.
# This allows one voluntary eviction at a time and blocks the second.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: zookeeper-pdb
  namespace: data
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: zookeeper

Two PodDisruptionBudgets: a percentage maxUnavailable that tracks a scaling Deployment, and an integer minAvailable sized to the quorum of a three-member ZooKeeper ensemble.

When PDBs block drains forever, and when to avoid them

Budgets that can never be satisfied

Three configurations make ALLOWED DISRUPTIONS zero by design: minAvailable equal to the replica count, maxUnavailable: 0, and any PDB covering a single-replica workload. Each eviction request is rejected with HTTP 429, and kubectl drain keeps retrying, so the drain hangs without failing cleanly. To find the cause, run kubectl get pdb -A and look for budgets showing 0 in the ALLOWED DISRUPTIONS column while a node sits cordoned.

The cost of strictness

A budget that blocks every drain does not stop maintenance. It postpones it, and the work usually gets finished by someone at an inconvenient hour. That person often reaches for --disable-eviction, deletes pods directly, or sets a drain timeout. All three bypass the PDB, so the protection is lost exactly when it was supposed to apply. Managed node upgrades and cluster autoscalers can also give up on graceful eviction after a grace period and replace the node anyway. The timeout and the fallback behavior vary by provider, so check your platform's documentation.

For a single replica, a PDB cannot make a disruption safe. You have two honest options: run more replicas, or accept downtime and schedule a maintenance window.

Readiness probes affect PDBs in both directions

A PDB counts only Ready pods as healthy. A probe that never passes keeps healthy pods below the threshold, which freezes evictions. A probe that passes before the app can serve traffic has the opposite effect: the budget counts the pod as available while users get errors, so evictions go through even though real capacity has dropped.

Copy#!/usr/bin/env bash
# Diagnose a drain that hangs because a PodDisruptionBudget allows zero evictions.
set -euo pipefail

NODE="worker-node-3"   # placeholder node name

# 1. Confirm the node is cordoned (STATUS shows SchedulingDisabled).
kubectl get nodes | grep SchedulingDisabled || echo "no cordoned nodes"

# 2. List every PDB with its allowed disruptions and healthy counts.
#    Rows where ALLOWED is 0 can block eviction of the pods they select.
kubectl get pdb -A -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
ALLOWED:.status.disruptionsAllowed,\
HEALTHY:.status.currentHealthy,\
DESIRED:.status.desiredHealthy \
  | awk 'NR==1 || $3==0'

# HEALTHY == DESIRED with ALLOWED 0 usually means the budget can never be met
# (minAvailable == replicas, maxUnavailable: 0, or a single replica).
# HEALTHY < DESIRED points at pods that are not Ready, often a readiness probe.

# 3. See which pods are still on the node and whether they are Ready.
kubectl get pods -A --field-selector spec.nodeName="$NODE" -o wide

# 4. Retry the drain with a timeout so it fails visibly instead of looping.
#    kubectl's --timeout only makes drain give up and exit non-zero; it does
#    not delete pods. Avoid --disable-eviction, which skips PDB checks entirely.
kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

A shell walkthrough that finds PDBs with zero allowed disruptions, separates impossible budgets from budgets blocked by pods that are not Ready, and retries the drain with a timeout that fails visibly instead of bypassing the budget.

Designing PDBs with replicas, topology spread and graceful shutdown

A budget counts healthy pods, not where they run. If two of three replicas share a node, the PDB still serializes drain evictions, but a node crash removes both at once and the budget can't prevent it. Spread replicas with topology spread constraints or pod anti-affinity on the hostname and zone labels. Also keep enough spare capacity for an evicted pod to schedule elsewhere. If it can't schedule, it stays Pending and the next eviction waits indefinitely.

Eviction triggers a normal pod deletion. The preStop hook runs first, then SIGTERM, and SIGKILL follows when terminationGracePeriodSeconds expires. A short preStop delay gives endpoint removal time to propagate. On SIGTERM, the app should stop accepting work and finish in-flight requests. The PDB knows nothing about leader handoff or data rebalancing. That logic belongs in an operator or the preStop path.

Worked example (illustrative estimate)

Assumed inputs:

• 3-replica quorum service with minAvailable: 2
• 5 nodes, one replica each on nodes A, B and C
• Per replica: 20 s shutdown, 15 s scheduling and image pull, 40 s startup, 25 s until readiness passes

Drain D and E first. They hold no replicas, so they finish quickly and free up capacity. Then drain A, B and C one at a time. Each move takes about 100 s before the budget allows the next eviction, so the quorum service needs about 5 minutes of serialized time. Slow startup or readiness extends every step.

Pre-upgrade checklist

• Replicas spread across nodes and zones
• Spare capacity for at least one rescheduled pod
• Readiness probes that reflect real serving ability
• Grace period longer than worst-case shutdown
• State handoff handled outside the PDB

CopyapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: quorum-pdb
spec:
  minAvailable: 2            # keep quorum: only one of 3 replicas may be evicted at a time
  selector:
    matchLabels: {app: quorum}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: quorum
spec:
  serviceName: quorum
  replicas: 3
  selector:
    matchLabels: {app: quorum}
  template:
    metadata:
      labels: {app: quorum}
    spec:
      terminationGracePeriodSeconds: 60   # must exceed worst-case shutdown (preStop + SIGTERM drain)
      topologySpreadConstraints:          # one replica per node, balanced across zones
        - {maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule, labelSelector: {matchLabels: {app: quorum}}}
        - {maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway, labelSelector: {matchLabels: {app: quorum}}}
      containers:
        - name: app
          image: registry.example.com/quorum:1.0.0
          ports: [{containerPort: 8080}]
          readinessProbe:                 # PDB counts a pod as healthy only once this passes
            httpGet: {path: /ready, port: 8080}
            periodSeconds: 5
            failureThreshold: 2
          lifecycle:
            preStop:                      # runs before SIGTERM; lets endpoint removal propagate
              exec:
                command: ["sh", "-c", "sleep 5"]
          # On SIGTERM the app should stop accepting work, finish in-flight requests,
          # and hand off leadership; the PDB does not coordinate that state.

A PodDisruptionBudget with minAvailable: 2 paired with a 3-replica StatefulSet that spreads pods across nodes and zones and sets a readiness probe, a preStop delay and a termination grace period so drains move one replica at a time without losing quorum.

Who owns PDBs and how to enforce a cluster-wide policy

Split authorship from execution

App teams should write their own PDBs because only they know the real availability constraints: how many replicas a quorum needs, how much headroom the service has at peak, and whether losing one pod during a deploy is harmless or triggers a leader election. The platform team should own drain execution: when nodes are cordoned, how many drain in parallel, and what happens when a drain stalls. Each side controls only what it understands, and the PDB is the contract between them.

Enforce the contract

Most stuck upgrades come from budgets that can never permit an eviction. Two checks catch them:

• At admission: a validating admission webhook or policy rule rejects PDBs with maxUnavailable: 0 or minAvailable: 100%. These values are always unsatisfiable, so they can be rejected without any other context.
• Continuously: a budget like minAvailable: 3 is only a problem if its selector matches three replicas, and admission can't see that when the PDB is created. A periodic audit should flag PDBs whose status.disruptionsAllowed stays at zero while all pods are healthy, and route the finding to the owning team.

Add review guidance alongside the checks. Every PDB should have an owner label, a short note explaining its quorum, and a replica count that leaves at least one disruption allowed.

Why it saves time

Without this contract, a blocked drain means the platform engineer has to find the right person and agree on whether to override the budget, often in the middle of a maintenance window. With admission checks and audits in place, unsatisfiable budgets are caught before the upgrade starts. Drains then proceed unattended, and manual escalation is limited to real capacity problems.

Alternatives and impact on cost and engineering time

PDBs are one option among several, and most teams combine them with the alternatives below.

Safe drains always need headroom. Each protected workload needs at least one replica beyond its minimum, and the cluster needs enough spare capacity to reschedule evicted pods. Without that headroom, every drain becomes a manual negotiation.

Illustrative estimate (assumed numbers, not a benchmark)

• Headroom cost per year = spare nodes × node cost per month × 12. Assuming 2 spare nodes at 250 per month gives 2 × 250 × 12 = 6,000.
• Engineering cost per year = stalled or forced drains per year × engineer-hours per event × hourly rate. Assuming 6 events, 8 hours each and 90 per hour gives 6 × 8 × 90 = 4,320, before counting any customer-facing incident.

Plug in your own figures. If headroom costs more than the time it saves, you can drop PDBs for non-critical workloads.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes pod disruption budget, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.