How does Karpenter handle spot interruptions?
Spot capacity is unused EC2 capacity sold at a steep discount, with one condition: AWS can reclaim it whenever it needs the capacity back. You get cheaper compute and accept that any node can be taken away on short notice.
AWS publishes several signals about instance health. The most important for spot is the interruption warning, sent roughly two minutes before reclamation. There are also rebalance recommendations, which indicate elevated interruption risk; scheduled change health events; and instance state changes such as stopping or terminating. EventBridge rules match these events and forward them to an SQS queue. Karpenter polls that queue, maps each event to the node it affects, and then taints and drains that node. At the same time it provisions replacement capacity so evicted pods have somewhere to land. This in-process handling is why a separate node termination handler DaemonSet is unnecessary when Karpenter owns the nodes. Running both means two controllers react to the same event.
The queue, the EventBridge rules, and the IAM permissions that let Karpenter read and delete messages are all infrastructure you have to maintain. They fail quietly. If a rule is missing or a policy is wrong, nothing errors. Nodes simply vanish without a drain, and you see it only as abrupt pod failures.
Treat two minutes as a ceiling, not a guarantee. The notice is best effort and can arrive late. Within the window that remains, Karpenter has to detect the event, evict pods, and let each pod's graceful shutdown complete before the instance disappears.
+---------------------------+ +---------------------------+
| AWS EC2 instance events | | IAM policy: Karpenter can |
| - spot interruption (~2m) | | receive + delete messages |
| - rebalance recommendation| +-------------+-------------+
| - scheduled change | |
| - state change (stop/term)| | fails
+-------------+-------------+ | quietly
| |
v |
+-------------+-------------+ |
| EventBridge rules | |
| match and forward events | |
+-------------+-------------+ |
v v
+---------------------------+ +---------------------------+
| SQS interruption queue |<-poll--| Karpenter controller |
| | | maps event -> node |
+---------------------------+ +------+-------------+------+
taint + drain | | provision
v v
+-------------------+ +------------------+
~2 min is a ceiling: | Affected node | | Replacement node |
detect + drain + shutdown | pods evicted | | evicted pods |
must all fit inside it | graceful shutdown | | reschedule here |
+-------------------+ +------------------+
EC2 instance events flow through EventBridge rules into an SQS queue that Karpenter polls. Karpenter then taints and drains the affected node while it provisions replacement capacity, and all of this has to finish within a best-effort window of about two minutes.
How do you configure a Karpenter NodePool for spot and on-demand?
The interruption path only has room to work if the NodePool lets Karpenter choose from many kinds of capacity. The core setting is a karpenter.sh/capacity-type requirement that allows both spot and on-demand. When both are permitted, Karpenter prefers spot. If it cannot launch spot capacity that satisfies the pending pods, for example because the cloud provider returns an insufficient-capacity error, it temporarily marks those offerings unavailable and launches on-demand instead.
Keep the other requirements wide. Allow several instance categories and families, a range of sizes, and every zone your subnets cover. On AWS, Karpenter requests spot capacity with the price-capacity-optimized allocation strategy. That strategy can only pick deep, less-contended pools if you give it many pools to choose from. If you pin the NodePool to one family in one zone, every spot node draws from the same pool. A single reclaim event can then take out a large share of your workers at once, and replacements compete for the same scarce capacity. That is how narrow requirements turn individual interruptions into correlated outages.
To make spot opt-in, split capacity into two NodePools. One is spot-only and carries a taint such as capacity-type=spot:NoSchedule. The other is on-demand and untainted. Only pods that add a matching toleration will land on spot.
Fallback to on-demand happens silently. If spot is constrained for days, your bill quietly moves back toward on-demand rates. Track node counts by the karpenter.sh/capacity-type label and alert when the spot share drops.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general-mixed
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
# Allow both: Karpenter prefers spot, falls back to on-demand
# when spot offerings are unavailable (e.g. insufficient capacity).
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
# Wide families and sizes = many spot pools for
# price-capacity-optimized to choose from.
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["4"]
- key: karpenter.k8s.aws/instance-size
operator: NotIn
values: ["nano", "micro", "small"]
# Every zone your subnets cover, so one zone's reclaim
# does not take out most of the fleet.
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a", "us-east-1b", "us-east-1c"]
# For opt-in spot: copy this pool, set capacity-type to ["spot"] only,
# and add the taint below; keep an untainted on-demand pool alongside.
# taints:
# - key: capacity-type
# value: spot
# effect: NoSchedule
limits:
cpu: "1000"
A Karpenter v1 NodePool that allows both spot and on-demand across several instance categories, sizes, and zones, with a commented taint showing how to turn a spot-only copy into an opt-in pool.
Which workloads should run on spot instances, and which should avoid them?
The deciding question is whether losing one pod with roughly two minutes' notice is routine or an incident.
| Workload | Spot fit | Why |
|---|---|---|
| Stateless replicated services | Good | Other replicas absorb traffic while the replacement schedules |
| Batch jobs with checkpointing | Good | A restart resumes from the last checkpoint |
| CI runners | Good | A failed job is retried, at worst costing one rerun |
| Singletons, leader-elected controllers | Poor | Each interruption causes downtime or a leader failover |
| Databases, long-running stateful pods | Poor | Volume reattachment and replica catch-up rarely finish inside the notice window |
| ML training without checkpoints | Poor | One interruption discards hours of compute |
To place workloads, add a nodeSelector on karpenter.sh/capacity-type: on-demand for anything in the bottom half of the table. Use required node affinity when you need to match several values. For spot-tolerant workloads, preferred affinity for spot lets them fall back to on-demand without failing to schedule. Karpenter does not taint spot nodes by default. If you add a taint to your spot NodePool, only pods that carry a matching toleration will land there. That makes spot opt-in, which is the safer default for shared clusters.
Small services are the trap. A service with one or two replicas has to over-provision to survive an interruption. The pods that remain must carry the full load until a replacement is ready.
Illustrative estimate: assume a spot discount of 60% and a service that needs one replica's worth of capacity. On spot you run three replicas across instance types and zones so one can disappear safely. That costs 3 × 0.4 = 1.2× the on-demand price, which is more than a single on-demand replica.
# Spot-tolerant, replicated service: prefers spot, falls back to on-demand.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-api
spec:
replicas: 3 # enough headroom to lose one pod to an interruption
selector:
matchLabels: {app: web-api}
template:
metadata:
labels: {app: web-api}
spec:
# Matches a custom taint you add to the spot NodePool (Karpenter adds none by default).
tolerations:
- {key: example.com/spot, operator: Equal, value: "true", effect: NoSchedule}
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- {key: karpenter.sh/capacity-type, operator: In, values: [spot]}
containers:
- name: web-api
image: registry.example.com/web-api:1.0.0
---
# Singleton controller: pinned to on-demand, never schedules on spot.
apiVersion: v1
kind: Pod
metadata:
name: leader-controller
spec:
nodeSelector:
karpenter.sh/capacity-type: on-demand
containers:
- name: controller
image: registry.example.com/controller:1.0.0
A replicated Deployment that tolerates a custom spot taint and prefers spot capacity with on-demand fallback, alongside a singleton Pod pinned to on-demand nodes using a nodeSelector on karpenter.sh/capacity-type.
How do you make pod drains safe inside the two-minute window?
Karpenter taints the node and evicts pods through the Eviction API, so PodDisruptionBudgets are respected during the drain. The instance still terminates when the notice expires, whether or not the drain has finished. Every setting below should help pods exit cleanly before that deadline.
- Set PDBs you can actually satisfy. With
maxUnavailable: 1on a three-replica Deployment, evictions proceed one at a time. WithminAvailableequal to the replica count, no eviction can proceed, so pods are killed at reclaim instead of drained. - Keep
terminationGracePeriodSecondswell under two minutes. Karpenter needs time to receive the notice, cordon the node and issue evictions, and that time comes out of the two-minute window. A 30–60 second grace period leaves margin. - Stop traffic before stopping the process. Add a short
preStopsleep so endpoint removal reaches kube-proxy and the load balancer before the app receives SIGTERM. Fail readiness when shutdown begins, then drain in-flight requests. - Spread replicas. Use
topologySpreadConstraintsontopology.kubernetes.io/zoneandkubernetes.io/hostnameso one reclaimed node, or one reclaimed pool, never holds most of a service's replicas.
Spot reclaims are often correlated. If many nodes of one instance type in one zone are reclaimed together, PDBs block most evictions, the nodes terminate anyway, and the replacement pods stay pending while Karpenter launches new capacity. Diversifying instance types reduces this risk better than any PDB setting.
Workloads with long shutdowns or large in-memory state will still lose work. If a job needs ten minutes to flush or hand off state, no grace period setting can fit it into two minutes. Checkpoint to external storage regularly, or run these workloads on on-demand capacity.
# PDB the drain can satisfy: evict one replica at a time
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
spec:
maxUnavailable: 1
selector:
matchLabels: {app: api}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
selector:
matchLabels: {app: api}
template:
metadata:
labels: {app: api}
spec:
# Covers preStop sleep + app drain; leaves margin inside the 2-minute notice
terminationGracePeriodSeconds: 45
topologySpreadConstraints:
# Spread across zones so one reclaimed pool never holds most replicas
- {maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule, labelSelector: {matchLabels: {app: api}}}
# Prefer one replica per node; soft, so replacements are not blocked
- {maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway, labelSelector: {matchLabels: {app: api}}}
containers:
- name: api
image: registry.example.com/api:1.4.2
readinessProbe: # app should return 503 here once shutdown starts
httpGet: {path: /ready, port: 8080}
periodSeconds: 5
lifecycle:
preStop:
# Delay SIGTERM so endpoint removal reaches kube-proxy and the LB
# (requires a sleep binary in the image)
exec: {command: ["sleep", "10"]}
A Deployment and PodDisruptionBudget set up for Spot drains: one eviction at a time, a 45-second grace period that includes the preStop sleep, a preStop delay so endpoint removal propagates before SIGTERM, a readiness probe, and replicas spread across zones and nodes.
How does node consolidation interact with spot interruptions?
Consolidation is voluntary disruption. Karpenter removes or replaces nodes it judges wasteful, and on a spot NodePool that churn adds to the involuntary reclamation you already absorb. Every consolidation event causes the same damage as an interruption: pods restart, in-process caches go cold, and the replacement node pulls images again before serving traffic. The difference is that you control consolidation.
WhenEmpty only removes nodes with no non-daemon pods. WhenEmptyOrUnderutilized also deletes or replaces nodes whose pods could fit elsewhere or on a cheaper instance. That bin-packs more tightly, but it moves running pods. The consolidateAfter setting sets how long a node must stay in a consolidatable state before Karpenter acts. Short values react to every brief traffic dip.
| Policy | What triggers removal | Churn on running pods | Fits |
|---|---|---|---|
WhenEmpty |
Node has no workload pods | Minimal | Latency-sensitive services, large images, warm caches |
WhenEmptyOrUnderutilized |
Node is empty, or its pods can be repacked more cheaply | Ongoing evictions | Stateless batch, fast-starting services |
Disruption budgets cap how many nodes Karpenter may voluntarily disrupt at once. You can scope a budget by reason and by schedule, for example blocking underutilization moves during business hours. Budgets do not slow spot reclamation, because the cloud provider takes the node regardless.
For latency-sensitive services, put them in a dedicated NodePool that uses WhenEmpty or a long consolidateAfter, and add a restrictive budget. Use the karpenter.sh/do-not-disrupt pod annotation sparingly, since it also blocks drift replacement. Let batch pools consolidate aggressively, where repacking costs little.
Does running Kubernetes workers on spot actually save money?
The model is simple: net savings = (on-demand price − spot price) × spot node-hours − interruption costs. The interruption costs are rescheduling delay, rerun batch work, extra headroom replicas, cold caches and image pulls, and engineering time. Spot prices, discounts, and interruption rates vary by region, zone, and instance type, so replace every input below with your own.
Illustrative estimate (assumed numbers, not a benchmark). Assume 20 nodes running 730 hours per month (14,600 node-hours), $0.40/hr on-demand, $0.14/hr spot, and 30 interruptions per month.
| Line item | Assumed input | Monthly |
|---|---|---|
| Gross savings | $0.26 × 14,600 node-hours | +$3,796 |
| Headroom replicas | 2 extra spot nodes × 730 h × $0.14 | −$204 |
| Rerun batch work | 30 × 1 node-hour × $0.14 | −$4 |
| Cold caches, image pulls | 30 × 0.5 node-hour × $0.14 | −$2 |
| Engineering and on-call | 8 h × $100/h | −$800 |
| Net | ≈ +$2,786 |
Compute costs barely register in this example. Engineering time is the cost that decides the outcome. At 3 nodes, the gross gap is about $569, and the same 8 hours of engineering time makes the result negative. Spot pays off at scale, or when the engineering cost is already paid for other reasons.
That engineering cost is often partly paid already. PodDisruptionBudgets, graceful SIGTERM handling, and idempotent, checkpointed jobs are also needed for consolidation, rolling deploys, and node upgrades. Spot consumes engineering time in other places:
- Debugging failures caused by interruptions that look like application bugs.
- Handling capacity shortages when instance types are narrowly constrained.
- Keeping latency-sensitive services stable when interruptions cluster in a single zone.
What are the alternatives to spot capacity for cutting Kubernetes costs?
Spot is the cost lever with the most variance. Cheaper, lower-risk changes often deliver much of the saving first, and they make any later spot discount apply to a smaller bill.
| Lever | Savings predictability | Operational effort | Main risk |
|---|---|---|---|
| Right-sizing requests | High once usage is measured | Moderate, ongoing | OOM kills or CPU throttling if cut too far |
| Consolidation on on-demand | Medium, tracks utilization | Low after tuning disruption budgets | Pod churn during repacking |
| Committed-use pricing | High, contractual | Low, but needs forecasting | Paying for unused commitment |
| Scaling non-prod to zero | High for idle hours | Low to moderate | Cold starts, broken schedules |
| Spot | Variable with market and availability | High | Interruptions, capacity gaps |
Right-size requests first. Karpenter provisions nodes against pod requests, not actual usage, so inflated requests waste money on every pricing model.
Illustrative estimate (assumed numbers, not a benchmark). Formula: reduction = 1 − (p95_usage × (1 + headroom)) / requested.
- Assumed inputs: 400 vCPU requested, 160 vCPU used at p95, and 30% headroom.
- Calculation: 160 × 1.3 = 208, so 1 − 208/400 = 48% fewer vCPU to provision.
- That reduction comes before any spot discount is applied.
A practical mix layers the levers:
- Use commitments to cover the steady on-demand floor that critical, stateful, and latency-sensitive services need.
- Let consolidation keep that floor packed tightly.
- Scale non-production to zero outside working hours.
- Use spot for stateless and burst capacity, where an interruption costs a retry rather than an incident.
How do you test and monitor spot interruption handling before relying on it?
Run interruption drills in staging before production traffic depends on spot capacity. The cheapest drill is to drain or delete a spot node and watch what happens. Deleting the Node object makes Karpenter cordon and drain it the same way it would for a real notice. A stronger test is a real interruption notice sent through your cloud provider's fault-injection tooling, if your account has it. Only that path checks the full chain: notice, then queue, then Karpenter, then drain. During each drill, compare error rate and latency against your SLOs. A drain that respects every PodDisruptionBudget but still causes 5xx spikes points to readiness probes or connection draining, not Karpenter.
Then track whether the cost model matches reality. The minimum signals are:
- Interruption events received by Karpenter. If this stays at zero for weeks while nodes still disappear, the interruption queue or its event routing is broken. Pods are then being killed with no warning.
- Node lifetimes by capacity type. Short spot lifetimes in one instance family suggest you should widen the instance selection.
- Pod restarts and time spent Pending after node loss. Rising pending time means replacement capacity is arriving too slowly for your drain window.
- Node-hours split between spot and on-demand. Your savings estimate depends on this ratio.
That last metric catches quiet drift. When spot capacity runs short, the NodePool falls back to on-demand, and nothing fails. The bill just grows. A weekly alert on the spot share of node-hours shows this within days, not at month-end. It also saves the debugging time of tracing a cost spike back after the fact.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to karpenter spot instances, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.