How does spot interruption work on AWS, and who should run workers on it?
Spot instances are spare EC2 capacity sold at a steep discount to On-Demand pricing. The trade is that AWS can reclaim the instance whenever it needs the capacity back. For Kubernetes workers this is often a good deal: the scheduler already treats nodes as replaceable, so a cluster built to survive node loss can run much of its fleet on cheaper hardware.
When AWS reclaims a Spot instance, it issues an interruption notice roughly two minutes before stopping or terminating it. The notice is exposed through instance metadata and emitted as an EventBridge event. AWS may also send an EC2 instance rebalance recommendation earlier, which signals elevated interruption risk for that instance. A rebalance recommendation is not a guaranteed precursor. It can arrive well ahead of an interruption, alongside one, or for an instance that is never reclaimed.
Treat the two-minute notice as best effort, not as a drain guarantee. Your handler has to receive the notice, cordon the node, and evict its pods. Each pod then has to shut down cleanly while replacement capacity launches and pulls images. Long termination grace periods, slow shutdown hooks, or blocking PodDisruptionBudgets can push that work past the window. When that happens, pods are killed mid-request.
Reclaims also tend to be correlated. A capacity crunch in one instance type or Availability Zone can interrupt many of your nodes at once, which turns a single-node event into a partial outage if you lack diversity.
Spot fits workloads that tolerate abrupt loss:
- stateless services
- horizontally scaled APIs
- queue workers that retry unfinished jobs
- CI runners
For these, interruption is a routine operating condition you design for, not a rare failure.
How does Karpenter's native interruption handling work?
Karpenter learns about interruptions from EC2 through an Amazon SQS queue that you provision. EventBridge rules match four event types and forward them to that queue:
- Spot interruption warnings, which give roughly two minutes' notice before EC2 reclaims the instance
- Rebalance recommendations, which signal elevated interruption risk
- Scheduled health events, such as planned instance retirement
- Instance state changes, such as stopping or terminating
The controller polls the queue and maps each event to a node it manages. For actionable events, it taints and drains the node, evicting pods so that PodDisruptionBudgets and graceful termination apply. While the drain runs, it launches replacement capacity. Starting the new node in parallel matters because a fresh instance takes time to boot and join the cluster, and two minutes of warning leaves little room for doing the steps one after another. Rebalance recommendations reach the queue, but Karpenter does not drain on them. They fire early and often, and acting on each one would cause needless churn.
This makes a separate node termination handler unnecessary. Running both means two controllers reacting to the same event, which can produce duplicate or conflicting drains.
Point Karpenter at the queue with the Helm value settings.interruptionQueue, set to the queue name. On the AWS side, two permissions are required:
- The controller's IAM role must be able to receive and delete messages on the queue.
- The queue's resource policy must allow EventBridge to send messages to it.
Get any of these pieces wrong and Karpenter reports no error. It never sees the event, so pods are not drained and the node simply disappears. To catch this early, check the queue's message metrics and the controller logs after you set it up, before you rely on spot capacity.
# Placeholder values; replace with your own.
CLUSTER_NAME="prod-eks"
QUEUE_NAME="prod-eks-karpenter-interruptions"
KARPENTER_NAMESPACE="kube-system"
# Point the controller at the interruption queue. Without this value,
# Karpenter does not handle interruptions at all.
# KARPENTER_VERSION must already be set to the chart version you run.
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
--version "${KARPENTER_VERSION}" \
--namespace "${KARPENTER_NAMESPACE}" \
--reuse-values \
--set settings.clusterName="${CLUSTER_NAME}" \
--set settings.interruptionQueue="${QUEUE_NAME}"
# Verify the pieces that fail quietly when they are wrong.
QUEUE_URL=$(aws sqs get-queue-url --queue-name "${QUEUE_NAME}" \
--query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "${QUEUE_URL}" \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
# 1. EventBridge rules must target this queue. Expect one rule per event type.
aws events list-rule-names-by-target --target-arn "${QUEUE_ARN}"
# 2. The queue policy must allow events.amazonaws.com to call sqs:SendMessage.
aws sqs get-queue-attributes --queue-url "${QUEUE_URL}" \
--attribute-names Policy --query Attributes.Policy --output text
# 3. A backlog that keeps growing means the controller is not consuming.
# Check its IAM role for sqs:ReceiveMessage and sqs:DeleteMessage.
aws sqs get-queue-attributes --queue-url "${QUEUE_URL}" \
--attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage
# 4. Look for interruption handling or queue access errors in controller logs.
kubectl logs -n "${KARPENTER_NAMESPACE}" deploy/karpenter --since=1h \
| grep -iE 'interruption|sqs|AccessDenied'
Sets settings.interruptionQueue through Helm, then checks the EventBridge rule targets, the queue policy, the queue backlog and the controller logs, because a broken link in any of these produces no error.
How do you configure a Karpenter NodePool for spot instances?
The NodePool's requirements list controls what Karpenter may launch. Set karpenter.sh/capacity-type to In with both spot and on-demand. When both are allowed, Karpenter launches spot first. If EC2 returns an insufficient-capacity error for every spot option that fits the pending pods, it launches on-demand instead.
Diversify the instance pool
Spot reliability depends mostly on how many capacity pools Karpenter can choose from. Each instance type in each availability zone is a separate spot pool with its own supply. Pinning three instance types in one zone leaves Karpenter few options when one pool runs short. Describe the shape of the node you need with these labels:
karpenter.k8s.aws/instance-categoryset toc,mandrkarpenter.k8s.aws/instance-generationgreater than a floor you acceptkarpenter.k8s.aws/instance-cpuandkarpenter.k8s.aws/instance-memorywithin a rangetopology.kubernetes.io/zoneset to several zones
A broader pool lets Karpenter's allocation strategy choose pools that are both cheap and deep. Deeper pools are reclaimed less often and run out of capacity less often.
| Choice | Benefit | Cost |
|---|---|---|
| Broad requirements | Fewer interruptions and capacity errors | Mixed CPU models make latency less consistent |
| Pinned types | Predictable performance | Shallow pools are reclaimed more often |
| Spot with on-demand fallback | Pods stay schedulable | On-demand launches can raise the bill without anyone noticing |
Mixed node types also cause problems for software licensed per core and make capacity planning per node harder. Track the karpenter.sh/capacity-type label across your nodes. If the share of on-demand nodes rises, spot capacity is running short, even though nothing has failed.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general-spot
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default # placeholder EC2NodeClass
requirements:
# Allow both: Karpenter tries spot first and falls back to on-demand
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
# Describe the node shape instead of pinning instance types
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["5"] # generation 6 and newer (Gt is exclusive)
- key: karpenter.k8s.aws/instance-cpu
operator: Gt
values: ["3"] # at least 4 vCPU
- key: karpenter.k8s.aws/instance-cpu
operator: Lt
values: ["33"] # at most 32 vCPU
- key: karpenter.k8s.aws/instance-memory
operator: Gt
values: ["8191"] # MiB: at least 8 GiB
# Each zone multiplies the number of spot pools available
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a", "us-east-1b", "us-east-1c"]
limits:
cpu: "1000" # cap total capacity, including on-demand fallback
A Karpenter v1 NodePool that allows spot with on-demand fallback and widens the spot pool by constraining instance category, generation, vCPU and memory ranges across three availability zones rather than listing specific instance types.
How do you make workloads tolerate a spot interruption?
Karpenter's drain only helps if pods shut down cleanly inside the two-minute notice. That budget is shared: Karpenter has to receive the event, cordon the node and evict pods before your containers even see SIGTERM. Set terminationGracePeriodSeconds well under two minutes so that shutdown finishes before the instance is reclaimed. If the grace period runs past the deadline, the kill is not graceful.
The Deployment below combines the settings that matter:
- Several replicas with topology spread constraints across zones and nodes. Spot reclamation tends to hit one instance pool at a time, so replicas packed onto similar nodes can all go at once.
- A PodDisruptionBudget so the drain evicts pods gradually rather than all together. The budget cannot stop AWS from reclaiming the instance.
- A preStop hook and SIGTERM handling. The preStop hook pauses briefly while endpoints update. On SIGTERM, the app stops accepting new requests, finishes the ones in flight and exits.
- A readiness probe that fails during shutdown, so the load balancer stops routing to the pod before the process exits.
These settings still have limits. Pods that take a long time to shut down, hold large state in memory or keep long-lived connections such as WebSockets or streaming consumers can lose work. Those workloads need checkpointing or client reconnect logic.
Size replicas and cluster headroom for losing several nodes at the same time, not one. Reclaims arrive in waves.
Getting shutdown right is a one-time task. Once it is in place, each interruption is a routine reschedule instead of a page that someone has to investigate.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api}
spec:
maxUnavailable: 1 # drain evicts one pod at a time; AWS can still reclaim the node
selector: {matchLabels: {app: api}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
replicas: 6 # sized to survive several spot nodes lost at once
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
terminationGracePeriodSeconds: 45 # covers preStop + drain; well under the 2-minute notice
topologySpreadConstraints:
- maxSkew: 1 # spread across zones so one reclaim wave can't take every replica
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector: {matchLabels: {app: api}}
- maxSkew: 1 # and across nodes within a zone
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector: {matchLabels: {app: api}}
containers:
- name: api
image: registry.example.com/api:1.4.2
ports: [{containerPort: 8080}]
lifecycle:
preStop: # give endpoints and the load balancer time to drop this pod
exec: {command: ["sleep", "10"]}
# On SIGTERM the app makes /readyz return 503, stops accepting new requests,
# finishes in-flight ones within ~30s, then exits.
readinessProbe:
httpGet: {path: /readyz, port: 8080}
periodSeconds: 5
failureThreshold: 1
A PodDisruptionBudget and Deployment that combine zone and node spread, a grace period that fits inside the spot notice, a preStop delay, and a readiness probe the app fails on SIGTERM so traffic moves away before the pod exits.
When should you avoid spot, and what are the alternatives?
Some workloads cost more on spot than they save. A database with an EBS volume can only restart in the volume's Availability Zone, and that zone may have no spot capacity left during a reclaim wave. A singleton controller, such as a leader-elected operator or a cron scheduler, stops working each time its node disappears. A batch job that runs for hours without checkpointing loses all of its progress on every interruption. You can engineer around each of these cases, but that work moves the cost into engineering time and incident risk.
Put these workloads on a separate on-demand NodePool. Set the NodePool's karpenter.sh/capacity-type requirement to on-demand and add a taint. Then give only the protected workloads a matching toleration and a node selector on that label. The taint keeps other pods off these nodes, and the selector keeps the protected pods off spot.
The karpenter.sh/do-not-disrupt annotation stops Karpenter from voluntarily removing a pod's node through consolidation or drift. It does not stop AWS from reclaiming a spot instance. After the two-minute notice, the node terminates whether the annotation is set or not.
| Option | Cost | Availability | Operational effort |
|---|---|---|---|
| On-demand only | Highest | Highest | Lowest |
| Mixed spot and on-demand NodePool | Lower | Good if workloads tolerate interruption | Moderate |
| Separate dedicated NodePools | Low for tolerant workloads, full price for protected ones | High for protected workloads | Moderate to high |
| Commitment-based discounts | Reduced on-demand rate | Same as on-demand | Low, but requires capacity forecasting |
# On-demand NodePool for workloads that should not run on spot (Karpenter v1 API).
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: on-demand-protected
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
# Only launch on-demand capacity from this pool.
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
# Keep pods without a matching toleration off these nodes.
taints:
- key: workload-class
value: protected
effect: NoSchedule
---
# Pod template fragment for a protected workload, such as a database StatefulSet.
template:
metadata:
annotations:
# Blocks voluntary disruption (consolidation, drift) only. It does not stop a spot reclaim.
karpenter.sh/do-not-disrupt: "true"
spec:
# The node selector keeps the pod off spot nodes in other NodePools.
nodeSelector:
karpenter.sh/capacity-type: on-demand
# The toleration lets the pod schedule onto the tainted on-demand nodes.
tolerations:
- key: workload-class
operator: Equal
value: protected
effect: NoSchedule
An on-demand Karpenter NodePool with a taint, and a pod template fragment that targets it with a matching toleration, a capacity-type node selector and the do-not-disrupt annotation.
How does node consolidation interact with spot interruptions?
Consolidation is Karpenter's other source of node churn. It runs independently of spot reclaims, so the two add together. The consolidationPolicy field on a NodePool's disruption block sets how aggressive it is:
WhenEmptyremoves only nodes that run no workload pods. DaemonSet pods don't count.WhenEmptyOrUnderutilizedalso drains nodes whose pods fit elsewhere, or replaces a node with a cheaper one.
consolidateAfter sets how long a node must stay eligible before Karpenter acts. A short value reclaims capacity quickly. It also lets Karpenter remove a node minutes after a scale-down, sometimes just before load returns.
Every consolidation evicts pods, the same as a spot interruption does. On a spot-heavy cluster, a single service might see an interruption-driven restart and then a consolidation-driven restart within the same hour. Each restart means cold caches, JIT warm-up, connection pool re-establishment and a latency spike for traffic hitting the new replicas. Disruption budgets limit this. A budget such as nodes: "10%" caps how many nodes Karpenter disrupts at once. You can add budgets that apply only on a schedule, for example to block consolidation during business hours.
The example that follows uses WhenEmptyOrUnderutilized with a longer consolidateAfter and a small percentage budget. You keep some idle capacity in exchange for fewer evictions on top of spot reclaims.
The opposite mistake also costs money. Two common culprits can stop consolidation on many nodes:
- PodDisruptionBudgets with
maxUnavailable: 0 karpenter.sh/do-not-disruptannotations applied by default
Karpenter respects both, so underutilized nodes keep running and billing. Reserve both for workloads that truly can't move.
How much can Karpenter spot instances cut EKS cost, and how do you validate it?
Illustrative estimate (assumed numbers, not a benchmark). Assume a baseline of 20 on-demand nodes at $0.40/hour, a spot price of $0.14/hour for the same shape, 20% of capacity landing on on-demand fallback, 15% extra headroom to absorb reschedules, and 730 hours per month.
Monthly cost = N × (1 + h) × [f × P_od + (1 − f) × P_spot] × 730
| Scenario | Nodes | Blended $/hour | Monthly cost | Savings vs. on-demand |
|---|---|---|---|---|
| All on-demand | 20 | 0.400 | $5,840 | — |
| Pure spot (headline) | 20 | 0.140 | $2,044 | 65% |
| Spot + 20% fallback + 15% headroom | 23 | 0.192 | $3,224 | ~45% |
Fallback and headroom take 20 points off the headline discount. Engineering time also counts. If hardening workloads takes 80 hours at an assumed $100/hour, the $8,000 cost needs about three months of the ~$2,616 monthly savings to pay back. Use your own fallback rate, measured from real capacity-type data, rather than guessing.
Validating it
- Disruption activity: Read the Karpenter controller logs and the Kubernetes events on nodes and NodeClaims. Check whether nodes are removed by interruption or by consolidation. A high consolidation rate points to NodePool tuning. A high interruption rate points to narrow instance diversity.
- Workload impact: Track pod restart counts and how long pods stay Pending after a node disappears. Rising pending time means your headroom or provisioning speed is too low.
- Capacity mix: Group node-hours by the
karpenter.sh/capacity-typelabel. If on-demand quietly grows, your actual savings shrink. - Deliberate tests: In a non-production cluster, trigger spot interruptions on purpose. AWS Fault Injection Service can send real interruption notices. Confirm that pods drain, reschedule, and meet their latency targets before you rely on the setup in production.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to karpenter spot instances, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.