How does spot reclamation reach a Kubernetes worker?

A spot instance is spare provider capacity sold at a discount, with the condition that the provider can take it back at short notice. In Kubernetes, that instance is an ordinary worker node. The scheduler does not treat it differently unless you add labels and taints.

Reclamation reaches your pods in this order:

  1. The provider issues a notice. Depending on the provider, this is a metadata endpoint signal or an event, and it arrives anywhere from tens of seconds to a couple of minutes before shutdown.
  2. Something has to act on that notice. Kubernetes does not watch for it. A node termination handler running on the node, or your managed platform's equivalent, detects the notice, cordons the node, and starts a drain.
  3. The drain evicts pods. Each pod runs its preStop hook, if it has one, and its containers then receive SIGTERM.
  4. Pods are killed by whichever comes first: terminationGracePeriodSeconds expiring, or the instance powering off. The provider's deadline does not depend on your grace period setting.

Treat the notice window as an upper bound, not a guarantee. Detection and the start of the drain take time, and pods evicted late in the drain get less of the window. A job that writes a large checkpoint during shutdown can run out of time partway through the write. Large images do not use up the window themselves, but they slow the replacement pod's start elsewhere. That adds to the time lost after every interruption.

Teams adopt spot for the lower hourly price. That price says nothing about savings on its own. The actual cost also includes retried work, idle time while pods reschedule, and the engineering effort needed to make jobs survive interruptions.

+---------------------+     notice (tens of s to ~2 min)
| Cloud provider      |------------------------------+
| (reclaims capacity) |                              |
+---------------------+                              v
          |                       +-------------------------------+
          | hard deadline:        | Node termination handler      |
          | instance powers off   | (or managed equivalent)       |
          | (ignores your grace   | detects the notice            |
          |  period setting)      +-------------------------------+
          |                                      | cordon + drain
          |                                      v
          |                       +-------------------------------+
          |                       | Spot worker node              |
          |                       |  pod A: evicted early         |
          |                       |  pod B: evicted late          |
          |                       +-------------------------------+
          |                                      | per pod
          |                                      v
          |                       +-------------------------------+
          |                       | preStop hook -> SIGTERM       |
          |                       | (graceful exit, checkpoint)   |
          |                       +-------------------------------+
          |                                      |
          v                                      v
+------------------------------------------------------------------+
| Pod killed at whichever comes first:                             |
|   terminationGracePeriodSeconds expires  OR  instance disappears |
+------------------------------------------------------------------+
 Window shrinks: detection delay + drain order + checkpoint writes

The spot reclamation path: the provider's notice reaches a node termination handler that cordons and drains the node. Each pod runs preStop and receives SIGTERM, then is killed when its grace period ends or the instance powers off, whichever comes first.

Do spot instances in Kubernetes actually save money?

A useful comparison has to include the time and work that interruptions add on top of the job itself:

  • Spot cost = spot price × (useful runtime + rework + rescheduling delay + startup overhead)
  • On-demand cost = on-demand price × useful runtime

The rework term depends on how the job recovers. A checkpointed job loses on average about half a checkpoint interval per interruption. A job that cannot resume loses everything it had done. If interruptions arrive at random at rate λ, a job that needs T hours of uninterrupted runtime takes about (e^{λT} − 1) / λ hours to finish, so its cost grows exponentially with job length.

Illustrative estimate (assumed numbers, not a benchmark). Assume on-demand costs $1.00/h and spot costs $0.35/h. Assume one interruption per 10 node-hours (λ = 0.1/h), and 15 minutes of rescheduling and startup per restart. The checkpointed job saves state every 5 minutes.

Job On-demand cost Expected spot hours Spot cost Saving
20 min, checkpointed $0.33 ~0.34 $0.12 ~64%
12 h, non-resumable $12.00 ~23.8 $8.33 ~31%
24 h, non-resumable $24.00 ~102.7 $35.95 −50%

All three jobs get the same 65% discount on the hourly price. The results differ because of rework. Savings estimates look better than reality when they leave out:

  • Rework: work that has to be redone after an interruption.
  • Replacement time: nodes that are billed while pods wait to be rescheduled.
  • The right baseline: comparing against list on-demand prices instead of what you already pay for on-demand capacity.

Engineering effort is also part of the cost. Making a job idempotent and adding checkpoints takes design, testing and ongoing maintenance. For a workload that uses only a few dozen node-hours a month, that one-time effort can cost more than several years of spot savings.

Which backend workloads should run on spot, and which should you avoid?

The deciding factor is interruption cost: how much work you lose, and what breaks, when a node disappears with about two minutes' notice.

Workload Interruption cost Placement
Stateless queue consumers Low. Unacked messages are redelivered. Spot
Idempotent batch jobs Low. A rerun produces the same result. Spot
Long jobs with checkpoints Bounded by the checkpoint interval Spot, if the interval is tuned
Long jobs without checkpoints All progress since the job started On-demand
Singleton schedulers (cron leaders, coordinators) Missed or duplicated triggers On-demand
Stateful services with unreplicated data Data loss or a long recovery On-demand

For checkpointing jobs, set the interval so the work lost per interruption stays small compared with the job's total runtime. Each checkpoint also has a cost in I/O and pause time, so checkpointing every few seconds can erase the savings.

Spot pays off for teams that run enough interruptible node-hours for the discount to outweigh the engineering effort, and whose jobs already tolerate retries. Wait until retries and idempotency are reliable before adopting it. If you add spot first, you will spend time debugging partial writes during reclamations.

Alternatives, with their trade-offs:

  • On-demand only: Predictable and needs no extra design work, but you pay full price.
  • Reserved or committed capacity: Discounts a steady baseline without interruptions. You pay for the commitment even when you don't use it, and it doesn't cover bursts.
  • Rightsizing and bin-packing: Tightening requests and consolidating nodes often removes more waste than spot does, at no reliability cost. It requires accurate usage data.
  • Off-peak scheduling: Shifting batch work into idle windows on capacity you already pay for costs nothing extra. It delays results, which deadline-bound jobs can't accept.

How should you lay out spot and on-demand node groups?

Run two kinds of node groups. A small on-demand baseline holds what must survive a reclamation wave: cluster DNS, ingress, the cluster autoscaler, job coordinators and anything stateful. A larger spot group carries the elastic batch capacity. Build it from several instance types of similar shape across at least three zones. Reclamation is driven by demand on a specific capacity pool, meaning one instance type in one zone. Spreading across pools makes it less likely that a single event drains the whole group.

Make spot opt-in

Taint every spot node, for example capacity=spot:NoSchedule. The scheduler will not place a pod there without a matching toleration, so a forgotten Deployment cannot drift onto spot. A toleration only permits placement and does not attract the pod. To steer opted-in jobs onto spot, pair the toleration with node affinity on a spot label.

Preferred versus required

Base the choice on whether the job can wait:

  • Required affinity: Use it when on-demand cost would break the budget and a delayed start is acceptable. Pods stay Pending until spot capacity returns.
  • Preferred affinity: Use it when deadlines matter. The scheduler favors spot but falls back to the baseline group, so the baseline needs headroom or room to autoscale.

Trade-offs

Diversification lowers correlated reclamation but does not eliminate it. A broad demand spike can hit several pools at once. Mixed instance types differ in CPU generation and memory bandwidth, so runtimes vary and resource requests must fit the smallest type. Capacity planning also gets harder, because available node counts depend on pools you do not control.

# Spot nodes carry taint capacity=spot:NoSchedule and label capacity=spot
# (set via the node group config). Only jobs that opt in can land there.
apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-reindex
spec:
  backoffLimit: 6            # retries absorb pods lost to reclamation
  template:
    spec:
      restartPolicy: Never
      tolerations:
        # Permits scheduling onto tainted spot nodes; does not attract.
        - key: capacity
          operator: Equal
          value: spot
          effect: NoSchedule
      affinity:
        nodeAffinity:
          # Deadline matters: favor spot, fall back to on-demand baseline.
          # Swap for requiredDuringSchedulingIgnoredDuringExecution if the
          # job can wait in Pending until spot capacity returns.
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: capacity
                    operator: In
                    values: ["spot"]
      containers:
        - name: worker
          image: registry.example.com/reindex:1.0
          resources:
            # Size requests to fit the smallest instance type in the spot group.
            requests:
              cpu: "2"
              memory: 4Gi

A batch Job that tolerates the spot taint and uses preferred node affinity to favor spot nodes while still falling back to the on-demand baseline.

How do you make a batch job interruption-safe in code?

When the node is drained, the kubelet sends SIGTERM to each container's main process and waits up to terminationGracePeriodSeconds before sending SIGKILL. If your entrypoint is a shell wrapper that does not forward signals, the worker never sees SIGTERM. Run the worker as PID 1 or use an init that forwards signals.

On SIGTERM, the worker should do four things in order:

  1. Stop pulling new work. Close the consumer loop first so no new message is leased.
  2. Finish or release the current unit. If it can finish inside the grace period, finish it. If not, release it explicitly with a nack or a zero visibility timeout so another worker picks it up quickly instead of waiting for the lease to expire.
  3. Write a checkpoint.
  4. Exit with a status your pod failure policy can recognize.

None of this guarantees exactly-once processing. With at-least-once delivery, a unit can be redelivered after a crash that happens between "write result" and "ack." Make every write idempotent: key outputs by a deterministic unit ID, use upserts or conditional writes, and only acknowledge after the result is durable.

Choosing a checkpoint interval

A checkpoint costs time C. With interruption rate λ and interval τ, two costs pull in opposite directions:

  • Checkpoint overhead is roughly C/τ.
  • Expected lost work per hour is roughly λτ/2.

The total is minimized near τ = √(2C/λ).

Illustrative estimate: assume C = 30 s and one interruption per 10 node-hours. Then τ ≈ 24 minutes. Checkpoint overhead and expected lost work each come to about 2%.

Checkpointing does not remove this cost. It moves it into object storage requests, network egress and disk I/O, so measure those costs as well.

import signal, sys, threading, time

stop = threading.Event()
signal.signal(signal.SIGTERM, lambda signum, frame: stop.set())

GRACE_BUDGET_S = 25       # stay under terminationGracePeriodSeconds (assumed 30)
CHECKPOINT_EVERY_S = 1440 # ~24 min, from the illustrative tau = sqrt(2C/lambda)
EXIT_INTERRUPTED = 143    # 128 + SIGTERM; match this in podFailurePolicy onExitCodes

def run(queue, store, process_step):
    # queue and store stand in for your own client wrappers.
    while not stop.is_set():                  # 1. stop leasing once SIGTERM arrives
        msg = queue.lease(timeout_s=5)
        if msg is None:
            continue
        ckpt = store.load_checkpoint(msg.unit_id) or {"offset": 0}
        last_ckpt, deadline = time.monotonic(), None
        while ckpt["offset"] < msg.total_steps:
            if stop.is_set() and deadline is None:
                deadline = time.monotonic() + GRACE_BUDGET_S
            if deadline and time.monotonic() > deadline:
                store.save_checkpoint(msg.unit_id, ckpt)  # 3. persist progress
                queue.nack(msg)                           # 2. release, don't wait for lease expiry
                sys.exit(EXIT_INTERRUPTED)                # 4. recognizable exit status
            process_step(msg, ckpt["offset"])
            ckpt["offset"] += 1
            if time.monotonic() - last_ckpt > CHECKPOINT_EVERY_S:
                store.save_checkpoint(msg.unit_id, ckpt)
                last_ckpt = time.monotonic()
        # Idempotent write keyed by a deterministic unit ID; a redelivery overwrites, never duplicates.
        store.upsert_result(key=msg.unit_id, value=ckpt)
        queue.ack(msg)                        # ack only after the result is durable
    sys.exit(EXIT_INTERRUPTED if stop.is_set() else 0)

A Python worker loop that stops leasing on SIGTERM, finishes the current unit within a grace budget or checkpoints and nacks it, writes results idempotently before acknowledging, and exits with a status a pod failure policy can match.

How do Job backoffLimit, pod failure policy and PodDisruptionBudgets behave on spot?

By default, a Job counts every failed pod against backoffLimit, whatever the cause. When a spot node is reclaimed, its pods are evicted or garbage-collected as failed. A Job with backoffLimit: 3 can therefore be marked Failed after a few reclamations even though the code never errored. The Job raises a failure alert, someone reruns it by hand, and any work it had not checkpointed is lost.

Separating disruptions from bugs

Pod failure policy, where your cluster supports it, lets the Job controller tell these cases apart. It requires restartPolicy: Never. A rule matching the DisruptionTarget pod condition with action Ignore stops reclamation-driven failures from counting toward backoffLimit. A second rule can map a known fatal exit code to FailJob, so real bugs fail fast instead of retrying. Check that your interruption path sets that condition. Eviction through the API sets it, and so does pod garbage collection after a node disappears. A process killed with no Kubernetes involvement may not set it.

What PodDisruptionBudgets can and cannot do

A PodDisruptionBudget only gates voluntary evictions made through the eviction API, such as drains, upgrades and autoscaler scale-down. It cannot stop the provider from reclaiming an instance. A strict PDB can even backfire: it can block a termination handler's drain until the notice window expires, and then the node is terminated anyway.

Illustrative estimate (assumed numbers, not a benchmark). The formula is false failures per month × minutes per triage and rerun = engineering time lost. With 40 false failures per month and 20 minutes each, that comes to 800 minutes, or about 13 hours a month of manual reruns and alert noise.

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-batch
spec:
  # Real application failures still get a few retries before the Job fails.
  backoffLimit: 3
  podFailurePolicy:
    rules:
      # Pod was disrupted (eviction, node loss, preemption): do not count it
      # against backoffLimit. The controller simply creates a replacement pod.
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
      # Exit code 42 is our known fatal error (e.g. bad input).
      # Fail the whole Job immediately instead of burning retries.
      - action: FailJob
        onExitCodes:
          containerName: worker
          operator: In
          values: [42]
  template:
    spec:
      # Required when podFailurePolicy is set.
      restartPolicy: Never
      containers:
        - name: worker
          image: registry.example.com/batch-worker:1.0.0
          args: ["--checkpoint-dir=/data/checkpoints"]
---
# Gates voluntary evictions only (drains, autoscaler scale-down).
# It cannot stop the provider from reclaiming a spot instance.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nightly-batch-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      batch.kubernetes.io/job-name: nightly-batch

A Job whose pod failure policy ignores disruption-caused pod failures and fails fast on a known fatal exit code, paired with a lenient PDB that limits only voluntary evictions.

What happens when spot capacity runs out, and how do you verify savings?

When the cloud has no spot capacity in your instance types and zones, the autoscaler's provisioning request fails. It then either retries later or tries an on-demand group, depending on how the groups are configured. Each failed attempt adds to the time before a replacement node joins and pods start, on top of the normal boot and image pull time. Pods with required spot-only affinity never fall back. They stay Pending, and the queue behind them stops moving.

Mass reclamation makes this worse. A single reclamation event can evict hundreds of pods. The scheduler then has to place all of them while the on-demand fallback is launching nodes, and other tenants are probably competing for the same on-demand pool at that moment. If a batch has a deadline, add an explicit escape hatch. One option is a second Job variant with on-demand affinity that a controller or an operator triggers after a set time-to-deadline. Another is preferred affinity in place of required affinity.

What to measure

Metric Why it matters
Interruptions per node group Shows which pools reclaim most often
Retries from node loss Separates spot cost from bugs
Rework time Compute lost since the last checkpoint
Queue latency during shortfalls Shows the cost of stalls
Cost per completed job, spot vs. on-demand The number that decides whether spot pays off

Illustrative estimate (assumed numbers, not a benchmark): cost per job = (node cost + rework cost + ops time cost) ÷ completed jobs. Assume:

  • Spot: $6,000 node cost + $900 rework + $1,300 ops time = $8,200 for 10,000 jobs, or $0.82 per job.
  • On-demand: $14,000 for 10,000 jobs, or $1.40 per job.

Count the extra moving parts as part of the risk. Termination handlers, extra node groups and fallback logic can each fail during the same incident they exist to handle.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spot instances kubernetes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.