How does bin packing work in Kubernetes?
The Kubernetes scheduler places a pod by comparing its resource requests against each node's allocatable capacity, minus the requests of pods already running there. Actual CPU and memory usage plays no part in that decision. A pod that requests 2 CPU but uses 200m still reserves 2 CPU, and no other pod can be scheduled into that space. A pod with no requests counts as zero, so the scheduler can stack it onto a node that is already busy.
This is why request accuracy decides packing quality before any autoscaler runs. Autoscalers provision and remove nodes based on what the scheduler can fit. If requests are inflated, nodes look full while sitting mostly idle. If requests are missing, nodes look empty while they are contended. Neither Cluster Autoscaler nor Karpenter can pack better than the requests allow.
Note also that the scheduler's default scoring spreads pods toward less-allocated nodes instead of filling nodes tightly. Dense packing over time therefore depends on removing nodes, not only on placing pods well.
What node consolidation means
Node consolidation is the process of removing nodes that are empty or underused, or replacing them with cheaper ones, and rescheduling their pods onto the remaining capacity. Teams use it to cut node spend because clusters drift. Deployments scale down, batch jobs finish, and the leftover pods end up spread across more nodes than they need.
This guide treats consolidation as one system with three parts: the consolidation policy, disruption budgets, and pod-level constraints such as PodDisruptionBudgets and affinity rules. Each part limits what the others can do.
What is the difference between Cluster Autoscaler and Karpenter?
Cluster Autoscaler works through node groups. Each group is a fixed template, typically one instance type or a small set of equivalent ones. When pods are pending, it simulates which group could fit them and increases that group's size. Scale-down runs separately. A node is a candidate when the requests of its pods fall below a utilization threshold and those pods can be rescheduled elsewhere. Its bin packing is therefore limited by the shapes you defined ahead of time. It can remove an underused node, but it cannot swap a large node for a smaller or cheaper one.
Karpenter skips node groups. It reads the requirements of pending pods (requests, node selectors, affinities, tolerations, topology spread), chooses an instance type from whatever the NodePool allows, and launches it directly. Consolidation then reviews running nodes and either deletes them or replaces them with a cheaper option that still fits their pods. Because it can change node shape after launch, the cluster's packing can improve over time instead of only shrinking.
| Aspect | Cluster Autoscaler | Karpenter |
|---|---|---|
| Capacity unit | Predefined node groups | Instance types chosen per launch |
| Scale-down | Removes nodes below a utilization threshold | Deletes or replaces nodes with cheaper ones |
| Instance flexibility | Fixed by group configuration | Broad, constrained by NodePool requirements |
| Tuning effort | Mostly at the node-group level | Mostly at the NodePool level |
Staying on Cluster Autoscaler is reasonable in several cases. Your workloads may be uniform and fit a few instance types well. Committed-use discounts may tie you to specific types. Your cloud provider may lack mature Karpenter support. In a small, stable cluster, periodically right-sizing node group instance types by hand may capture most of the savings.
Karpenter gives the largest benefit to teams that run varied workloads and can accept many instance types. That flexibility is what consolidation needs in order to find cheaper placements.
What are the alternatives to Karpenter consolidation?
Karpenter is not the only way to improve bin packing. The main alternatives work at different layers, and several can be combined.
- Cluster Autoscaler with tuned node groups. Scale-down removes underused nodes once their pods fit elsewhere. You can lower the utilization threshold or define node groups in more sizes so that scale-up picks a better-fitting shape. It cannot replace a running node with a cheaper one, so the shape of the fleet only changes when you change the groups.
- Manual right-sizing of node groups. Periodically review request totals per group and move groups to smaller or cheaper instance types. This is low-risk and predictable, and in a small, stable cluster it may be enough. It does not react to day-to-day drift.
- Scheduler scoring for tighter packing. The kube-scheduler's
NodeResourcesFitplugin supports aMostAllocatedscoring strategy, which prefers nodes that are already more allocated instead of spreading pods out. New pods then land on busy nodes, which leaves other nodes empty and easier for any autoscaler to remove. It only affects new placements and does not move running pods. - The descheduler. The Kubernetes descheduler project evicts running pods according to policies, such as pods on underused nodes, so the scheduler can place them again. Paired with
MostAllocatedscoring and an autoscaler that removes empty nodes, it approximates consolidation. It needs careful policy configuration and still respects PodDisruptionBudgets. - Request right-sizing. Tools such as the Vertical Pod Autoscaler, in recommendation mode or applied automatically, lower inflated requests. This is not consolidation itself, but because every packer works on requests, it often recovers capacity on its own and makes any of the options above more effective.
The trade-off across these options is control versus automation. Manual and node-group approaches keep the fleet predictable. Karpenter consolidation and descheduler-based setups react continuously, but they evict pods more often and need the disruption controls described below.
How does Karpenter consolidation work?
Consolidation tries to find a cheaper shape for the same work. Karpenter continuously simulates whether the current set of nodes could run its pods more cheaply. For each candidate node, or group of nodes, it checks whether the pods on it could be scheduled onto other existing nodes or onto a cheaper set of new nodes. It uses the same constraints the scheduler would apply: requests, selectors, affinities, tolerations, topology spread and PodDisruptionBudgets. If the simulation finds a valid and cheaper result, it acts in one of two ways.
Deletion. If every pod on a node can be rescheduled onto spare capacity in other existing nodes, Karpenter drains the node and terminates it. Nothing new is launched. Empty nodes are the simplest case.
Replacement. If the pods cannot fit elsewhere as-is, Karpenter checks whether one or more nodes could be swapped for fewer or cheaper nodes. It launches the replacement first, waits for it to become ready, then drains the old nodes. Pod disruption budgets and the karpenter.sh/do-not-disrupt annotation are respected throughout.
| Before | After | |
|---|---|---|
| Node A (large, 30% used): pods 1, 2 | → | Node C (medium, cheaper): pods 1, 2, 3, 4 |
| Node B (large, 25% used): pods 3, 4 | → | Nodes A and B terminated |
Replacement only finds savings if the NodePool allows instance types that fit the leftover pods. If only one large size is permitted, the cheapest replacement for two half-empty nodes is still one large node, and sometimes no cheaper option exists at all. A list that spans several sizes and families lets the scheduler match capacity closely to actual requests.
The trade-off is heterogeneity. A very broad list produces a fleet of mixed CPU generations, architectures, and network or storage characteristics. Latency can then vary between replicas of the same service, and debugging gets harder because "the node" is no longer a known quantity. A common compromise is to allow several sizes within a few families you have tested, rather than everything the cloud provider offers.
BEFORE AFTER
+----------------------+
| Node A (large) |
| 30% used |
| [pod 1] [pod 2] |--+
+----------------------+ | +---------------------------+
+-----> | Node C (medium, cheaper) |
+----------------------+ | | [pod 1] [pod 2] |
| Node B (large) | | | [pod 3] [pod 4] |
| 25% used |--+ +---------------------------+
| [pod 3] [pod 4] | ^
+----------------------+ |
|
+-----------------------------------------------+------------+
| NodePool allowed instance types |
| [small] [medium] [large] ... within a few tested families |
| Replacement chooses the cheapest type that fits pods 1-4 |
+------------------------------------------------------------+
Order of operations:
1. Launch Node C
2. Wait until Node C is Ready
3. Drain Nodes A and B (respects PDBs and do-not-disrupt)
4. Terminate Nodes A and B
Replacement consolidation: Karpenter moves the pods from two underused large nodes onto one cheaper medium node chosen from the NodePool's allowed instance types. It launches the new node and waits for it to be ready before it drains and terminates the old ones.
How do you configure consolidation policy and disruption budgets on a NodePool?
Once the instance families are settled, the NodePool's disruption settings decide how aggressively Karpenter repacks onto them. The NodePool example that accompanies this section uses one release's field names. Consolidation policy values, the delay field and budget syntax have been renamed or changed meaning between Karpenter releases, so check every field and default against the documentation for the version you run.
| Setting | Empty-only | Empty and underutilized |
|---|---|---|
| What gets removed | Nodes with no non-daemon pods | Empty nodes, plus nodes whose pods fit elsewhere or on a cheaper node |
| Pod evictions | None | Yes, on every replacement |
| Best fit | Stateful or restart-sensitive services | Stateless, horizontally scaled workloads |
Consolidation delay. The delay sets how long a node must stay a candidate before Karpenter acts on it. Bursty workloads need a longer delay. Without it, Karpenter removes a node after traffic dips, then launches another when the next burst arrives, and you pay for boot time and cold caches twice.
Disruption budgets. NodePool budgets cap how many nodes, as a count or a percentage, can be disrupted at once. They can also restrict disruption to scheduled time windows, for example blocking it during business hours. These budgets sit above application PodDisruptionBudgets rather than replacing them. Karpenter honours PDBs during eviction, so a PDB with zero allowed disruptions stalls the drain and leaves the node running. That node keeps costing money until you fix the PDB.
Spot and on-demand. Consolidation can move pods from on-demand nodes to cheaper spot nodes. That trades lower cost for a higher chance of interruption. Interruption handling is a separate mechanism, and it only works if you configure the interruption queue. Spot-to-spot replacement has been feature-gated in some releases, so verify how your version behaves before you rely on it.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general-stateless
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
# Allow both capacity types. Consolidation may move pods onto spot.
# Spot-to-spot replacement is controlled separately (a feature gate in
# some releases). Verify its behaviour for your version.
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
disruption:
# Use WhenEmpty for restart-sensitive workloads. This value also repacks
# underutilized nodes, which evicts pods.
consolidationPolicy: WhenEmptyOrUnderutilized
# A node must stay a candidate this long before Karpenter acts. Raise it for
# bursty traffic so nodes are not removed between bursts.
consolidateAfter: 10m
budgets:
# Disrupt at most 10% of this pool's nodes at once.
- nodes: "10%"
# No voluntary disruption on weekday business hours (cron is in UTC).
- nodes: "0"
schedule: "0 9 * * mon-fri"
duration: 8h
A Karpenter v1 NodePool that mixes spot and on-demand capacity, consolidates empty and underutilized nodes after a 10-minute delay, and uses disruption budgets to cap concurrent disruptions at 10% and block them during weekday business hours.
Why isn't Karpenter consolidating my nodes?
Before tuning consolidateAfter or budgets further, check whether individual pods are vetoing the move. Karpenter only consolidates a node when every pod on it can be evicted and rescheduled somewhere else. One pinning pod is enough to keep an otherwise idle node alive.
| Constraint | Why it blocks consolidation |
|---|---|
| PDB with zero allowed disruptions | Eviction would violate the budget, so Karpenter skips the node |
karpenter.sh/do-not-disrupt annotation |
Explicit opt-out, honored for voluntary disruption |
| Pod without a controller | Nothing would recreate it after eviction |
| Local storage (local PVs, node-bound volumes) | Data cannot follow the pod to a new node |
| Required affinity or anti-affinity | No other node satisfies the rule |
| Topology spread constraints | Removing the node would break the skew limit |
Topology spread deserves extra attention. A maxSkew of 1 across three zones means a small workload still needs a node in each zone, even when one node could hold all the pods by CPU and memory. That is why the resource math says two nodes while the cluster keeps five.
To find the culprit, start with kubectl get events -A --field-selector involvedObject.kind=Node. Karpenter records events that state why disruption was blocked. Run kubectl describe node <name> to list the pods on a stubborn node. Then read kubectl logs -n <karpenter-namespace> deployment/karpenter for the consolidation decisions themselves.
This order, starting with events, then nodes, then logs, usually gives you a named pod and a specific reason within minutes. Without it, a cost investigation tends to turn into a long debate about instance types.
#!/usr/bin/env bash
set -euo pipefail
NODE="${1:?usage: $0 <node-name>}"
KARPENTER_NS="${KARPENTER_NS:-kube-system}" # set to wherever Karpenter runs
kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name="$NODE" \
| grep -i disrupt || echo "no disruption events for $NODE"
echo "--- pods on $NODE (namespace/name, owner kind, do-not-disrupt) ---"
kubectl get pods -A --field-selector spec.nodeName="$NODE" -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.metadata.ownerReferences[0].kind}{"\t"}{.metadata.annotations.karpenter\.sh/do-not-disrupt}{"\n"}{end}'
echo "--- PDBs with 0 allowed disruptions ---"
kubectl get pdb -A \
-o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed \
| awk 'NR==1 || $3==0'
kubectl describe node "$NODE"
kubectl logs -n "$KARPENTER_NS" deployment/karpenter --since=1h \
| grep -i -E 'consolidat|disrupt' | tail -n 50
A bash script that follows the events, then node, then logs order to name the pod blocking consolidation on one node, flagging controller-less pods, do-not-disrupt annotations and PDBs with zero allowed disruptions.
How much can Karpenter consolidation save on node costs?
Consolidation packs pods by their requests, not their actual usage. If requests are inflated, Karpenter will faithfully pack padding. Right-sizing therefore comes first. Compare each workload's requests against observed CPU and memory usage from the metrics you already collect, looking at peaks as well as averages. Lower requests with care. CPU set below real demand causes throttling under a limit, memory set too low causes OOM kills, and tight packing makes noisy neighbors more likely.
Illustrative estimate (assumed numbers, not a benchmark)
Formula: nodes after ≈ nodes before × (utilization before ÷ utilization after) Monthly savings = (nodes before − nodes after) × hourly price × 730
| Input | Before | After |
|---|---|---|
| Node count | 40 (assumed) | 40 × 0.45 ÷ 0.75 = 24 |
| Request utilization per node | 45% (assumed) | 75% (assumed target) |
| Per-node hourly price | $0.40 (assumed) | $0.40 |
| Monthly node cost | 40 × 0.40 × 730 = $11,680 | 24 × 0.40 × 730 = $7,008 |
In this example, removing 16 nodes saves about $4,672 per month. Replace these inputs with your own values. The result assumes pods can be rescheduled freely and that the remaining instance types are priced the same.
The savings come with ongoing engineering cost. Someone has to audit requests, set disruption budgets, and investigate evictions and latency regressions. Weigh that time against the recurring reduction. A few days of tuning pays off quickly on a large fleet. On a ten-node cluster, it may never pay off.
How can Karpenter consolidation save engineering time?
Consolidation costs engineering time to set up, but once it is tuned it removes several recurring manual tasks.
- Less node-group maintenance. With Cluster Autoscaler, every new workload shape can mean a new node group, or a decision about which existing group to stretch. A NodePool with a list of allowed instance types covers many shapes at once, so engineers edit requirements instead of maintaining many fixed templates.
- No manual repacking. Without consolidation, recovering capacity after a scale-down usually means someone spots underused nodes, cordons them, drains them and checks that the pods landed safely. Karpenter does the simulation, the drain ordering and the termination itself, and it respects PDBs while doing so.
- Fewer periodic right-sizing reviews of the fleet. Replacement moves pods onto cheaper shapes as the workload mix changes. Engineers still need to review pod requests, but they no longer need to keep deciding which instance types the node groups should use.
- Disruption policy written down once. Budgets and schedules encode rules such as "no voluntary disruption during business hours" in the NodePool. That replaces informal agreements about when node maintenance is allowed, and it is reviewed like any other configuration change.
- Faster cost investigations. When a node will not go away, Karpenter's events and logs name the blocking pod and the reason, as described in the troubleshooting section. That is quicker than reconstructing scheduling decisions by hand.
Illustrative estimate (assumed numbers, not a measurement)
Assume a team spends 4 hours a month reviewing underused nodes and draining them by hand, and 2 hours a month adjusting node groups for new workloads. That is 6 hours a month, or about 72 hours a year. Assume initial Karpenter tuning takes 24 hours and ongoing review of disruption events and budgets takes 2 hours a month, or 24 hours a year. With these assumed inputs, the first year saves about 72 − 24 − 24 = 24 hours, and later years save about 48 hours. Replace the assumed hours with your own. If your cluster rarely changes, the manual work may already be close to zero and there is little time to save.
The time saved only holds if the setup is stable. A NodePool that evicts latency-sensitive pods too often turns saved maintenance hours into incident hours, which is why the rollout below is staged.
When should you avoid aggressive consolidation, and how should you roll it out?
On small clusters the cost reduction may never cover the hours spent tuning, which is a hint that consolidation trades stability for spend. Some workloads pay more for that trade than their nodes cost.
Where to limit consolidation
- Latency-sensitive services. Every consolidation reschedules pods. The drain and the cold start on the new node show up as p99 spikes, even with healthy replica counts.
- Long warm-up workloads. JIT-compiled runtimes, model loading and cache priming can take minutes. Each move removes that capacity until the replacement pod is warm.
- Connection-heavy workloads. WebSocket servers, long-lived gRPC streams and connection poolers force every client to reconnect on eviction. Frequent moves cause reconnect storms.
- Spot-heavy NodePools. Spot interruptions and consolidation stack on each other. Their combined disruption rate can exceed what your replica counts and PDBs were sized for.
- Thin headroom. Consolidation removes the slack that absorbs bursts. The next spike then waits for a node to provision and pull images. If that scale-up latency is unacceptable, keep some buffer.
Put these workloads on their own NodePool with empty-only consolidation or a tight budget.
The opposite failure is quiet. A PDB that allows zero disruptions, or a karpenter.sh/do-not-disrupt annotation, makes Karpenter skip the node entirely. A large node can sit there indefinitely, running two small pods. Audit these settings periodically.
Staged rollout
- Enable empty-node consolidation on a non-critical NodePool.
- Track three signals per service: Karpenter disruption events, pod restart rates and request latency.
- Switch to underutilized consolidation. Start with a budget of one node at a time and a conservative
consolidateAfter. - Widen the budget and extend the policy to more NodePools only after a stable observation period.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to karpenter consolidation cost savings, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.