What does an ephemeral preview environment actually cost?

A per-PR preview environment is a Kubernetes namespace created when a pull request opens. It runs some or all of the application stack built from that branch. Reviewers, QA and product can test a change against running services before merge without competing for a shared staging cluster.

Most of the cost comes from what the namespace reserves, not from what it does. The scheduler places pods according to their CPU and memory requests. The cluster autoscaler adds nodes when pending pods' requests don't fit on existing ones. Neither looks at actual usage. A preview pod that requests one core holds that core on a node all night, even if it sits near zero utilisation.

Component What drives the charge Billed while idle?
CPU and memory requests Node capacity reserved by requests Yes
Persistent volumes and snapshots Provisioned size per month Yes. With a Retain reclaim policy, volumes can outlive the namespace
Load balancers and public IPs Hourly charge per LoadBalancer Service or address Yes
Per-environment databases Instance or pod requests plus storage Yes
Egress and log volume Traffic, log ingestion and retention Mostly no. Scales with activity

Illustrative estimate (assumed numbers, not a benchmark): monthly compute = (cores × core-hour price + GiB × GiB-hour price) × hours alive × concurrent previews.

Assume each preview requests 2 cores and 4 GiB, at $0.03 per core-hour and $0.004 per GiB-hour, for 720 hours, with 20 previews open:

  • Hourly cost per preview: (2 × 0.03) + (4 × 0.004) = $0.076
  • Monthly cost per preview: $0.076 × 720 = $54.72
  • Total for 20 previews: $54.72 × 20 ≈ $1,094

Treat per-namespace figures derived from requests as directional, not exact billing. Bin-packing leaves unallocated headroom on nodes, and several namespaces share each node. Your real invoice will not split cleanly along namespace lines.

Why is idle time the biggest driver of preview environment cost?

Those caveats change the totals, but they don't change where the money goes. A simple model makes that clear.

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs:

  • 30 open PRs on average
  • Each environment requests 2 vCPU, with memory sized proportionally
  • An environment lives 120 hours, the PR's open lifetime
  • Reviewers actively use it for 6 of those hours
  • A node costs $0.16/hour for 4 allocatable vCPU, so $0.04 per requested vCPU-hour

Formula:

monthly cost = open PRs × vCPU per env × 730 h × (billable hours ÷ lifetime hours) × rate

Baseline: 30 × 2 × 730 × 1.0 × $0.04 = $1,752/month. Only 6 of 120 hours are active use, so about 95% of that spend is idle capacity.

Change Billable fraction Monthly cost Saving
Baseline 120/120 $1,752 —
Trim requests 25% (1.5 vCPU) 120/120 $1,314 25%
48-hour TTL, recreate on demand 48/120 $701 60%
Scale to zero outside 12 h of use 12/120 $175 90%

Trimming requests scales every hour down by the same percentage, so the idle share stays at 95%. Lifecycle controls remove idle hours, which is where most of the cost sits. For the same reason, the number of namespaces matters less than you might expect. Doubling the number of PRs doubles the bill, but leaving each environment running all week multiplies it by twenty.

The development cost depends on how you implement this. A TTL or scale-to-zero policy only saves money if bringing an environment back is fast and reliable. If reviewers wait minutes for a cold start or lose seeded data, you pay back the savings in engineer time.

How do per-PR namespaces work with sensible defaults?

The basic pattern is simple. When a pull request opens, CI creates a namespace named after the PR, such as pr-1423, and labels it with the PR number, repository, owner, and creation time. Those labels let a cleanup job, a cost report, or an engineer select every preview resource with one query, so they are worth making mandatory. CI then deploys only stateless services into the namespace, using the same manifests or chart as production with preview-specific values.

Routing goes through a single shared ingress controller. Each preview gets a hostname like pr-1423.preview.example.internal, or a path prefix if wildcard DNS and TLS are not available. Do not give each namespace a LoadBalancer Service. On most cloud providers that provisions a separate load balancer for every PR, which bills per hour whether or not anyone opens the preview. That cost also tends to outlive the namespace when teardown fails.

Every preview namespace should get a ResourceQuota and a LimitRange when it is created. The quota caps total CPU, memory, pod count, and persistent volume claims, so one misconfigured PR cannot use a node pool's capacity. The LimitRange sets small default requests and limits for containers that do not declare their own. Preview traffic is a few reviewers clicking around, not production load.

The trade-off is real. Low limits can cause OOM kills during startup or migrations, and tight requests make pods easy targets for eviction under node pressure. Small allocations also hide problems such as memory growth or CPU-bound endpoints. A passing preview shows the code works. It does not show the code behaves like production under production resources.

# Applied by CI right after it creates namespace pr-1423.
# Caps the total footprint of one PR's preview.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: preview-quota
  namespace: pr-1423
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.cpu: "4"
    limits.memory: 8Gi
    pods: "20"
    persistentvolumeclaims: "0"   # previews stay stateless
    services.loadbalancers: "0"   # traffic goes through the shared ingress
---
# Defaults for containers that omit requests/limits.
# Sized for a few reviewers, not production load.
apiVersion: v1
kind: LimitRange
metadata:
  name: preview-defaults
  namespace: pr-1423
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 50m
        memory: 128Mi
      default:            # default limits; low values risk OOM kills on startup or migrations
        cpu: 500m
        memory: 512Mi
      max:                # stops one container from claiming the whole quota
        cpu: "1"
        memory: 2Gi

A ResourceQuota and LimitRange for one PR namespace that cap total CPU, memory, and pods, block PVCs and LoadBalancer Services, and set small default requests and limits for preview containers.

How do you implement namespace TTLs and guaranteed teardown?

Each preview namespace should carry its own metadata, so cleanup never depends on a CI system's memory. Put the values you filter on in labels: a marker such as preview=true, the PR number, a sanitized repository name, and the owning team. Put timestamps in annotations, such as created-at and expires-at in RFC 3339 format. Annotations are the right place for timestamps because label values cannot contain colons and are limited to 63 characters. To extend an environment, a developer or bot updates the expires-at annotation. The sweeper only ever reads that value, so an extension is explicit and leaves an audit trail.

Two triggers, because one is not enough

The first trigger is the PR closed or merged event, which deletes the namespace right away. Webhooks get dropped, pipelines fail halfway, and someone eventually force-pushes a deleted branch. A scheduled sweep, run as a CronJob or a CI scheduled pipeline, catches whatever the event missed. It lists namespaces labeled preview=true and deletes any that are past expiry or whose PR is no longer open according to the Git host's API. If that API call fails, the sweeper should skip deletion for that run rather than treat "unknown" as "closed".

Limiting the blast radius

Namespaces are cluster-scoped, and RBAC cannot restrict deletion by label. A ClusterRole that allows namespace deletion therefore allows deletion of any namespace. Use these mitigations:

  • Give the sweeper a dedicated service account that has only namespace list, get, and delete permissions.
  • Have the sweeper re-check the preview label on each namespace immediately before deleting it.
  • Add an admission policy that rejects namespace deletes from that service account unless the target namespace carries the preview label.
# Deny namespace deletes made by the sweeper's service account
# unless the target namespace carries preview=true.
# Requires Kubernetes 1.30+ (ValidatingAdmissionPolicy GA).
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: preview-sweeper-guard
spec:
  failurePolicy: Fail            # if the policy can't be evaluated, block the delete
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["DELETE"]
        resources: ["namespaces"]
  matchConditions:
    # Apply only to the sweeper; other users are governed by their own RBAC
    - name: is-preview-sweeper
      expression: >-
        request.userInfo.username ==
        'system:serviceaccount:preview-system:preview-sweeper'
  validations:
    # On DELETE, 'object' is null; the namespace being removed is in 'oldObject'
    - expression: >-
        has(oldObject.metadata.labels) &&
        'preview' in oldObject.metadata.labels &&
        oldObject.metadata.labels['preview'] == 'true'
      message: "preview-sweeper may only delete namespaces labeled preview=true"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: preview-sweeper-guard
spec:
  policyName: preview-sweeper-guard
  validationActions: [Deny]      # enforce, not just audit or warn

A ValidatingAdmissionPolicy and its binding that reject namespace deletions from the sweeper's service account unless the target namespace is labeled preview=true, which limits the blast radius that RBAC alone cannot.

Why does Kubernetes cleanup leave namespaces stuck or resources billing?

A namespace delete is a request for the namespace controller to empty the namespace. The namespace stays in Terminating until every object inside is gone, and an object with finalizers is not removed until the controller responsible for each finalizer clears it. Common blockers in preview environments:

  • Custom resources whose operator is gone or broken. If the operator was uninstalled first, or its webhook is failing, nothing clears its finalizer.
  • PersistentVolumeClaims. The kubernetes.io/pvc-protection finalizer holds the claim while any pod still uses it.
  • LoadBalancer Services. The service.kubernetes.io/load-balancer-cleanup finalizer waits for the cloud controller to delete the external load balancer. Cloud API errors or missing permissions keep the Service alive.
  • An unavailable aggregated API. If an API service is down, the controller cannot list every resource type and stalls.

To diagnose, start with kubectl get namespace pr-123 -o jsonpath='{.status.conditions}'. The conditions NamespaceContentRemaining, NamespaceFinalizersRemaining and NamespaceDeletionDiscoveryFailure name the blocking resources or API groups. Then run kubectl api-resources --verbs=list --namespaced -o name and query each type in the namespace. Inspect .metadata.finalizers on what remains. Fix the owning controller rather than removing finalizers by hand. Stripping a finalizer skips the cleanup it was guarding, which is how cloud resources get orphaned.

Even a clean delete only removes Kubernetes objects. PersistentVolumes with a Retain reclaim policy, disk snapshots, DNS records, load balancers created outside a Service, and per-PR cloud databases keep billing. Tag every such resource with the PR number and repository when you create it. Then run a separate scheduled audit that lists resources by tag and deletes any whose PR is closed.

#!/usr/bin/env bash
# Diagnose a preview namespace stuck in Terminating.
# Usage: ./inspect-stuck-ns.sh pr-123
set -uo pipefail
NS="${1:-pr-123}"

echo "== Namespace conditions (which content, finalizers or API groups block deletion)"
kubectl get namespace "$NS" \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

echo "== Objects still in the namespace, with their finalizers"
# Walk every listable namespaced type, including custom resources.
# Types that fail to list usually point to an unavailable aggregated API.
kubectl api-resources --verbs=list --namespaced -o name | while read -r type; do
  out=$(kubectl get "$type" -n "$NS" --ignore-not-found \
    -o jsonpath='{range .items[*]}{.kind}/{.metadata.name}{"\t"}{.metadata.finalizers}{"\n"}{end}' 2>&1) \
    || { echo "cannot list $type: $out" >&2; continue; }
  [ -n "$out" ] && echo "$out"
done

echo "== PersistentVolumes that were bound to this namespace"
# Cluster-scoped PVs with reclaim policy Retain survive the namespace
# and their cloud disks keep billing until deleted separately.
kubectl get pv \
  -o jsonpath="{range .items[?(@.spec.claimRef.namespace==\"$NS\")]}{.metadata.name}{\"\t\"}{.spec.persistentVolumeReclaimPolicy}{\"\t\"}{.status.phase}{\"\n\"}{end}"

# Next step: fix the controller that owns each remaining finalizer
# (reinstall the operator, restore cloud permissions) instead of
# patching finalizers away, then audit cloud resources by PR tag.

A kubectl script that reads a stuck namespace's status conditions, lists every remaining object with its finalizers, and finds retained PersistentVolumes that will outlive the namespace.

When should you scale previews to zero, and when should you avoid it?

Teardown handles merged and abandoned PRs. Scale-to-zero handles open PRs that nobody is looking at. A scheduled job can set every preview Deployment to zero replicas outside working hours, or a request-based scaler can do it after a period with no traffic. The environment wakes when a reviewer opens the URL or when a new push triggers a redeploy.

This saves money only if nodes actually go away. Idle pods at zero replicas free capacity, but the bill drops only when the cluster autoscaler can drain and remove the underlying nodes. If previews share nodes with long-running services, freed capacity just sits there. A dedicated preview node pool, selected with node selectors and taints, lets that pool shrink to its minimum overnight. Because previews are disposable, that pool is also a good fit for spot or preemptible capacity. Label its nodes and every preview object with team and PR so cost attribution stays accurate.

The cost moves to people:

  • Sleeping previews: A reviewer who hits a sleeping or torn-down preview waits, retries, or gives up on the review.
  • Cold starts: Startup time plus reseeding test data can take minutes, and an engineer waits through each one.
  • Spot reclamation: When the provider reclaims a spot node mid-test, the preview flakes and CI reports a failure that wasn't caused by the code. Engineers then spend time chasing it.

Aggressive policies fit stateless services with fast startup and small seed data, and teams whose reviews happen asynchronously. Avoid them for stateful previews with expensive data setup, for environments that run end-to-end suites gating merges, and for previews that product or QA reviewers open on short notice.

# Weekday 20:00: scale every preview namespace to zero. A new push redeploys and wakes it.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: preview-sleep
  namespace: preview-system
spec:
  schedule: "0 20 * * 1-5"
  timeZone: "Europe/Berlin"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: preview-scaler  # RBAC: list namespaces, patch deployments/scale
          restartPolicy: OnFailure
          containers:
            - name: scale
              image: registry.example.com/tools/kubectl:pinned
              command: ["/bin/sh", "-c"]
              args:
                - |
                  for ns in $(kubectl get ns -l env=preview -o jsonpath='{.items[*].metadata.name}'); do
                    kubectl scale deployment --all --replicas=0 -n "$ns"
                  done
---
# Excerpt of a preview Deployment. Pin pods to the tainted preview pool so the
# autoscaler can empty and remove those nodes. Labels carry team and PR for cost attribution.
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: pr-1234, labels: {team: checkout, pr: "1234"}}
spec:
  template:
    metadata:
      labels: {app: web, team: checkout, pr: "1234"}
    spec:
      nodeSelector: {pool: preview}  # dedicated (e.g. spot) node pool
      tolerations:
        - {key: pool, operator: Equal, value: preview, effect: NoSchedule}

A CronJob that scales all preview namespaces to zero each weekday evening, plus a Deployment excerpt that pins preview pods to a dedicated, tainted node pool and carries team and PR labels, so freed nodes can be removed and their cost attributed.

What should previews share, and what are the alternatives to full per-PR isolation?

Not every preview needs its own copy of the whole system. Most of the cost sits in stateful components and in services the PR never touched, so the useful question is which parts to share.

Option Cost Isolation Main risk
Fully isolated per-PR stack Highest Complete Slow startup, idle spend
Stateless services per PR, shared or seeded DB Medium Partial Migrations and test data collide
Per-PR schema or logical database in one instance Low–medium Data-level Shared instance load, cleanup gaps
Shared message broker Low Weak Consumers steal each other's messages unless queues or topics are namespaced
Stubbed third-party integrations Low n/a Stubs drift from real API behaviour
Partial preview with header-based routing Lowest Changed services only Routing complexity, harder debugging

Sharing reintroduces the coupling that previews exist to remove. A destructive migration in one PR can break every other preview using the same database, and two PRs seeding the same fixture IDs will overwrite each other's data. Per-PR schemas avoid most of this, as long as teardown drops the schema too.

Partial previews deploy only the changed services and send all other traffic to a shared baseline. This depends on every hop propagating the routing header, including async consumers and background jobs. When a request lands on the wrong version, the cause is hard to find.

Use full isolation for PRs that change schemas, cross-service contracts, or event formats. For UI-only and single-service changes, a partial preview or a per-PR schema costs much less and catches nearly the same defects.

How do you make preview costs visible to save engineering time?

Previews stay alive because nobody can see what they cost. The fix is to report that cost where the decision gets made: the pull request. A bot comment that updates on each deploy with the environment's age, its TTL expiry, and an estimated running cost lets the author decide whether to keep it. Asking people to go check a dashboard does not work as well.

To calculate that estimate, label every preview namespace with team, repository, and pr. Schedule previews onto a dedicated node pool using a taint and toleration. The labels let you attribute requested CPU and memory to individual PRs. The node pool gives you a separate line on the cloud bill, so you can compare the per-PR totals against what you actually paid.

Illustrative estimate (assumed numbers, not a benchmark): cost = (vCPU requested × price per vCPU-hour + GiB requested × price per GiB-hour) × hours alive. With 2 vCPU at $0.04, 4 GiB at $0.005, and 120 hours: (0.08 + 0.02) × 120 = $12.

Treat these figures as directional. They are based on requests, not actual usage. They also leave out shared overhead such as the ingress, logging, and idle node headroom. Their purpose is to help engineers compare previews, not to reconcile invoices.

Set up two alerts:

  • A namespace has outlived its TTL.
  • A namespace carries a pr label that does not match any open PR.

Both alerts point at a failure in teardown, not at a single forgotten preview.

This visibility saves engineering time in three ways:

  • Fewer manual cleanup sweeps.
  • Fewer debugging sessions spent on environments running week-old code.
  • Faster decisions about which previews deserve to keep running.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to ephemeral preview environments cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.