Deployment Rollback Strategy for Code, Config and Database

What a deployment rollback has to cover: code, config and data

A deployment rollback is three separate problems that happen to share a name. Each part of a release has its own lifecycle, its own storage and its own failure modes. A strategy that only handles one of them will fail partway through an incident. This guide is for backend and platform engineers who own deploy pipelines, database schemas or runtime configuration.

Code is the easy case. A built artifact, such as a container image or a versioned package, is immutable. Rolling back means redeploying the previous one. Config is harder because it often skips the release pipeline. Feature flags, environment variables and service settings get edited in a console or a separate repository, with no artifact version tying them to the code that reads them. Data is the hardest. A dropped column, a backfill that overwrote values, or rows written in a new format by the new release cannot be undone by redeploying anything.

Rollback has to be designed before you ship. During an incident you cannot make an old binary understand a new schema, and you cannot recover a config value that nobody saved.

How rollbacks work when two versions share one database

During a rolling or canary deploy there is no single "current version". For minutes or hours, old and new instances read and write the same database rows, cache keys and queue topics. A rollback goes back to that mixed state. Every change therefore has to work in both directions: old code must survive data written by new code, and new code must survive data written by old code.

The usual failures come from data the old code has never seen:

• New enum values. New code writes status = 'partially_refunded'. Old code hits an exhaustive switch or a strict deserializer and throws.
• Changed serialization formats. A field moves from a string to a structured object, or a timestamp changes from epoch seconds to milliseconds. The old parser fails or silently misreads the value.
• New message schemas. Events published by new producers sit in a queue after the rollback. Old consumers keep receiving them and keep failing on them.
• Cache entries. New code caches an object in a new shape under an existing key. After the rollback, old code reads that shape until the entry expires.

The defence is the tolerant reader. Ignore unknown fields. Map unknown enum values to an explicit fallback instead of crashing. Put a version number in serialized payloads and messages so a reader can recognise data it doesn't understand and skip it, route it elsewhere or degrade gracefully. A version in the cache key also helps: old and new code then use separate entries instead of overwriting each other's.

Blue-green deployment doesn't fix any of this. It swaps the stateless tier, but if both environments point at one database, the data blue reads after a switch-back is whatever green has already written.

Copy// Tolerant reader for order events consumed by both old and new code.
const KNOWN_STATUSES = ['pending', 'paid', 'refunded'] as const;
type Status = (typeof KNOWN_STATUSES)[number] | 'unknown';

// Unknown enum values (e.g. 'partially_refunded' from newer code)
// map to an explicit fallback instead of throwing.
function parseStatus(raw: unknown): Status {
  return typeof raw === 'string' && (KNOWN_STATUSES as readonly string[]).includes(raw)
    ? (raw as Status)
    : 'unknown';
}

interface Order { id: string; status: Status; amountCents: number }

// Highest payload version this build understands.
const MAX_SUPPORTED_VERSION = 1;

function readOrderEvent(json: string): Order | null {
  const msg = JSON.parse(json) as { v?: number; data?: Record<string, unknown> };
  // Payloads written before versioning was added count as v1.
  const version = msg.v ?? 1;
  // Data from a newer producer: skip it (or send it to a parking queue)
  // rather than misread a format this build has never seen.
  if (version > MAX_SUPPORTED_VERSION || !msg.data) return null;

  const d = msg.data;
  // Copy only the fields we know. Extra fields added by new code are ignored.
  return {
    id: String(d.id),
    status: parseStatus(d.status),
    amountCents: Number(d.amountCents),
  };
}

// Schema version in the cache key: old and new builds keep separate
// entries, so a rollback never reads an object cached in the new shape.
const CACHE_SCHEMA = 'v1';
const orderCacheKey = (id: string) => `order:${CACHE_SCHEMA}:${id}`;

A TypeScript tolerant reader that maps unknown enum values to a fallback, ignores unknown fields, skips payloads with a newer version, and puts a schema version in cache keys so old and new code can run against the same data.

Rolling back code and config

Code: redeploy the artifact, don't rebuild it

A code rollback should put back the exact bytes that ran before. Rebuilding an old commit doesn't guarantee that, because base images, dependency resolution and build tooling may have changed since then. Deploy the previous artifact by image digest or build ID, not by a mutable tag such as latest. Set registry retention so the last several production artifacts can't be garbage-collected. Make "deploy N-1" a named pipeline action that runs the same health checks as a forward deploy, and use it often enough that people trust it during an incident.

Config: versioned, diffed, observable

Keep config in version control and ship it through a pipeline that shows the diff before it is applied. That gives you an audit trail and turns a config rollback into a revert. Each instance should expose which config version it has loaded, for example through a health endpoint or a startup log line. Without that, you can't tell whether a rollback has taken effect. You also need to know which settings are read on every request and which are read only at startup, since the startup-only ones won't change until the process restarts.

Feature flags as a kill switch

Flags separate deploying code from releasing behavior. Turning a flag off disables a bad code path in seconds and needs no deploy. Every flag needs an owner, a default that is safe when the flag service can't be reached, and a removal date. Old flags pile up and create code paths nobody has tested.

Keep code and config paired

Rolling back code while new config stays in place can break in either direction. Old code may not recognize a new key, or it may read a renamed setting as missing and fall back to a default. Record which config version each artifact was tested with, and roll the two back together.

Copy#!/usr/bin/env bash
# Roll back to the previous known-good release (N-1) by digest. Never rebuild an old commit.
set -euo pipefail

NAMESPACE="prod"
DEPLOYMENT="checkout"
CONTAINER="checkout"
HEALTH_URL=""
# Appended by the forward-deploy pipeline, newest last. Each line is "<image@digest> <config_version>",
# so code and config are always rolled back as the pair that was tested together.
RELEASES="releases.log"

# N-1 is the second-to-last recorded release.
read -r IMAGE CONFIG_VERSION < <(tail -n 2 "$RELEASES" | head -n 1)

# Refuse mutable tags like :latest. Only a digest guarantees the same bytes that ran before.
if [[ "$IMAGE" != *@sha256:* ]]; then
  echo "refusing: $IMAGE is not pinned by digest" >&2
  exit 1
fi

echo "Rolling back $DEPLOYMENT to $IMAGE with config $CONFIG_VERSION"

# Restore the paired config first. Startup-only settings take effect when the
# image change below restarts the pods.
kubectl -n "$NAMESPACE" apply -f "config/${DEPLOYMENT}-${CONFIG_VERSION}.yaml"

kubectl -n "$NAMESPACE" set image "deployment/$DEPLOYMENT" "$CONTAINER=$IMAGE"
# Same gate as a forward deploy: fail the rollback if pods don't become ready.
kubectl -n "$NAMESPACE" rollout status "deployment/$DEPLOYMENT" --timeout=5m

# Verify that instances report the config version we expect. Assumes the service
# exposes it on its health endpoint as {"config_version": "..."}.
LOADED=$(curl -fsS "$HEALTH_URL" | jq -r '.config_version')
if [[ "$LOADED" != "$CONFIG_VERSION" ]]; then
  echo "config mismatch: instance reports $LOADED, expected $CONFIG_VERSION" >&2
  exit 1
fi
echo "Rollback complete: $IMAGE running with config $LOADED"

A 'deploy N-1' script that redeploys the previous artifact by image digest together with the config version it was tested with, waits on the same rollout health gate as a forward deploy, and checks the health endpoint to confirm which config version is loaded.

Backward compatible migrations with expand/contract

Schema changes need the opposite discipline: each step must leave the database usable by both the current and the previous release. Expand/contract does this by splitting one breaking change into several compatible ones.

1. Expand. Add new columns as nullable, or add new tables. Old code ignores them.
2. Dual write. Deploy code that writes both the old and new shapes but still reads the old one.
3. Backfill. Copy historical rows in small, resumable batches keyed on primary key, so a failure partway through restarts from a checkpoint rather than from zero.
4. Switch reads. Move reads to the new shape and keep dual writes on.
5. Contract. Drop the old structure in a later release, once you no longer need to roll back to code that depends on it.

With this sequence, a schema rollback means redeploying the previous artifact against the expanded schema, which it still understands. Down migrations are a poor substitute. Teams rarely test them. They can delete data written after the deploy, such as values that exist only in a new column. They can also lock large tables at the moment you most need the database responsive. Fixing forward avoids all three problems.

Dual writes and backfills drift. A write path can miss a code branch, or a batch can fail silently. Before switching reads, run a reconciliation job that compares row counts and per-row checksums between the old and new shapes, and repair any mismatches. Keep running it until the contract step, because a rollback re-enables only the old write path, and the new shape falls behind again.

Copy-- PostgreSQL. Example: replacing orders.amount (numeric) with orders.amount_cents (bigint).

-- 1. Expand: add a nullable column. The previous release ignores it.
ALTER TABLE orders ADD COLUMN amount_cents bigint;

-- 2. Dual write happens in application code (it writes amount and amount_cents).

-- 3. Backfill one batch keyed on primary key. The job stores the last id it
--    processed as a checkpoint, so a failed run restarts from that id instead of 0.
--    Replace 400000 with the stored checkpoint value.
UPDATE orders
SET    amount_cents = round(amount * 100)::bigint
WHERE  id > 400000 AND id <= 400000 + 5000
  AND  amount_cents IS NULL;

-- 4. Reconcile before switching reads, and again after any rollback.
--    IS DISTINCT FROM treats NULLs as values, so it catches rows that
--    the backfill missed and rows where a dual-write code path disagrees.
SELECT count(*)                                              AS total_rows,
       count(*) FILTER (WHERE amount_cents IS NULL
                          AND amount IS NOT NULL)            AS missing,
       count(*) FILTER (WHERE amount_cents IS DISTINCT FROM
                          round(amount * 100)::bigint)       AS drifted
FROM   orders;

-- 5. Repair drift in bounded id ranges, so no single statement holds row locks
--    on the whole table. The old column stays the source of truth
--    until reads switch over.
UPDATE orders
SET    amount_cents = round(amount * 100)::bigint
WHERE  id > 0 AND id <= 50000
  AND  amount_cents IS DISTINCT FROM round(amount * 100)::bigint;

-- 6. Contract happens in a later release, once no deployable artifact reads amount:
-- ALTER TABLE orders DROP COLUMN amount;

A PostgreSQL expand/contract sequence that adds a nullable column, backfills it in resumable primary-key batches, and uses a reconciliation query to find and repair rows where dual writes or the backfill drifted, before the old column is dropped in a later release.

When to roll back, when to fix forward and when to avoid relying on rollback

The decision to roll back should be made before the incident. Write down the health signals that trigger it: error rate, latency at the percentiles your users feel, saturation (CPU, connection pools, queue depth) and at least one business metric, such as completed checkouts. Business metrics catch bugs that return a clean 200 with wrong results. Name who can trigger a rollback without asking anyone. The on-call engineer should not need a manager's approval at 3 a.m. Where you run canaries, let the pipeline abort automatically when the canary's signals diverge from the baseline.

Fix-forward is often the wrong choice. If your pipeline takes 40 minutes from merge to production, every fix attempt costs that long while users sit in a degraded state. Subtle bugs are worse, because the first fix is often wrong. A tested down migration and a known-good build usually get you out sooner.

What rollback cannot undo

Some effects escape the deploy boundary:

• Emails, SMS and push notifications already sent
• Payments already charged
• Data deleted or transformed in place
• Calls to external APIs that changed state elsewhere

These need compensating designs, such as idempotency keys, refund paths and soft deletes. The alternative is a gated release behind a flag, so the change reaches only a small slice of traffic first.

Automated rollback has its own failure modes. Noisy signals and normal traffic shifts can trigger false aborts. A rollback that happens to coincide with recovery can also hide the real cause, such as a failing upstream dependency, which will still be broken on the next deploy.

Alternatives and complementary techniques

No single technique covers code, config and data, so most teams combine several.

Feature flags deserve extra caution. Ten independent boolean flags produce 1,024 possible states, and nobody tests all of them. A kill switch can therefore move the system into a state it has never run in before. Keep flags short-lived, and test the off path explicitly.

The flag service also becomes a critical dependency. If it is unreachable, every client falls back to its default value. Choose defaults that are safe, and cache the last known values locally.

Rehearsing rollbacks and the impact on cost and engineering time

A rollback path that only runs during incidents is untested code. Make "deploy N, then roll back to N-1" a standard pipeline stage in staging. Apply N's migration, run smoke tests, redeploy N-1's artifact and config against the migrated schema, and run the same tests again. If N-1 fails against the new schema, the migration was not backward compatible and the release stops there. The rollback path then gets exercised on every release, as often as the forward path.

The costs are real. Expand/contract turns one schema change into three or four releases spread over days or weeks. It leaves dual-write code and unused columns behind, and someone has to schedule their removal. Backfills and dual writes add load to the database. Retaining old artifacts and running a parallel staging environment also costs storage and compute.

The return comes during incidents. When N-1 is known to run against the current schema, nobody has to debate whether a rollback is safe. That decision was made in the pipeline. Nobody writes data repair scripts under pressure either.

Illustrative estimate (assumed numbers, not a benchmark). Annual saving = incidents per year × hours saved per incident × engineers involved.

Assume:

• 6 incidents per year
• 2 hours saved per incident
• 4 engineers involved

That gives 6 × 2 × 4 = 48 engineer-hours saved per year. Compare that figure with your own extra release work and cleanup hours to see whether the trade pays off for your team.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to deployment rollback, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.