CI/CD Pipeline Design: Ordering Stages, Gates and Rollbacks

What CI/CD pipeline design is and who needs which kind

Pipeline design is risk management. Each stage and gate exists to catch one class of failure: a compile error, a broken contract, a bad migration, a latency regression. The design goal is to catch each one at the earliest and cheapest point. A pipeline that finds a type error in ten seconds on a laptop costs far less than one that finds it during a production canary. Before you add a stage, name the failure it catches. If you can't, it probably shouldn't be there.

The three practices are often blurred together, but each one makes a different promise:

Choosing per service

Continuous deployment depends on trusting automated signals. A high-traffic API produces enough requests to show a regression within minutes. A low-traffic internal service may not, so a bad build can sit unnoticed for hours. Platform teams that ship shared infrastructure often keep a manual release step, because one bad change affects every team that depends on them. Regulated systems may need a recorded approval that automation can't stand in for.

Make the choice for each service on purpose. If you copy one template across the organisation, the low-risk services get too many gates and the high-risk ones get too few.

How a pipeline works: ordering stages by cost and feedback speed

Order the stages by how fast and how cheaply they can reject a bad change. Lint, compilation and unit tests run first. They need no infrastructure and finish quickly, and they catch most mistakes. Integration and contract tests come next because they need databases, queues or stubbed dependencies. End-to-end tests run last, since they are the slowest and fail most often for reasons unrelated to the change. With this order, a typo fails the build in minutes, not after a long end-to-end run, and the expensive stages only spend time on changes that are already likely to be correct.

Build once, promote everywhere

The build stage produces one immutable artifact, such as a container image identified by its digest or a package tagged with the commit SHA. That exact artifact then moves through every environment. Each environment changes only the configuration it injects. The artifact itself stays the same. If you rebuild for each environment, base images, unpinned dependencies or build flags can drift between builds, so production ends up running something your tests never saw. A digest also tells you exactly what to roll back to.

The flow from commit to production

1. Commit → fast checks → build and record the digest
2. Integration and contract tests against that digest
3. Gate: deploy to staging and run end-to-end tests
4. Gate: promotion criteria met, or approval given
5. Progressive rollout to production

The rollback path starts at step 5. The previous known-good digest stays recorded and deployable, so rolling back means redeploying that artifact. You never need to rebuild anything to roll back.

Copy commit
   |
   v
+------------------------------+
| Fast checks: lint, compile,  |   cheapest and fastest; most failures
| unit tests (no infra)        |   stop here within minutes
+------------------------------+
   |
   v
+------------------------------+      +--------------------------+
| Build once -> one immutable  |----->| Artifact registry        |
| artifact, record digest      |      | sha256:<new>             |
+------------------------------+      | sha256:<prev known-good> |
   |  same digest from here on        +--------------------------+
   v                                         |
+------------------------------+             |
| Integration + contract tests |             |
+------------------------------+             |
   |                                         |
   v                                         |
[GATE 1] staging: deploy digest + E2E tests  |
   |                                         |
   v                                         |
[GATE 2] promotion criteria or approval      |
   |                                         |
   v                                         |  rollback: redeploy
+------------------------------+             |  previous digest,
| Progressive rollout to prod  |<------------+  no rebuild
| same digest, prod config only|
+------------------------------+

Stages run from cheapest to most expensive, one immutable artifact identified by digest is promoted through both gates to production, and rollback redeploys the previous known-good digest from the registry without rebuilding.

Designing deployment gates with clear pass/fail criteria

A gate is a rule a machine checks. It should be a fixed threshold, not someone's judgment call. Each gate needs one named signal, a threshold and an action to take when the threshold is crossed. "Tests look fine" is not a gate. "Zero failing tests in the unit and integration suites" is.

Typical pre-deploy gates:

• Test results: any failure blocks promotion.
• Security and dependency scans: block on findings at or above a severity level you have agreed on. Keep a small, reviewed allowlist for accepted risks, and give each entry an expiry date.
• Migration dry run: apply the schema change to a recent copy of production-shaped data. It must succeed within a time limit.

Post-deploy gates compare the new version against a baseline, either the previous release or the stable fleet running next to it. Error rate and p99 latency are common signals. For example, if the canary's error rate goes above the baseline plus an agreed margin and stays there for a full observation window, the pipeline stops and redeploys the previous digest. A comparison against a baseline catches regressions that a fixed absolute number would miss when traffic is unusually high or low.

A gate nobody trusts is worse than no gate. Engineers learn to click "retry" or "override" without reading the result, so a real failure gets the same treatment as noise. Meanwhile the dashboard still says the gate exists, which suggests a level of safety that isn't there.

Flaky tests are the main way that trust gets lost. When a test is shown to be flaky, either fix it right away or quarantine it. Quarantining means moving it to a non-blocking suite, assigning an owner and setting a deadline to fix it. Do not allow automatic reruns until the test passes. That changes the gate's pass condition from "passes" to "passes eventually," and it hides real race conditions.

Copy# Generic pipeline-as-code sketch (tool-agnostic schema).
# Every stage promotes the same immutable digest, never a mutable tag.
artifact:
  image: registry.example.com/shop/api@sha256:3f9a1c0e7b2d4a6f8e1c5b9d0a7e2f4c6b8d1e3a5c7f9b0d2e4a6c8e0f1a3b5c

stages:
  - name: verify
    gates:
      - signal: failing_tests        # unit + integration suites
        threshold: 0                 # any failure blocks promotion
      - signal: scan_findings
        min_severity: high           # agreed blocking level
        threshold: 0
        allowlist: security/accepted-risks.yaml  # reviewed; each entry has an expiry date
      - signal: migration_dry_run    # run against production-shaped copy
        must_succeed: true
        timeout: 15m
    on_fail: stop                    # no manual override, no auto-rerun

  - name: canary
    deploy: { digest: "${artifact.image}", traffic_percent: 5 }
    compare_to: stable_fleet         # baseline running alongside the canary
    window: 10m                      # breach must persist for the full window
    gates:
      - signal: error_rate
        max: baseline + 0.5%         # agreed margin over baseline
      - signal: p99_latency
        max: baseline * 1.20
    on_fail: rollback                # redeploy previous known-good digest

  - name: production
    deploy: { digest: "${artifact.image}", traffic_percent: 100 }

# Flaky tests live in a separate non-blocking suite with an owner and deadline.
quarantine:
  suite: tests/quarantine
  blocking: false

Illustrative, tool-agnostic pipeline definition that promotes one artifact by digest through explicit pre-deploy gates and a baseline-relative canary gate that rolls back when error rate or p99 latency exceeds an agreed margin for a full window.

Progressive delivery and its alternatives: canary, blue-green and feature flags

Pre-production gates test what you predicted. Progressive delivery lets a gate judge what actually happens: a new version takes real production traffic while only a small share of users is exposed, and the gate compares its error rate and latency against the current version before more traffic shifts.

Each approach has a specific failure mode. A canary on a low-traffic service may serve too few requests in its window to tell a real regression from noise. A passing result there gives false confidence, so set a minimum request count before the gate can pass. Blue-green doubles capacity during cutover. Long-lived connections, in-memory sessions, and background jobs running in both environments can cause dropped requests or duplicate work. Feature flags add runtime configuration that needs an owner. Stale flags pile up, and with each new flag the number of possible flag combinations grows faster than anyone tests them.

For internal tools, low-traffic services, or teams that ship weekly, a rolling update with health checks, or a scheduled manual release with someone watching dashboards, is often enough.

Designing rollback before the first deploy

Whichever strategy you pick, the path back has to exist before the path forward is used. A rollback should redeploy the previous known-good artifact, the same immutable build that already passed every gate. It should not rebuild from an older commit, because a rebuild pulls in new base images and dependency resolutions and produces an artifact nobody has tested. Keep several prior versions in the registry so you are not depending on a single one. Run the rollback job on a schedule, for example in staging, because an untested rollback path usually fails when you need it.

Matching the response to the failure

Rollback cannot undo everything. It cannot reverse a destructive migration, records already written in a new format that old code cannot read, sent emails, or messages already consumed by other services. Those cases need a fix that moves forward.

Automatic rollback triggered by metrics has its own risks. It can oscillate between versions when error rates hover near the threshold. It can also revert a healthy release because a downstream outage pushed your error rate up. Tune thresholds and evaluation windows against real traffic, and give on-call engineers a way to pause or override the automation.

Implementing safe schema changes and secure pipeline definitions

Automated rollback only helps if the previous release can still run against the current database. Redeploying old code is easy. Reverting a migration is not, because the migration may already have rewritten or dropped data. The expand-and-contract pattern avoids this by splitting a breaking change into steps that each stay compatible with the version before them. For example, to rename users.email to users.email_address:

1. Expand: run ALTER TABLE users ADD COLUMN email_address TEXT;. The new column is nullable, so the running code ignores it.
2. Dual-write: deploy code that writes to both columns but still reads email.
3. Backfill: run UPDATE users SET email_address = email WHERE email_address IS NULL; in small batches so it doesn't hold long locks.
4. Switch reads: deploy code that reads email_address and still writes to both columns.
5. Contract: in a later deploy, after you no longer need to roll back to any version that reads email, run ALTER TABLE users DROP COLUMN email;.

At every step, rolling back one release leaves a schema the older code can use. The cost is real, though. One rename turns into four or five deploys, which may be spread over several days. Someone also has to track which step each table is on.

Pipeline definitions need the same discipline as application code. A pipeline change can ship anything to production, so keep definitions in version control and review changes before merging. Inject secrets at runtime from a secrets store instead of committing them or saving them as static pipeline variables. For deploy steps, issue short-lived credentials to each job instead of keeping long-lived access keys in the pipeline. A leaked token expires within minutes, but a leaked key keeps working until someone notices and revokes it.

Copy-- Expand-and-contract rename of users.email -> users.email_address (PostgreSQL).
-- Each step ships in its own deploy, so the previous release always matches the schema.

-- Deploy 1 (EXPAND): add a nullable column. Running code ignores it.
ALTER TABLE users ADD COLUMN email_address TEXT;

-- Deploy 2 (DUAL-WRITE): no SQL here. The app writes both columns, still reads email:
--   INSERT INTO users (email, email_address) VALUES ($1, $1);
--   UPDATE users SET email = $1, email_address = $1 WHERE id = $2;

-- BACKFILL: copy old rows in small batches so no single statement holds locks for long.
-- Re-run until it reports UPDATE 0. Rows with a NULL email are skipped so the loop ends.
UPDATE users
SET email_address = email
WHERE id IN (
  SELECT id FROM users
  WHERE email_address IS NULL
    AND email IS NOT NULL
  LIMIT 1000
);

-- Check before switching reads. This should return 0.
SELECT count(*) FROM users
WHERE email_address IS DISTINCT FROM email;

-- Deploy 3 (SWITCH READS): no SQL. The app reads email_address and keeps writing both,
-- so rolling back to Deploy 2 code still finds email populated.

-- Deploy 4+ (CONTRACT): only after no release you might roll back to reads email.
ALTER TABLE users DROP COLUMN email;

A PostgreSQL expand-and-contract sequence for renaming a column. It adds the new column, backfills it in batches, verifies the copy and drops the old column in a later deploy, with comments marking which steps change only application code.

When to avoid heavy gating, and the impact on cost and engineering time

Every gate is a queue. A manual approval that waits half a day doesn't just delay one change. It teaches the team to hold changes until several are ready and ship them together. That batch is harder to review, harder to bisect when it fails, and harder to roll back cleanly, so a gate meant to reduce risk ends up raising it.

A lighter pipeline is often the right call for prototypes, internal tools with a small and forgiving user base, and services that change a few times a year. For these, a build, the fast test tier, and a one-step deploy with a known way to revert are usually enough. Canary analysis and approval chains cost more than the failures they would prevent.

Heavier pipelines also have a development cost that recurs. Someone maintains the pipeline definitions, triages flaky tests, removes stale feature flags, and runs multi-step migrations across several releases. Count that work before adding another stage.

The return comes from what a well-ordered pipeline makes cheap. Fast feedback catches mistakes while the author still has context. Changes stay small. Recovery is quicker because the rollback path has already been exercised on ordinary deploys.

Illustrative estimate (assumed numbers, not a benchmark)

Weekly hours lost to gates = deploys per week × manual gates per deploy × average wait per gate (hours)

Assume 20 deploys, 2 manual gates, and a 0.5-hour wait per gate: 20 × 2 × 0.5 = 20 engineer-hours per week. Compare that figure with the incident time those gates actually prevent.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to ci cd pipeline design, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.