Backward Compatible API Changes for Independently Deployed Services

What backward compatibility means when services deploy independently

A change is backward compatible only if it keeps working through the whole rollout, not just once every service has the new version. When services deploy independently, there is no moment when everything switches over at once. A rolling deployment replaces instances a few at a time. Consumers owned by other teams upgrade on their own schedule. A message written to a queue or log yesterday may be read by code released today. So for long stretches, old and new versions of producers and consumers are live together.

That means every change to an API, RPC contract or event schema has to hold up in four situations:

• New producer, old consumer. The consumer gets fields, values or shapes it has never seen. It must not crash or quietly misread them.
• Old producer, new consumer. The consumer cannot rely on data that older producers never send.
• Producer rollback. The new consumer is suddenly receiving old-format data again, sometimes after it has already relied on the new format.
• Consumer rollback. Old code is now reading data, including stored records and queued events, that the new producer has already written.

Rollback is the case teams most often forget, and it is the one that turns a routine revert into an incident.

Teams accept this discipline because the alternative is lockstep releases: coordinated deploy windows, cross-team scheduling, and a single rollback that drags several services with it. This matters most to backend and platform engineers who own interfaces other teams consume, because they cannot see when those consumers will upgrade and cannot control it.

Copy PRODUCERS                SHARED CONTRACT        CONSUMERS
 (rolling deploy)         API / RPC / events     (other teams' pace)

 +---------------+                                      +---------------+
 | Producer v1   |--+                                +-->| Consumer v1   |
 | (old format)  |  |    +---------------------+    |   | (old reader)  |
 +---------------+  +--->|  requests, replies, |----+   +---------------+
        ^                |  queued events,     |               ^
        | (3) producer   |  stored records     |               | (4) consumer
        |     rollback   |                     |               |     rollback
 +---------------+  +--->|  old AND new shapes |----+   +---------------+
 | Producer v2   |  |    |  coexist here       |    |   | Consumer v2   |
 | (new format)  |--+    +---------------------+    +-->| (new reader)  |
 +---------------+                                      +---------------+

 Every change must survive all four pairings:
 (1) new producer -> old consumer : unseen fields/values must not break it
 (2) old producer -> new consumer : cannot rely on fields never sent
 (3) producer rolls back v2 -> v1 : new consumer gets old-format data again
 (4) consumer rolls back v2 -> v1 : old code reads data v2 already wrote

During an independent rollout, old and new producers and consumers share the same requests, queues and stored records, so a change must work in all four pairings, including after either side rolls back.

Who should use these practices

Use this approach whenever the producer and its consumers cannot be released as a single atomic unit. In practice, that covers:

• Teams that own an API, RPC service or event topic consumed by other teams. You don't control when those teams deploy, so every change you ship meets old readers.
• Any service deployed with rolling, canary or blue-green releases. Even with a single consumer, a rolling deploy creates a window where old and new instances serve traffic side by side, and a canary that gets reverted creates the rollback cases above.
• Producers and consumers of event streams or queues. Messages outlive the code that wrote them, so the readers of today must handle the writers of every version still within retention.
• Owners of data that is stored in serialized form. Caches, document stores, and blobs written in a contract format are read back by later versions and, after a rollback, by earlier ones.
• Providers of APIs to clients you cannot force to upgrade, such as installed mobile or desktop apps, partner integrations, and scripts written by other teams.

The people doing the work are mostly backend and platform engineers, but reviewers and tech leads need to know the rules too, because the decision to classify a change as safe or breaking usually happens in code review.

The exemption is narrow: one consumer, same repository, single atomic deploy, no stored or queued data in the contract format, and no mixed-version window. If any of those conditions fails, the practices in this article apply.

How it works

The approach rests on one rule: every individual release must be compatible with the versions deployed on either side of it, in both directions. Instead of changing a contract in one step, you change it through a sequence of releases, each of which can be deployed or reverted on its own without breaking anything that is still running. Four mechanisms make that possible:

1. Tolerant readers. Consumers ignore fields they don't recognize, apply explicit defaults when optional fields are missing, and map unknown enum values to a fallback. This makes additive producer changes safe, because old readers simply skip what they don't understand.
2. Ordering of changes. Readers learn to handle a new shape before any writer emits it, and writers stop emitting an old shape only after every reader has stopped depending on it. Expand-and-contract is this ordering applied to renames and type changes: add the new field, migrate readers, migrate writers, then remove the old field in a later release.
3. Format-specific rules. Each schema format defines what counts as compatible on the wire. In Protobuf, tag numbers identify fields, so tags must never be reused. In JSON, compatibility depends entirely on how readers treat unknown keys. In event streams, compatibility must hold for the full retention period.
4. Automated enforcement and usage evidence. CI compares each contract change against the released version and blocks structural breaks. Consumer-driven contract tests check known consumers. Runtime metrics on deprecated fields and endpoints show when old usage has actually stopped, so removal is based on data rather than memory.

The checklists below walk through each mechanism in the order you apply it: classify the change, make readers tolerant, sequence breaking changes, follow the format's rules, and automate detection.

Checklist: classify the change as safe or breaking

Before writing any code, decide which category the change falls into. The test is simple: can a consumer built against the old contract keep working against the new one, and can an old producer keep serving a new consumer during rollout? If either answer is no, the change is breaking.

Run through these checks for every change:

• Does any existing field disappear or change shape? Renames count as removals. Old clients look for the old name, and a strict parser may reject the new one.
• Does a request that succeeded yesterday now fail? Newly required fields and stricter validation both turn working clients into failing ones with no code change on their side.
• Did the meaning change while the schema stayed the same? Switching a duration from seconds to milliseconds, redefining what status means, or changing result ordering will pass every schema diff tool. Consumers still compute wrong answers, and nothing reports an error. Review these changes by hand.
• Does a default now behave differently? Clients that omit a parameter are relying on the old default, often without knowing it.

You can skip the rest of this process when the API has exactly one consumer, both sides live in the same repository, and they ship in a single atomic deploy with no mixed-version window. Once a second consumer appears or the deploys separate, the exemption no longer applies.

Checklist: build tolerant readers and handle enums safely

The consumer side carries most of the compatibility burden, because a producer can add fields far more safely when every reader already expects things it doesn't recognize.

• Ignore unknown fields. A consumer that rejects unrecognized properties turns every additive producer change into a breaking one. Check your deserializer's defaults. Some JSON libraries fail on unknown properties unless you configure them not to, and strict schema validation on input has the same effect.
• Read only what you need. Bind to the fields your logic actually uses instead of mirroring the producer's full model. Every field you parse is a field whose removal or retyping can break you.
• Apply explicit defaults for missing optional fields. Decide what absence means, such as zero, empty, or "feature off," and write it down in code. A default left to the language runtime is a guess, and it can differ between consumers.
• Define behavior for unknown enum values. Map them to an explicit fallback such as UNKNOWN, and decide whether that record is skipped, queued for review, or processed with conservative handling. A producer must not emit a new value until every consumer is running a version that tolerates it.
• Instrument the tolerance. Ignoring unexpected input can hide producer bugs or payloads sent to the wrong endpoint. Count unknown fields, unknown enum values, and applied defaults, labeled by producer, and alert when a count that should be near zero starts climbing.

The example below shows a consumer that ignores unknown properties, maps an unrecognized enum value to a fallback, and increments a counter each time it does so.

Copyimport json
from collections import Counter
from enum import Enum

metrics = Counter()

class Status(Enum):
    PENDING = "pending"
    SHIPPED = "shipped"
    UNKNOWN = "unknown"  # explicit fallback, never sent by producers

KNOWN_FIELDS = {"order_id", "status", "priority"}

def parse_status(raw, producer):
    try:
        return Status(raw)
    except ValueError:
        # New or garbage value: degrade instead of failing, but make it visible.
        metrics[("unknown_enum_value", producer, "status")] += 1
        return Status.UNKNOWN

def parse_order(body, producer):
    data = json.loads(body)

    # Tolerate extra fields, but count them so misrouted payloads show up.
    extra = set(data) - KNOWN_FIELDS
    if extra:
        metrics[("unknown_fields", producer)] += len(extra)

    # Absent priority means "normal" (0) by decision, not by runtime accident.
    if "priority" not in data:
        metrics[("default_applied", producer, "priority")] += 1

    return {
        "order_id": data["order_id"],  # required: a missing ID should still fail
        "status": parse_status(data.get("status"), producer),
        "priority": data.get("priority", 0),
    }

order = parse_order('{"order_id": "ord-123", "status": "returned", "gift_wrap": true}', "orders-api")
print(order["status"], dict(metrics))

A Python consumer reads only the fields it needs and ignores unknown properties. It applies an explicit default for a missing optional field, maps an unrecognized enum value to UNKNOWN, and counts each of these cases per producer so they can be alerted on.

Checklist: run breaking changes as expand-and-contract sequences

A rename or type change is a breaking change in disguise. Run it as a sequence of releases, each independently safe. The running example below renames price: "12.50" (string) to price_cents: 1250 (integer).

1. Expand the schema. Add price_cents as optional and leave price untouched. State: {price}, with price_cents declared but absent. Nothing reads or writes the new field yet, so rolling this back changes no behavior.
2. Deploy consumers that prefer the new field. Readers use price_cents when present and fall back to price otherwise. Ship this first so that consumers are ready before any producer emits the new shape.
3. Migrate producers to dual-write. State: {price: "12.50", price_cents: 1250}. Write down which field is authoritative, normally the old one until cutover. Two fields can disagree after a partial deploy or a bug, and readers need one rule for resolving the conflict.
4. Check for drift. On read, or in a sampled background job, compare the two values and count mismatches. A nonzero count means the conversion is wrong. Fix it before going further.
5. Flip authority and stop reading the old field. Consumers read only price_cents, and producers still write both. Rollback stays safe because older consumers still find price.
6. Verify that old usage has stopped. Use traffic and logs, not memory, to confirm that no reader still touches price.
7. Contract in a separate release. State: {price_cents: 1250}. Remove price only after the previous release has been stable long enough that you will not need to roll back past it.

At every step, ask what breaks if this release is reverted while its neighbors stay deployed. If anything does, split the step.

Copy// Reader for the price -> price_cents migration (steps 2-5).
// Authority flag: 'legacy' until cutover (step 5), then 'cents'.
type Authority = 'legacy' | 'cents';

interface PricePayload {
  price?: string;        // old shape, e.g. "12.50"
  price_cents?: number;  // new shape, e.g. 1250
}

let driftCount = 0; // export this as a metric; it must stay at zero

function legacyToCents(price: string): number {
  // Parse decimal string without float math: "12.50" -> 1250
  const [whole, frac = ''] = price.split('.');
  return Number(whole) * 100 + Number(frac.padEnd(2, '0').slice(0, 2));
}

export function readPriceCents(p: PricePayload, authority: Authority): number {
  const fromLegacy = p.price !== undefined ? legacyToCents(p.price) : undefined;
  const fromNew = p.price_cents;

  // Step 4: during dual-write both exist; count disagreements.
  if (fromLegacy !== undefined && fromNew !== undefined && fromLegacy !== fromNew) {
    driftCount++;
  }

  // One explicit rule decides conflicts; fall back if the preferred field is absent
  // so the reader survives rollback of either producer or consumer.
  const preferred = authority === 'legacy' ? fromLegacy : fromNew;
  const fallback = authority === 'legacy' ? fromNew : fromLegacy;
  const value = preferred ?? fallback;
  if (value === undefined) throw new Error('payload has neither price nor price_cents');
  return value;
}

A consumer-side reader that applies an explicit authority rule during dual-write, falls back to whichever field exists so rollback stays safe, and counts drift between price and price_cents.

Checklist: follow the rules of your schema format

Each serialization format has its own compatibility rules. Learn the rules of the formats you actually ship.

Protobuf

On the wire, a field is identified by its tag number. The field name does not appear in the binary encoding. This makes three rules important:

• Never reuse or renumber a tag. An old binary that still has a deployed reader will decode the new data as the old field. It will not raise an error.
• Reserve tags and names when you remove a field. For example, reserved 4, 7; and reserved "legacy_id";. The compiler then rejects any later attempt to reuse them. Reserving the name matters because the JSON mapping and text format both use names.
• Treat a rename as breaking if any consumer uses the JSON mapping. The binary format is unaffected by a rename. The JSON mapping is not.

JSON

JSON has no schema enforcement on the wire. Adding a field is safe only when every consumer ignores unknown keys. A strict deserializer, or a validator configured to reject additional properties, will fail on the first new field. Audit your consumers before you assume an addition is harmless.

Event streams

A message is written once and read much later. Old message shapes must stay readable for the full retention period of the topic. Count every way old data comes back into use:

• Replays
• Backfills
• New consumers that start reading from the earliest offset
• Caches that hold serialized payloads

Upgrading the producer does nothing to messages already stored.

Copy// order.proto (v2): proto3 schema after removing two fields from v1.
// v1 had: string legacy_id = 4; and int32 discount_pct = 7;
syntax = "proto3";

package shop.v1;

message Order {
  // Reserve the removed tag numbers. Old binaries still decode tag 4 as
  // legacy_id and tag 7 as discount_pct. Reusing either tag would make
  // those readers misread new data without raising an error.
  reserved 4, 7;

  // Reserve the removed names too. The JSON mapping and text format
  // identify fields by name, so a later field reusing a name would
  // collide in those encodings.
  reserved "legacy_id", "discount_pct";

  string order_id = 1;
  string customer_id = 2;
  int64 total_cents = 3;

  // Existing tags keep their numbers. Never renumber them.
  string currency = 5;
  int64 created_at_unix = 6;

  // A new field takes a never-used tag. Old readers skip it as an
  // unknown field. New readers see the default value (0 for int64)
  // when they read old stored messages.
  int64 discount_cents = 8;

  // The compiler rejects either of these lines if uncommented:
  // string legacy_id = 9;      // name is reserved
  // int64 discount_amt = 7;    // tag is reserved
}

A proto3 message that reserves the tag numbers and names of removed fields and adds a replacement field under a new tag, so the compiler blocks any later reuse that old readers would misinterpret.

Checklist: automate detection in CI and make removal evidence-based

Reviewers miss these rules under deadline pressure, so the build should enforce them.

• Diff every contract change against the last released version. Compare against the tag or artifact that is actually deployed, not the main branch. A field removed and re-added across two unreleased commits is harmless, while the same change against production is a break. Most schema formats have breaking-change linters. Run one as a required check so a breaking diff blocks the merge unless someone approves an explicit override.
• Add consumer-driven contract tests for known consumers. Each consumer publishes the requests it sends and the response fields it reads. The producer's pipeline replays them, so a change that breaks a real consumer fails the producer's build, before deployment.
• Know what the tooling misses. Linters catch structural breaks: removed fields, changed types, renumbered tags. They miss behavioral breaks, such as a field that keeps its type but changes units, starts returning nulls, or sorts differently. Contract tests only cover consumers that registered contracts. They don't protect unknown clients, third-party integrations, or scripts written by other teams.
• Instrument before you deprecate. Record which client identity calls each deprecated endpoint or sends each deprecated field. Without that data, you can't tell whether a removal is safe.
• Remove only after a sustained zero. Usage should stay at zero for a window longer than your slowest consumer's cycle. Monthly reconciliations, quarterly reports, and annual audits show zero traffic for weeks and then fail on the day they run. Check job schedules with their owners before you treat a quiet period as proof that a field is unused.

How backward compatible changes save engineering time

Each individual change takes more releases under this discipline, but the time it saves comes from work that no longer has to happen elsewhere:

• No coordinated release windows. When every release is compatible with its neighbors, teams deploy on their own schedules. Nobody spends time negotiating a shared deploy slot, waiting on another team's release train, or holding a finished change until a consumer is ready.
• Rollbacks stay local. A revert affects only the service being reverted. Without compatibility, rolling back one service can force rollbacks of every service that depends on its new contract. Those multi-team reverts are slow, error-prone, and often happen during an incident.
• Breaks are caught at merge time instead of in production. A breaking-change linter or a failing consumer contract test points at the exact field and the exact consumer while the author still has the change in mind. The same problem found in production arrives as a symptom in another team's service, and someone has to trace it back across service boundaries first.
• Fewer silent data bugs. Tolerant readers with counters, explicit defaults, and drift checks during dual-write turn quiet misreads into visible metrics. Silent corruption is expensive to clean up because it is usually discovered late, after bad values have spread into stored records and downstream systems.
• Removal decisions without archaeology. Usage metrics on deprecated fields and endpoints answer "is anyone still using this?" directly. Without them, engineers search code they don't own, send messages to other teams, and wait for replies, or they skip the removal and keep maintaining the old field indefinitely.
• Less cross-team communication per change. Additive changes to an API with tolerant readers need no coordination at all. Only true breaking changes need a planned sequence, and the sequence is the same every time, so teams don't redesign the process for each migration.

The savings grow with the number of consumers, because coordination costs and the chance of an incident both scale with how many teams depend on a contract, while the per-change overhead of expand-and-contract stays roughly fixed. The next section works through that trade-off with assumed numbers.

Alternatives, when to avoid expand-and-contract, and impact on cost

Some changes cannot be expressed additively: a different identity model, a split resource, or semantics that conflict with the old field. For these, publish an explicit new version, such as a new path, a new RPC method, or a new topic. Treat it as a second API with its own owners, SLOs, and a dated deprecation plan. If you treat it as a temporary alias, it tends to become permanent.

Expand-and-contract is slower because every step needs a deploy and a confirmation from the consuming team. A field rename can take three or four deploy cycles. Weigh that against the incidents it prevents.

Illustrative estimate (assumed numbers, not a benchmark). Assume:

• Expand-and-contract adds 6 engineer-hours.
• A breaking deploy has a 20% chance of causing an incident.
• Each incident costs 5 engineers × 3 hours, including a failed rollback.

The expected incident cost is 0.20 × 15 = 3 engineer-hours. Here the overhead of 6 hours exceeds the risk of 3 hours, so a lockstep deploy is defensible.

The result changes with the number of consumers. Each consumer adds risk, while the overhead stays roughly fixed. With four consumers:

• Probability of at least one incident: 1 − 0.8⁴ ≈ 0.59
• Expected incident cost: 0.59 × 15 ≈ 8.9 engineer-hours, which exceeds the 6-hour overhead.

To compare the approaches with your own figures, use this formula:

Copyoverhead  <  (1 − (1 − p)^consumers) × incident_cost

If the inequality holds, expand-and-contract is the cheaper choice.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to backward compatible api changes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.