What feature flags do in a monolith and why branches hurt more there
A feature flag is a runtime switch that decides which code path executes, read from configuration rather than fixed at build time. Its job is to separate deploy from release. You ship code to production with the new path dormant, then turn it on for some or all traffic without another deploy. If something breaks, you flip the switch back instead of reverting a commit and waiting for a pipeline.
Long-lived branches are expensive in any codebase, and a monolith makes them worse. Many teams edit the same modules: the order service, the shared data-access layer, the request middleware. A branch that lives for three weeks diverges from changes made by every other team in that time, not just its own. Conflicts compound, and they surface at merge time, which is usually the week the feature was meant to ship. The worst failures are often not textual conflicts but semantic ones: two changes that merge cleanly but break each other at runtime. Late merges also mean CI tests a combination of code that nobody has run together before.
Flags let every team commit small, incomplete changes to trunk daily, so integration happens continuously. That safety is conditional, though. Flags only make trunk-based development safe if each one is treated as short-lived code with an owner, a typed definition, a known evaluation point and a removal date. Without that discipline, branch complexity moves into runtime conditionals.
This guide is for backend and platform engineers working in a shared monolith with several teams committing to it.
A flag taxonomy: release, experiment, ops and permission flags
Flags look identical in code: a named condition evaluated at runtime. They differ in why they exist, and that purpose determines how long one should live and who is accountable for deleting it. Classify each flag when you create it, because a flag's type sets its removal rules.
| Type | Expected lifetime | Owner | Removal expectation |
|---|---|---|---|
| Release | Days to weeks | Team shipping the feature | Delete once fully rolled out and stable |
| Experiment | Length of the experiment | Team running the experiment | Delete when a decision is made, win or lose |
| Ops / kill switch | Long-lived on purpose | Service or platform owner | Review periodically; never auto-expire |
| Permission / entitlement | Indefinite | Product or billing | Usually move to product configuration |
Release flags hide unfinished or risky code on trunk. They are meant to be temporary. Each one that outlives its rollout doubles the code paths someone has to reason about.
Experiment flags split traffic so you can compare outcomes. They need stable bucketing and a defined end. The experiment's conclusion is the removal trigger.
Ops flags let on-call degrade or disable expensive paths under load. Their value is that they persist. Treat them as part of the runbook and exercise them, because a kill switch nobody has flipped in a year may not work when you need it.
Permission flags gate features by plan or customer. These are business data. If a value changes when a contract changes, store it in product configuration with an audit trail, not in a flag system built for short-lived toggles.
How flag evaluation works: boundaries, context and consistent bucketing
Evaluate each flag once, at a boundary where the full context is available: request entry, job start, or the seam between a controller and a service. Pass the result down, either as a boolean or, preferably, as the implementation to use. A flag check buried in a shared library reads global state that callers cannot see, and two checks in the same request can disagree if the flag changes between them.
Branch by abstraction gives this a fixed shape. Put the old and new code paths behind one interface, choose the implementation at the boundary, and inject it. Downstream code stays flag-free. Cleanup then means deleting one class and one line of selection logic.
Percentage rollouts need deterministic bucketing. Hash the flag name together with a stable key, such as the user, account or tenant ID, into a fixed range, and compare the result to the rollout percentage. The same tenant then gets the same answer on every node and every request. Raising the percentage only adds tenants and never moves existing ones out. Bucket on the tenant when users share data, so that one account does not see two behaviours at once.
Background work must carry the decision with it:
Request → evaluate flags at entry → decision passed to services → decision serialized into job/message payload → worker reads payload, not the flag
If the worker re-evaluated the flag, the sync and async halves of one operation could take different paths. Recording the decision in the payload also defines what happens to in-flight work: jobs enqueued before a flag change finish on the path they started with. This means the old path must stay deployable until those queues drain.
import hashlib
import json
def in_rollout(flag: str, tenant_id: str, percent: int) -> bool:
# Hash flag name + stable key: same answer on every node, independent buckets per flag
digest = hashlib.sha256(f"{flag}:{tenant_id}".encode()).digest()
bucket = int.from_bytes(digest[:4], "big") % 100 # 0..99
return bucket < percent # raising percent only adds tenants, never removes them
class InvoiceRenderer: # one interface for old and new paths
def render(self, invoice: dict) -> str: raise NotImplementedError
class LegacyRenderer(InvoiceRenderer):
def render(self, invoice): return f"legacy:{invoice['id']}"
class NewRenderer(InvoiceRenderer):
def render(self, invoice): return f"new:{invoice['id']}"
# Cleanup later = delete one class and one entry here
RENDERERS = {"legacy": LegacyRenderer(), "new": NewRenderer()}
def handle_request(tenant_id: str, invoice: dict, enqueue) -> str:
# Evaluate once at the request boundary; downstream code stays flag-free
path = "new" if in_rollout("new-invoice-renderer", tenant_id, 20) else "legacy"
html = RENDERERS[path].render(invoice)
# Serialize the decision into the job so the async half takes the same path
enqueue(json.dumps({"invoice": invoice, "renderer": path}))
return html
def worker(message: str) -> str:
job = json.loads(message)
# Read the recorded decision, not the live flag: jobs enqueued before a
# flag change finish on the path they started with
return RENDERERS[job["renderer"]].render(job["invoice"])
Deterministic tenant bucketing, a branch-by-abstraction interface chosen once at request entry, and the chosen path carried in the job payload so the worker follows the same path as the request.
Implementing a typed flag registry and gradual rollout
Every flag should be declared in one typed registry module rather than as string literals scattered through the codebase. Each entry records the flag's name, owning team, creation date, expected removal date and safe default. Referencing a flag through a typed symbol means a deleted flag breaks the build instead of silently evaluating to its default. CI can then fail any change that adds a flag without an owner, or that keeps a flag past its removal date. The safe default is the value the system uses when nothing else can be decided, so it should almost always be the old behaviour.
Ramping by stable key
Start with an internal-only cohort, such as staff accounts or a test tenant, so real traffic exercises the new path before customers do. Then ramp by percentage. Hash the flag name together with a stable key (account ID, not request ID) and map the result to a bucket from 0 to 99. Including the flag name means each flag splits users differently. Using a stable key means a user stays in the same bucket as the percentage grows, so nobody flips back and forth between paths.
Write down each step before you take it, for example 1%, 10%, 50% and 100%. For each step, name the metric that must hold and the threshold that triggers a rollback. This turns the decision to proceed or roll back into a lookup instead of a debate.
Kill switches
A kill switch is checked on hot paths. It must be read from an in-process cache and must fall back to the safe default if the flag store is unreachable. The cache TTL sets the trade-off. A shorter TTL makes the switch take effect faster but puts more load on the store. Choose the TTL deliberately and write the resulting worst-case propagation delay into the runbook.
import hashlib
from dataclasses import dataclass
from datetime import date
@dataclass(frozen=True)
class Flag:
name: str
owner: str
created: date
remove_by: date # CI fails the build once this date has passed
safe_default: bool # old behaviour; used whenever we cannot decide
# Single registry: call sites use Flags.NEW_CHECKOUT, never a string literal,
# so deleting a flag here breaks every remaining reference at build time.
class Flags:
NEW_CHECKOUT = Flag("new_checkout", "payments-team",
date(2026, 9, 1), date(2026, 12, 1), safe_default=False)
def bucket(flag: Flag, stable_key: str) -> int:
# Flag name + account ID: each flag splits users independently, and an
# account keeps the same bucket (0-99) as the rollout percentage grows.
digest = hashlib.sha256(f"{flag.name}:{stable_key}".encode()).digest()
return int.from_bytes(digest[:8], "big") % 100
def is_enabled(flag: Flag, account_id: str, is_staff: bool, cache) -> bool:
try:
# In-process cache refreshed from the flag store at most once per TTL;
# the TTL is the worst-case delay before a kill switch takes effect.
cfg = cache.get(flag.name)
except Exception:
return flag.safe_default # store unreachable: fail to old behaviour
if cfg is None or cfg.get("killed"):
return flag.safe_default
if is_staff and cfg.get("internal"):
return True # internal cohort sees the new path first
return bucket(flag, account_id) < cfg.get("percent", 0)
An illustrative typed flag registry with owner and removal metadata, stable hash-based bucketing by account ID, and a cached evaluation path that falls back to the safe default when the flag store is unavailable or the kill switch is set.
Schema changes, testing and operational visibility in a flagged codebase
Keep both paths valid against one schema
A flag lets the old and new code paths run at the same time, so the database has to support both at once. Use expand/contract migrations. First, expand: add the new column or table without removing anything. Then ship code that dual-writes to old and new structures, or backfill existing rows so the new path reads complete data. Contract by dropping the old structure only after the flag is fully on and removed. Never put a destructive migration behind a reversible flag. If you flip the flag back after the old column is gone, the old path fails in production, and turning a flag off is supposed to be the safe move.
Test states, not combinations
For each active flag, write unit tests for both the on and off states. CI should also run against the default production configuration, because that is what users actually get. Skip the full combinatorial matrix. Ten boolean flags produce 1,024 combinations, and most of them will never occur in production. Keep the number of interacting flags small, and when one flag depends on another, write that dependency down in the registry.
Make decisions observable
Record flag decisions alongside traces and logs so that an incident timeline shows which path each request took. Keep an audit trail of who changed which flag and when. Production flag flips skip code review and the deploy pipeline, so put access control on them.
-- Migration 0042_expand: additive only, safe with the flag on or off.
-- Old code ignores the new column; new code reads and writes it.
ALTER TABLE orders ADD COLUMN amount_cents BIGINT; -- nullable, no default
-- Backfill existing rows in small batches so the new path sees complete data.
-- Run repeatedly until it updates 0 rows. While the flag is active, the
-- application dual-writes amount and amount_cents, so new rows stay in sync.
UPDATE orders
SET amount_cents = ROUND(amount * 100)
WHERE id IN (
SELECT id FROM orders
WHERE amount_cents IS NULL AND amount IS NOT NULL
ORDER BY id
LIMIT 10000
);
-- Gate check before ramping the flag: this should return 0.
SELECT count(*) FROM orders
WHERE amount_cents IS NULL AND amount IS NOT NULL;
-- ---------------------------------------------------------------
-- Migration 0057_contract: destructive. A flag flip cannot undo it.
-- Ship this only after the orders_amount_cents flag is fully on AND its
-- old code path has been deleted. Otherwise turning the flag off would
-- send traffic to code that reads a column that no longer exists.
-- ---------------------------------------------------------------
ALTER TABLE orders DROP COLUMN amount;
A PostgreSQL expand/contract sequence in which an additive migration and batched backfill keep both flag states working against one schema, and the column drop ships as a separate migration only after the flag and the old code path are removed.
Flag cleanup and when to avoid feature flags
Flag debt builds up because adding a flag is cheap and removing one belongs to nobody. Once a release flag reaches 100%, the old path still compiles, still passes its tests, and looks like live code. A few months later nobody is sure it can go, so everyone who reads the module has to reason about both behaviours. Each flag that lingers also multiplies the states the system can be in, and most of those combinations are never tested.
Make removal part of done
- Owner and expiry date. Give every flag both in the registry. Release and experiment flags should expire within weeks. Ops and permission flags are long-lived by design, so mark them explicitly rather than giving them a far-off date.
- CI check. Add a check that compares the registry against references in code. It should report flags past expiry, flags defined but never referenced, and references to undefined flags. Start with warnings, then fail the build.
- One change. Delete the losing code path, its tests, and the flag definition together. If you split the work, you leave either a dead branch or a definition nothing reads.
When not to use a flag
- Irreversible data migrations. A flag implies you can turn the change off. If the new path has already rewritten data, the old path may not be able to read it.
- External API contracts and message formats. Consumers don't see your flag, and queued messages outlive a flip. Use versioning or expand/contract.
- Tightly coupled code. If a change needs flag checks in many places, those checks can disagree within a single request. Refactor behind an abstraction first, then flag the one point where the implementation is selected.
# ci/check_flags.py: compare the flag registry against flag references in code.
# Registry (flags.json), for example:
# {"checkout_v2": {"owner": "payments-team", "expires": "2026-11-15"},
# "maintenance_mode": {"owner": "sre", "permanent": true}}
import json, re, sys
from datetime import date
from pathlib import Path
REGISTRY = json.loads(Path("flags.json").read_text())
# Only literal names are found, so keep calls like flags.is_enabled("checkout_v2")
# greppable and never build flag names at runtime.
FLAG_CALL = re.compile(r"is_enabled\(\s*[\"']([a-z0-9_]+)[\"']")
STRICT = "--strict" in sys.argv # warn first; turn on --strict once the backlog is cleared
referenced = set()
for path in Path("src").rglob("*.py"):
referenced.update(FLAG_CALL.findall(path.read_text(encoding="utf-8")))
problems, today = [], date.today()
for name, meta in sorted(REGISTRY.items()):
if not meta.get("owner"):
problems.append(f"{name}: no owner")
# Long-lived ops/permission flags are marked explicitly, not given a far-off date.
if not meta.get("permanent"):
if "expires" not in meta:
problems.append(f"{name}: no expiry date")
elif date.fromisoformat(meta["expires"]) < today:
problems.append(f"{name}: expired {meta['expires']}, remove code path and definition")
if name not in referenced:
problems.append(f"{name}: defined but never referenced, delete the definition")
for name in sorted(referenced - REGISTRY.keys()):
problems.append(f"{name}: referenced in code but not in flags.json")
for p in problems:
print(("ERROR: " if STRICT else "WARNING: ") + p)
sys.exit(1 if STRICT and problems else 0)
A minimal CI script that checks a JSON flag registry against is_enabled() calls in the source tree. It reports missing owners, expired flags, unreferenced definitions and undefined references, warning by default and failing the build when run with --strict.
Alternatives and the impact on cost and engineering time
Flags are one way to separate merging code from releasing it. The alternatives make that separation at a different layer, and each has a different cost.
| Approach | Rollback path | Main cost |
|---|---|---|
| Long-lived feature branch | Revert the merge | Merge conflicts that grow with branch age |
| Short-lived branches, frequent merges | Revert and redeploy | Incomplete work must stay hidden or inert |
| Branch by abstraction, no toggle | Redeploy the old implementation | Switching requires a deploy |
| API or message versioning | Clients stay on the old version | Supporting several versions in parallel |
| Separate deploy environments | Route traffic back | Duplicated infrastructure and drift between environments |
Building flags in-house looks cheap at first, because a config table and a lookup function are enough. Over time, teams add targeting rules, a UI for non-engineers and audit logs for compliance, and the system becomes a product the team has to maintain. A dedicated flag service provides those features from the start. In exchange, you take on a runtime dependency, a recurring fee and a network path that the flag evaluation code has to tolerate when it fails.
Flags save time in three places: fewer merge conflicts, rollback without a redeploy, and release timing that does not depend on deploy timing. They cost time in runtime branching, testing both paths, cleanup work and keeping the flag store reliable.
Illustrative estimate (assumed numbers, not a benchmark). Net hours saved per month = (conflict hours avoided + redeploy-rollback hours avoided) − (flag test hours + cleanup hours + store maintenance hours). Assume 20 + 6 − (8 + 5 + 4). That gives a net saving of 9 hours per month. Replace each term with your team's own figures.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to feature flags monolith, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.