Why do feature flags disagree across microservices?
Cross-service flag consistency means that a single request, for a single user, gets the same flag decision from every service that checks the flag. If the checkout service shows the new pricing page, the pricing service must compute the new prices and the invoicing service must format them the new way. When they disagree, users don't see a clean "old" or "new" experience. They see a broken hybrid, and the bug reports are hard to reproduce because they depend on timing and on which service answered.
Disagreement usually comes from one of four places:
- Different evaluation context. One service targets on
userId, another onaccountId, and a third only has a session cookie. Each one buckets a different identity, so a 10% rollout selects a different 10% of traffic in each service. - Different SDKs or hashing schemes. Two services can use the same targeting key and still disagree if one SDK hashes it with a different algorithm, salt, or bucket count than the other.
- Stale local caches. Services poll or stream flag configuration on their own schedules. For a while after a change, some instances are serving the old rules and others the new ones.
- Mid-request changes. A flag flips after the gateway has evaluated it but before a downstream service does, so one request is handled under two configurations.
The common thread is that consistency requires three things to match in every service: the same flag definition, the same context, and the same bucketing. If any one of them differs, the decisions can diverge.
What does OpenFeature standardize, and what does it leave to the provider?
OpenFeature is a vendor-neutral specification and set of language SDKs for evaluating flags. Application code calls a single API. A provider, a plugin that talks to the actual flag backend, does the work behind it. Swapping backends means swapping the provider. Call sites stay the same.
The specification defines a few building blocks. The evaluation context carries attributes such as a user ID or tenant, and the provider uses them for targeting. Hooks run at defined stages of an evaluation (before, after, on error, finally), which makes them the natural place for logging, metrics, and context enrichment. Evaluation is typed: you ask for a boolean, string, number, or object, and you always pass a default. If the provider fails or the type doesn't match, the caller gets the default and no exception escapes.
| Concern | Defined by OpenFeature | Left to the provider and backend |
|---|---|---|
| Evaluation API and types | Yes | — |
| Context shape and hooks | Yes | — |
| Default-on-error behaviour | Yes | — |
| Flag storage and distribution | No | Yes |
| Targeting rule syntax | No | Yes |
| Bucketing and hashing algorithm | No | Yes |
That last row matters most for the consistency problem described above. OpenFeature makes call sites uniform. It does not make bucketing uniform. Suppose two services use different providers, or the same provider pointed at different backends. They can hash the same user into different percentages even when the code at each call site is identical. Consistent bucketing still requires every service to use one provider and one source of flag definitions.
Platform teams that support many services in several languages get the most from OpenFeature. They can standardize context fields, hooks, and provider configuration once and apply them everywhere.
How does an OpenFeature flag evaluation work?
A single call such as client.getBooleanValue('new-checkout', false, ctx) goes through a fixed sequence that is the same in every language SDK:
- Context merge. The SDK combines evaluation context from several levels: context set globally on the API, context set on the client, and context passed at the call site. More specific levels override less specific ones, so a value passed at the call site wins over a global one. This lets a service set stable attributes such as
regiononce and add per-request attributes such as the targeting key at each call. beforehooks. Registered hooks run in a defined order. They can inspect the flag key and context, and they can add attributes to the context before resolution.- Provider resolution. The SDK hands the flag key, the requested type, the default, and the merged context to the provider. The provider applies the backend's targeting rules and bucketing, either locally from a cached rule set or by calling a remote service, and returns a value with metadata: the variant name, a reason (for example
TARGETING_MATCH,SPLIT,DEFAULT, orERROR), and an error code if something went wrong. afterorerrorhooks. On success,afterhooks receive the evaluation details. If the provider throws, returns an error, or returns the wrong type,errorhooks run and the SDK substitutes the caller's default.finallyhooks. These run in both cases, which makes them the place for cleanup and for timing measurements.- Return. The
get…Valuemethods return only the value. The matchingget…Detailsmethods return the value plus flag key, variant, reason, and error information, which is what you want when debugging cross-service disagreement.
Providers also have a lifecycle. The SDK tracks whether a provider is not yet ready, ready, in an error state, or serving stale data, and it emits events when that changes, including when flag configuration changes. Evaluations made before the provider is ready return the default, which is why the wrapper later in this article waits for readiness at startup.
The consistency implication is that steps 1 and 3 decide the outcome. The SDK guarantees the same pipeline everywhere, but the merged context and the provider's bucketing must still match across services for decisions to agree.
When should you use OpenFeature for cross-service flags?
Use it when most of the following are true:
- Several services in more than one language read the same flags. A shared API and hook model is most valuable when you would otherwise maintain different flag-calling conventions per language.
- Rollouts need runtime control. You want percentage ramps, per-tenant targeting, or a kill switch that takes effect without a redeploy.
- Teams release independently. Flags let a producer and consumer ship code on different schedules and enable behaviour later, in a controlled order.
- You want to keep the backend replaceable. Because call sites depend only on the OpenFeature API, changing the flag backend is a provider swap rather than a rewrite of every call site.
- You need consistent telemetry about flag decisions. Hooks give one place to emit flag key, variant, and reason from every service.
It is a weaker fit when a single service owns the behaviour, when services always deploy together, or when the change cannot be safely turned off, such as an irreversible data migration. The rollout section below covers those cases in more detail.
How do you get the same bucketing for a user in every service?
Most providers implement percentage rollouts by hashing the targeting key together with the flag key into a bucket. Because the hash is deterministic, consistent bucketing comes down to two conditions. Every service must send the same targeting key, and every service must resolve against the same flag definitions. If either differs, the same subject can land in different variants.
Choose the key based on what must switch together. Use a user ID when the feature affects one person's experience. Use a tenant ID when an entire account has to move as a unit, for example when a change affects shared data or billing. Never use session or request IDs. They change constantly, so users would be reshuffled between variants.
The entry service should resolve identity once and forward only the identifiers, not the full context. You can carry them in a dedicated header or in tracing baggage. Baggage has an advantage here: instrumented clients propagate it automatically, including across message queues. It also has a cost. Baggage is often forwarded to every downstream hop, including third-party APIs, and often ends up in logs, so treat its contents as public.
Propagate only opaque, stable, non-sensitive values such as pseudonymous user IDs, tenant IDs and region codes. Re-derive everything else server-side. Email addresses, IP addresses and names should never travel in headers. Roles and plan tiers should also be re-derived, because a client could forge them; strip any such incoming values at the edge.
In a downstream service, building the context works like this. Read the user and tenant IDs from incoming baggage and set the user ID as the targeting key. Look up the tenant's plan from the service's own cache, add it as an attribute, and evaluate.
import { OpenFeature, EvaluationContext } from '@openfeature/server-sdk';
import { context, propagation } from '@opentelemetry/api';
// Placeholder: this service's own tenant cache, never values sent by the caller.
declare function getTenantPlan(tenantId: string): Promise<string | undefined>;
const flags = OpenFeature.getClient();
async function buildFlagContext(): Promise<EvaluationContext | undefined> {
// Read only the opaque identifiers the entry service put in baggage.
const baggage = propagation.getBaggage(context.active());
const userId = baggage?.getEntry('app.user_id')?.value; // e.g. 'u_8f3a91c2'
const tenantId = baggage?.getEntry('app.tenant_id')?.value; // e.g. 't_4417'
if (!userId || !tenantId) return undefined;
// Re-derive the plan server-side. Ignore any plan or role found in
// baggage or headers, because a client could forge it.
const plan = (await getTenantPlan(tenantId)) ?? 'unknown';
return {
targetingKey: userId, // same key in every service => same hash bucket
tenantId,
plan,
};
}
export async function useNewCheckout(): Promise<boolean> {
const ctx = await buildFlagContext();
// Without identity, take the safe default instead of bucketing on a random key.
if (!ctx) return false;
return flags.getBooleanValue('new-checkout', false, ctx);
}
A downstream service reads only the pseudonymous user and tenant IDs from OpenTelemetry baggage, re-derives the tenant's plan from its own cache, and evaluates a flag with the user ID as the targeting key so the percentage bucket matches every other service.
Should you evaluate once at the edge or in every service?
Once every service can build the same context, the open question is where evaluation happens. There are three patterns, and they fail in different ways.
| Approach | Consistency within a request | Main cost | Fits when |
|---|---|---|---|
| Evaluate at the edge, propagate the result | Guaranteed: one decision per request | Downstream services depend on an upstream decision; headers grow; receivers must trust or validate the value | Several services must act on the same variant within one request path |
| Each service evaluates via a remote flag service | Can briefly disagree while a rule change propagates | A network hop and a runtime dependency on every evaluation | Low request volume, or rules that change often |
| Each service evaluates in-process from a synced rule set | Agrees only if every service holds identical rules | Staleness window between syncs; rule versions must match everywhere | Latency-sensitive paths where a few seconds of skew is acceptable |
Edge propagation is a good fit for a checkout redesign where the API gateway, pricing service, and payment service must all agree on the variant. It is a poor fit for most other flags. If a flag only affects one service, or affects several services that never share a request, propagating it adds header weight and coupling for no benefit. A propagated value also becomes an input that callers control, so validate it against the allowed variants. For anything security-relevant, re-evaluate it rather than trusting it.
Use per-service evaluation by default. Reserve edge propagation for the small set of flags whose correctness depends on agreement within a single request.
A) Evaluate once at the edge, propagate the decision
+-----------+ variant=B +-----------+ variant=B +-----------+
| Gateway |------------>| Pricing |------------>| Payment |
| evaluates | | validates | | validates |
+-----------+ +-----------+ +-----------+
B) Each service evaluates via a remote flag service
+-----------+ +-----------+ +-----------+
| Gateway |------------>| Pricing |------------>| Payment |
+-----------+ +-----------+ +-----------+
| | |
v network hop v per evaluation v
+---------------------------------------------------------------+
| Remote flag service (single rule set) |
+---------------------------------------------------------------+
C) Each service evaluates in-process from a synced rule set
+-----------+ +-----------+ +-----------+
| Gateway |------------>| Pricing |------------>| Payment |
| rules v42 | | rules v41 | | rules v42 |
+-----------+ +-----------+ +-----------+
^ ^ stale until ^
| sync | next sync | sync
+---------------------------------------------------------------+
| Rule source (publishes v42) |
+---------------------------------------------------------------+
Three places to evaluate a flag: (A) the edge decides once and downstream services validate the propagated variant, (B) every service calls a remote flag service on each evaluation, and (C) every service evaluates locally from synced rules and can briefly disagree while one copy is stale.
How do you build a shared OpenFeature wrapper for all teams?
Whichever evaluation model you choose, each team should not wire OpenFeature by hand. A thin internal library keeps the decisions that must match across services in one place.
What the wrapper owns
- Provider setup. One initialization call configures the provider, its connection settings, and its startup wait behavior, so no service starts serving with an uninitialized provider.
- Context construction. A single function builds the evaluation context from request data: the targeting key, tenant, region, and any propagated attributes. If the targeting key comes from a different field in each service, bucketing drifts, and no other fix can compensate.
- Defaults and failure behavior. OpenFeature returns the caller's default value when evaluation fails. The wrapper should define defaults centrally, per flag, and set them to the safe, pre-rollout state. That way an unreachable backend produces the same result in every service, instead of
truein one andfalsein another. - Standard hooks. An
after/errorhook records flag key, variant, and reason as log fields, metric labels, and span attributes. With these, you can join a trace and see that service A goton/TARGETING_MATCHwhile service B gotoff/ERROR. At high volume, sample logs and span attributes, and aggregate metrics by flag, variant, and reason rather than by user. Always emitERRORreasons unsampled.
The cost trade-off
The wrapper removes the need for each team to reimplement this setup and to debug the inconsistencies that follow. It also becomes a cross-team dependency. Every change to defaults or context shape needs a coordinated upgrade, and in a polyglot stack you maintain parallel implementations that must stay behaviorally identical. Keep the wrapper small, version it explicitly, and test every language implementation against a shared conformance suite.
import { OpenFeature, Hook, Provider, EvaluationContext } from '@openfeature/server-sdk';
import { trace, metrics } from '@opentelemetry/api';
// Central defaults: the safe, pre-rollout state used whenever evaluation fails.
const DEFAULTS = { 'checkout.new-pricing': false, 'search.v2': false } as const;
type FlagKey = keyof typeof DEFAULTS;
const evals = metrics.getMeter('flags').createCounter('feature_flag.evaluations');
const LOG_SAMPLE_RATE = 0.01; // sample routine logs; errors are always logged
const telemetryHook: Hook = {
after(_ctx, d) {
const attrs = { 'feature_flag.key': d.flagKey, 'feature_flag.variant': d.variant ?? 'none', 'feature_flag.reason': d.reason ?? 'UNKNOWN' };
evals.add(1, attrs); // aggregated by flag/variant/reason, never by user
trace.getActiveSpan()?.setAttributes(attrs); // cost follows the tracer's span sampling
if (d.reason === 'ERROR' || Math.random() < LOG_SAMPLE_RATE) console.info('flag_evaluated', attrs);
},
error(ctx, err) {
console.error('flag_error', { 'feature_flag.key': ctx.flagKey, error: String(err) }); // unsampled
},
};
export async function initFlags(provider: Provider): Promise<void> {
OpenFeature.addHooks(telemetryHook);
try {
await OpenFeature.setProviderAndWait(provider); // don't serve before the provider is ready
} catch (err) {
console.error('flag_provider_init_failed', { error: String(err) }); // keep serving on DEFAULTS
}
}
// One rule for every service: targetingKey is always the user ID, so bucketing matches.
export function buildContext(req: { userId: string; tenantId: string; region: string }): EvaluationContext {
return { targetingKey: req.userId, tenant: req.tenantId, region: req.region };
}
const client = OpenFeature.getClient();
export function isEnabled(flag: FlagKey, ctx: EvaluationContext): Promise<boolean> {
return client.getBooleanValue(flag, DEFAULTS[flag], ctx); // callers cannot pick their own default
}
A minimal TypeScript OpenFeature wrapper that centralizes provider startup, context construction, per-flag safe defaults, and a telemetry hook emitting flag key, variant and reason to metrics, spans and sampled logs.
How should you plan a progressive rollout across dependent services?
Start from the dependency graph, not the flag. If service A will emit a new event field or call a new endpoint on service B, B must accept the new shape before A sends it. Deploy and enable the consumer side first, at 100%, while the producer flag stays off. Then ramp the producer. Reversing the order means a partial rollout sends requests that some replicas cannot parse.
Treat every flagged path as an expand-and-contract change. Additive schema changes, optional fields, and tolerant readers let old and new code run side by side. Until the flag is fully on and removed, the old path is still production code. Drop it only after cleanup.
Ramp in fixed steps, for example 1%, 10%, 50%, 100%. Gate each step on evaluation telemetry: the share of requests that actually resolved to the new variant in each service, plus error rates for that cohort. Divergent variant counts between services mean bucketing is broken, and you should stop there.
A kill switch counts only if you have exercised it. Flip it in staging under load, confirm every service falls back cleanly, and write down how long propagation takes.
| Change type | Cross-service flag? | Better tool |
|---|---|---|
| New behavior, compatible payloads | Yes | — |
| Additive schema field | Yes, consumer first | — |
| Irreversible data migration | No | Migration with backups and a forward-fix plan |
| Breaking contract change | No | Versioned API, deprecate the old version |
Flags assume you can turn them off. If turning one off would corrupt data or break a contract, the flag is not a real rollback path.
What are the alternatives to cross-service feature flags with OpenFeature?
Runtime flags are one way to separate deploying code from releasing behaviour. Depending on the change, these alternatives may be simpler or safer:
- Calling a flag backend's own SDK directly. This removes one abstraction layer and can expose backend-specific features. The cost is that every call site is tied to that backend, and each language may follow different conventions for context, defaults, and error handling. You also lose the standard hook stages unless you build an equivalent yourself.
- A homegrown flag service or shared config table. This works for a small number of boolean switches. Percentage rollouts, deterministic bucketing, caching, and audit history then become your team's code to write and maintain, and the consistency issues described earlier still apply.
- Static configuration and environment variables. Turning behaviour on means changing config and redeploying or restarting. This is predictable and needs no extra runtime dependency, but it cannot target individual users or tenants, and a rollback is as slow as a deploy.
- Infrastructure-level traffic splitting. Canary and blue-green deployments, using a load balancer or service mesh, route a share of traffic to a new version of a service. This tests a whole build rather than one feature, needs no code branches, and suits changes that are hard to isolate behind a flag. It is harder to keep one user on the same version across several services unless routing uses a sticky key consistently at every hop.
- Versioned APIs and coordinated deploys. For breaking contract changes, or services that always ship together, a new API version or a single coordinated release is clearer than a flag that cannot be safely turned off.
Teams often combine these: canary deployments to verify a build is healthy, and flags to control who sees a feature once the build is out.
How can a shared OpenFeature setup save engineering time?
The time savings come from removing repeated work and shortening debugging, not from evaluation itself:
- One integration instead of many. Provider setup, context construction, defaults, and telemetry are written once in the wrapper per language, rather than by each team.
- Shorter investigation of inconsistent behaviour. Because every service emits flag key, variant, and reason with the same attribute names, a single trace shows which service resolved which variant and why. Without that, engineers reconstruct the answer from differently formatted logs across services.
- Fewer release coordination meetings. Producers and consumers can deploy on their own schedules with code dark, then enable behaviour in dependency order.
- Faster rollback. Turning a flag off is a configuration change, not a revert, build, and redeploy across several services.
- Cheaper backend changes. Switching flag backends means changing provider configuration in the wrapper rather than editing every call site.
As an illustrative estimate with explicitly assumed inputs, not a measured result: assume 12 services, each team spending 2 days to integrate a flag SDK, define context, and add telemetry, compared with 6 days for a platform team to build the wrapper in 3 languages plus half a day per service to adopt it. Per-team integration totals 24 engineer-days; the shared approach totals 6 + 6 = 12 engineer-days. If you also assume 4 cross-service flag incidents a year, each taking 2 engineers a day to diagnose without consistent telemetry and half a day with it, that is another 6 engineer-days saved per year. Your inputs will differ, and the wrapper's ongoing maintenance and upgrade coordination, described above, should be counted against these savings.
How do you keep cross-service flags from turning into debt?
A flag read by one service is a branch you delete in one pull request. A flag read by six services means six removals, often owned by six teams with different release schedules. Until the last one ships, the flag cannot be archived in the provider. Every flag left live also doubles the number of possible behavior combinations, and most of those combinations are never exercised in staging.
Hygiene rules that hold up
- One named owner per flag. The owner should be a person or team, not "platform." That owner decides when the rollout is finished and tracks removal in every consuming service.
- Names that encode type and domain. A pattern such as
checkout.release.split-tax-calcorbilling.ops.disable-retriestells a reader who owns the flag and whether it is meant to be temporary. Release flags should be short-lived. Ops and permission flags can be long-lived. - Expiry dates on release flags. Store the date as flag metadata, and have CI or a scheduled job report flags that are past expiry, along with the repositories that still reference them.
- Delete dead paths, not just the flag. Removing the flag while leaving the
elsebranch in place leaves untested code that can still run.
When the overhead pays off
Cross-service flags are worth the cost when several teams ship independently, rollouts need percentage targeting, and a fast kill switch is valuable. If two services always deploy together, a coordinated deploy is simpler. If one service can decide and pass the result downstream, keep the flag there and nowhere else.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to feature flags microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.