Why do microservices return different feature flag values for the same user?

A feature flag is consistent when every service that evaluates it for the same request returns the same variant. Across services, that breaks in three ways:

Cause What diverges Typical symptom
Divergent evaluation context Missing targeting key, different attribute names or types Same user lands in different rollout buckets
Divergent flag definitions Stale caches, out-of-sync config files Services apply different rules
Evaluation timing Rules change mid-request during a rollout Early hops see the old rule, later hops see the new one

Consider a checkout request. The frontend evaluates new-pricing and gets on. The pricing service evaluates the same flag and gets off. The customer sees one price on the cart page and is charged another.

Each cause can produce that result:

  • Context. The frontend uses userId as the targeting key. The pricing service never receives it and buckets on a request ID. A rule matching plan == "pro" also fails when pricing reads plan as a numeric tier.
  • Definitions. The pricing service caches rules for minutes and still holds the 10% rollout after the frontend loaded 50%.
  • Timing. An operator widens the rollout between the frontend's evaluation and the pricing service's.

This guide is for backend and platform engineers running several services that share flags.

Installing the same SDK everywhere does not fix any of these problems. The SDK does not choose your targeting key, carry context between services or synchronize rules. Consistency needs three things:

  • a standard evaluation API,
  • a shared context that every service propagates, and
  • one source of truth for flag rules.

The rest of this guide builds each of them.

What is OpenFeature and how does its evaluation model work?

OpenFeature is a vendor-neutral specification and set of SDKs for flag evaluation. Application code calls the API to get a client, and asks it for typed values: boolean, string, number, or object. Every call passes a flag key, a default value defined in code, and an evaluation context. The SDK delegates the actual decision to a provider, which is the adapter for a specific backend such as flagd. Hooks run before and after evaluation, and on errors. Teams use them for logging, metrics, and context enrichment.

This split matters for polyglot systems. A Go service and a Python service both express a check as "evaluate new-checkout as a boolean, default false, for this context." Swapping the backend means changing the provider registration, not the call sites. The code default also gives every evaluation a defined fallback when the provider is unreachable or the flag is missing.

The evaluation context holds a targeting key, usually a stable user or account ID, plus attributes such as plan, region, or app version. In flagd, percentage rollouts are deterministic. The targeting key is hashed together with the flag key into a bucket, and the bucket maps to a variant. Any service that holds the same definition and receives the same key computes the same answer locally. No central call per request is needed, and services do not need to coordinate.

Keep three limits in mind:

  • The spec standardizes the API, not behavior. Provider features, hook support, context-propagation helpers, and SDK maturity differ by language.
  • Bucketing inputs are part of the contract. Changing the targeting key, renaming a flag, or altering the bucketing inputs reshuffles users in the middle of a rollout.
import { OpenFeature, Hook } from '@openfeature/server-sdk';
import { FlagdProvider } from '@openfeature/flagd-provider';

// Swapping backends means changing only this registration, not call sites.
await OpenFeature.setProviderAndWait(
  new FlagdProvider({ host: 'localhost', port: 8013 }),
);

// Hooks see every evaluation. Here they log the variant and reason.
const auditHook: Hook = {
  after(hookContext, details) {
    console.log(hookContext.flagKey, details.variant, details.reason);
  },
  error(hookContext, err) {
    console.warn(`flag ${hookContext.flagKey} fell back to default`, err);
  },
};

const client = OpenFeature.getClient('checkout-service');
client.addHooks(auditHook);

export async function useNewCheckout(userId: string, plan: string) {
  // targetingKey drives deterministic bucketing in flagd percentage rollouts.
  // Keep it stable (same ID format in every service) or users get reshuffled.
  const context = { targetingKey: userId, plan, region: 'eu-west' };

  // Typed call with a code default: `false` is returned if the provider
  // is unreachable or the flag does not exist.
  return client.getBooleanValue('new-checkout', false, context);
}

A TypeScript service registers the flagd provider, attaches a logging hook, and evaluates a boolean flag with a code default and an evaluation context keyed on a stable user ID.

How do you design and propagate a shared evaluation context across services?

Consistency starts with the evaluation context. If two services hand the provider different inputs, identical rules produce different answers. For a sticky rollout, the targeting key must be stable across the whole journey. Use a user or account ID. Do not use a session or request ID, because those values change and reshuffle users into different buckets.

Treat attribute names and types as a shared schema with an owner. Suppose one service sends plan: "pro" and another sends tier: "PRO". No error is raised in that case. The rule simply fails to match and the flag returns its default. Version the schema, review changes to it, and lint flag rules against it.

Where context is set

  • API-level (global) context: static facts such as service name, region and environment, set once at startup.
  • Per-request context: set at the call site as invocation context. Where the SDK supports it, set it once in transaction context so every evaluation in that request inherits it.

Propagation

Carry the targeting key and a few agreed attributes in request headers or W3C baggage, next to trace context. In receiving-side middleware, rebuild the evaluation context from those values before any handler runs.

Attribute Propagate? Reason
Targeting key (account ID) Yes, verified against auth token Required for consistent bucketing
Region, app version Yes Low risk, not personal
Plan or entitlement Re-derive server-side A client can spoof it
Email, IP address Re-derive or omit Personal data leaks into baggage and logs

Re-deriving attributes costs an extra lookup per hop. It also adds a risk: if the source of truth changes mid-request, a downstream service can see a different value from the one upstream evaluated against.

import express from 'express';
import { OpenFeature, AsyncLocalStorageTransactionContextPropagator, EvaluationContext } from '@openfeature/server-sdk';
import { FlagdProvider } from '@openfeature/flagd-provider';
import { context, propagation } from '@opentelemetry/api';
import { lookupPlan } from './entitlements'; // placeholder: reads plan from the source of truth

// API-level context: static facts, set once at startup.
OpenFeature.setProvider(new FlagdProvider({ host: 'localhost', port: 8013 }));
OpenFeature.setContext({ service: 'checkout', region: 'eu-west-1', environment: 'prod' });
OpenFeature.setTransactionContextPropagator(new AsyncLocalStorageTransactionContextPropagator());

const app = express();
// Assumes OTel instrumentation has already extracted W3C baggage, and that auth
// middleware has put the verified account ID in res.locals.accountId.
app.use(async (req, res, next) => {
  const bag = propagation.getBaggage(context.active());
  const claimedKey = bag?.getEntry('ff.targeting_key')?.value;
  const accountId = res.locals.accountId as string | undefined;
  // Never trust the propagated key on its own: it must match the auth token.
  if (claimedKey && claimedKey !== accountId) console.warn('targeting key mismatch, using token value');

  const evalCtx: EvaluationContext = {};
  if (accountId) {
    evalCtx.targetingKey = accountId;           // stable ID keeps bucketing sticky
    evalCtx.plan = await lookupPlan(accountId); // spoofable, so re-derive server-side
  }
  const appVersion = bag?.getEntry('ff.app_version')?.value;
  if (appVersion) evalCtx.appVersion = appVersion; // low risk, safe to propagate
  // Email and IP are deliberately not read from baggage.

  // Every evaluation inside this request inherits evalCtx.
  OpenFeature.setTransactionContext(evalCtx, () => next());
});

app.get('/checkout', async (_req, res) => {
  const enabled = await OpenFeature.getClient().getBooleanValue('new-checkout', false);
  res.json({ newCheckout: enabled });
});

app.listen(3000);

Express middleware rebuilds the OpenFeature evaluation context from W3C baggage. It verifies the targeting key against the auth token and re-derives the spoofable plan attribute, then uses transaction context so every flag evaluation in the request gets the same inputs.

Which flagd deployment topology should you use for distributed feature flags?

The same staleness risk shows up at the infrastructure layer. Where flagd runs determines how many copies of your flag definitions exist at any moment, and each copy is a chance for services to disagree.

flowchart LR
    S[(Sync source:<br/>Git repo / config store)]
    S --> C[Central flagd service]
    S --> SC1[flagd sidecar, pod A]
    S --> SC2[flagd sidecar, pod B]
    S --> IP[In-process provider<br/>in service C]
    C -- network call per evaluation --> SVC1[Service A, Service B]
    SC1 -- localhost --> A[Service A]
    SC2 -- localhost --> B[Service B]
Topology Consistency Evaluation cost Failure impact
Central flagd One revision at any instant Network round trip per evaluation Outage or slowness reaches every caller
Sidecar per pod Eventually consistent across pods Local call, no cross-host hop Contained to one pod
In-process Eventually consistent across instances Function call, no extra process Contained to one instance

A central service gives you one point of truth: two services asking the same question at the same moment get the same answer. The cost is that every evaluation now waits on the network, and flagd becomes a dependency in the request path. Its latency adds to yours, and its outage triggers your fallback defaults.

Sidecars and in-process providers remove that hop. Each one pulls definitions from the sync source on its own schedule, so while a change propagates, instances can briefly hold different revisions. Rule changes become eventually consistent, and a single request crossing several services can see a mix of old and new definitions.

There is also operational cost. A central deployment is one more service to scale, make highly available, authenticate, and upgrade. Sidecars multiply that work across every pod and add resource overhead to each one. In-process evaluation adds no extra process, but each service's SDK and provider must be upgraded on its own release cycle.

+--------------+
| Sync source  |
| Git repo /   |
| config store |
+--------------+
  |
  |   CENTRAL: one copy, network hop per evaluation
  +--->+---------------+   remote call   +-----------+
  |    | Central flagd |<----------------| Service A |
  |    +---------------+<----------------| Service B |
  |                                      +-----------+
  |   SIDECAR: one copy per pod, localhost call
  +--->+---------------------------------------------+
  |    | Pod A   +-------+  localhost  +-----------+ |
  |    |         | flagd |<------------| Service A | |
  |    |         +-------+             +-----------+ |
  |    +---------------------------------------------+
  +--->+---------------------------------------------+
  |    | Pod B   +-------+  localhost  +-----------+ |
  |    |         | flagd |<------------| Service B | |
  |    |         +-------+             +-----------+ |
  |    +---------------------------------------------+
  |   IN-PROCESS: one copy per instance, function call
  +--->+---------------------------------------------+
       | Service C process                           |
       |  +-----------------+  fn call  +--------+   |
       |  | provider: sync  |<----------| app    |   |
       |  | + local eval    |           | code   |   |
       |  +-----------------+           +--------+   |
       +---------------------------------------------+

One sync source feeds a central flagd service, per-pod flagd sidecars, and an in-process provider; each arrow from the source is a separate copy of the flag definitions that can briefly lag behind the others.

How do you keep flag definitions in a single source of truth?

Treat flag definitions like any other production configuration. Keep the flagd JSON files in one repository, change them only through pull requests, and require review from the team that owns the affected flag. A flag such as checkout-redesign should show its full decision logic in one file: its state, its variants, a defaultVariant, and a targeting rule that uses flagd's fractional operation to split users. For example, the rule might send 10% of users to on and 90% to off, bucketed on targetingKey. A reviewer can see the exact split and the bucketing key before the change merges. Reviewing a vague dashboard toggle gives them much less to check.

In CI, validate every definition file against the published flagd JSON schema with a standard JSON Schema validator, and fail the build on any error. A typo in an operator name or a variant that doesn't exist then fails in CI, before a single instance loads it.

Publish validated definitions to exactly one sync source, such as a file mount, a ConfigMap, or an HTTP endpoint. Every flagd instance and in-process provider should read from it. If you have two sources, you have two possible truths.

Tag each published artifact with its revision, typically the commit SHA, for example in a ConfigMap label or the object name. Have services log that revision on startup and on every sync. Also export it as a metric label, for instance a gauge set to 1 with the SHA as a label. When two services disagree, comparing revisions takes seconds and tells you whether you are looking at drift or a logic bug.

#!/usr/bin/env bash
# CI step: validate flagd definitions, then publish them to the single sync source
# (a ConfigMap) tagged with the commit SHA so every instance can report its revision.
set -euo pipefail

# Example definition under review (flags/checkout.json):
# {
#   "flags": {
#     "checkout-redesign": {
#       "state": "ENABLED",
#       "variants": { "on": true, "off": false },
#       "defaultVariant": "off",
#       "targeting": {
#         "fractional": [
#           { "var": "targetingKey" },
#           ["on", 10],
#           ["off", 90]
#         ]
#       }
#     }
#   }
# }

# 1. Fail the build on any schema error (bad operator, malformed variant, etc.).
#    The flagd schema is vendored into the repo so CI does not depend on the network.
check-jsonschema --schemafile schemas/flagd-flags.json flags/*.json

# 2. The commit SHA is the revision every service will log and export as a metric label.
REVISION="$(git rev-parse --short HEAD)"

# 3. Publish to exactly one sync source: one ConfigMap that all flagd instances mount.
kubectl create configmap flag-definitions \
  --from-file=flags/ \
  --dry-run=client -o yaml | kubectl apply -f -

# 4. Tag the published artifact so drift is a one-command comparison.
kubectl label configmap flag-definitions flags-revision="${REVISION}" --overwrite
echo "published flag definitions at revision ${REVISION}"

A CI step that schema-validates a flagd definition with a 10/90 fractional rollout, publishes it to one ConfigMap, and labels it with the commit SHA so instance revisions can be compared.

When should you evaluate once and propagate the decision instead of re-evaluating?

Shared context and synchronized rules make agreement likely, not guaranteed. A flag can change between the moment the checkout service evaluates it and the moment the payment service does, and a single request can straddle that change. When a flag gates a contract between services, such as a new payload shape or a pricing rule, that small window is enough to cause a failure.

For those flags, evaluate once at the edge and pass the result downstream, for example as a header or a message field. Every hop sees the same answer because nobody asks the question twice. This has costs. Downstream services now depend on an upstream contract. The flag disappears from the owning team's view, so they can't target it themselves. Async consumers act on a decision that may have been reversed hours ago. If a flag was turned off because of an incident, a queue of messages still carrying "on" will keep the incident going.

A useful rule: propagate when the decision describes the shape of work already started, and re-evaluate when it describes current policy, such as kill switches, rate limits, or rollout percentages.

Approach Use when Main cost
Per-service re-evaluation Flags are local to one service, or must react to changes immediately Can briefly disagree mid-request
Decision propagation Several hops must agree within one request or workflow Coupling, hidden flags, stale decisions in async work
Central evaluation service You need one audited answer and can afford a network hop Latency and a shared failure point
Config or environment toggle The value changes only at deploy time Changing it requires a redeploy

How do you detect flag disagreement and fail safely?

Disagreement is only visible if every evaluation leaves a record that can be compared across a trace. Write an OpenFeature hook that runs in the after and error stages and adds the flag key, variant, reason, and targeting key to the active span. If the targeting key is a user ID or email, hash it so traces can be correlated without exposing the raw identifier. When two services in the same trace report different variants for the same key and targeting key, the trace shows it directly.

Use the before stage to check that required context attributes, such as tenant or region, are present. A missing attribute does not raise an error. It silently skips targeting rules, so the hook should log it and increment a counter.

Define failure behavior on purpose:

  • Code defaults must be safe when the provider is unreachable or not yet initialized. That usually means the existing behavior, not the new feature.
  • Startup readiness: wait for the provider to become ready before serving traffic, or treat PROVIDER_NOT_READY evaluations as expected and bounded.
  • Sync outages: decide whether services keep serving the last synced definitions when the sync source is down, and confirm what your flagd mode actually does. Watch for stale or error provider events.

Fallback defaults hide outages. If one service loses its provider while the others stay healthy, it returns defaults while its peers return targeted variants. That produces the cross-service inconsistency this setup is meant to prevent, and it is invisible unless you monitor it. Alert on the rate of evaluations whose reason is DEFAULT or ERROR, broken down by service.

When should you avoid OpenFeature and flagd, and what do they cost?

The machinery described so far has real costs, and for some systems it is the wrong trade.

Skip it if you run a few services with simple on/off flags that only change at deploy time. In that case flagd plus a sync pipeline adds a process to operate and a new failure mode to monitor, and it doesn't fix any problem you currently have. Environment variables or a static config file shipped with each release are consistent by construction, because every replica reads the same value.

Concern Env vars / static config OpenFeature + flagd
Per-user targeting No Yes
Change without redeploy No Yes
Cross-service consistency Only per release Shared definitions and context
Operational surface None added flagd, sync source, middleware

The development cost falls into four areas:

  • Infrastructure: flagd instances and their sync source need deployment, upgrades and health checks.
  • Schema governance: teams must agree on targeting keys and attribute names, and they must keep agreeing as services change.
  • Propagation middleware: every language and transport needs its own propagation code.
  • Uneven SDKs: feature support varies across languages in a polyglot fleet, so expect gaps.

The return is engineering time saved in three ways. You get fewer bugs where services disagree on a flag. Debugging is faster because evaluations show up in traces with reason codes. And you can switch flag backends behind a standard API without rewriting call sites.

Incremental adoption checklist

  • Pick one flag shared by two services on one request path.
  • Define the targeting key and a minimal context schema in a shared repository.
  • Run flagd with a single file-based source before adding other sources.
  • Add context propagation and an evaluation hook that emits spans.
  • Expand to more services only after DEFAULT/ERROR alerting is live.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to openfeature microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.