What is reference data in microservices, and why does its classification matter?

Reference data is the small, slowly changing set of values that other data points at: ISO country codes, currency codes, product categories, tax classes, order status enumerations. It typically has hundreds or thousands of rows rather than millions. It changes on a scale of weeks or months, and almost every service reads it. Its job is to constrain and label other records. An order row stores currency = "EUR", and the reference table says what EUR means, how many minor units it has, and whether it is still accepted.

It is easy to confuse with neighbouring categories, but they behave differently:

Property Reference data Master data Transactional data
Examples Currencies, tax classes, statuses Customers, products, suppliers Orders, payments, shipments
Volume Small, bounded Medium to large, growing Large, unbounded
Change rate Rare, often scheduled Frequent, per entity Constant, append-heavy
Readers Nearly every service Several domains Mostly the owning service
Tolerance for staleness Usually high, with limits Moderate Low

The classification matters because it determines your options. A dataset that is small, rarely written and broadly read can be copied in full into every consumer and refreshed through events. Each service can then join locally without a network call. Master data is too large and too volatile for that approach. Transactional data should not leave its owner at all.

The category also sets your consistency budget. A one-minute lag on a new product category is harmless. A one-minute lag on a tax rate change at midnight may not be. If your service joins against shared lookup tables, you need to know which kind you are holding before you choose how to replicate it.

Who should use replicated reference data?

The pattern in this article fits teams that already run several independently deployed services and an event broker, and where several of these conditions hold:

  • Many services read the same lookup tables. If three or more services join against currencies, countries or tax classes, the coupling costs described below start to compound.
  • Reads are on hot paths. Order placement, pricing and checkout should not wait on a network call to translate a code into a label.
  • Different teams own producers and consumers. When the team that maintains a lookup table cannot coordinate every deploy with its readers, a versioned event contract replaces that coordination.
  • The data tolerates seconds to minutes of lag. Most reference data does, especially when future changes are published ahead of time with an effective_from date.

It is a poor fit for a single service or a small monolith, where a plain table and a foreign key are simpler. It is also a poor fit for teams without an event broker or the capacity to operate consumers, monitor lag and run rebuilds. The last section covers specific data shapes where replication is the wrong tool.

Why do shared tables, synchronous lookups and shared libraries create coupling?

Before replicating anything, it helps to see why the three usual shortcuts break down. Each one works with two consumers and gets harder with every consumer you add.

A shared table or schema feels like the simplest option: one currencies table, many readers. But each reader now depends on the column names, types and constraints. Renaming a column or splitting a table means finding every service that queries it, changing them in lockstep and deploying them in a careful order. The schema has become a public API that nobody versioned.

A synchronous lookup avoids schema sharing by putting a service in front of the data. The cost moves to runtime. If an order request triggers five lookups, its latency includes five network round trips. Its availability is also capped by the product of each dependency's availability. When the reference service slows down, every caller's thread pool fills up, and the failure spreads to services that only needed a country name.

A shared library with static data removes the network call. It ties data changes to code releases. Adding a new tax code means publishing a new library version and then redeploying every consumer. Until that finishes, different services hold different versions of the data.

Approach Coupling type Hidden cost as consumers grow
Shared table/schema Schema and deployment Coordinated migrations across teams
Synchronous lookup Runtime and latency Cascading outages, amplified tail latency
Shared library Release Fleet-wide redeploys for data changes

What are the alternatives to event-carried replication?

The three approaches above are the common alternatives, and each remains a reasonable choice in narrow cases. A few more options sit between them and full event-carried replication:

  • Constants in code. For a tiny enumeration that only changes alongside code, such as a fixed set of internal states, a constant or enum is simpler than any replication. The release coupling is acceptable because a change needs a deploy anyway.
  • Synchronous lookup with a local cache. Caching responses with a time-to-live removes most round trips and softens owner outages. It still leaves cold-start and cache-miss paths dependent on the owner, and the TTL bounds staleness without telling you when data actually changed.
  • Change data capture from the owner's database. Streaming row changes from the owner's transaction log produces events without changing application code. The risk is that the event shape becomes the owner's internal table schema. If you use it, put a mapping step in front of consumers so that the published contract stays stable when the owner's schema changes.
  • Periodic bulk export. The owner publishes a full snapshot on a schedule, and consumers reload it. This is simple and easy to reason about, but freshness is bounded by the schedule, and every consumer reloads the whole table even when one row changed.
  • Event-carried state transfer. The owner publishes the full current row on every change, and consumers keep a local read-only copy. This is the approach the rest of the article describes. It costs more to operate than the options above, and it removes runtime, schema and release coupling.

How does reference data replication work, end to end?

At a high level, the flow has five steps:

  1. One owner writes. A single service holds the system of record for a dataset and is the only place where changes are made.
  2. The owner publishes full-state events. Every insert, update or retirement produces an event containing the complete current row, a per-entity version and validity dates. The events go to a topic keyed by entity ID.
  3. Consumers project events into local tables. Each consumer runs a handler that upserts events into its own read-only table, applying only versions newer than the one it holds.
  4. Consumers read locally. Application code joins against the local table with no network call to the owner.
  5. New or broken consumers bootstrap and reconcile. A new consumer loads the current state from a compacted topic or a snapshot. Periodic reconciliation checks that each replica still matches the owner.

The following sections cover each step in detail: event design and consumer tables, bootstrap and ordering, versioning and temporal validity, and drift detection.

How does event-carried state transfer replicate lookup tables?

Event-carried state transfer starts with ownership. Each reference dataset has exactly one system of record: the service that exposes the write API, validates changes and publishes them. No other service writes to that data. If a consumer finds a wrong tax rate or a misspelled country name, the fix goes through the owner's API. Consumers never patch their copy. A local edit would be silently overwritten by the next event, or it would persist and diverge from every other replica.

The owner publishes an event for every change, and the event carries the full current row: the key, all attributes, a per-entity version number, and the effective_from and effective_to timestamps. A thin "row 42 changed" notification is not enough, because every consumer then has to call the owner to fetch the new state. That brings back the synchronous dependency the replica was meant to remove, and a single change becomes a burst of callbacks to the owner. A full-state event is self-sufficient. Applying it requires nothing beyond the event itself.

Each consumer upserts these events into a local table that its own code treats as read-only. The consumer chooses the shape: it can drop unused attributes, denormalise columns into the tables it joins against, or add indexes that suit its own queries. The owner's schema is a contract on the event, not on how consumers store it.

Ownership should follow business domains. Currency codes belong to finance, product categories to catalogue. A single catch-all "reference data service" tends to turn into a dumping ground for loosely related tables, and every team's change ends up waiting in that service's queue.

-- Consumer side (PostgreSQL): a local, read-only replica of finance's currency data.
-- Example event from the owner, which carries the full current row:
-- {"currency_code":"EUR","name":"Euro","minor_units":2,"numeric_code":"978",
--  "version":7,"effective_from":"2026-01-01T00:00:00Z","effective_to":null}

-- The consumer chooses its own shape. It keeps only the columns it queries
-- and drops numeric_code because it never uses it.
CREATE TABLE currency_replica (
  currency_code  char(3)     PRIMARY KEY,
  name           text        NOT NULL,
  minor_units    smallint    NOT NULL,
  version        bigint      NOT NULL,  -- per-entity version from the owner
  effective_from timestamptz NOT NULL,
  effective_to   timestamptz             -- NULL means open-ended
);

-- Index chosen for this consumer's own lookups ("currencies valid at time T").
CREATE INDEX currency_replica_validity
  ON currency_replica (effective_from, effective_to);

-- Application code may only read. Only the event handler's role can write.
REVOKE INSERT, UPDATE, DELETE ON currency_replica FROM orders_app;
GRANT SELECT ON currency_replica TO orders_app;

-- The event handler runs this upsert for each event. It needs nothing beyond
-- the event payload and makes no callback to the owner.
INSERT INTO currency_replica AS r
  (currency_code, name, minor_units, version, effective_from, effective_to)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (currency_code) DO UPDATE
  SET name           = EXCLUDED.name,
      minor_units    = EXCLUDED.minor_units,
      version        = EXCLUDED.version,
      effective_from = EXCLUDED.effective_from,
      effective_to   = EXCLUDED.effective_to
  -- Apply only newer versions, so redelivered or out-of-order events are no-ops.
  WHERE r.version < EXCLUDED.version;

A consumer uses PostgreSQL to keep its own read-only copy of an owner's currency data, shaped for its own queries, and upserts each full-state event with a version check so that duplicate or older events are ignored.

How do you bootstrap, order and recover a reference data replica?

The simplest bootstrap is a compacted log keyed by entity ID. Compaction keeps the latest record per key, so a new consumer that reads from the earliest offset ends up with the full current table, without needing a separate export path. If the topic is not compacted, or the table is too large to replay, load a snapshot or bulk export that records the log position it reflects, then replay the stream from that position.

Reconciling snapshot and stream

The snapshot and the stream will overlap. Some records will appear in both. Give every entity a monotonic version, and apply each record with a conditional upsert that writes only when the incoming version is higher than the stored one. This makes the overlap harmless. Replaying a duplicate does nothing, and starting slightly early never leaves a gap. The same rule handles ordering. Most brokers guarantee order only within a partition, so events for different keys can interleave, and redeliveries can bring old versions back. A consumer that discards anything older than what it already holds does not depend on delivery order.

Deletes

Do not hard-delete rows in the replica. Orders and invoices that referenced the old key will fail to resolve in historical joins. Instead, mark the row inactive with a soft-delete flag and set an effective-to date. On a compacted log, publish a tombstone as well.

Edge cases

  • Tombstone retention: Tombstones are themselves removed after a retention window. A consumer that is offline longer than that window never sees the delete.
  • Detecting missed deletes: After a long outage, re-bootstrap the replica or diff it against a fresh snapshot.
  • Compaction timing: Compaction runs asynchronously, so a bootstrapping consumer may still see several versions of the same key.
  • Snapshot consistency: Take the snapshot from a single consistent read, such as one transaction, not a series of paginated queries. Otherwise the recorded log position does not match the data.
-- Replica table: version is the per-key monotonic version from the source of truth.
CREATE TABLE currency_ref (
  code         text PRIMARY KEY,
  name         text NOT NULL,
  version      bigint NOT NULL,
  is_active    boolean NOT NULL DEFAULT true,
  effective_to timestamptz          -- NULL while the row is current
);

-- Apply an upsert from either the snapshot or the stream.
-- The WHERE clause skips duplicates and stale or out-of-order versions,
-- so snapshot and stream overlap is harmless.
INSERT INTO currency_ref (code, name, version, is_active, effective_to)
VALUES ($1, $2, $3, true, NULL)
ON CONFLICT (code) DO UPDATE
  SET name         = EXCLUDED.name,
      version      = EXCLUDED.version,
      is_active    = EXCLUDED.is_active,
      effective_to = EXCLUDED.effective_to
  WHERE currency_ref.version < EXCLUDED.version;

-- Apply a delete event as a soft delete, never a hard DELETE,
-- so historical joins on old orders and invoices still resolve.
-- $3 is the event's own timestamp, not now(), so replays give the same result.
UPDATE currency_ref
   SET is_active    = false,
       effective_to = $3,
       version      = $2
 WHERE code = $1
   AND version < $2;  -- ignore a delete older than the state already held

-- Record the consumed log position in the same transaction as the writes,
-- so a restart resumes exactly where the applied state ends.
INSERT INTO replica_offsets (topic, part, next_offset)
VALUES ('currency-ref', $4, $5)
ON CONFLICT (topic, part) DO UPDATE SET next_offset = EXCLUDED.next_offset;

This PostgreSQL example applies reference data idempotently. It uses a version-guarded conditional upsert, a version-guarded soft delete with an effective-to date, and an offset checkpoint committed in the same transaction (it assumes a replica_offsets table keyed by topic and partition).

How should you version reference events and handle temporal validity?

The reference event schema is a public contract. It should not mirror the owner's internal table. If the owner splits a column, renames a field or normalizes a join, consumers should not notice. Map internal rows to a stable event shape at the publishing boundary, and evolve that shape on its own schedule.

Within that contract, prefer additive changes. Adding an optional field or a new code is safe. Renaming, removing or changing the meaning of a field requires a new event version that runs in parallel with the old one. Never reuse a code. If GB-SCT once meant one tax region, assigning it to another silently corrupts every historical record that references it. Retired codes stay in the stream marked as deprecated, so old transactions can still be resolved.

Consumers must read tolerantly. They should ignore fields they don't recognize. They should store unknown codes rather than reject them, so the owner can add values without coordinating a release.

Treat time as part of the data. Give each entry an effective_from and an optional effective_to, so consumers can answer "what was valid on this date?" and the owner can publish future changes ahead of time.

Transactions should record either the reference key plus its version or the denormalized value they used. Recording only the key means history shifts whenever the row changes. Denormalized values keep history accurate, but they do not pick up corrections. When the owner fixes a wrong value, someone must decide whether the fix applies retroactively and, if it does, run a deliberate backfill.

from dataclasses import dataclass
from datetime import date

@dataclass(frozen=True)
class TaxRegion:
    code: str
    version: int
    rate: str                  # decimal string, avoids float rounding
    effective_from: date
    effective_to: date | None  # exclusive end; None means open-ended
    deprecated: bool = False   # retired codes stay resolvable

def parse_event(evt: dict) -> TaxRegion:
    # Tolerant reader: read only known fields and ignore any new ones.
    # Codes are not checked against a fixed list, so new codes are stored, not rejected.
    end = evt.get("effective_to")
    return TaxRegion(
        code=evt["code"],
        version=evt["version"],
        rate=evt["rate"],
        effective_from=date.fromisoformat(evt["effective_from"]),
        effective_to=date.fromisoformat(end) if end else None,
        deprecated=evt.get("deprecated", False),
    )

def valid_on(entries: list[TaxRegion], code: str, on: date) -> TaxRegion | None:
    # Answers "what was valid on this date?" and also matches future-dated entries.
    for e in entries:
        if e.code == code and e.effective_from <= on and (e.effective_to is None or on < e.effective_to):
            return e
    return None

def record_sale(entries: list[TaxRegion], code: str, sold_on: date, amount: str) -> dict:
    ref = valid_on(entries, code, sold_on)
    if ref is None:
        raise LookupError(f"no {code} entry valid on {sold_on}")
    # Store key + version and the value actually used. Later corrections to the
    # rate do NOT flow into this record; applying a fix retroactively is a deliberate backfill.
    return {"amount": amount, "sold_on": sold_on.isoformat(), "tax_region": ref.code,
            "tax_region_version": ref.version, "tax_rate_applied": ref.rate}

A consumer reads reference events tolerantly, resolves the entry that was valid on a given date, and stamps each transaction with the reference key, its version and the denormalized rate it used.

What happens when a consumer sees an unknown key, and how do you detect drift?

Replication is eventually consistent, so a request can carry a code the owner published seconds ago that the replica has not yet applied. Choose the policy per use case before production chooses it for you.

Policy Behaviour Cost
Reject Return a validation error; the caller retries False rejections during lag; acceptable when retries are cheap
Park and retry Move the message to a delay queue and reprocess later Added latency; needs a retry limit and dead-letter queue so a truly invalid key does not loop forever
Accept raw key, enrich later Store the code now, fill in descriptive fields when the replica catches up Readers must tolerate incomplete rows; you need a backfill job
Synchronous fallback Call the owner for the missing key only Reintroduces runtime coupling on that path; rate-limit it and fail closed if the owner is down

Making staleness visible

Track three signals per table. The first is replica lag, measured as the time between the owner's publish timestamp and the consumer's apply time. The second is the last-applied version, compared against the owner's latest. The third is a counter of unknown-key lookups. Read them together. Unknown keys rising while lag stays flat usually means bad input. Unknown keys rising alongside lag means the consumer is stuck.

Reconciliation

These metrics show that the replica is moving. They do not show that it is correct. A skipped poison message or a field-mapping bug lets the replica diverge silently, and the first symptom is wrong business output. Run a periodic job in which the owner exposes a row count and a checksum over sorted keys, values and versions, and the consumer computes the same values locally. On a mismatch, alert, diff per key, and repair or rebuild from a snapshot.

import hashlib
import json


def table_digest(rows):
    """Row count and SHA-256 over rows sorted by key.
    rows: list of (key, value_dict, version). Owner and consumer must
    run this same function so the serialization matches byte for byte."""
    h = hashlib.sha256()
    for key, value, version in sorted(rows, key=lambda r: r[0]):
        # Canonical JSON: sorted fields, no whitespace, so it is deterministic
        line = json.dumps([key, value, version], sort_keys=True, separators=(",", ":"))
        h.update(line.encode("utf-8") + b"\n")
    return len(rows), h.hexdigest()


def reconcile(table, owner_summary, local_rows, fetch_owner_rows, alert):
    """Compare cheap summaries first. Fetch full rows only on a mismatch."""
    count, checksum = table_digest(local_rows)
    if count == owner_summary["count"] and checksum == owner_summary["checksum"]:
        return []  # replica matches owner; nothing to repair

    alert(f"{table}: replica drift (local count={count}, owner count={owner_summary['count']})")

    # Per-key diff tells you whether to repair individual rows or rebuild the table
    owner = {k: (v, ver) for k, v, ver in fetch_owner_rows(table)}
    local = {k: (v, ver) for k, v, ver in local_rows}
    diffs = []
    for key in owner.keys() - local.keys():
        diffs.append(("missing", key))   # e.g. a skipped poison message
    for key in local.keys() - owner.keys():
        diffs.append(("extra", key))     # e.g. a missed delete
    for key in owner.keys() & local.keys():
        if owner[key] != local[key]:
            diffs.append(("changed", key))  # e.g. a field-mapping bug
    return diffs

A reconciliation check in which the consumer computes a row count and checksum over sorted keys, values and versions, compares them with the owner's summary, and produces a per-key diff only when they disagree.

When should you avoid replicating reference data, and what does it cost?

Replication is the default, not a rule. Some data is better kept in code or behind a synchronous call.

Data shape Better fit Why
Tiny, truly static enumeration (days of week, a fixed status set) Constant in code A change already requires a deploy, so a projector adds nothing
Must reflect the owner's latest write (read-your-writes) Synchronous call with cache, authorization and a defined fallback A replica is stale by design
Too large to copy, or under regulatory or residency constraints Synchronous call scoped to the rows needed Copying it multiplies storage and compliance scope
Everything else Event-carried replica Consumers read locally and fail independently

Replication has a running cost. Every consumer runs a projector, keeps a local table, monitors lag and maintains a rebuild procedure.

Illustrative estimate (assumed numbers, not a benchmark). Assume 50,000 rows at 1 KB each, 2,000 changes per day, 1 KB per event and 10 consumers.

  • Storage = rows × row size × consumers = 50,000 × 1 KB × 10 ≈ 500 MB
  • Daily fan-out = changes × event size × consumers = 2,000 × 1 KB × 10 ≈ 20 MB/day
  • Full rebuild = rows × row size ≈ 50 MB per consumer

Substitute your own inputs. At numbers like these, storage and bandwidth are rarely the constraint. Most of the cost is operational.

That cost buys engineering time back. The owner can change its schema without a cross-team migration. An outage in the owner does not take down consumers' hot paths. Nobody waits on a shared-library release train.

Choose synchronous access or a shared table only after you have measured that replica lag is unacceptable for a specific read path. "It might be stale" is not enough reason on its own.

How can replicated reference data save engineering time?

Most of the savings come from removing coordination between teams, which is usually more expensive than writing code:

  • Schema changes stay local. With a shared table, renaming a column means finding every reader, changing them together and sequencing their deploys. With a stable event contract, the owner changes its internal schema and updates one mapping at the publishing boundary. Consumers do nothing.
  • Data changes stop requiring deploys. With a shared library, a new tax code means a library release and a redeploy of every consumer. With replication, the owner publishes an event and every replica picks it up. Future-dated entries let the owner publish scheduled changes in advance, instead of coordinating a release at the moment the change takes effect.
  • Fewer cross-service incidents. Consumers read locally, so a slow or failed owner does not exhaust their thread pools. On-call engineers spend less time tracing cascading failures back to a lookup service.
  • Faster onboarding of new consumers. A new service bootstraps from the compacted topic or a snapshot and owns its local table shape. It does not need the owner team to add endpoints, raise rate limits or add indexes for its queries.
  • Cheaper debugging. Version-guarded upserts make replays safe, offset checkpoints make restarts predictable, and reconciliation produces a per-key diff. Recovering from a bad deploy becomes "rebuild the replica", not a manual data repair.

The trade is a fixed operational investment per consumer: a projector, lag and unknown-key metrics, and a rebuild procedure. This is worth making when many services read the same tables and are owned by different teams. With one or two consumers owned by the same team, the coordination costs above are small, and a shared table or a cached synchronous call may cost less time overall.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reference data microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.