Monolith Caching Strategies: In-Process vs Shared Cache

In-process vs shared cache: how each one works

Teams add a cache to a monolith for two reasons: to take repeated reads off the database and to return responses faster than a query can. The choice is where the cached data lives, and that choice determines most of the trade-offs that follow.

An in-process cache keeps objects in the application's own heap, usually in a bounded map with an eviction policy such as LRU. A lookup is a memory access. Nothing is serialized and nothing crosses the network, so the caller gets back the same object the loader produced. The cost is isolation. Each instance has its own copy, and that copy is lost when the process restarts.

A shared cache such as Redis runs as a separate service. Every application instance reads and writes the same keyspace over the network. Each access pays a network round trip, and values must be serialized on write and deserialized on read. In return, all instances see the same entries, and the cache survives application deploys.

What horizontal scaling does to hit rates and backend load

Each instance builds its own cache from nothing. One process that has learned a hot key does not help its neighbours, so every instance pays the miss separately. This has two effects. First, the effective hit rate drops, because a load balancer spreads requests for the same key across instances. Second, backend load scales with instance count whenever caches are cold, which happens on every deploy, autoscaling event and crash restart.

Illustrative estimate

The following inputs are assumed for this example:

• 10 instances behind round-robin load balancing
• 50,000 hot keys
• 20 ms of database time per cache miss
• Each key requested 10 times per TTL window

With in-process caching, a key requested 10 times per TTL reaches each instance about once. Almost every request is a miss. A rolling deploy also clears all 10 caches within minutes, so the database absorbs up to 10× the warm-up load while serving live traffic.

A shared cache keeps one copy of each key, so all instances see the same hit rate. The cost is a network round trip per read. The shift is justified once you run several instances and either the hit rate falls as you add instances, or deploys produce database load spikes, or instances must agree on the same cached value. A single instance, or data that rarely changes, does not need it yet.

Keeping caches consistent: invalidation options and their limits

When one instance handles a write, it can update or evict its own in-process entry. The other instances keep serving their copies until those expire. With a load balancer spreading requests, a user who just changed their display name can see the new value on one request and the old one on the next. A shared cache has no such divergence, because there is only one copy. In-process caches need an explicit plan for this.

Short TTLs bound how long drift can last, with no coordination between instances. The cost is a lower hit rate and more backend reads. A TTL short enough to hide drift from users may also be too short to absorb much load.

Pub/sub invalidation broadcasts "key X changed" so every instance evicts its copy. It narrows the inconsistency window, but most brokers deliver messages to whoever is subscribed at that moment. An instance that is restarting, reconnecting, or briefly partitioned misses the message and keeps its stale entry. Treat broadcasts as an accelerator, and keep a TTL on every entry as the guarantee that stale data eventually ages out.

Versioned keys move the source of truth to a small counter stored in the shared cache or the database. Readers build keys such as user:42:v7, and a write increments the version. Old entries are never read again and expire on their own. The trade-off is one extra lookup per read to fetch the current version.

Accepting bounded staleness is often the right choice for feature flags, product catalogs, or aggregate counts. State the bound explicitly for each data class, for example "up to 60 seconds old," and set that class's TTL to match.

Copyimport time
import redis

# Version counters live in the shared cache, so every instance agrees on them.
r = redis.Redis(host="cache.internal", port=6379, decode_responses=True)

LOCAL_TTL_SECONDS = 300  # backstop: entries age out even if never superseded
_local = {}  # in-process cache: versioned key -> (expires_at, value)


def _versioned_key(user_id: int) -> str:
    # One extra round trip per read: fetch the current version.
    version = r.get(f"user:{user_id}:version") or "0"
    return f"user:{user_id}:v{version}"


def get_user(user_id: int) -> dict:
    key = _versioned_key(user_id)
    hit = _local.get(key)
    if hit and hit[0] > time.monotonic():
        return hit[1]
    value = db_load_user(user_id)  # placeholder for the real database read
    _local[key] = (time.monotonic() + LOCAL_TTL_SECONDS, value)
    return value


def update_user(user_id: int, fields: dict) -> None:
    db_save_user(user_id, fields)  # placeholder: commit to the database first
    # Bumping the version invalidates every instance's copy at once:
    # readers now build user:<id>:v<n+1>, and old keys are never read again.
    r.incr(f"user:{user_id}:version")
    # Old entries stay in each process's dict until removed; a real cache
    # needs a size bound or periodic sweep of expired entries.

An in-process cache keyed by a version counter held in Redis, where a write increments the counter so every instance stops reading the old entry, with a local TTL as the backstop.

Preventing cache stampedes

A stampede happens when many requests miss on the same key at once and all go to the backend to rebuild it. There are three common triggers. A hot key expires under load. A cold start or deploy brings up instances with empty in-process caches. Many keys written together share one TTL and expire together.

Per-key single-flight (request coalescing) inside each process lets the first miss on a key run the loader while concurrent callers wait on that same result. This limits concurrent rebuilds of a key to one per instance. Across a fleet of N instances, though, you can still get N rebuilds at once. Waiters also inherit the leader's latency and errors, so give the wait a timeout.

Lease-based locking in the shared cache coordinates the whole fleet. One caller gets a lock key with a short expiry and rebuilds. The others briefly serve stale data or retry. The expiry matters. Without it, a holder that crashes leaves the lock in place indefinitely and nobody refreshes the key. Set the lease above typical rebuild time but well below the TTL.

TTL jitter adds a random offset to each expiry, for example ±10% of the base TTL. This prevents keys written in the same batch from expiring together. The cost is less predictable freshness.

Stale-while-revalidate and early probabilistic refresh start a refresh before expiry, either in the background or on a request chosen at random as expiry approaches. Callers rarely block. In exchange, they deliberately receive stale data within the bound you defined above.

Copyimport asyncio, random, time

BASE_TTL = 300   # seconds
JITTER = 0.10    # +/-10% of base TTL

cache = {}       # key -> (value, expires_at)
inflight = {}    # key -> Future for the rebuild in progress

def jittered_ttl():
    # Spread expiries so keys written in one batch don't expire together
    return BASE_TTL * random.uniform(1 - JITTER, 1 + JITTER)

async def get(key, loader, wait_timeout=2.0):
    entry = cache.get(key)
    if entry and entry[1] > time.monotonic():
        return entry[0]

    fut = inflight.get(key)
    if fut is None:
        # First miss becomes the leader. There is no await between the
        # check and the insert, so on one event loop only one leader exists.
        fut = asyncio.get_running_loop().create_future()
        inflight[key] = fut
        try:
            value = await loader(key)
            cache[key] = (value, time.monotonic() + jittered_ttl())
            fut.set_result(value)
        except Exception as exc:
            fut.set_exception(exc)   # waiters inherit the leader's error
        finally:
            del inflight[key]        # the next miss can start a fresh rebuild
        return await fut

    # Followers wait on the leader's result. shield() keeps a follower's
    # timeout from cancelling the shared future for everyone else.
    return await asyncio.wait_for(asyncio.shield(fut), wait_timeout)

Per-key single-flight on one asyncio event loop: the first miss runs the loader, concurrent callers wait on its result with a timeout, and each stored entry gets a TTL with ±10% jitter.

The two-tier pattern: in-process L1 in front of shared L2

The two tiers split the work. Each instance keeps a small in-process L1 with an entry limit and a short TTL, typically seconds. All instances share one L2 with longer TTLs.

Copy ┌──────────── Instance A ───────────┐   ┌──────────── Instance B ───────────┐
 │ app ──► L1 (bounded, TTL ~secs)   │   │ app ──► L1 (bounded, TTL ~secs)   │
 └──────────┬──────────────▲─────────┘   └──────────┬──────────────▲─────────┘
            │ miss         │ invalidate             │ miss         │ invalidate
            ▼              │                        ▼              │
     ┌────────────────┐    │                 ┌────────────────────┴───┐
     │ Shared L2      │◄───┼─────────────────│ pub/sub channel        │
     └───────┬────────┘    └─────────────────┴────────────────────────┘
             │ miss
             ▼
     ┌────────────────┐
     │ Database       │
     └────────────────┘

Read path: L1, then L2, then the database. On the way back, populate L2 first and L1 second.

Write path: commit to the database, delete the key in L2, then publish the key on the channel. Every instance subscribes and evicts the key from its own L1. Pub/sub delivery is usually at-most-once, so a subscriber that is disconnected misses the message. The short L1 TTL limits how long that stale entry can survive. It is the safety net, so do not lengthen it to raise hit rates.

The cost is that a stale value can now hide in two places. When a user reports old data, you have to determine which tier served it. Tag responses or logs with the tier that returned the value, and test invalidation on both tiers.

When L2 is slow or down: use a tight timeout, then fail open to the database. Put a concurrency limit or circuit breaker in front of that fallback, and reject excess requests instead of letting every instance hit the database at once. L1 continues to absorb hot reads while L2 is unavailable.

Copy// Two-tier cache sketch. L2, bus and db are thin wrappers over your own clients.
interface L2 { get(k: string): Promise<string | null>; set(k: string, v: string, ttlSec: number): Promise<void>; del(k: string): Promise<void>; }
interface Bus { publish(ch: string, msg: string): Promise<void>; subscribe(ch: string, fn: (msg: string) => void): void; }
declare const l2: L2, bus: Bus, db: { load(k: string): Promise<string>; save(k: string, v: string): Promise<void> };

const L1_TTL_MS = 5_000, L1_MAX = 10_000, L2_TTL_SEC = 300, L2_TIMEOUT_MS = 50, MAX_DB_FALLBACK = 20;
const l1 = new Map<string, { v: string; exp: number }>(); // Map keeps insertion order, so this is a cheap FIFO bound
let fallbackInFlight = 0;

const withTimeout = <T>(p: Promise<T>, ms: number) =>
  Promise.race([p, new Promise<never>((_, rej) => setTimeout(() => rej(new Error("l2 timeout")), ms))]);

function setL1(k: string, v: string) {
  if (l1.size >= L1_MAX) l1.delete(l1.keys().next().value!); // evict the oldest entry to stay bounded
  l1.set(k, { v, exp: Date.now() + L1_TTL_MS });
}

export async function read(k: string): Promise<string> {
  const hit = l1.get(k);
  if (hit && hit.exp > Date.now()) return hit.v;
  let v: string | null;
  try { v = await withTimeout(l2.get(k), L2_TIMEOUT_MS); }
  catch {
    // L2 is slow or down: fail open to the DB, but shed requests beyond a fixed concurrency budget
    if (fallbackInFlight >= MAX_DB_FALLBACK) throw new Error("503: shedding load");
    fallbackInFlight++;
    try { return await db.load(k); } finally { fallbackInFlight--; }
  }
  if (v === null) { v = await db.load(k); await l2.set(k, v, L2_TTL_SEC).catch(() => {}); } // populate L2 first
  setL1(k, v); // then L1
  return v;
}

export async function write(k: string, v: string) {
  await db.save(k, v);                 // 1. commit to the source of truth
  await l2.del(k);                     // 2. drop the shared copy
  l1.delete(k);                        //    and the local one, without waiting for the message
  await bus.publish("cache-inval", k); // 3. tell every instance to evict from its L1
}

// Delivery is at-most-once, so a missed message is covered only by the short L1 TTL.
bus.subscribe("cache-inval", (k) => { l1.delete(k); });

A minimal TypeScript sketch of the two-tier read path, the commit-delete-publish write path with L1 eviction via pub/sub, and a timeout-bounded L2 lookup that fails open to the database behind a concurrency limit.

Choosing a tier by data type, and when to avoid caching

The right tier depends on how often the data changes, who it belongs to, and whether a slightly stale read is acceptable.

When to avoid caching, or a given tier

An in-process map with no size bound or eviction policy is a memory leak that grows until something breaks. Even a bounded L1 has a cost. A larger heap gives the garbage collector more to scan, which can lengthen pauses, and a cache sized without the container's memory limit in mind invites an OOM kill. If the data set is large, keep L1 small and push volume to L2.

Caching can also hide a slow query or a missing index. Latency looks fine until a cold start or a mass expiry sends traffic straight to the database. Before adding a cache, check whether one of these fixes the cause:

• Rewrite the query or add the missing index.
• Route read-heavy traffic to read replicas.
• Precompute aggregates in materialized views.
• Use HTTP caching headers or a CDN for responses that are the same for every user.

Impact on cost and engineering time, and what to measure

A shared cache is a production service, not a library. Someone has to size it, pick an eviction policy, decide whether persistence is worth the write overhead or whether a cold restart is acceptable, set up failover, and lock down network access and authentication. Eviction policy matters more than it looks: evicting any key under memory pressure behaves very differently from evicting only keys that have a TTL. The shared cache also adds a dependency on the request path. If it is slow or unreachable and the monolith has no defined fallback, the cache becomes the outage.

The payoff is deferred work. A cache that absorbs most reads can postpone a database scale-up or a service split, which lets the team make those changes deliberately instead of under load. The cost shows up in debugging. A "stale data" report now means checking L1, L2 and the database, and tier-specific bugs are hard to reproduce locally.

Illustrative estimate (assumed numbers, not a benchmark): backend reads/s = total reads/s × (1 − combined hit ratio). At an assumed 2,000 reads/s and a 90% hit ratio, the database sees 200 reads/s. At 80%, it sees 400. A ten-point drop doubles backend load, which is why you should watch hit ratio closely.

What to measure, before and after

• Hit ratio per tier (L1, L2 and combined)
• Backend load, as queries per second and CPU
• p50 and p99 latency, including the network hop to the shared cache
• Eviction rate, where a sustained rise signals undersizing
• Memory per instance, since L1 is duplicated on every replica

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith caching strategies, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.