Why do cron jobs run multiple times when a monolith has multiple replicas?

An in-process scheduler, whether a @Scheduled-style annotation, a timer library, or a background thread started at boot, runs inside the application process and has no knowledge of other copies. Every replica loads the same schedule, so when the expression matches, every replica fires. With N replicas you get N runs.

N is also not stable, which makes the duplicates hard to reason about:

  • Autoscaling changes the replica count between triggers. A job that ran twice yesterday may run five times today.
  • Rolling deploys overlap old and new pods. Surge capacity adds extra triggers, and one run can execute under old code while another executes under new code.
  • Clock skew spreads the firings across milliseconds or seconds. That breaks naive "did this already run?" checks, because each replica reads the guard before any other has written it.

In practice this shows up as customers getting the same email twice, invoices or charges created twice, and reports regenerated N times. Concurrent copies also contend for the same rows, which causes lock waits, deadlocks and connection-pool pressure that look like unrelated database problems.

Illustrative estimate (assumed numbers, not a benchmark). Extra runs per trigger = (replicas alive at trigger time − 1). Assume a daily job, 4 steady replicas, 30 days, and one deploy that surges to 6 pods at trigger time. Extra runs = 29 × 3 + 1 × 5 = 92.

This affects any team running a horizontally scaled monolith with scheduled work inside the app. It often goes unnoticed until the first move from one replica to two.

Who should use lock-based scheduling across replicas?

The approach in this article is aimed at teams that match most of these conditions:

  • You run a monolith (or a large service) with more than one replica, or you plan to, and scheduled jobs live inside the application code as annotations, timer libraries or boot-time threads.
  • The replicas share a relational database or another store that supports an atomic conditional write. The lock table and the idempotency markers live there.
  • The jobs are short to moderate in length, from seconds to tens of minutes, and run at intervals of minutes, hours or days.
  • You want to keep job code in the same repository and deploy pipeline as the app, rather than standing up and operating a separate scheduling service.

It is a poor fit for teams that already have a dedicated scheduler or workflow engine that owns these jobs, for deployments that will stay at one replica with no surge during deploys, and for workloads covered in "When not to do this" below: jobs that run for hours, fire every few seconds, or fan out large batches.

What are the alternatives for coordinating scheduled jobs across replicas?

There are four common ways to make sure a job fires once. They differ mainly in where the "only one" guarantee lives.

Approach Isolation Failure behaviour Deployment surface Cost
Dedicated scheduler instance / worker deployment High: jobs never compete with request traffic Jobs stop if the single instance dies, until it is restarted Extra deployment and config to maintain Idle compute between runs
Kubernetes CronJob calling an internal endpoint Medium: the work still runs on a serving replica Missed or retried by the platform; the endpoint needs auth and timeouts Second codepath (manifest and endpoint) that must stay in sync with the app Pod startup on each trigger adds cold-start latency
Leader election Low: the leader serves traffic and runs every job Handover gap while a new leader is elected; long jobs can be cut off mid-run Small: a library plus a lease store One replica carries all scheduled load
Per-job distributed lock (e.g. ShedLock) Low to medium Lock expiry bounds the outage; jobs spread across replicas Small: an annotation plus a lock table One lock write per trigger

Per-job locks are the least disruptive choice for a monolith that already has a shared database. Leader election suits schedules with many small, frequent jobs where lock churn is a concern. Use a CronJob when the platform team already owns scheduling and you want a single audit point.

Avoid in-process scheduling entirely when jobs are CPU-heavy, run longer than your deploy cadence, or need different resource limits from the web tier. In those cases a separate worker deployment is worth the overhead.

When should you introduce distributed locking for scheduled jobs?

Add it before duplicates become possible, not after the first incident. The concrete trigger points are:

  • Before scaling from one replica to two. This is the moment every in-process schedule starts firing twice.
  • Before enabling autoscaling or a minimum replica count above one. The number of duplicate runs then changes from day to day.
  • Before switching to rolling deploys with surge capacity. Even a single-replica service briefly runs two pods during a surge, and a trigger that lands in that window fires twice.
  • When you already see symptoms: duplicate emails, double-created records, reports generated more than once, or unexplained lock contention and deadlocks at round clock times.

If you are already past these points, the migration path at the end of this article lets you add locking one job at a time without stopping the system.

How does it work, end to end?

The design has four layers, each covered in its own section below:

  1. Trigger. Every replica keeps its in-process scheduler and fires on the same cron expression. Nothing about the schedule changes.
  2. Coordination. Before doing any work, each replica tries to take a time-limited lease on a per-job lock row with an atomic conditional update. One replica wins and runs the job; the others skip that slot.
  3. Safety. Because a lease can expire while its holder is paused, a second run is still possible. The job itself is made idempotent with natural work keys and a marker table, so a duplicate run produces the same end state.
  4. Completeness and visibility. A job-state table records the last successful interval, so runs dropped by a crash are detected and handled by a per-job catch-up, run-once or skip policy. Lock decisions, durations and staleness are exported as signals you can alert on.

The lock reduces duplicate runs, idempotency makes the remaining ones harmless, and the state table catches the runs that never happened.

How does a lock-based scheduler like ShedLock work?

Each job gets exactly one row (or one key, in a key-value store), identified by the job name. The row records who holds the lock and a lock_until timestamp. Every replica's local scheduler fires on the same cron expression, and each tries to claim the row before doing any work:

  1. Ensure the row exists. Insert it once and ignore the duplicate-key error if another replica got there first.
  2. Acquire with a conditional update. For example, UPDATE job_lock SET lock_until = now() + max_hold, locked_by = 'replica-7' WHERE name = 'invoice_rollup' AND lock_until <= now(). The database applies this atomically, so at most one replica sees "1 row affected." Computing time with the database's now() rather than each replica's clock removes one source of skew.
  3. Run or skip. The winner runs the job. Every other replica gets zero rows and returns immediately.
  4. Release. On completion, the winner sets lock_until to whichever is later: now, or the acquire time plus a minimum hold.

The minimum hold matters for short jobs. Suppose a job finishes in 200 ms and releases right away. A replica whose clock runs a second behind then fires late, finds the lock free, and runs the job a second time.

Losers skip rather than queue because the lock exists to collapse N triggers into one execution. A queued attempt would just be the duplicate run arriving late.

The max_hold value makes this a lease, not mutual exclusion. If the winner stalls for longer than max_hold, through a GC pause, a frozen VM, or a slow network call, the lease expires. Another replica acquires it, and the stalled node then wakes up and continues without knowing it lost the lock. Both are now running the job, which is why the job itself must tolerate duplicates.

-- PostgreSQL syntax; other databases need equivalent upsert and timestamp functions.
-- One row per job name. lock_until is a lease expiry, not a guarantee of exclusivity.
CREATE TABLE job_lock (
  name        VARCHAR(64)  PRIMARY KEY,
  lock_until  TIMESTAMPTZ  NOT NULL,
  locked_at   TIMESTAMPTZ  NOT NULL,
  locked_by   VARCHAR(255) NOT NULL
);

-- 1. Ensure the row exists. If another replica inserted it first, do nothing.
INSERT INTO job_lock (name, lock_until, locked_at, locked_by)
VALUES ('invoice_rollup', now(), now(), '')
ON CONFLICT (name) DO NOTHING;

-- 2. Acquire. Only a row with an expired lease matches, so at most one
--    concurrent caller gets 1 row affected. The database clock is used, not the replica's.
UPDATE job_lock
   SET lock_until = now() + INTERVAL '10 minutes',  -- max_hold: lease length
       locked_at  = now(),
       locked_by  = 'replica-7'
 WHERE name = 'invoice_rollup'
   AND lock_until <= now();
-- 0 rows affected: another replica holds the lease, so skip this run (do not queue it).
-- 1 row affected: this replica runs the job.

-- 3. Release on completion. Keep the lease until at least locked_at + min_hold
--    so a replica whose trigger fires slightly late does not rerun a short job.
--    The locked_by check prevents overwriting a lease that expired and was
--    taken by another replica while this one was stalled.
UPDATE job_lock
   SET lock_until = GREATEST(now(), locked_at + INTERVAL '30 seconds')  -- min_hold
 WHERE name = 'invoice_rollup'
   AND locked_by = 'replica-7';
-- This check does not make the job exclusive. If this replica stalled past
-- max_hold, another replica may already have run it, so the job must be idempotent.

A PostgreSQL lock table with an idempotent row insert, an atomic conditional UPDATE that grants a time-limited lease to at most one replica, and a release step that enforces a minimum hold time.

How should you choose a lock backend and size lock durations?

No backend removes the pause problem. A lease only tells you who held the lock most recently. The backends differ in how reliably that answer survives failures.

Backend Node crash Backend failover / network loss Cost
Existing relational DB Lease expires; another replica takes over Durable and consistent, but if the DB is down, no job runs Extra writes on the primary; scheduling depends on DB health
Redis (single node or async failover) Lease expires normally A lock written just before failover can vanish, so two holders are possible Cheap and fast, with weaker guarantees
Coordination service (consensus-based) Session or lease expiry is well defined Survives minority failures without losing the lock Another system to run and secure

For most monoliths the database you already run is the pragmatic choice. The jobs usually write to that same database, so a DB outage stops them anyway.

Size two durations:

  • Maximum lock time must exceed the worst realistic run, not the average one. If a job outlives its lock, another replica can start it while the first is still running. Idempotency, covered in the next section, is what keeps that safe.
  • Minimum lock time must exceed the expected clock drift between replicas. Without it, a short job finishes quickly and a replica whose clock is behind fires the same schedule slot again.

Illustrative estimate (assumed inputs):

  • The job usually takes 2 minutes and takes 9 minutes at month-end.
  • Clock drift is under 5 seconds.

Set the maximum to about 15 minutes and the minimum to about 30 seconds.

Keep the minimum small. A long minimum also blocks legitimate reruns and manual retriggers until it expires.

How do you make scheduled jobs idempotent so duplicate runs are harmless?

A lock lowers the odds of two replicas running the same job. It cannot rule it out. A paused JVM, a lock that expires mid-run, or a manual rerun can still produce a second execution. The goal is to make that second run harmless.

Key the work on natural identifiers. A billing job should process "invoice for account 42, period 2026-11", not "the next 500 rows." When each unit of work has a stable identity, the database can enforce uniqueness for you.

Use a processed-marker table with a unique constraint. Create a table such as job_runs (job_name, work_key, processed_at) with a unique constraint on (job_name, work_key). Before or alongside the side effect, write the marker with INSERT ... ON CONFLICT DO NOTHING (PostgreSQL) or the equivalent in your database. If the insert affects zero rows, another run already claimed that key, so skip it. For writes to business tables, prefer upserts over blind inserts so a repeat produces the same end state.

Checkpoint in small batches. Commit every few hundred keys, each batch in its own transaction together with its markers. A crash then loses one batch rather than an hour of work, and the next run resumes where the last one stopped. One long transaction also holds locks and bloats the database.

This has a real cost. Jobs that used to be a single UPDATE now need a new table, a key design, and batching code. The payoff is that a duplicate run no longer causes an incident. You don't have to reconcile double charges or duplicate emails, and you don't have to rebuild a timeline after the fact.

-- Marker table: one row per unit of work a job has claimed.
CREATE TABLE job_runs (
  job_name     text        NOT NULL,
  work_key     text        NOT NULL,  -- natural key, e.g. 'acct:42:2026-11'
  processed_at timestamptz NOT NULL DEFAULT now(),
  CONSTRAINT job_runs_job_key_uniq UNIQUE (job_name, work_key)
);

-- Business table keyed on the same natural identity.
CREATE TABLE invoices (
  account_id   bigint NOT NULL,
  period       text   NOT NULL,
  amount_cents bigint NOT NULL,
  PRIMARY KEY (account_id, period)
);

-- One batch = one transaction. The job loops over account_id ranges.
BEGIN;

WITH claimed AS (
  -- Claim keys. A key already claimed by another run returns no row here.
  INSERT INTO job_runs (job_name, work_key)
  SELECT 'monthly_billing', 'acct:' || u.account_id || ':2026-11'
  FROM usage_totals u
  WHERE u.period = '2026-11'
    AND u.account_id BETWEEN 1 AND 500   -- batch boundary
  ON CONFLICT (job_name, work_key) DO NOTHING
  RETURNING work_key
)
-- Upsert, so a repeat converges on the same end state.
INSERT INTO invoices (account_id, period, amount_cents)
SELECT u.account_id, u.period, u.total_cents
FROM usage_totals u
JOIN claimed c ON c.work_key = 'acct:' || u.account_id || ':2026-11'
WHERE u.period = '2026-11'
ON CONFLICT (account_id, period)
  DO UPDATE SET amount_cents = EXCLUDED.amount_cents;

COMMIT;  -- markers and invoices commit together; a crash loses only this batch

This PostgreSQL batch claims natural work keys in a marker table that has a unique constraint, using INSERT ... ON CONFLICT DO NOTHING, and writes invoices with an upsert in the same transaction, so a duplicate or resumed run skips keys that were already processed.

How should jobs handle missed runs after deploys or outages?

Healing duplicates is only half the problem. The other half is runs that never happen at all, and a lock-based scheduler can drop them without any error.

Here is how it happens. The replica holding the lock crashes halfway through a job. Every other replica sees an unexpired lock and skips that slot. When the lock finally expires, the scheduled time has passed, so nothing retries. No error is logged and no replica ran the job to completion. The lock table cannot detect this, because it records how long a lock is held, not whether the work finished.

So record success explicitly. Keep a job-state table with one row per job. It should hold job_name, last_success_at (the logical interval the job completed, not wall-clock time), last_attempt_at, and last_error. Update last_success_at only after the work commits. Do not infer it from logs or lock rows.

On each trigger, compare last_success_at with the schedule. Then apply the policy you chose for that job:

  • Catch up: process every missed interval. Use this when each interval has its own data, such as daily settlement or per-hour aggregates.
  • Run once: collapse all missed intervals into a single run. This fits cache rebuilds, cleanup sweeps, and syncs where only the current state matters.
  • Skip: drop missed intervals and wait for the next slot. Use this for time-sensitive work, like a "starts in 10 minutes" reminder that is useless an hour late.

Catch-up logic needs a bound. After a long outage, every catch-up job wakes at once and sends days of backlog to the database and downstream APIs. That can cause a second outage. Cap how many intervals a job processes per tick and go oldest first. Advance last_success_at after each interval so progress survives a crash, and add jitter so jobs don't start in lockstep.

import random, time
from datetime import timedelta

MAX_INTERVALS_PER_TICK = 6        # bound catch-up so an outage doesn't become a burst
SKIP_GRACE = timedelta(minutes=5)  # how late a 'skip' job may still run

def save(conn, job, slot=None, error=None):
    with conn, conn.cursor() as cur:  # psycopg2: commits on exit
        cur.execute("""UPDATE job_state SET last_attempt_at = now(), last_error = %s,
                         last_success_at = GREATEST(last_success_at, COALESCE(%s, last_success_at))
                       WHERE job_name = %s""", (error, slot, job))

def run_due(conn, job, policy, interval, now, work):
    with conn, conn.cursor() as cur:
        cur.execute("SELECT last_success_at FROM job_state WHERE job_name = %s", (job,))
        last = cur.fetchone()[0]
    missed, t = [], last + interval
    while t <= now:                    # every slot due since the last recorded success
        missed.append(t)
        t += interval
    if not missed:
        return
    if policy == "catch_up":
        targets = missed[:MAX_INTERVALS_PER_TICK]  # oldest first; rest waits for next tick
    elif policy == "run_once":
        targets = [missed[-1]]                     # collapse the backlog into one run
    elif now - missed[-1] <= SKIP_GRACE:           # skip: only the current slot, if still fresh
        targets = [missed[-1]]
    else:
        return save(conn, job, slot=missed[-1])    # too late: advance marker without running
    for slot in targets:
        try:
            work(slot)                 # idempotent work for this interval, committed by work()
        except Exception as e:
            save(conn, job, error=str(e)[:500])
            raise
        save(conn, job, slot=slot)     # record success per interval so progress survives a crash
        time.sleep(random.uniform(0.5, 2.0))  # jitter so caught-up jobs don't hit downstream in lockstep

A job-state table plus per-job catch-up, run-once and skip policies, with catch-up capped per tick, success recorded after each interval, and jitter between runs.

How can this approach save engineering time?

The time savings come from removing recurring manual work, not from making jobs run faster:

  • Fewer duplicate-run incidents to clean up. Without coordination, every double charge or duplicate email means finding affected records, reversing side effects, contacting customers and writing a postmortem. Locks plus idempotent jobs turn most of those events into a skipped run or a no-op.
  • Less time diagnosing misleading symptoms. Concurrent copies of a job show up as deadlocks, lock waits and connection-pool exhaustion that look like database problems. Once only one copy runs, those false leads disappear from investigations.
  • No separate scheduler to build and operate. Jobs stay in the same codebase, test suite and deploy pipeline as the app. There is no second deployment, manifest or internal endpoint to keep in sync.
  • Silent misses are caught by an alert, not by a customer. The last_success_at staleness check surfaces a dropped run within a few intervals, instead of weeks later during a reconciliation.
  • Scaling decisions stop requiring a job audit. Once every job is wrapped and idempotent, changing replica counts or autoscaling settings no longer means re-checking every scheduled task.

Illustrative estimate (assumed inputs, not a measured result): assume a team has one duplicate-run incident per quarter, and each takes two engineers one day to investigate and reconcile. That is about 8 engineer-days a year. Assume the one-time work of adding a lock table, idempotency markers and a job-state table across the existing jobs takes 5 engineer-days. Under those assumptions the work pays back within the first year, before counting time saved on misleading database investigations. Substitute your own incident rate and cleanup cost; the result is only as good as the inputs.

Observability and a no-flag-day migration path for multi-replica cron

What to measure

Log every lock decision, not just job output. Each tick should emit one of three events: acquired, skipped because another replica holds the lock, or failed to reach the lock store. A steady stream of skips is normal. A replica that never acquires, or a spike in lock-store errors, is not.

For each run, record start, finish, duration, and outcome. Compare duration against the lock's hold time. If runs approach it, the lock can expire mid-run and a second replica can start.

Alert on staleness, not on errors. A run that never started throws nothing, so per-failure alerts miss the worst case. Export last_success_at per job and page when it is older than two or three intervals.

Migrating without a flag day

  1. Inventory every crontab entry and annotated method. Note each job's schedule, typical runtime, and side effects.
  2. Add idempotency first, while the job still runs on one node. Duplicates are then harmless before they become possible.
  3. Create the lock table and deploy the library with no jobs wrapped.
  4. Wrap one job per deploy. Confirm acquire and skip events look right before you move to the next.
  5. Keep a kill switch per job, such as a config flag that disables it without a redeploy.
  6. Scale replicas only after every job is wrapped and verified.

When not to do this

Jobs that run for hours, run every few seconds, or fan out large batches belong in a queue or workflow system. Those systems provide retries, progress tracking, and parallelism that a lock around a timer cannot.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cron jobs multiple replicas, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.