Why do cron jobs run multiple times when you add replicas?
A monolith with in-process scheduling carries its schedule inside the application. When the app boots, the scheduler starts and registers every job. On one replica that works as intended. On three replicas, three schedulers fire at the same instant, and nothing in the code tells them about each other. One job definition becomes N concurrent runs, so you get three invoice emails, three cleanup passes racing on the same rows, and three calls to a rate-limited partner API.
N is also hard to predict. An autoscaler adds replicas under load, so a job that ran twice at 03:00 may run five times at 14:00. A rolling deploy briefly runs old and new pods side by side, and during that window the old version may still schedule a job the new version renamed, reshaped or removed. Pods that restart mid-schedule can fire a run late or skip it.
Illustrative estimate (assumed numbers, not a benchmark): extra runs per day = (replicas − 1) × scheduled runs per day. With an assumed 4 replicas and a job every 15 minutes (96 runs per day), that is 3 × 96 = 288 duplicate executions per day for that one job.
This guide is for teams moving a monolith from one replica to several. The approach throughout is to pick one explicit coordination strategy and make every job idempotent anyway. Locks and leaders reduce duplicates, but none of them guarantees exactly-once execution.
one job definition in the app image
job: send_invoices schedule: daily 03:00
|
+-----------------------+-----------------------+
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| replica 1 (v1) | | replica 2 (v1) | | replica 3 (v2) |
| in-proc scheduler | | in-proc scheduler | | in-proc scheduler |
| 03:00 -> run job | | 03:00 -> run job | | 03:00 -> run job |
+---------+---------+ +---------+---------+ +---------+---------+
| | |
v v v
+-------------------------------------------------------------------+
| shared resources: one DB, one email provider, one partner API |
| no lock, no leader: schedulers never see each other |
| result: 3 invoice emails, 3 cleanups racing on the same rows, |
| 3 calls to a rate-limited partner API |
+-------------------------------------------------------------------+
N changes at runtime:
autoscaler adds/removes pods ---> N = 2 at 03:00, N = 5 at 14:00
rolling deploy (v1 + v2 live) --> v1 pods fire jobs v2 renamed/removed
pod restart mid-schedule -------> run fires late or is skipped
Every replica boots its own in-process scheduler, so one job definition fires N times against the same shared resources. Autoscaling and rolling deploys change N and can leave old and new versions scheduling jobs side by side.
Who should use these patterns?
These patterns apply to you if your application schedules work from inside its own process and you run, or plan to run, more than one copy of it. Typical cases:
- Teams scaling a monolith horizontally. The app uses an in-process scheduler, such as a framework's scheduled-method annotation or a cron library started at boot, and is moving from one instance to several for availability or load.
- Teams turning on autoscaling or rolling deploys. Even an app that "normally" runs as one replica briefly runs two during a rolling update, and an autoscaler can raise the count at any time.
- Teams whose jobs have external side effects. Sending email, charging cards, calling partner APIs with rate limits or quotas, generating invoices or exports, and deleting data are the jobs where a duplicate run turns into a customer-visible problem.
- Teams whose jobs write to shared tables. Cleanup, aggregation and reconciliation jobs that update the same rows will race when they run concurrently.
You may not need in-app coordination if the app genuinely runs as one instance with no plans to scale, or if the platform already triggers every job from a single external scheduler. Even then, the idempotency techniques later in this article are worth applying, because external schedulers and orchestrators can also produce duplicate or overlapping runs. Jobs that are cheap and harmless to repeat, such as warming a cache, can sometimes tolerate duplicates. Make that a deliberate decision per job, not an accident.
What are the options for coordinating scheduled jobs across replicas?
There are three common ways to coordinate scheduled jobs, and each moves the "who runs this?" decision to a different place.
External scheduler
A single trigger outside the application, such as a Kubernetes CronJob, a dedicated scheduler instance or system cron on one host, calls an internal endpoint or command. The replicas do no scheduling of their own. The trigger is now a separate system with its own semantics. A Kubernetes CronJob's concurrency policy determines whether a slow run overlaps the next one. Its starting deadline determines whether a delayed run is skipped. If the controller is unavailable, schedules can be missed. Job pods also pay a cold-start cost on every run.
Database-backed lock
In the ShedLock style, each job name has a lock row with a locked_until timestamp. Every replica's scheduler fires, and the replica whose conditional update succeeds runs the job. Because coordination happens through the database the app already uses, no new infrastructure is needed. The cost is that scheduled work now depends on the primary database being reachable, and each tick adds write traffic from every replica.
Leader election
One replica holds a lease, and only that leader runs scheduled work. The lease can be a Kubernetes Lease object, a database advisory lock or a lease row. When the leader dies, nothing runs until its lease expires and another replica acquires it. All scheduled load also lands on a single replica.
| Approach | Where the decision lives | Main failure modes |
|---|---|---|
| External scheduler | Outside the app | Concurrency policy, missed schedules, deadlines, cold starts |
| DB lock | Per-job lock row | Primary DB dependency, extra write load |
| Leader election | One lease for all jobs | Expiry gaps, load concentrated on the leader |
How does coordinated scheduling work end to end?
Whichever option you choose, a scheduled run passes through the same four stages. The rest of this article covers each stage in detail.
- Trigger. Something decides it is time to run. With in-process scheduling, every replica's scheduler fires on the same tick. With an external scheduler, a single caller invokes a command or endpoint.
- Gate. A coordination mechanism decides which caller proceeds. With a database lock, every replica attempts one conditional write on the job's lock row, and only the replica whose write succeeds continues. The others see zero affected rows and return immediately. With leader election, a replica runs scheduled work only while it holds the lease. With an external scheduler, the gate is the fact that there is only one caller, plus the scheduler's own concurrency settings.
- Execute safely. The winning replica runs the job. Because a gate can fail, for example when a paused process outlives its lock, the job body is written so that a second concurrent or repeated run does no harm. It claims individual rows, uses conditional state transitions and relies on unique constraints so that duplicate writes become no-ops. Long jobs renew their lock with a heartbeat and commit progress in batches.
- Record and release. The job releases its lock (respecting a minimum hold time so a slightly late replica still skips), and logs and metrics record who ran, who skipped and when the last successful run happened. An alert on the age of the last successful run catches the cases where the gate silently blocks every replica.
The key idea is defense in depth. The gate removes most duplicates cheaply, idempotent job bodies make the remaining duplicates harmless, and monitoring tells you when either layer stops working.
How do you implement a database-backed job lock in SQL?
The lock lives in one table with a row per job: name as the primary key, locked_until as the expiry, and locked_at and locked_by for debugging and safe release. When a scheduled tick fires, every replica runs the same conditional update: set locked_until = now() + max_duration, locked_at = now() and locked_by = <replica id>, but only WHERE name = :job AND locked_until <= now(). The database serializes concurrent updates to a row, so only one statement finds the condition true. Each replica checks the affected-row count. If it is 1, that replica runs the job. If it is 0, it skips the run. On the first run no row exists yet, so the replica attempts an INSERT. A primary-key violation means another replica won, and you treat it as not acquired.
A SELECT followed by an UPDATE is a race. Two replicas can both read an expired lock before either writes, and both then proceed. The check and the claim must be a single statement.
Compare against the database's now(), not a timestamp each replica computes, because replica clocks drift and a skewed node could see an active lock as expired. This makes the database the single source of time and the single source of truth. A failover that loses recent writes, or a lock table split across databases, breaks the guarantee.
To release, update with WHERE name = :job AND locked_by = <me> and set locked_until to the later of now() and locked_at + min_duration. The max_duration value in the acquire is lock-at-most-for. It frees the lock if the holder crashes. The min_duration floor is lock-at-least-for. It stops a fast job from running again on a replica whose scheduler fires slightly late. Libraries such as ShedLock implement this pattern, so you don't have to maintain it yourself.
-- PostgreSQL. :'job' and :'replica' are psql variables, e.g. 'nightly-report' and 'worker-7f9c'.
CREATE TABLE job_lock (
name text PRIMARY KEY, -- one row per job
locked_until timestamptz NOT NULL, -- lock expiry, always set from the DB clock
locked_at timestamptz NOT NULL, -- when the current holder acquired it
locked_by text NOT NULL -- holder id, used for debugging and safe release
);
-- Acquire: check and claim in ONE statement. Every replica runs this on the tick.
-- Row updates are serialized, so only one replica sees 'locked_until <= now()' as true.
-- 1 row affected = acquired; 0 rows = someone else holds it, skip this run.
UPDATE job_lock
SET locked_until = now() + interval '10 minutes', -- lock-at-most-for (crash safety)
locked_at = now(),
locked_by = :'replica'
WHERE name = :'job'
AND locked_until <= now(); -- DB clock, not the replica's clock
-- First run only: no row exists yet, so the UPDATE above affected 0 rows.
-- ON CONFLICT DO NOTHING turns the primary-key collision into 0 rows affected,
-- which means another replica inserted first and won.
INSERT INTO job_lock (name, locked_until, locked_at, locked_by)
VALUES (:'job', now() + interval '10 minutes', now(), :'replica')
ON CONFLICT (name) DO NOTHING;
-- Release: only the holder may release. Keep the lock until at least
-- locked_at + 30s (lock-at-least-for) so a fast job is not re-run by a
-- replica whose scheduler fires slightly late.
UPDATE job_lock
SET locked_until = GREATEST(now(), locked_at + interval '30 seconds')
WHERE name = :'job'
AND locked_by = :'replica';
A PostgreSQL job-lock table with a single-statement conditional UPDATE for acquire, an INSERT for the first run, and a holder-only release that applies the lock-at-least-for floor, all timed by the database's now().
How should you choose lock durations for long-running jobs?
Lock-at-most-for is a bet on your worst-case runtime. If you set it too short, a slow day lets the lock expire mid-run and another replica starts a second copy. A bigger table or a contended database is enough to cause that. If you set it too long, a holder that crashes or gets OOM-killed blocks the job until the lock expires on its own. As an illustrative estimate, take a job that normally takes 4 minutes, runs every 15 minutes and has a 60-minute at-most-for: one crash can cost up to four runs.
Lock-at-least-for is the smaller setting. Keep it just above the scheduler skew you expect between replicas, which is usually seconds. Keep it below the schedule interval. A job that finishes in 200 ms then still holds the lock when a slightly late replica fires, and that replica skips the run.
Heartbeats
When runtime is unpredictable, keep at-most-for short and have the worker extend its lease while it makes progress. Every minute or so, it runs UPDATE job_locks SET locked_until = now() + interval '5 minutes' WHERE name = 'nightly_export' AND locked_by = 'replica-3' AND locked_until > now() (PostgreSQL syntax). The locked_by condition stops a worker from renewing a lock that another replica has already taken over. If the update affects zero rows, the worker has lost the lock and must stop.
Batches
Split large jobs into batches that each commit their progress, for example as a cursor position or a processed flag. An interrupted run then resumes from the last checkpoint instead of starting over. At-most-for only has to cover one batch, not the whole job.
-- PostgreSQL. One row per job; locked_until is the lease deadline.
CREATE TABLE IF NOT EXISTS job_locks (
name text PRIMARY KEY,
locked_by text NOT NULL,
locked_at timestamptz NOT NULL,
locked_until timestamptz NOT NULL
);
-- Acquire: short at-most-for (5 min). Succeeds only if no live lease exists.
-- Zero rows returned means another replica holds the lock: skip this run.
INSERT INTO job_locks (name, locked_by, locked_at, locked_until)
VALUES ('nightly_export', 'replica-3', now(), now() + interval '5 minutes')
ON CONFLICT (name) DO UPDATE
SET locked_by = EXCLUDED.locked_by,
locked_at = EXCLUDED.locked_at,
locked_until = EXCLUDED.locked_until
WHERE job_locks.locked_until <= now()
RETURNING locked_by;
-- Heartbeat (about every minute while batches progress).
-- locked_by guards against renewing a lease another replica took over.
-- Zero rows updated means the lease is lost: stop before the next batch.
UPDATE job_locks
SET locked_until = now() + interval '5 minutes'
WHERE name = 'nightly_export'
AND locked_by = 'replica-3'
AND locked_until > now();
-- Release: honor at-least-for (30 s) so a slightly late replica still
-- sees a held lock and skips, even if the job finished in 200 ms.
UPDATE job_locks
SET locked_until = GREATEST(locked_at + interval '30 seconds', now())
WHERE name = 'nightly_export'
AND locked_by = 'replica-3';
PostgreSQL statements for acquiring a lease with a short at-most-for, renewing it with a holder-conditioned heartbeat, and releasing it no earlier than the at-least-for window.
Why isn't a lock enough, and how do you make jobs idempotent?
A lock bounds how long a job should run. It cannot stop a process that has paused. A long garbage-collection pause, a network partition or a frozen VM can last longer than the lock's TTL. The lock expires, a second replica acquires it and starts, and the first process resumes without knowing it lost ownership. Both now write to the same resources.
Fencing tokens close this gap. Each lock acquisition returns a number that only ever increases. The job sends that number with every write, and the target resource rejects any write whose token is lower than the highest one it has already accepted. This only works if the resource itself enforces the check, so most jobs also need idempotency.
Practical techniques:
- Claim rows, not the whole job. Where the database supports it,
SELECT ... FOR UPDATE SKIP LOCKEDlets each worker claim a batch of pending rows. Rows that another worker already holds are skipped instead of waited on or processed twice. - Use state transitions. Write updates as conditional moves, such as
UPDATE ... SET status = 'done' WHERE id = ? AND status = 'processing'. If another worker already made the change, zero rows match, and the duplicate becomes a no-op. - Put unique constraints on outputs. A unique key on fields like
(invoice_id, period)makes the database reject a second insert. - Record completed work so a rerun can check what is already finished and skip it.
Idempotency takes design work up front. That work replaces the incident response and manual cleanup that duplicate runs cause later.
-- PostgreSQL. Assumed tables: billing_tasks(id, invoice_id, period, status,
-- claimed_by, claimed_at, completed_at) and invoices(invoice_id, period, amount_cents).
-- One-time setup: the database rejects a second invoice for the same period.
ALTER TABLE invoices
ADD CONSTRAINT invoices_invoice_period_uniq UNIQUE (invoice_id, period);
-- 1. Claim a batch. Rows locked by another worker are skipped, not waited on.
-- The row lock lasts until COMMIT. The 'processing' status records the claim after that.
WITH batch AS (
SELECT id
FROM billing_tasks
WHERE status = 'pending'
ORDER BY id
LIMIT 50
FOR UPDATE SKIP LOCKED
)
UPDATE billing_tasks t
SET status = 'processing', claimed_by = 'worker-2', claimed_at = now()
FROM batch
WHERE t.id = batch.id
RETURNING t.id, t.invoice_id, t.period;
-- 2. Write the output. A duplicate run hits the unique key and inserts nothing.
INSERT INTO invoices (invoice_id, period, amount_cents)
VALUES ($1, $2, $3)
ON CONFLICT (invoice_id, period) DO NOTHING;
-- 3. Conditional state transition. If this task was already completed or
-- reclaimed by another worker, 0 rows match and the update is a no-op.
UPDATE billing_tasks
SET status = 'done', completed_at = now()
WHERE id = $4
AND status = 'processing'
AND claimed_by = 'worker-2';
A PostgreSQL example where a worker claims pending rows with FOR UPDATE SKIP LOCKED, writes output that a unique constraint protects from duplicates, and marks completion with a conditional state transition that does nothing on a rerun.
How do you monitor scheduled jobs for skipped and missed runs?
A lock that works correctly produces no errors. When a replica fails to acquire it, the job returns without doing anything, and most lock libraries, ShedLock included, treat that as normal. The silence is a problem when a lock gets stuck. Suppose a crashed holder left a lock with a long expiry, or a clock or row is wrong. Every replica then skips the job, nothing fails, and no error alert fires. You find out from a user.
Log each run with the job name, the replica identity (pod name or hostname), the outcome and the lock's expiry time. This answers "who ran it?" without guesswork. Then emit metrics per job:
| Metric | Type | What it catches |
|---|---|---|
| Runs started | Counter | Baseline cadence |
| Runs skipped (lock held) | Counter | Contention, or a stuck lock if skips never stop |
| Runs failed | Counter | Errors inside the job |
| Run duration | Histogram | Jobs drifting toward the lock duration |
| Last successful run | Timestamp gauge | Missed runs of any cause |
The last-success timestamp is the most important of these. Alert when now - last_success exceeds the schedule interval plus a margin. That single rule fires on failures, stuck locks, a scheduler that never started, and a misconfigured cron expression. A failure-only alert misses all but the first.
Rolling deploys need the same signals. During a rollout, old and new versions run side by side. If a release renames a lock, both versions hold different locks and run the job twice. If a release changes the job's behavior, the two versions may process the same data differently. Duplicate starts across replicas and an unexpected rise or drop in skipped runs are how you see these problems before they cause damage.
Which approach should you use: external scheduler, DB lock or leader election?
Once you can see these problems, the next step is to pick one coordination model and apply it to every job. The right choice depends mostly on what your platform already provides.
| Approach | Use it when | Avoid it when |
|---|---|---|
| External scheduler (Kubernetes CronJob, system cron on one host) | The platform already runs one, and jobs can be triggered as a CLI command or an internal endpoint | Jobs need in-process state or tight coupling to app internals |
| DB lock library (ShedLock-style) | You have an existing monolith that already depends on a primary database | The database is a fragile dependency, because lock queries add load and a DB outage stops every job |
| Leader election | Many jobs need continuous single-owner coordination, and the team can operate a consensus mechanism | Request-serving capacity is tight, because the leader carries all job load on top of its share of traffic |
For most monoliths, the DB lock is the pragmatic default. It adds no new infrastructure, and the failure mode is easy to reason about: if the database is unavailable, the job is skipped.
If a single external trigger is enough, skip in-app coordination entirely. One caller means there is nothing to coordinate.
The larger cost is mixing approaches. A codebase where some jobs use a lock, some check a hostname, and one relies on a replica count of one makes every incident start with archaeology. Engineers first have to work out which mechanism a job uses before they can ask why it ran twice. A single explicit strategy, written down and enforced in code review, turns that question into a lookup. New engineers learn one pattern instead of reverse-engineering several.
How can this approach save engineering time?
Duplicate and missed scheduled runs are expensive mainly because of the human work they create, not the compute they waste. A coordinated, idempotent setup removes or shortens several kinds of that work:
- Less cleanup after duplicate runs. Without a gate and idempotent writes, a duplicate run means finding and deleting duplicate rows, reversing double charges, apologizing for repeated emails and explaining quota overruns to a partner. With unique constraints and conditional state transitions, most duplicates become no-ops that need no cleanup at all.
- Faster diagnosis. Per-run logs with replica identity, plus the started, skipped, failed and last-success metrics, answer "did it run, where, and why not?" directly. Without them, engineers reconstruct the answer from scattered application logs across pods.
- Earlier detection. The last-success alert surfaces a stuck lock or a broken schedule within one interval plus a margin, instead of when a user reports missing data days later. Problems found early are smaller to fix.
- Safer scaling and deploys. Once jobs are coordinated and idempotent, changing the replica count or running a rolling deploy no longer requires checking every scheduled job by hand or temporarily scaling down to one replica.
- Less code to own. Using an established lock library instead of a hand-written mechanism means the acquire, release and expiry logic is maintained once, not re-implemented per job.
- One pattern to learn. A single documented strategy shortens onboarding and code review, because reviewers check new jobs against one known checklist: is the job gated, is its output idempotent, and does it emit the standard metrics.
Illustrative estimate (assumed inputs, not a measured result): assume a team sees 2 duplicate- or missed-run incidents per month, and each takes an assumed 4 engineer-hours to investigate, clean up and communicate. That is 8 engineer-hours per month, or 96 per year. If adopting a lock library, adding idempotency to existing jobs and wiring up the metrics takes an assumed 3 engineer-days (24 hours), the work pays for itself within about 3 months under these assumptions. Substitute your own incident rate and effort figures; the shape of the calculation matters more than these particular numbers.
Alternatives to consider
The three main approaches cover most systems, but a few others are worth knowing, mainly because they move the problem elsewhere.
A dedicated single-replica scheduler process
Split the scheduler out of the web tier into its own deployment pinned to one instance. It reads the schedule and enqueues work, and the existing replicas consume it. This keeps the monolith's code in one repository while removing the "every replica fires" problem at the source. The weakness is availability. During a restart or node failure there is no scheduler, so the missed-run detection from your monitoring has to catch the gap. Orchestrators can also briefly run two copies during a rollout, so idempotency still matters.
Durable job queues with unique-job constraints
Many background job libraries let you enqueue a job with a uniqueness key, such as the job name plus the scheduled timestamp, and reject duplicates at insert time. Every replica can try to enqueue, and only one entry survives. This works well if you already run a queue. It also gives you retries and visibility you would otherwise build yourself.
Database advisory locks
Some databases offer session-scoped advisory locks that are released automatically when the connection drops. This avoids stale lock rows and expiry tuning. In exchange, the lock is tied to connection lifetime, so it behaves badly behind connection poolers that reassign sessions.
Locks in an in-memory store
A key-value store with set-if-absent and a TTL is fast and simple. However, a failover can lose a lock that was just acquired, so treat it as best-effort deduplication rather than mutual exclusion.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to cron jobs multiple replicas, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.