Why do scheduled jobs run multiple times when a monolith scales horizontally?

A scheduler embedded in your application has no idea other copies of the application exist. When you register a job such as "send the daily digest at 08:00," every replica that boots loads the same code, starts the same in-process scheduler, and registers the same trigger. If three instances are running at 08:00, the job runs three times. Nothing failed: each scheduler did what it was built to do, and none of them had a way to coordinate with the others.

The number of duplicates also changes over time. During a rolling deploy, old and new instances overlap, so the count briefly goes up. Autoscaling adds replicas under load and removes them later. A job that ran twice yesterday might run four times during today's deploy window.

Illustrative estimate (assumed numbers, not a benchmark): extra runs per day = (instances alive at trigger time − 1) × triggers per day. Assume an hourly job (24 triggers) with 3 steady instances, plus 2 triggers that fall during a deploy when 5 instances overlap. That gives (2 × 22) + (4 × 2) = 52 unwanted runs per day.

This article is for teams running a horizontally scaled monolith with an embedded scheduler. The problem matters most for jobs with external side effects, such as sending email, charging cards, or pushing exports to partners. You cannot take those actions back once they happen.

The thesis of this article is that you need two layers. A lock reduces how often duplicates happen. Idempotency makes sure the duplicates that still get through do no harm.

            job registered in code: "daily digest @ 08:00"
                                    |
        +---------------------------+---------------------------+
        |                           |                           |
        v                           v                           v
+----------------+          +----------------+          +----------------+
|  Replica A     |          |  Replica B     |          |  Replica C     |
|  monolith      |          |  monolith      |          |  monolith      |
| +------------+ |          | +------------+ |          | +------------+ |
| | in-process | |          | | in-process | |          | | in-process | |
| | scheduler  | |          | | scheduler  | |          | | scheduler  | |
| +-----+------+ |          | +-----+------+ |          | +-----+------+ |
+-------|--------+          +-------|--------+          +-------|--------+
        | 08:00 fire                | 08:00 fire                | 08:00 fire
        v                           v                           v
+------------------------------------------------------------------------+
|   External side effects: email / billing / partner export (no undo)    |
|             result: 3 replicas -> each user gets 3 digests             |
+------------------------------------------------------------------------+

  no shared state between schedulers  -->  runs = instances alive at 08:00
  rolling deploy / autoscaling        -->  that instance count changes

Each replica boots the same in-process scheduler with the same trigger, so one registered job fires once per live instance and repeats an irreversible side effect, and deploys and autoscaling change how many times that happens.

How do Postgres advisory locks prevent duplicate cron runs?

Postgres advisory locks are application-defined locks keyed by a 64-bit integer. Postgres attaches no meaning to the number. It only guarantees that one session holds a given key at a time. That is enough to make every instance agree on which one runs a job.

The flow for each instance at tick time is simple. Call pg_try_advisory_lock(key). If it returns false, log "skipped, held elsewhere" and exit without retrying. If it returns true, run the job, then call pg_advisory_unlock(key) in a finally block. The try variant matters because the blocking pg_advisory_lock would queue every instance and run the job once per instance, one after another. If the work fits inside one transaction, pg_try_advisory_xact_lock(key) is simpler. Postgres releases the lock at commit or rollback, so there is no unlock call to forget.

Deriving the key

Hash the job name to a bigint in application code, for example by taking the first eight bytes of a SHA-256 of "billing.nightly-invoices". Avoid relying on internal Postgres hash functions, whose output is not promised to stay stable across upgrades.

Pitfalls

Postgres releases the lock automatically when the connection dies, which is why a crashed instance cannot leave the lock held forever. The same session binding causes three problems:

  • Transaction-mode poolers can run your lock and unlock on different backend connections. The lock silently stops protecting anything.
  • Long-running jobs pin a dedicated connection for their entire runtime.
  • Failover drops the lock mid-run. Another instance can then acquire it while the first instance is still working.
import hashlib
import logging
import psycopg  # psycopg 3

log = logging.getLogger("cron")

def lock_key(job_name: str) -> int:
    # Deterministic on every host: first 8 bytes of SHA-256 as a signed bigint.
    digest = hashlib.sha256(job_name.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big", signed=True)

def run_exclusive(dsn: str, job_name: str, job) -> bool:
    key = lock_key(job_name)
    # Connect directly, not via a transaction-mode pooler, so lock and unlock share one session.
    with psycopg.connect(dsn, autocommit=True) as conn:
        got = conn.execute("SELECT pg_try_advisory_lock(%s)", (key,)).fetchone()[0]
        if not got:
            log.info("%s skipped, held elsewhere", job_name)
            return False  # skip this tick, no retry
        try:
            job()  # this connection stays pinned until the job finishes
        finally:
            # If the connection died, Postgres already released the lock and this call fails.
            conn.execute("SELECT pg_advisory_unlock(%s)", (key,))
        return True

def run_in_tx(dsn: str, job_name: str, job) -> bool:
    key = lock_key(job_name)
    with psycopg.connect(dsn) as conn, conn.transaction():
        # Released automatically at commit or rollback; no unlock call needed.
        got = conn.execute("SELECT pg_try_advisory_xact_lock(%s)", (key,)).fetchone()[0]
        if not got:
            log.info("%s skipped, held elsewhere", job_name)
            return False
        job(conn)  # all work must happen inside this transaction
        return True

run_exclusive("postgresql://app:changeme@db.internal:5432/app", "billing.nightly-invoices", lambda: None)

Hashing a job name to a stable bigint key, then using session-level pg_try_advisory_lock with an unlock in finally, or transaction-scoped pg_try_advisory_xact_lock, so that only one instance runs the job and the others skip the tick.

How does a ShedLock-style lock table work, and how do you set lockAtMostFor?

A lock table avoids pinned sessions entirely. Each job gets one row with four columns: name (primary key), lock_until, locked_at and locked_by (usually the hostname). To acquire a lock, an instance tries an INSERT for a new job name. If the row exists, it runs a conditional UPDATE shedlock SET lock_until = :until, locked_at = :now, locked_by = :me WHERE name = :job AND lock_until <= :now. Exactly one instance sees one affected row and runs the job. The others see zero and skip.

Because this is a single-row compare-and-set, it works on any SQL store with atomic updates. It also works on Redis with SET key value NX PX ttl. Nothing holds a connection open between acquiring and releasing the lock. Each job also has its own row, so you can tune it independently.

With ShedLock, you annotate the method with @SchedulerLock(name = "nightlyInvoices", lockAtMostFor = "PT30M", lockAtLeastFor = "PT1M").

Sizing the two durations

  • lockAtMostFor sets when the lock expires on its own. If it is shorter than the job's worst-case runtime, a second instance acquires the lock mid-run and the runs overlap. If it is too long, a crashed holder blocks the job until the lock expires. Size it from observed p99 runtime plus headroom.
  • lockAtLeastFor keeps the lock held after a fast job finishes. Without it, an instance whose clock runs a few seconds behind can start the same scheduled run after the first instance has already released the lock.

The costs

Every trigger writes to the primary database, which adds load as job frequency grows. Scheduling also now depends on that database being available. If it is down, no jobs run.

-- One row per job. The primary key on name guarantees a single lock row per job.
CREATE TABLE shedlock (
    name       VARCHAR(64)  NOT NULL,
    lock_until TIMESTAMP    NOT NULL,  -- lock expires on its own after this (lockAtMostFor)
    locked_at  TIMESTAMP    NOT NULL,  -- when the current holder acquired it
    locked_by  VARCHAR(255) NOT NULL,  -- usually the hostname of the holder
    PRIMARY KEY (name)
);

-- First acquisition for a new job name. If the row already exists,
-- the primary key violation tells the caller to fall through to the UPDATE.
INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('nightlyInvoices', :until, :now, 'app-node-2');

-- Compare-and-set. Only an expired lock can be taken over.
-- Exactly one instance gets 1 affected row and runs the job.
-- Every other instance gets 0 rows and skips this trigger.
UPDATE shedlock
   SET lock_until = :until,        -- :now + lockAtMostFor, e.g. 30 minutes
       locked_at  = :now,
       locked_by  = 'app-node-2'
 WHERE name = 'nightlyInvoices'
   AND lock_until <= :now;

-- Release after the job finishes. The lock is not cleared outright.
-- lock_until moves back to locked_at + lockAtLeastFor, or stays at now
-- if that point has already passed, so a peer whose clock lags slightly
-- cannot rerun the same scheduled slot.
UPDATE shedlock
   SET lock_until = :release_at    -- GREATEST(locked_at + lockAtLeastFor, now)
 WHERE name = 'nightlyInvoices'
   AND locked_by = 'app-node-2';

This example shows the lock-table schema and the full cycle for one job: the INSERT for a new job, the conditional UPDATE that lets only one instance take an expired lock, and the release that keeps the lock held for at least lockAtLeastFor.

When should you use leader election instead of per-job locks?

With leader election, instances compete for a single lease, and only the holder runs the scheduler. The lease can be a Kubernetes Lease object, a key in etcd or Consul tied to a session, or a database row with an owner and an expiry timestamp.

Lease timing

The lease duration sets how long the system goes without a scheduler when a leader dies. The leader renews on a shorter interval, often a third of the duration or less, so a slow renewal does not cost it the lease.

A new leader must not just start the clock. It should read the last recorded run time for each job, decide whether a missed window needs a catch-up run or can be skipped, and look for runs the old leader started. For each one, it either resumes the run or marks it abandoned.

When it fits

Leader election suits a monolith with many jobs, where one lease is cheaper than a lock per trigger. It also suits schedulers that hold state in memory, such as queues or backoff timers, across runs.

The costs

All job load lands on one instance. Deploys move leadership, which can leave a gap. Misconfigured timing, such as a duration close to the renewal interval or clock skew between nodes, can cause brief double leadership.

A lease alone cannot guarantee mutual exclusion. A GC pause, a network partition, or a run that outlives the lease can let a second leader start while the first still believes it holds the lease. To prevent this, use fencing tokens. Each lease grant carries a number that always increases. Every write sends the token with it, and the storage layer rejects any token lower than the highest one it has already seen.

What are the alternatives to in-app scheduling locks?

Another option is to take the schedule out of the application. The app instances then don't decide when a job runs. An external scheduler fires once, and the application only does the work.

Kubernetes CronJob

A Kubernetes CronJob can run the job as its own container, or as a short-lived pod that calls an internal endpoint. Two fields matter in the manifest:

  • concurrencyPolicy: Forbid skips a new run if the previous Job is still active. This prevents overlap.
  • startingDeadlineSeconds limits how late a missed run may start. If the controller is down or slow beyond that window, the run is skipped. Without this field, a backlog of missed runs can trigger an unexpected catch-up.

The Kubernetes documentation states that a CronJob creates a Job approximately once per scheduled time. In edge cases it can create two Jobs or none. Pods are also retried according to backoffLimit, and a pod on an unreachable node may be replaced while the original is still running. The platform scheduler reduces duplicates, but it does not remove them.

A single cron host

A dedicated host can run plain cron that calls an endpoint or enqueues a message. This is simpler, but that host becomes a single point of failure. A missed run there fails silently unless you alert on it.

The cost

Job logic lives in the application repository, while schedules live in manifests or crontabs owned by another pipeline. Renaming an endpoint or changing a job's arguments now requires coordinated changes in two deploys.

This approach fits best when the operations team already owns cron and the jobs are triggered rather than stateful. In that case, a job is a thin call into code that is already idempotent, as covered in the next section.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-invoice-run
spec:
  schedule: "0 2 * * *"            # 02:00 every day (controller time zone)
  concurrencyPolicy: Forbid         # skip this run if the previous Job is still active
  startingDeadlineSeconds: 300      # if it can't start within 5 minutes, skip it; no catch-up
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1               # each retry is another call; the endpoint must be idempotent
      activeDeadlineSeconds: 900    # kill a hung run so Forbid doesn't block later runs
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: trigger
              image: curlimages/curl:8.5.0
              # Thin trigger: the monolith does the work; this pod only fires the request.
              # The run key lets the app deduplicate if Kubernetes starts two Jobs.
              command: ["sh", "-c"]
              args:
                - >
                  curl --fail --silent --show-error --max-time 600
                  -X POST
                  -H "Authorization: Bearer $JOB_TOKEN"
                  -H "Idempotency-Key: invoice-run-$(date -u +%Y-%m-%d)"

              env:
                - name: JOB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: job-trigger-token
                      key: token

A Kubernetes CronJob that uses concurrencyPolicy: Forbid and startingDeadlineSeconds to call an internal endpoint once per night, and sends a per-day idempotency key so the application can drop the occasional duplicate Job.

How do you make scheduled jobs idempotent so duplicates are harmless?

Idempotency is the layer that keeps the whole design safe when any of those mechanisms fails. Start by deriving a logical run window from the schedule instead of from now(). An hourly job truncates its intended fire time to the hour. Two instances whose clocks differ by a few seconds then compute the same scheduled_window and compete for the same unit of work, rather than for two neighbouring ones.

Record each window in a run ledger. The table holds job_name, scheduled_window, a sequence-backed fencing_token, status, and finished_at, with a unique constraint on (job_name, scheduled_window). An instance claims a window with INSERT ... ON CONFLICT (job_name, scheduled_window) DO NOTHING RETURNING fencing_token. If the statement returns zero rows, another instance owns that window. Inside the job, write results with upserts and set processed-marker columns such as processed_at, so reprocessing a row is a no-op.

Fencing tokens handle a worker that stalls and resumes after its claim has expired. Each downstream UPDATE includes WHERE last_token < :token and sets last_token = :token, so a stale writer matches nothing.

Lock-and-skip semantics also have a gap. If the instance that holds the lock crashes, a duplicate becomes a missed run. Add catch-up logic that checks recent windows on every tick. It processes any window that has no ledger row, and it reclaims, with a fresh token, any window stuck in running past a timeout.

This adds a table, a migration, and extra branching code. You pay that cost once. The alternative is repeated incident work: hand-diffing duplicate invoices, writing one-off repair scripts, and reconstructing which window actually ran.

Which approach should you choose, and how do you know it is working?

Start from what you already run. If Postgres is your only data store and you have a handful of jobs, advisory locks add no new tables or services. If you need per-job control or your jobs touch several stores, use a lock table or ShedLock, because the lock state is visible and you can query it. If you have many jobs or a stateful scheduler that runs for a long time, use leader election so you coordinate once rather than per job. If your operations team already owns cron or a workflow platform, move the trigger out of the app.

Option Failure mode Dependency Dev / ops cost Avoid when
Advisory locks Lock is lost when the session drops, and poolers can break it Postgres Low / low Using transaction-mode pooling or non-Postgres stores
Lock table / ShedLock Job runs again early if lockAtMostFor is too short Any shared DB Low / medium Job duration is unpredictable
Leader election Split brain during failover Consensus store or DB lease Medium / medium You only have a few short jobs
External scheduler The trigger fires but the app is unreachable Platform scheduler Low / owned elsewhere Jobs need in-process state

Every one of these options can fail silently by skipping runs. A stale lock or a lost leader looks the same as a quiet night.

  • Log every attempt with the holder and the result, for example {"job":"invoice_sweep","window":"2026-11-10T02:00Z","holder":"web-3","lock":"acquired","duration_ms":4120}.
  • Count outcomes with scheduler_job_runs_total{job,result="executed|skipped|failed"}. Each window should have exactly one executed.
  • Alert on absence with scheduler_job_last_success_timestamp_seconds. Fire when the gap since the last success exceeds twice the schedule interval.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to prevent duplicate cron jobs multiple instances, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.