Background Jobs in a Monolith: In-Process vs a Separate Queue

Three ways to run background jobs in a monolith

Most monoliths end up choosing among three designs. Two separate decisions sit underneath them. The first is where the job code runs: inside the web server process or in a separate worker process. The second is where pending jobs are stored: in memory, in your existing database, or in a dedicated broker. A separate worker process can read from a table in your primary database, so a separate process does not require a separate queue technology.

In-process work covers threads, thread pools or async tasks started by the web server. It costs nothing to set up. The job shares memory, CPU and lifetime with request handling, so a deploy or crash discards whatever was queued.

A worker process with a durable job queue runs the same codebase as a different command. It polls a jobs table, usually the same database the app writes to. Because the enqueue is just an insert, it can commit in the same transaction as the business data.

An external message broker is a separate system built for delivery, routing and multiple consumers. It cannot join your database transaction, so you need extra patterns to keep the two consistent.

In-process work suits small apps and work you can afford to lose. A worker with a database queue suits most teams running a monolith. A broker earns its cost when throughput or fan-out exceeds what one database can comfortably serve.

Who should use a worker with a database queue

This design fits teams that already run a monolith on a relational database and have background work that must not be lost: receipts, invoices, webhooks, exports, payment follow-ups, data syncs. The typical profile looks like this:

• One codebase, one primary database. The jobs table lives next to the business tables, and the worker is the same application started with a different command.
• Jobs tied to business writes. If a job should exist exactly when a row commits ("send a receipt for this order"), transactional enqueue removes a whole class of consistency bugs.
• Moderate job volume. The claim queries and row churn fit within the database's existing headroom. Capacity planning has to include them, but they don't dominate.
• Small teams without dedicated platform staff. There is no new system to deploy, secure, back up or page on. The people who already operate the database operate the queue.
• Teams that want to debug with SQL. Pending, running and failed jobs are rows you can query, count, inspect and replay with the tools you already use.

If your background work is purely best-effort, in-process is enough. If several services need to consume the same events, look at a broker. Everything in between is the territory of the database queue.

When to avoid a database-backed queue

A jobs table is the wrong choice in several situations:

• The primary database is already the bottleneck. If it is near its write, connection or vacuum limits, adding constant polling and row churn makes request latency worse. Fix the database first, or put job load elsewhere.
• Many independent consumers need the same event. A table works well for "one worker pool runs this job." It handles "five services each react to order_placed" poorly. That is publish/subscribe, and brokers do it natively.
• Other services or languages must enqueue or consume. Sharing your primary database with outside services couples their deploys and schema to yours.
• Very low pickup latency is required. Polling adds up to one poll interval of delay. If jobs must start within milliseconds, polling is a poor fit. PostgreSQL's LISTEN/NOTIFY can wake workers sooner, but it adds its own complexity.
• Very high sustained throughput. Each job writes several row versions. At high rates, the dead rows and index bloat become a vacuum and storage problem on the database that serves your users.
• Work that is genuinely losable. Cache warming or best-effort pings don't need durability. A table adds writes for no benefit.
• Long, multi-step workflows with human waits or compensation logic. A single jobs table can express these, but you end up building a workflow engine by hand. Consider a tool designed for durable workflows.

How it works: the lifecycle of one job

Here is the full path of one job, before the details in the next section:

1. Enqueue. Application code inserts a row into jobs in the same transaction as the business write. If the transaction rolls back, the job disappears with it.
2. Claim. A worker process polls for rows that are pending and due. It locks one with FOR UPDATE SKIP LOCKED so concurrent workers never claim the same row, marks it running and sets a lease (locked_until).
3. Execute. The worker loads current state by ID from the payload and runs the handler. Long jobs extend their lease with a heartbeat.
4. Finish. On success the row is marked done. On failure the attempt count increases and the row is rescheduled with backoff, or moved to failed once it reaches the attempt cap.
5. Recover. If the worker dies, its lease expires and a reaper or the next claim query returns the job to the queue.
6. Clean up. A scheduled delete or archive removes old finished rows so the table and its indexes stay small.

Scheduled tasks plug into the same flow: a single scheduler only performs step 1, guarded by a unique run key.

Why in-process jobs lose work and slow down requests

An in-process job lives only in the memory of the process that accepted it. If that process stops, the job is gone. Nothing on disk records that it existed, so nothing can retry it.

Processes stop more often than most teams expect:

• Deploys. A rolling restart sends a termination signal. Graceful shutdown waits for in-flight requests, but it rarely waits for a thread pool working through a backlog of queued emails.
• Crashes. An unhandled exception or a segfault in a native library takes every pending job with it.
• OOM kills. The kernel or container runtime kills the process without warning. There is no shutdown hook and no chance to flush anything.
• Autoscaler scale-in. The autoscaler removes instances because request traffic is low. It does not know that one of those instances holds 400 unsent webhooks.

The second problem is contention. The job runs on the same CPU, in the same heap, and against the same database connection pool as your request handlers. Suppose a report export holds three connections for forty seconds and the pool has ten. Requests now compete for seven, and when those run out they wait. Latency rises even though request traffic hasn't changed. A memory-heavy job also adds garbage-collection pressure, which pauses request threads too. None of this appears in job logs. It shows up as unexplained tail latency on endpoints that have nothing to do with the job.

In-process work is still a reasonable choice when losing a job costs nothing. Cache warming, best-effort analytics pings, and non-critical metrics flushes all qualify. If a missed run will quietly fix itself on the next request or interval, keep it in-process. If someone would notice or complain about a missed run, the job needs durable storage.

How a database-backed job queue works

The common next step is a jobs table in the database you already run. Each row carries a payload plus status, attempts, run_at (earliest execution time) and locked_until (a lease that expires if a worker dies mid-job).

Copy+-------------+        +-------------------------+        +----------------+
| web process | -----> | database                | <----- | worker process |
|             |  one   |  business tables        | claim  |  poll, run,    |
|             |  txn   |  jobs table             | update |  mark done     |
+-------------+        +-------------------------+        +----------------+

A worker claims work with SELECT ... FOR UPDATE SKIP LOCKED, filtered on status = 'pending' AND run_at <= now(). This is PostgreSQL syntax. MySQL 8 supports a similar clause, and other databases need different locking. SKIP LOCKED lets concurrent workers pass over rows another worker already holds, so they don't block each other. The worker sets locked_until, commits, and runs the job. When the queue is empty it polls again after an interval, typically one to a few seconds, which trades pickup latency against idle queries. A partial index on run_at WHERE status = 'pending' keeps each claim cheap no matter how many finished rows exist. Delete or archive completed rows on a schedule so the table stays small.

The main advantage is transactional enqueue. The job row is inserted in the same transaction as the business write, so both commit or neither does. A job can't exist for an order that rolled back, and an order can't commit without its job. With an external broker you get the same guarantee only by adding an outbox table and a relay that forwards rows to the broker.

The cost lands on your primary: extra writes, constant row churn, and dead tuples that put pressure on vacuum and cause index bloat.

Copy-- PostgreSQL. Job table with a lease column for crash recovery.
CREATE TABLE jobs (
  id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind         text        NOT NULL,
  payload      jsonb       NOT NULL,
  status       text        NOT NULL DEFAULT 'pending', -- pending | running | done | failed
  attempts     int         NOT NULL DEFAULT 0,
  run_at       timestamptz NOT NULL DEFAULT now(),     -- earliest execution time
  locked_until timestamptz                             -- lease; NULL when unclaimed
);

-- Index only claimable rows, so finished rows don't slow down claims.
CREATE INDEX jobs_claimable ON jobs (run_at) WHERE status = 'pending';

-- Transactional enqueue: the order and its job commit or roll back together.
BEGIN;
INSERT INTO orders (id, customer_id, total_cents) VALUES (1042, 7, 4999);
INSERT INTO jobs (kind, payload) VALUES ('send_receipt', '{"order_id": 1042}');
COMMIT;

-- Worker claim: SKIP LOCKED lets concurrent workers pass over held rows.
UPDATE jobs
SET status = 'running',
    attempts = attempts + 1,
    locked_until = now() + interval '5 minutes'
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_at <= now()
  ORDER BY run_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, kind, payload;

-- Periodic reaper: return jobs whose worker died (lease expired) to the queue.
UPDATE jobs SET status = 'pending', locked_until = NULL
WHERE status = 'running' AND locked_until < now();

-- Scheduled cleanup to limit row churn and table growth.
DELETE FROM jobs WHERE status = 'done' AND run_at < now() - interval '7 days';

A PostgreSQL jobs table with a partial index, a transactional enqueue alongside a business write, a SKIP LOCKED claim that sets a lease, a reaper for expired leases, and scheduled cleanup of finished rows.

Idempotency, retries and failure handling

Whichever option you pick, delivery is at least once. A worker can finish the work and crash before marking the job done, and the job runs again. Design every handler to tolerate a second run.

Making handlers idempotent

• Idempotency keys. Derive a key from the job's intent, such as invoice:8812:send, and store it under a unique constraint. A duplicate insert fails cleanly instead of doing the work twice.
• State checks before writes. Update with a guard such as WHERE status = 'pending', so a replay changes zero rows.
• Record external side effects first. Charges, emails and third-party calls cannot be rolled back. Write an "attempted" row before the call and pass the same key to providers that accept one. On retry, check that row instead of charging again.

Retries and failure

Use exponential backoff with jitter. Without jitter, jobs that failed together retry together and hit the struggling dependency all at once. Cap the number of attempts, then move the job to a failed state that keeps the last error and stack trace so someone can inspect it and replay it.

Lock or visibility timeouts recover jobs from crashed workers. A timeout shorter than the job's real runtime is dangerous: a second worker picks up a job that is still running. Either heartbeat to extend the lock, or split long jobs into resumable chunks that checkpoint their progress.

Strict ordering or per-entity serialization requires partitioning or single-consumer lanes. Either one limits throughput, and one slow entity can stall its whole lane.

Copy-- PostgreSQL. Ledger of external side effects, keyed by intent.
CREATE TABLE side_effects (
  idem_key   text PRIMARY KEY,            -- e.g. 'invoice:8812:send'
  status     text NOT NULL DEFAULT 'attempted',
  created_at timestamptz NOT NULL DEFAULT now()
);

-- Record the attempt BEFORE calling the provider. Zero rows inserted means
-- an earlier run got here first: check its status, don't send/charge again.
INSERT INTO side_effects (idem_key) VALUES ('invoice:8812:send')
ON CONFLICT (idem_key) DO NOTHING;

-- Guarded transition: a replay matches no row and changes nothing.
UPDATE invoices SET status = 'sent' WHERE id = 8812 AND status = 'pending';

-- Claim one job. Expired locks are reclaimed from crashed workers.
-- The 5-minute timeout must exceed real runtime (or be extended by heartbeat).
UPDATE jobs SET status = 'running', locked_until = now() + interval '5 minutes'
WHERE id = (
  SELECT id FROM jobs
  WHERE (status = 'queued' AND run_at <= now())
     OR (status = 'running' AND locked_until < now())
  ORDER BY run_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, payload;

-- Heartbeat from a long-running handler to keep its lock alive.
UPDATE jobs SET locked_until = now() + interval '5 minutes'
WHERE id = $1 AND status = 'running';

-- On error: exponential backoff with full jitter (capped at 1h), then 'failed'.
UPDATE jobs SET
  attempts     = attempts + 1,
  last_error   = $2,                      -- keep message + stack trace for replay
  status       = CASE WHEN attempts + 1 >= max_attempts THEN 'failed' ELSE 'queued' END,
  run_at       = now() + random() * least(3600, 2 ^ (attempts + 1)) * interval '1 second',
  locked_until = NULL
WHERE id = $1;

PostgreSQL statements for a side-effect ledger keyed by idempotency key, a guarded state update, job claiming with a reclaimable lock timeout and heartbeat, and capped exponential backoff with jitter that ends in a failed state holding the last error.

Scheduled tasks without duplicate runs

Scheduled tasks are a different problem from queued jobs. A queued job exists because something created it once. A scheduled task exists because a clock fired, and if every app instance carries the same crontab, the clock fires once per instance. Scale to four instances and the nightly invoice run sends four invoices.

Separate scheduling from execution

Keep one scheduler, and limit it to enqueuing. It inserts a row into the job queue and returns. Workers do the actual work, with the retries and idempotency they already have. Because the scheduler does almost nothing, it is easy to reason about and cheap to restart.

To make each run happen exactly once, give it an identity: a unique constraint on (job_name, scheduled_for). If two schedulers race, or one restarts mid-tick, the second insert fails on the constraint and is discarded. The database enforces uniqueness, so you don't depend on timing for it.

Decide what downtime means

If the scheduler is down from 01:00 to 04:00, you need to choose what happens to the hourly job's missed runs:

• Skip them. This suits cache warming.
• Run once to catch up. This suits "sync everything changed since last run."
• Backfill each missed slot. This suits per-period reports.

Pick one per job and write it down. Otherwise whatever the code happens to do becomes the policy.

The single scheduler problem

A single scheduler is a single point of failure. Running several with leader election or a lock fixes availability but introduces new failure modes. In a split-brain, a paused leader resumes believing it still leads. A lock can also expire during a slow run, so a second node starts the same job. The unique run key catches the duplicate enqueue. Long jobs still need fencing or idempotent writes.

Copy-- PostgreSQL. One row per (job, slot): the run's identity lives in the database.
CREATE TABLE scheduled_runs (
  id            bigserial   PRIMARY KEY,
  job_name      text        NOT NULL,
  scheduled_for timestamptz NOT NULL,
  enqueued_at   timestamptz NOT NULL DEFAULT now(),
  UNIQUE (job_name, scheduled_for)
);

-- Scheduler tick for an hourly job. The scheduler only enqueues; workers do the work.
-- If a second scheduler, or a restarted one, runs the same tick, the insert is a
-- no-op and RETURNING yields zero rows.
INSERT INTO scheduled_runs (job_name, scheduled_for)
VALUES ('hourly_sales_report', date_trunc('hour', now()))
ON CONFLICT (job_name, scheduled_for) DO NOTHING
RETURNING id;

-- Backfill policy for per-period jobs: after downtime, enqueue every missed slot
-- since the last recorded run. Slots that already exist are skipped by the
-- constraint, so running this repeatedly is safe.
-- If the job has no rows yet, max() returns NULL, generate_series yields no slots,
-- and nothing is backfilled.
INSERT INTO scheduled_runs (job_name, scheduled_for)
SELECT 'hourly_sales_report', slot
FROM generate_series(
  (SELECT max(scheduled_for) + interval '1 hour'
     FROM scheduled_runs
    WHERE job_name = 'hourly_sales_report'),
  date_trunc('hour', now()),
  interval '1 hour'
) AS slot
ON CONFLICT (job_name, scheduled_for) DO NOTHING;

-- For the skip policy, run only the first INSERT. For run-once, enqueue only the
-- current slot and let the worker process everything changed since the last run.

A PostgreSQL unique constraint on (job_name, scheduled_for) makes duplicate enqueues from racing or restarted schedulers harmless no-ops, and the same table supports an explicit backfill policy for runs missed during downtime.

Running and monitoring the worker process

Ship the worker from the same build artifact as the web app, with the same configuration source and a different entrypoint. One image means one dependency set and one config loader, so the worker cannot quietly run different code from the web tier.

Shutdown and scaling

On SIGTERM, the worker should stop claiming new jobs first. It then lets in-flight jobs finish within the deploy grace period. Any job still running when time runs out should have its claim released explicitly, so it becomes ready again at once rather than waiting for a lease timeout. Run workers as their own process group with their own replica count and memory and CPU limits. A memory-hungry export job should never evict web pods.

Deploy hazards

During a rolling deploy, old and new workers run side by side. Old workers will pick up jobs enqueued by new code, and new workers will pick up jobs enqueued by old code. Apply migrations before any code that depends on them, and keep schema changes backward compatible for one release. Keep payloads small and include a version field. Pass record IDs instead of serialized objects, so the worker reads current state rather than a stale snapshot written by older code.

What to monitor

Alert on the age of the oldest ready job. Depth alone misleads: 10,000 fast jobs can clear in seconds, while a single job that has waited 20 minutes means a user is still waiting for it.

Copyimport os, signal, threading
from concurrent.futures import ThreadPoolExecutor, wait

from app.config import load_config      # the loader the web tier uses
from app.jobs import queue, handlers    # app's queue client and handler registry

GRACE_SECONDS = 25  # keep below the orchestrator's termination grace period (e.g. 30s)
stopping = threading.Event()
signal.signal(signal.SIGTERM, lambda signum, frame: stopping.set())

def run(job):
    try:
        # Payload holds a version and an ID, not a serialized object.
        handler = handlers[(job.type, job.payload["v"])]
        handler(job.payload["record_id"])   # handler loads current state by ID
        queue.complete(job.id)
    except Exception as exc:
        queue.fail(job.id, error=repr(exc))  # counts toward retries / dead jobs

def main():
    config = load_config()
    pool = ThreadPoolExecutor(max_workers=config.worker_concurrency)
    in_flight = {}
    while not stopping.is_set():            # 1. claim only while not shutting down
        job = queue.claim(timeout=1)
        if job:
            in_flight[pool.submit(run, job)] = job
        in_flight = {f: j for f, j in in_flight.items() if not f.done()}

    # 2. Give in-flight jobs the rest of the grace window to finish.
    _, unfinished = wait(in_flight, timeout=GRACE_SECONDS)
    # 3. Release what is left so it is ready again now, not after a lease timeout.
    for fut in unfinished:
        queue.release(in_flight[fut].id)
    # Exit without joining stuck threads; their claims are already released.
    os._exit(0)

if __name__ == "__main__":
    main()

A worker entrypoint that stops claiming on SIGTERM, waits for in-flight jobs within the grace window, then releases unfinished claims; the queue client and its methods are assumed app interfaces.

When to move to a dedicated message broker, and what it costs

Polling is cheap until the rate rises. A database queue's load comes from two sources: claim queries, which run even when there is no work, and row writes, which create dead rows that vacuum or compaction must clean up on your primary.

Illustrative estimate (assumed numbers, not a benchmark). Assume 20 workers, a 1-second poll interval when idle, 50 jobs/s sustained, and 3 writes per job (insert, claim, mark done).

• Idle claim queries/s = workers ÷ poll interval = 20 ÷ 1 = 20/s, or 72,000/hour doing nothing
• Busy row writes/s = jobs/s × writes per job = 50 × 3 = 150/s, around 13 million row versions a day

These run on the same disk, WAL and connection pool that serve your requests. That is acceptable if your capacity planning includes it. It is a signal to move if you see it in your latency graphs.

A broker has real costs. It is another system to deploy, patch, secure and back up. It adds a failure domain. You also lose transactional enqueue, so you need an outbox table to recover it, which means a database queue still exists. Starting with a table costs one migration and one extra entrypoint. Move when one of these signals actually appears, not in anticipation.

Alternatives to a database-backed queue

The jobs table is a middle option. These are the realistic alternatives and where each fits:

• In-process threads or async tasks. These need no setup and no extra process. Use them for work that is losable and short, and accept that deploys, crashes and scale-in discard pending work.
• An external message broker. A dedicated system for delivery, routing and fan-out. It suits high throughput, multiple independent consumers, and cross-service events. It costs another system to operate and needs an outbox to stay consistent with your database.
• A key-value or in-memory data store used as a queue. Many job frameworks use one. It moves polling and churn off the primary database and offers low pickup latency, but it is a second datastore to operate, its durability depends on how it is configured, and enqueue is not transactional with your business writes.
• The job library your framework already provides. Most mature web frameworks have a background job library, and several support a database backend. Using one gets you retries, scheduling and an admin view without writing the claim loop yourself. The design in this article is still the mental model for how it behaves and fails.
• OS cron or the platform's scheduled-job feature. This is enough for a few periodic tasks if exactly one instance runs them. It does not give you retries, backoff, or protection against duplicate runs unless you add them, for example by having cron only enqueue into the jobs table.
• Database notifications plus a table. In PostgreSQL, LISTEN/NOTIFY can wake idle workers when a job is inserted, which cuts pickup latency and idle polling. Keep the table as the source of truth, because notifications are not stored for disconnected listeners.
• A durable workflow engine. For long, multi-step processes with waits, timers and compensation, a workflow engine stores progress for you. It is more machinery than a jobs table and only pays off when workflows are genuinely complex.

How this approach saves engineering time

Most of the savings come from problems you never have to solve rather than code you write faster.

• No consistency reconciliation. Transactional enqueue means no "order committed but email never queued" bugs, no outbox relay to build, and no scripts that compare business tables against broker state after an incident.
• No new system to run. There is nothing extra to provision, patch, secure, back up, monitor or learn. On-call engineers debug the queue with the database knowledge they already have.
• One codebase and one deploy. The worker ships from the same artifact with a different entrypoint, so there is no second repository, build pipeline or dependency set, and no version drift between web and worker code.
• Debugging with SQL. "Why didn't this customer get a receipt?" becomes a query against jobs filtered by payload, not a search across broker dashboards and log streams. Failed rows keep the last error, so replaying is an UPDATE.
• Fewer lost-work incidents. Moving work out of in-process threads removes the class of bugs where deploys, OOM kills or scale-in silently drop jobs, along with the investigation time each of those incidents costs.
• Fewer latency mysteries. Running jobs in a separate process with its own resource limits stops background work from causing unexplained tail latency on unrelated endpoints.
• A cheap path forward. If you later outgrow the table, the jobs table becomes your outbox. Handlers that are already idempotent and payloads that already carry IDs and versions move to a broker with little rework.

Illustrative estimate (assumed numbers, not a measurement). Assume an in-process setup loses work during deploys about twice a month, and each incident takes an engineer 3 hours to notice, trace and fix by hand. That is about 6 hours a month, or roughly 72 hours a year. Assume moving to a worker with a jobs table takes about 2 to 3 days of work. Under these assumptions the change pays for itself within the first few months, before counting the cost of the missed emails or webhooks themselves. Substitute your own incident rate and effort to see whether the same holds for you.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to background jobs in monolith, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.