How to Speed Up a Monolith Test Suite: Parallel Tests and Module Splits

Where does the time in a slow monolith test suite actually go?

Before adding workers or CI machines, find out where the minutes go. Most test runners can emit JUnit-style XML reports. Each testcase element records a time attribute along with a classname (and sometimes a file). Collect these reports from several recent CI runs on the main branch, sum durations per file or class, and rank them. In large monoliths a small set of files often accounts for a disproportionate share of runtime. These are frequently integration tests that hit a real database or sleep on timeouts.

The XML only covers time spent inside tests. Compare its total against the CI job's wall-clock time. The gap is fixed cost: container boot, dependency install, asset compilation, database schema creation and seed loading. Each job pays fixed costs once, so splitting the suite across machines multiplies them, while per-test costs shrink with parallelism.

Illustrative estimate (assumed numbers, not a benchmark)

Assume fixed cost F = 6 min, serial test time T = 40 min, and N shards.

• Wall time ≈ F + T / N
• Total machine time ≈ N × F + T

With N = 1: 46 min wall time, 46 machine-minutes. With N = 8: 11 min wall time, 88 machine-minutes. Fixed cost now makes up more than half of the compute you pay for.

This is why profiling comes first. Backend and platform engineers who own large shared suites need to know whether they are fighting slow tests or slow setup, because each problem has a different fix.

Copy#!/usr/bin/env python3
"""Rank the slowest test files/classes across several JUnit-style XML reports.
Usage: python3 rank_slow.py reports/run-*/junit*.xml
"""
import sys
import xml.etree.ElementTree as ET
from collections import defaultdict

totals = defaultdict(float)
for path in sys.argv[1:]:
    root = ET.parse(path).getroot()
    # iter() finds every <testcase>, whether the root is <testsuites> or <testsuite>
    for case in root.iter("testcase"):
        # Group by file when the runner emits it; otherwise fall back to classname
        key = case.get("file") or case.get("classname") or "unknown"
        totals[key] += float(case.get("time") or 0)

grand = sum(totals.values())
if grand == 0:
    sys.exit("no testcase time found")

print(f"in-test time: {grand / 60:.1f} min summed over {len(sys.argv) - 1} report files")
print("Compare against CI wall-clock time. The difference is fixed cost (boot, install, schema).")

cumulative = 0.0
for key, secs in sorted(totals.items(), key=lambda kv: kv[1], reverse=True)[:20]:
    cumulative += secs
    # The cumulative share shows how few files account for most of the runtime
    print(f"{secs:9.1f}s  {100 * secs / grand:5.1f}%  cum {100 * cumulative / grand:5.1f}%  {key}")

A standard-library Python script that sums testcase durations from JUnit-style XML reports by file or class, ranks the 20 slowest, and prints the in-test total so you can compare it with CI wall-clock time and see how much is fixed cost.

Who should use these techniques?

The techniques fit teams that share one large codebase and one test suite, where many people merge into the same branch and wait on the same CI pipeline. In practice that means:

• Backend and platform engineers who own the shared suite. They decide how the runner is configured, how databases are provisioned for tests, and how CI jobs are laid out.
• CI and build maintainers. They control runner sizes, caching, artifacts, and job fan-out, which is where sharding and setup reductions live.
• Feature teams on a monolith. They benefit from module-based selection on pull requests, and they are usually the ones who must fix tests that break under parallel execution.

The techniques fit less well in two situations. The first is a small suite whose runtime is mostly setup. Parallelism cannot shrink that setup, so profiling and setup reduction are the only useful steps. The second is a suite that depends heavily on shared mutable state, such as one database, fixed file paths, or global singletons. That team should do the isolation work in "Making tests safe to run concurrently" before adding workers. Otherwise the first result of parallelism is a wave of flaky failures.

When should you parallelize, shard, or select tests?

Start when CI feedback time changes how people work: developers switch to other tasks while waiting, batch unrelated changes to avoid extra runs, or merge queues back up behind the test job. The profile from the previous section tells you which technique to use first.

• Test time T is much larger than fixed setup S, and the runner has idle cores. Add multiple workers on one machine. This is the cheapest step and needs no CI restructuring.
• The runner's cores or memory are saturated and T still dominates. Shard across CI jobs, using historical timings to balance them.
• Most pull requests touch one or two modules. Add module-based selection for pull requests and keep full runs on main.
• S is a large share of each job, or grows as you add shards. Stop adding shards and cut setup: caching, prebuilt images, schema dumps, build-once artifacts.
• A few files account for most of the runtime. Fix those tests directly before scaling infrastructure around them.

Avoid sharding when setup dominates. Avoid selection when the module map would send most changes to "run everything." In both cases the extra machinery costs more than it returns.

A workable order is: profile, isolate, multi-worker, shard, select on pull requests, then cut setup. Re-profile after each step, because the bottleneck moves.

How does it work?

All of these techniques reduce the time between a push and a result. Each one does it by changing which tests run, and where, without changing what the tests check.

1. Multi-worker execution. The test runner starts N worker processes on one host and hands tests to them as they free up. With pytest and the pytest-xdist plugin, for example, pytest -n 4 starts four workers named gw0 to gw3. Each worker exposes its ID so fixtures can create per-worker databases and directories. Setup runs once per machine, and test time divides across workers.
2. Sharding. The CI system starts N jobs from the same commit. Each job receives its shard index and the same inputs: the file list, a timings artifact from the latest main build, and a seed. It runs the same deterministic planning algorithm. All jobs compute the identical split, so no coordinator is needed, and each job runs only the files assigned to its index. Per-shard reports are merged afterward into one result.
3. Module-based selection. On a pull request, CI diffs the branch against its merge base. A checked-in map translates the changed paths into test directories, and CI runs only those directories. Unmatched paths and changes to shared infrastructure fall back to the full suite. Main-branch runs still execute everything.
4. Setup reduction and tiering. Caches, prebuilt images, schema dumps, and build artifacts shrink the fixed cost that every shard repeats. Splitting unit tests from integration and end-to-end tests into separate jobs lets fast results arrive without waiting for slow ones.

The rest of this article covers each mechanism in that order, starting with the difference between the two levels of parallelism.

Two levels of parallelism: multi-worker on one machine vs sharding across CI jobs

Each lever attacks a different cost.

Multi-worker runs N test processes on one machine. Setup happens once: dependencies install once, the database container starts once, and the app boots once per worker. Workers share CPU, memory, and often a database server, so they need isolation from each other at the data level.

Sharding splits the suite across N CI jobs. Each job is a fresh machine that repeats checkout, dependency install, and service startup before running its slice.

CopyMulti-worker (1 runner)           Sharding (N jobs)
+----------------------+          +---------+  +---------+  +---------+
| setup (once)         |          | setup   |  | setup   |  | setup   |
| w1 | w2 | w3 | w4    |          | shard 1 |  | shard 2 |  | shard 3 |
+----------------------+          +---------+  +---------+  +---------+
wall = S + T/N                    wall = S + T/N, billed = N x (S + T/N)

Use multi-worker first: it is cheap and usually gives most of the gain. Shard only when one machine's cores are saturated and test time T still far exceeds setup time S.

Avoid sharding when setup dominates. Illustrative estimate: if S is 4 minutes and T is 6, four shards cut wall time from 10 to 5.5 minutes but raise billed time from 10 to 22. A small suite gains even less, because no amount of splitting shrinks S.

Copy MULTI-WORKER: 1 runner               SHARDING: N CI jobs (N = 3)
+-----------------------------+       +---------+ +---------+ +---------+
| checkout + install + db (S) |       | job 1   | | job 2   | | job 3   |
+-----------------------------+       |---------| |---------| |---------|
       |    |    |    |               | setup S | | setup S | | setup S |
       |    |    |    |               |    |    | |    |    | |    |    |
       v    v    v    v               |    v    | |    v    | |    v    |
     +----+----+----+----+            | slice 1 | | slice 2 | | slice 3 |
     | w1 | w2 | w3 | w4 |            | T/N     | | T/N     | | T/N     |
     +----+----+----+----+            +---------+ +---------+ +---------+
       |    |    |    |                    |           |           |
       v    v    v    v                    v           v           v
  +--------------------------+        +---------------------------------+
  | shared DB server: needs  |        | collect and merge shard reports |
  | schema or DB per worker  |        +---------------------------------+
  +--------------------------+

 wall   ~ S + T/N                     wall   ~ S + T/N
 billed ~ S + T                       billed ~ N x S + T
 fixed setup S paid once              fixed setup S paid N times
 isolation: workers share one host    isolation: separate machines

One runner pays setup once and splits tests across N workers that share a host and database, while sharding repeats the fixed setup in every CI job, so both reach about S + T/N wall time but sharding bills roughly N x S + T.

Making tests safe to run concurrently

Parallel workers expose every assumption that tests run alone. The fix is to give each worker its own copy of anything mutable, keyed by a worker index that the test runner exposes.

Databases. Create one database or schema per worker, named from the worker index, and run migrations once per worker at startup. Inside each worker, choose a cleanup strategy:

Default to rollback and fall back to truncation only for tests that need real commits.

Filesystem and network. Write to a per-test temporary directory, never a fixed path like /tmp/output. Bind servers to port 0 and read back the assigned port, so two workers never race for the same one.

Process state. Global singletons, module-level caches, and mutated configuration let one test change another's outcome. Reset them in teardown or inject them as dependencies.

Run locally with random ordering and several workers before touching CI. Expect a first wave of failures. Each one is a real hidden dependency, so fix it rather than adding retries, which only hide it.

Per-worker databases and services also compete for CPU, memory, and connection limits. Size worker count to the host, not to core count alone.

Copyimport threading
from http.server import HTTPServer, SimpleHTTPRequestHandler

import psycopg
import pytest
from psycopg import sql

BASE_URL = "postgresql://app_test:example-password@localhost:5432"

@pytest.fixture(scope="session")
def db_name(worker_id):
    # worker_id is "gw0", "gw1", ... under xdist; "master" when run serially
    name = f"app_test_{worker_id}"
    with psycopg.connect(f"{BASE_URL}/postgres", autocommit=True) as admin:
        admin.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(name)))
        admin.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(name)))
    # apply migrations here, once per worker, not once per test
    return name

@pytest.fixture
def db(db_name):
    conn = psycopg.connect(f"{BASE_URL}/{db_name}")  # implicit transaction
    yield conn
    conn.rollback()  # discard everything this test wrote
    conn.close()

@pytest.fixture
def http_server(tmp_path):
    # tmp_path is unique per test; port 0 lets the OS choose a free port
    handler = lambda *a: SimpleHTTPRequestHandler(*a, directory=str(tmp_path))
    server = HTTPServer(("127.0.0.1", 0), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    host, port = server.server_address  # read back the port the OS assigned
    yield f"{host}:{port}"
    server.shutdown()
    server.server_close()

A pytest conftest that gives each xdist worker its own PostgreSQL database, rolls back each test's transaction, and starts per-test servers on OS-assigned ports that serve unique temporary directories.

Splitting tests into balanced shards with historical timing data

Splitting by file count assumes every file costs the same, and in a monolith that is rarely true. One integration file that boots the full application can outweigh fifty unit files, so a count-based split leaves one shard running long after the others finish. The CI run ends only when its slowest shard ends.

Record per-file durations from each main-branch run and upload them as a CI artifact. Have the next run download the latest copy before planning shards. Sort files from longest to shortest, then assign each file to whichever shard currently has the smallest total. This greedy approach is simple and deterministic, and it usually gets close to an even split. Files with no recorded timing get a default estimate, such as the median file duration, so new tests are spread across shards instead of all landing on one.

Timings go stale as tests change. If you plan from old data, a file that grew from seconds to minutes can turn one shard into the bottleneck. Refresh the artifact on every main-branch build, and alert when the gap between the fastest and slowest shard grows past a threshold you choose.

Sharding should not make failures harder to debug. Merge the per-shard results into one report so engineers see a single list of failures. At the start of each job, log the random seed, the shard index, and the exact list of files assigned to it. Provide one local command that takes a shard index and seed and runs that same file list in the same order, so a failure seen in CI can be reproduced locally.

Copyimport heapq, json, random, statistics, sys

def plan(files, timings, shards):
    known = [timings[f] for f in files if f in timings]
    default = statistics.median(known) if known else 1.0  # new files get the median estimate
    cost = {f: timings.get(f, default) for f in files}
    # Longest first; ties broken by name so every job computes the identical plan
    ordered = sorted(files, key=lambda f: (-cost[f], f))
    heap = [(0.0, i) for i in range(shards)]  # (estimated seconds, shard index)
    buckets = [[] for _ in range(shards)]
    for f in ordered:
        total, i = heapq.heappop(heap)  # lightest shard so far
        buckets[i].append(f)
        heapq.heappush(heap, (total + cost[f], i))
    totals = [t for t, _ in sorted(heap, key=lambda x: x[1])]
    return buckets, totals

if __name__ == "__main__":
    # CI and local repro use the same call, e.g.:
    #   python shard_plan.py timings.json files.txt 4 2 81723 | xargs pytest
    timings_path, list_path, shards, index, seed = sys.argv[1:6]
    shards, index = int(shards), int(index)
    with open(timings_path) as fh:  # artifact from the latest main build: {"tests/test_x.py": 12.4}
        timings = json.load(fh)
    with open(list_path) as fh:
        files = [line.strip() for line in fh if line.strip()]
    buckets, totals = plan(files, timings, shards)
    mine = buckets[index]
    random.Random(int(seed)).shuffle(mine)  # same seed gives same order in CI and locally
    log = sys.stderr
    print(f"seed={seed} shard={index}/{shards} est={totals[index]:.1f}s", file=log)
    if max(totals) - min(totals) > 120:  # 120 s is an arbitrary example threshold; tune for your suite
        print("WARNING: shard imbalance; timings may be stale", file=log)
    for f in mine:
        print(f"  assigned: {f}", file=log)  # exact file list for this shard
    print("\n".join(mine))  # stdout is only the file list, for the test runner

A greedy longest-first shard planner that reads timings from a CI artifact, gives new files the median estimate, warns on imbalance, and logs seed and file assignment so one shard can be rerun locally.

Selecting tests by module boundary on pull requests

Sharding makes the full suite faster, but most pull requests touch one or two areas of the codebase. They don't need the full suite at all. Module-based selection runs only the tests that a change could plausibly affect.

Build the mapping

Start with a checked-in file that maps source directories to test directories. For example, billing/ maps to tests/billing/, and shared/ maps to every test directory. In CI, diff the pull request against its merge base and collect the changed paths. Look up the affected modules and run only their tests. Treat any path that matches no rule as "run everything." Unknown paths then fail safe rather than silently skipping tests. Changes to dependency manifests, build configuration, or the test harness itself should also trigger the full suite.

Keep full runs on the main branch after merge, on a schedule, or both. Selection trades certainty for speed on pull requests. The main-branch run catches what the mapping missed, and each miss tells you which rule needs fixing.

When not to rely on it

A directory map only sees dependencies that follow file layout. It misses implicit coupling:

• two modules reading and writing the same database tables
• shared configuration keys
• reflection or dependency-injection wiring resolved at runtime
• global state or event handlers registered from anywhere

If writing the map forces you to send most directories to "run everything," the monolith is too tightly coupled for selection to help. That result is still useful: it is an architecture problem, and better test tooling cannot fix it.

Cutting fixed per-job costs and separating slow test tiers

Sharding divides test execution time across jobs, but every shard still pays the full setup cost. A rough model for one pipeline run:

• wall time ≈ setup + (test time ÷ N)
• compute time ≈ N × setup + test time

Illustrative estimate (assumed numbers, not a benchmark): setup = 6 min, test time = 40 min, N = 8 shards. Wall time is 6 + 5 = 11 min, and setup alone consumes 8 × 6 = 48 compute minutes. Cut setup to 2 min and wall time drops to 7 min while setup compute falls to 16 min. Once tests are well sharded, setup is often the largest remaining cost.

Reduce setup that sharding repeats

• Cache dependencies and key the cache on the lockfile hash. A shard then installs packages only when the dependencies actually change.
• Prebuild a CI image that already contains the toolchain and system packages, so shards don't run the same apt steps on every job.
• Load a schema dump instead of replaying every migration. Keep one separate job that runs migrations from scratch so they are still verified.
• Build once, fan out. Compile code and assets in one upstream job and pass the output to shards as an artifact.

Separate slow tiers

Give integration and end-to-end tests their own job, with their own shard count and failure policy. Unit tests should report within minutes and fail fast. Slower tiers can run in parallel with the unit job and still gate the merge, but they should not delay unit results.

Retries need to be visible. Record every test that passed only on retry, because silent retries hide flakiness. A flaky test should move to a non-blocking quarantine job with a named owner and a fix-by date. If nobody fixes it by that date, delete it.

Alternatives, cost impact, and engineering time saved

Sharding trades compute for wall time. Illustrative estimate (assumed numbers, not a benchmark): a 40-minute suite has S = 3 minutes of fixed setup and T = 37 minutes of test execution, split across N = 8 shards. Wall time ≈ S + T/N = 3 + 37/8 ≈ 7.6, or about 8 minutes. Billed job-minutes ≈ N × (S + T/N) = N·S + T. That is about 61 minutes, or about 64 if each job rounds up to 8 minutes, compared with 40 minutes for one serial job. Doubling to 16 shards brings wall time down to only about 5.3 minutes, while the setup overhead doubles. Once S is larger than T/N, adding shards mostly buys more setup runs.

The extra compute usually pays for itself in waiting time. Using the same assumed numbers, cutting about 32 minutes from each run means a developer who pushes 5 times a day waits about 160 minutes less, roughly 2.7 hours, and is less likely to switch tasks and lose focus.

Track these over time:

• p50 and p95 wall time
• Job-minutes per run
• Setup share of each shard
• Shard imbalance
• Retry rate

A rising setup share is the signal to stop adding shards and fix setup instead.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to speed up monolith test suite, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.