What are Nx flaky tasks and how does Nx detect them?
In Nx, a task such as api:test is identified by a hash. Nx computes that hash from everything the task declares as input: the project's source files, the files of its dependencies, the relevant configuration, the runtime and environment values you list, and the target's options. Two runs with the same hash should be interchangeable, which is why Nx can replay a cached result instead of executing the task again.
A flaky task is one where that promise fails. If runs with an identical hash have both passed and failed, the outcome depends on something the hash does not capture. Common examples are timing, test ordering, a shared database, a network call, or the state of the agent. Nx Cloud tracks run outcomes per hash, and a pass and a fail on the same hash are enough evidence to mark the task flaky.
hash a91f… (api:test)
agent-1 ──► PASS ┐
agent-3 ──► FAIL ├──► same inputs, different outcomes ──► task marked FLAKY
agent-2 ──► PASS ┘
│
later run, hash a91f…, FAIL ──► eligible for automatic retry
The signal applies to the whole task, not to individual tests. Nx knows that api:test is unstable, but it does not know which of the 400 test cases inside it caused the failure. You have to narrow that down yourself.
Detection also depends on history. Nx can only classify a task as flaky after it has seen several runs with the same hash. New tasks, tasks that rarely run, and tasks whose inputs change on almost every commit produce few matching-hash runs. These tasks can be flaky for a long time before Nx flags them.
declared inputs
┌──────────────────────────────┐
│ sources · deps · config │
│ env/runtime · target options │
└──────────────┬───────────────┘
▼
┌───────────────┐
│ hash a91f… │
│ (api:test) │
└───────┬───────┘
┌──────────┼──────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│agent-1 │ │agent-3 │ │agent-2 │
│ PASS │ │ FAIL │ │ PASS │
└───┬────┘ └───┬────┘ └───┬────┘
└──────────┼──────────┘
▼
┌────────────────────────────────────┐
│ Nx Cloud: run outcomes per hash │
│ same inputs, different outcomes │
│ ──► api:test marked FLAKY │
│ (whole task, not a test case) │
└──────────────┬─────────────────────┘
▼
┌────────────────────────────────────┐
│ later run, hash a91f…, FAIL │
│ ──► eligible for automatic retry │
└────────────────────────────────────┘
Needs enough same-hash runs: new, rare or churning tasks lag.
Runs of api:test with the same input hash pass on some agents and fail on another. Nx Cloud then marks the whole task flaky, and a later failure on that hash becomes eligible for retry. The diagram does not identify which test case failed.
How do Nx Cloud retries work, and when should you avoid them?
Once Nx Cloud has marked a task as flaky, a failure of that task no longer has to end the pipeline. Nx Cloud can re-run the task, and if the retry passes, the run continues as if the first attempt had succeeded. Retry counts, opt-in behavior, and plan availability can change, so check the current Nx Cloud documentation and your workspace settings rather than assuming any default.
The trade-off is signal for throughput. A retry that passes only shows the task can pass, not that the failure was harmless. A race condition in a connection pool or a missing transaction boundary in production code fails intermittently in exactly the way a timing-sensitive test does. Retries will convert both into green builds, and only one of them is a test problem.
Retries also run at task granularity. For a backend suite that starts containers, applies migrations, and seeds fixtures, one retry repeats all of that setup to re-check a single unstable assertion. The pipeline then also depends on a hosted service: its availability, your workspace configuration, and the features in your plan all shape what CI does.
Retries pay off most in monorepos with shared CI and many backend projects. There, one team's intermittent failure would otherwise block unrelated merges from other teams. Avoid retries for tasks that became flaky right after a specific merge, because that merge is the lead to investigate. Also avoid them for tasks with no owner, since nobody will ever act on a failure that keeps being retried away.
Why are backend tests so prone to flakiness?
Backend tests touch shared state that unit tests in the frontend rarely see. That includes databases, sockets, clocks and other services. When Nx runs several test targets in parallel on one agent, or splits work across agents, the test order and timing change on every run. Hidden coupling then shows up as intermittent failures. The useful skill is matching the log signature to the cause before you reach for a retry.
| Cause | Why it is intermittent | What CI logs show |
|---|---|---|
| Shared database state between test files | Pass or fail depends on which file ran first or concurrently | Unique constraint violations, unexpected row counts |
| Port collisions under parallel execution | Two processes on one agent bind the same fixed port | address already in use (EADDRINUSE) |
| Wall-clock time or timezone reliance | Results change with the date, hour or agent's timezone | Failures near midnight UTC or DST transitions, off-by-one dates |
Queries without ORDER BY |
Row order is unspecified and changes with query plans or vacuuming | Array equality assertions failing with the same elements reordered |
| Network calls to real dependencies | Latency, rate limits and outages outside your control | Timeouts, connection resets, HTTP 429 |
| Containers or migrations not ready | Tests start before the service accepts connections or the schema exists | connection refused, missing relation or table errors |
Two patterns matter most here. The first is that the error often points at the victim rather than the culprit. A constraint violation in one file usually means another file left rows behind. The second is that a retry tends to "fix" every one of these causes by changing the timing. That is exactly why a retry tells you nothing about which cause you have.
Are your inputs wrong before your tests are? Auditing a flaky task's inputs
Before blaming the test, check whether the task's hash describes everything the test depends on. Nx computes a hash from the task's declared inputs. If a test reads something the hash doesn't include, two runs with the same hash can execute against different conditions. The task then fails "randomly" even though the test code is deterministic.
Common undeclared inputs in backend projects:
- Environment variables, such as
DATABASE_URL, feature flags, orTZ. - Generated files, such as ORM clients, OpenAPI stubs, or protobuf output.
- Config outside the project root, such as a shared
docker-composefile, a root.env, or migration scripts in another folder.
The opposite failure is quieter. A migration changes, the hash stays the same, and Nx restores a cached pass for tests that should have run again.
To fix this, declare those inputs explicitly. You can do it per target in project.json. For backend test targets, a reusable named input in nx.json usually works better. It combines default file inputs with {workspaceRoot}/-prefixed paths for shared config, env entries for each variable the tests read, and runtime inputs where a tool version matters.
Then compare what the hash covers with what the test reads:
- List the target's resolved inputs with
nx show project <name>. - Search the test setup and helpers for
process.env, file reads, and config loaders. - Treat anything that appears in the code but not in the inputs as a gap.
Expect your cache hit rate to drop after you close these gaps. That is the correct result. The hits you lose were runs where Nx could not know whether the old result still applied.
{
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"sharedGlobals": [],
"production": ["default", "!{projectRoot}/**/*.spec.ts"],
"backendTest": [
"default",
"^production",
"{workspaceRoot}/docker-compose.test.yml",
"{workspaceRoot}/.env.test",
"{workspaceRoot}/db/migrations/**/*",
"{workspaceRoot}/proto/**/*.proto",
{ "env": "DATABASE_URL" },
{ "env": "FEATURE_FLAGS" },
{ "env": "TZ" },
{ "runtime": "node --version" }
]
},
"targetDefaults": {
"test": {
"inputs": ["backendTest"],
"cache": true
}
}
}
An nx.json named input for backend test targets that adds shared config, migrations, proto sources, environment variables and the Node version to the task hash.
How do you narrow a flaky task down to a flaky test?
Step 1: Measure the failure rate
Run the target in a shell loop, 20 to 50 times, calling nx run <project>:<target> --skip-nx-cache and appending each exit code to a log. Skipping the cache matters here. A cached success replays the old result and tells you nothing about the current run. The pass/fail count gives you a baseline, so you can later check whether a fix actually lowered the rate. Without it, a fix that "seems better" may only reflect a lucky streak. Save the full output of each failed iteration. The test name that fails most often is your first suspect.
Step 2: Shake the ordering
Many runners can shuffle test order or repeat tests on their own. Go's go test supports -count and -shuffle=on. Jest and Vitest have equivalent options; check your version's documentation for the exact flag. A test that fails only in certain orders depends on state left behind by an earlier test. When the runner prints a seed, record it so you can reproduce the failing order exactly.
Step 3: Run files serially
Next, run the suite without parallel workers. In Jest, that is --runInBand. If the failures stop, the cause is contention between workers over a shared database, port, or temp directory. If they continue, look for a logic or timing problem inside the test.
Step 4: Consider splitting the target
Breaking a large test target into several smaller ones (for example unit, repository, and API tests) isolates failures to a narrower scope. It also makes each retry cheaper, because only the failing piece reruns. The cost is more configuration to maintain: each new target needs correct inputs, its own setup, and possibly duplicated fixtures. Split along boundaries where the setup actually differs, not just to cut the file count.
#!/usr/bin/env bash
# Measure how often an Nx target fails across repeated, uncached runs.
# Usage: ./flake-loop.sh api:test 30 [extra args passed to nx run]
set -u
TARGET="${1:-api:test}" # <project>:<target>
RUNS="${2:-30}" # 20-50 runs gives a usable baseline
shift 2 2>/dev/null || true
EXTRA_ARGS=("$@") # e.g. --runInBand when using the Jest executor
LOG_DIR="flake-logs/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$LOG_DIR"
pass=0
fail=0
for i in $(seq 1 "$RUNS"); do
out="$LOG_DIR/run-$i.log"
# --skip-nx-cache forces a real run; a cached pass would hide the flake.
if npx nx run "$TARGET" --skip-nx-cache "${EXTRA_ARGS[@]}" >"$out" 2>&1; then
pass=$((pass + 1))
rm "$out" # keep only failing output for triage
else
fail=$((fail + 1))
echo "run $i: FAIL (see $out)"
fi
done
echo "$TARGET: $pass passed, $fail failed out of $RUNS" | tee "$LOG_DIR/summary.txt"
# Most frequent failing test names are the first suspects.
# Adjust the pattern to your runner's failure marker (Jest prints lines starting with '●').
grep -h '●' "$LOG_DIR"/run-*.log 2>/dev/null | sort | uniq -c | sort -rn | head -n 10
A shell loop that runs an Nx target repeatedly with --skip-nx-cache, records a pass/fail baseline, keeps only failing logs, and ranks the most frequently failing tests; rerun it with extra args such as --runInBand to compare parallel and serial failure rates.
How do you fix flaky backend tests instead of retrying them?
Once a flaky task is narrowed to a specific test, fix the source of nondeterminism directly. Each common cause has a corresponding change:
- Shared database state: Give each test its own schema, or open a transaction in the fixture's setup and roll it back in teardown. Every test then starts from the same state, whatever ran before it or in parallel. Rollback is usually cheaper than truncating tables. It does not work when the code under test commits explicitly or opens its own connections.
- Port collisions: Bind servers to port 0 and read back the port the OS assigned. Hardcoded ports collide when Nx runs several tasks at once on one agent.
- Wall-clock time: Pass a clock interface into the code instead of reading system time directly. Production uses the real clock, and tests use a fixed one that they advance explicitly. This removes failures around midnight, DST changes, and token expiry.
- Startup races: Before the first test runs, poll for actual readiness. Check that a query succeeds and that migrations have reached the expected version. A container being in the "running" state does not mean it is ready.
- Generated data: Seed the random generator and log the seed, so you can replay any failure exactly.
- Row ordering: Add
ORDER BYto any query whose result order is asserted. SQL does not guarantee order without it. - Third-party calls: Replace them with fakes or contract stubs. Network latency and rate limits are outside your control.
Each fix is a one-time cost. A retry adds minutes to every run, and every unexplained red build makes engineers trust CI a little less.
// Injected clock: production passes systemClock, tests pass a FixedClock.
interface Clock { now(): Date }
const systemClock: Clock = { now: () => new Date() };
class FixedClock implements Clock {
constructor(private t: Date) {}
now() { return new Date(this.t.getTime()); }
advance(ms: number) { this.t = new Date(this.t.getTime() + ms); } // time moves only when the test says so
}
function isTokenExpired(expiresAt: Date, clock: Clock = systemClock): boolean {
return clock.now().getTime() >= expiresAt.getTime();
}
// Minimal driver shape; adapt to your database client.
interface Conn { query(sql: string, params?: unknown[]): Promise<{ rows: any[] }>; release(): void }
interface Pool { connect(): Promise<Conn> }
// Transaction-rollback fixture: each test sees the same starting state.
// Only works if the code under test uses this conn and does not COMMIT itself.
async function withRollback<T>(pool: Pool, fn: (conn: Conn) => Promise<T>): Promise<T> {
const conn = await pool.connect();
await conn.query("BEGIN");
try { return await fn(conn); }
finally { await conn.query("ROLLBACK"); conn.release(); }
}
// Example test bodies (wire into your test runner of choice).
function expiryAcrossMidnight() {
const clock = new FixedClock(new Date("2026-01-01T23:59:59Z"));
const expiresAt = new Date("2026-01-02T00:00:00Z");
if (isTokenExpired(expiresAt, clock)) throw new Error("expired too early");
clock.advance(1_000);
if (!isTokenExpired(expiresAt, clock)) throw new Error("should be expired");
}
async function insertsAreIsolated(pool: Pool) {
await withRollback(pool, async (conn) => {
await conn.query("INSERT INTO users (email) VALUES ($1), ($2)", ["b@example.test", "a@example.test"]);
const { rows } = await conn.query("SELECT email FROM users ORDER BY email"); // order is asserted, so ORDER BY
if (rows[0].email !== "a@example.test") throw new Error("unexpected order");
});
}
A runtime-neutral sketch of an injected clock that tests advance explicitly, and a fixture that wraps each test in a transaction and rolls it back, with an ORDER BY on the asserted query.
What are the alternatives to Nx Cloud retries, and which layer should retry?
Each layer that can retry a failure trades CI time against visibility differently. The table compares them.
| Layer | Granularity | CI time per retry | Instability hidden | Needs hosted service |
|---|---|---|---|---|
| Nx Cloud task retry | Whole task (all tests in the project) | Full setup plus full suite | High: the task log shows a pass | Yes |
| Test-runner retry | Single test case | One test, usually without setup | Medium: visible only if reports record retries | No |
| CI job rerun | Entire pipeline | Every task in the job | High, and usually triggered manually | No |
| Quarantine | Named tests excluded from gating | None | Low: the list is explicit and reviewable | No |
| No retries | None | None | None | No |
Illustrative estimate. Assume a backend project's suite takes 8 minutes, container and database setup takes 2 minutes, one flaky test takes 5 seconds, and 5% of 100 daily runs hit the flake. A whole-task retry costs 5 × 10 minutes = 50 minutes of CI per day. A single-test retry costs 5 × 5 seconds, which is under half a minute.
Stacking layers multiplies both cost and concealment. With two runner retries inside two task retries, a test can execute 3 × 3 = 9 times before the build goes red. A test that fails half the time then passes the build with probability 1 − 0.5⁹, about 99.8%. That test is broken, but the stacked retries make it look stable.
Pick one layer on purpose. For most backend suites, use runner-level retries that are recorded in test reports, together with an explicit quarantine list. Use task retries only as a short-term fallback while you fix the test.
A flaky task policy that saves engineering time
The policy below makes that fallback temporary by default.
Sample policy checklist
- Retries are allowed only for tasks already flagged as flaky. Any other task fails on its first red run. A new failure is treated as signal, not noise.
- Every flaky task has an owner and a tracking issue. Tag each project with its owning team, for example
"tags": ["team:payments", "type:backend"]in the project configuration. Mirror that mapping in a CODEOWNERS-style file so the issue lands with the right people automatically. - Track retry frequency per task over time. A task whose retry count is rising needs attention before it starts failing outright.
- Treat newly flaky tasks as likely regressions. If a task was stable before a merge and is flaky after it, start the investigation from that diff. Do not assume an infrastructure problem.
- Make a time-boxed fix-or-quarantine decision. The owner chooses within a fixed window, such as one sprint, and records the choice on the issue.
- Keep quarantined tests running. Move them into a non-blocking CI job that still reports results. This way, coverage is not silently lost, and a test that starts passing reliably can return to the blocking job.
- Review the flaky list on a regular cadence. Weekly or biweekly works for most teams. Close fixed items and escalate items that have gone stale.
Why this saves time
Engineers stop rerunning pipelines by hand. They also stop debugging failures that belong to someone else's project, because the owner is already known.
The time box and the review cadence keep the flaky list short. A long, unowned flaky list is what teaches teams to ignore red builds. Once that habit forms, real regressions go unnoticed, and the retries that were meant to save time end up costing far more.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx flaky tasks, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.