How does Nx detect flaky tests from task hashes?
Before Nx runs a task, it computes a hash from everything that is supposed to determine the task's result. That includes the project's source files, relevant configuration such as nx.json, project configuration and tool configs, the hashes of dependent projects, the external dependency versions in your lockfile, and any environment variables or runtime values you have declared as inputs. Nx uses this hash for caching. If the hash matches a previous run, the output should match too.
Flaky detection relies on the same idea. If two executions of a task have identical hashes, nothing Nx was told to track has changed. When one of those executions passes and the other fails, the difference must come from outside the declared inputs. Common sources are timing, test ordering, shared state, network calls, or an undeclared environment variable. Nx Cloud records task results by hash across CI runs. When it sees both outcomes for one hash, it marks the task as flaky.
run A: api:test hash=7f3c... ──► PASS ──┐
├──► same hash, different
run B: api:test hash=7f3c... ──► FAIL ──┘ outcomes => FLAKY
The signal is only as good as your input definitions. A missing input can cause a real regression to be labeled flaky. That happens when a change Nx doesn't track alters the result while the hash stays the same.
Flaky detection and automatic retries are Nx Cloud features, not part of core Nx. Their behavior and settings have changed between releases. For that reason, this guide describes the mechanism rather than a specific dashboard or configuration screen.
┌──────────────── declared task inputs ────────────────┐
│ source files · nx.json / project / tool config │
│ dependent project hashes · lockfile versions │
│ declared env vars and runtime inputs │
└───────────────────────────┬──────────────────────────┘
│ hashed by Nx
▼
┌─────────────────────┐
│ api:test hash=7f3c… │
└──────────┬──────────┘
┌───────────┴────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ CI run A │ │ CI run B │
│ result: PASS │ │ result: FAIL │
└────────┬─────────┘ └────────┬─────────┘
└───────────┬────────────┘
▼
┌────────────────────────────────────┐
│ Nx Cloud: results stored by hash │
│ 7f3c… → { PASS, FAIL } │
└─────────────────┬──────────────────┘
▼
┌────────────────────────────────────┐
│ api:test marked FLAKY │
│ cause: outside declared inputs │
│ (timing, test order, shared state, │
│ network, undeclared env var) │
└────────────────────────────────────┘
Declared inputs produce one task hash for api:test, and when Nx Cloud records both a pass and a fail across CI runs with that same hash, it marks the task flaky because the varying factor lies outside the declared inputs.
Which backend flakes does hash-based detection catch, and which does it miss?
Detection works on one rule: the same task hash produced different outcomes. That catches any cause where the declared inputs stay the same but the runtime environment varies.
| Cause | Caught? | Why |
|---|---|---|
| Shared database state | Yes | Leftover rows change results, but no input changes |
| Port collisions under parallel runs | Yes | Scheduling varies, the hash does not |
| Time and timezone dependence | Yes, unless the clock leaks into inputs | Wall-clock time is not hashed by default |
| Test-order coupling | Yes | Order shifts with sharding and parallelism, not inputs |
| External service reliance | Yes | Remote availability is invisible to the hash |
| CI agent resource limits | Yes | Memory and CPU pressure are not inputs |
| Undeclared env var or file read | No | Its effect on the result is never recorded, so failures cannot be traced to it |
| Over-broad inputs (timestamps, volatile env vars) | No | Every run hashes differently, so runs are never compared |
The two blind spots fail in opposite ways. An undeclared dependency leaves runs grouped under one hash, but the real cause stays hidden. If that dependency changes the result, Nx reports flakiness that is actually a missing input. An over-declared input does the reverse. A value like a build timestamp or CI_JOB_ID gives every run a unique hash, so Nx never sees two runs with the same hash and cannot flag any task as flaky.
To fix this, define namedInputs in nx.json and reference them from each test target in project.json. Use { "env": "DATABASE_URL" } entries only for variables that change behavior. Use file globs such as {projectRoot}/**/* or {workspaceRoot}/docker-compose.test.yml for fixtures and config the tests read. Leave out values that change on every run.
{
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"sharedGlobals": [
"{workspaceRoot}/docker-compose.test.yml",
"{workspaceRoot}/tsconfig.base.json"
],
"production": [
"default",
"!{projectRoot}/**/*.spec.ts",
"!{projectRoot}/test/**/*"
],
"backendTestInputs": [
"default",
"{workspaceRoot}/fixtures/db/**/*.sql",
{ "env": "DATABASE_URL" },
{ "env": "TZ" },
{ "env": "PAYMENTS_API_MODE" }
]
},
"targetDefaults": {
"test": {
"cache": true,
"inputs": ["backendTestInputs", "^production"]
}
}
}
An nx.json that hashes the test fixtures, compose file and behavior-changing env vars like DATABASE_URL and TZ. It leaves out per-run values such as CI_JOB_ID, so repeated runs share a hash and flaky outcomes can be compared.
What does an Nx flaky task retry actually do?
When Nx Cloud has marked a task as flaky and that task fails, it runs the task again. The retry happens at the task level. For a backend project that usually means the entire test target: every suite in the project, along with whatever setup it needs, such as starting containers, running migrations and seeding data. One intermittent assertion failure costs you the runtime of the whole target a second time.
Test-runner retries work at a smaller scope. Jest's jest.retryTimes and pytest rerun plugins such as pytest-rerunfailures rerun only the failing test inside the same process. That is cheap, but it changes what Nx sees. The task exits successfully, so Nx records a pass for that hash. It never sees mixed outcomes, and hash-based detection never fires. The flakiness is still there, but it only shows up in runner output that most people don't read.
Retries at different layers multiply. Suppose a CI job retries twice, Nx retries the task once, and Jest retries each test twice. A single bad test can then trigger many runs, and the final report shows a pass without saying which layer recovered it.
| Retry layer | Scope | Cost per retry | Visible to Nx | Reporting clarity |
|---|---|---|---|---|
| CI job retry | Whole pipeline or job | Highest | Only as separate runs | Poor |
| Nx Cloud task retry | One task (project target) | Medium to high | Yes | Good |
| Test-runner retry | Single test | Lowest | No, task passes | Poor outside runner logs |
Pick one primary layer. If you rely on Nx flaky detection, keep runner retries off or confine them to explicitly quarantined tests. That way the failure signal reaches Nx and is recorded against the hash.
When should you avoid relying on retries?
A retry answers one question: "can this test pass at all?" It does not answer "why did it fail?" Every time a retry turns red into green, the evidence of an intermittent failure disappears from the build result. That matters most when the flake lives in production code rather than in the test. A race between two writers, a connection pool that leaks under load, or a missing lock around a cache refresh will fail tests intermittently for exactly the same reason it will fail in production intermittently. Blanket retries make these defects look like test noise and ship them.
Treat retries as acceptable only under narrow conditions:
- Short-term unblocking. A known flake is blocking merges, and someone owns the fix with a deadline.
- Clear test-side cause. The failure is a fixed sleep, a shared port, or ordering between tests, and you can see it in the test code.
- Every retry is visible. Passes that needed a retry are reported, not silently absorbed.
Disable retries, or require triage before any retry, for tests that cover concurrency, transactions, idempotency, or retry logic in the service itself. A flake in those areas should be treated as a defect report until someone proves otherwise.
If you don't use Nx Cloud, you can build the same signal from run history. Record the task hash and outcome for every CI run. A hash that has both passed and failed is a flaky candidate, because the inputs were identical and only the run changed. Keep that list small and reviewed. Its job is to start investigations, not to automatically approve retries.
How do you split test targets so flaky signals are cheaper to act on?
Flakiness is recorded per task, so the size of the task decides how precise the signal is. If a service has a single test target that runs every unit and integration test, one intermittent failure in a database test marks the whole target as flaky. A retry then reruns all of it, including hundreds of fast, deterministic unit tests that never failed. You pay for the full suite on every retry, and the flaky report points at the service when it should point at a much smaller area.
Split the target along the line where nondeterminism usually lives. In the example below, test-unit reads only source and spec files and needs no external services. test-integration adds its own inputs, such as migration files, Docker Compose definitions, and fixture data, so changes to those files invalidate only the integration cache. When the integration target goes flaky, the unit target keeps its cache hits and its clean history. If integration tests in one domain are much less stable than the rest, you can move them into a separate project so they get their own graph node and their own flakiness record.
Where your test runner plugin supports task splitting, Nx can break a target into per-file tasks. This narrows the signal further, and a retry reruns only the file that failed.
Finer granularity has costs:
- Graph and configuration: each new target or project needs its own inputs, outputs, and dependencies kept correct.
- Repeated setup: if every integration task starts its own database container or runs migrations, splitting into ten tasks can mean ten startups. Measure that setup time before you split further.
{
"name": "orders-service",
"$schema": "../../node_modules/nx/schemas/project-schema.json",
"sourceRoot": "apps/orders-service/src",
"projectType": "application",
"targets": {
"test-unit": {
"executor": "nx:run-commands",
"cache": true,
"inputs": [
"{projectRoot}/src/**/*.ts",
"{projectRoot}/jest.unit.config.ts",
"^production"
],
"outputs": ["{workspaceRoot}/coverage/apps/orders-service/unit"],
"options": {
"command": "jest --config apps/orders-service/jest.unit.config.ts"
}
},
"test-integration": {
"executor": "nx:run-commands",
"cache": true,
"inputs": [
"{projectRoot}/src/**/*.ts",
"{projectRoot}/test/integration/**/*",
"{projectRoot}/migrations/**/*.sql",
"{projectRoot}/docker-compose.test.yml",
"{projectRoot}/jest.integration.config.ts",
"^production"
],
"outputs": ["{workspaceRoot}/coverage/apps/orders-service/integration"],
"options": {
"command": "jest --config apps/orders-service/jest.integration.config.ts --runInBand"
}
}
}
}
A project.json with separate unit and integration test targets: the integration target also tracks migrations, Compose files and fixtures, so changes to those, or flaky failures there, leave the unit target's cache and history untouched.
How do you set up a test quarantine process with entry and exit criteria?
Nx has no single "quarantine" switch, so quarantine has to be a process you define. There are three common mechanisms:
- Exclusion by convention: a naming convention or tag (for example, files matching
*.quarantine.spec.ts) that the blockingtesttarget's runner config excludes. - A separate target: a non-blocking
test-quarantinetarget that runs only those files and reports results without failing the pipeline. - The runner's own features: the test runner's skip or tag mechanisms.
The separate target is usually the best choice, because quarantined tests keep running and keep producing data you can use to decide when to release them.
Entry criteria. A test enters quarantine only when it has both passed and failed against the same task hash. Record those run IDs in the linked issue. Without matching hashes, you may be quarantining a real regression.
Ownership. Every quarantined test needs three things recorded next to the exclusion: an owner, an issue link and a removal date. A reviewer should be able to read all three without leaving the diff.
Reproducing locally. Run nx run-many -t test --projects=orders-api --skip-nx-cache --parallel=4 in a loop, using the same parallelism as CI. Contention for ports, database schemas or temporary directories often appears only at that concurrency.
Exit criteria. A test leaves quarantine when two conditions hold:
- A fix has been merged.
- The test then passes an agreed number of consecutive repeated runs at CI parallelism.
Risks. Each quarantined test is coverage you no longer enforce. Without expiry dates, the quarantine list turns into a graveyard. The non-blocking target also costs CI minutes, and teams quickly learn to ignore its failures. To prevent both problems, have a scheduled check fail when any test is past its removal date.
#!/usr/bin/env bash
# repro-flake.sh: rerun one project's tests uncached at CI parallelism.
# Use it to confirm a flake before quarantine (entry) and to prove a fix (exit).
# Usage: ./repro-flake.sh [runs] [project] [parallel]
set -u
RUNS="${1:-20}" # exit criterion: agreed number of consecutive passes
PROJECT="${2:-orders-api}"
PARALLEL="${3:-4}" # must match the --parallel value CI uses
LOG_DIR="flake-logs/${PROJECT}"
mkdir -p "$LOG_DIR"
fails=0
for i in $(seq 1 "$RUNS"); do
# --skip-nx-cache forces real execution; a cache hit would replay an old result.
if npx nx run-many -t test --projects="$PROJECT" \
--skip-nx-cache --parallel="$PARALLEL" > "$LOG_DIR/run-$i.log" 2>&1; then
echo "run $i: pass"
else
fails=$((fails + 1))
# Keep the log so the failure can be attached to the quarantine issue.
echo "run $i: FAIL (see $LOG_DIR/run-$i.log)"
fi
done
echo "${PROJECT}: ${fails}/${RUNS} runs failed at parallel=${PARALLEL}"
# A non-zero exit means the test is still flaky and stays in quarantine.
# Zero failures across all runs, after a merged fix, meets the exit criteria.
[ "$fails" -eq 0 ]
A shell loop that reruns a project's test target with --skip-nx-cache at CI parallelism, counts failures and keeps per-run logs, so the same script produces evidence for both entering and leaving quarantine.
How much do flaky task retries cost, and how do you track flakiness over time?
Illustrative estimate (assumed numbers, not a benchmark). Assume a 6-minute integration target, 200 runs per day, and a 5% flaky failure rate. Each flaky failure triggers one whole-task rerun.
- Reruns per day = runs per day × flake rate = 200 × 0.05 = 10
- Agent minutes per day = reruns × task duration = 10 × 6 = 60
An hour of agent time is cheap compared with the alternative. Assume that without automatic retries, each flaky failure costs a developer 15 minutes to notice it, open the log, decide it is noise, and re-trigger the job. Ten failures then cost 150 developer minutes a day. That time is spread across people and interrupts other work.
| Assumed flake rate | Reruns/day | Agent min/day | Developer min/day if manual (15 min each) |
|---|---|---|---|
| 1% | 2 | 12 | 30 |
| 5% | 10 | 60 | 150 |
| 10% | 20 | 120 | 300 |
| 20% | 40 | 240 | 600 |
Cost grows linearly with the flake rate, and the table understates it at higher rates, because a retry can also flake and need another run. Retries keep the pipeline moving, but they hide the growth. That is why every retried task needs a matching entry in the fix queue.
For observability, export or review the tasks Nx Cloud flags as flaky on a regular cadence, and bring the list to a short weekly triage meeting. Track the count of flaky tasks across the whole workspace and treat it as a platform health metric, like build duration. A rising count usually points to shared causes such as test containers, fixtures or network mocks. Fixing one shared cause clears flakes for several teams at once, which saves more engineering time than having each team chase its own symptoms.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx flaky tests, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.