Nx Self Hosted Remote Cache vs Distributed Task Execution

Remote caching vs distributed task execution: what you can actually self-host

Nx has two features that are often conflated. Only one of them can be self-hosted.

Remote caching extends the local computation cache to a shared store. Before running a task, Nx computes a hash from its inputs: source files, dependency outputs, configuration, environment variables you declare, and the command itself. If another machine has already stored outputs under that hash, Nx downloads and replays them instead of running the task. The protocol is a simple put/get keyed by hash, so the backing store can be a bucket or file share that you operate.

Distributed task execution (DTE) is an orchestration service. A coordinator reads the task graph, assigns individual tasks to a pool of CI agents as their dependencies finish, and moves build artifacts between agents so downstream tasks can start. This is part of Nx Cloud. Outside Nx's enterprise offerings there is no open, self-hosted equivalent.

This article is for platform teams that cannot or will not send build data to Nx Cloud. For most of these teams the cache is enough. On a typical pull request most projects are unchanged, so their tasks are cache hits on every agent. DTE mainly spreads the remaining cache misses across machines. Static sharding, covered later, can approximate that.

Self-hosted cache options and why custom task runners are a risk

Since cache hits do most of the work, the cache backend matters more than how misses are scheduled. Nx has published official storage-backed cache packages, historically sold under the Nx Powerpack license, plus a documented HTTP remote cache interface. Licensing terms, package names and environment variables have all changed over time. Check the current Nx docs before you commit to one.

Why custom task runners were deprecated

Before these packages existed, community plugins added remote caching by replacing the Nx task runner. That approach had two problems.

• Fragility. The task runner sat on an internal surface that Nx kept changing, so plugins broke on upgrades.
• Security. A replacement runner decides what gets written to the cache and what gets restored from it. Nx itself has no control over that.

The security problem is cache poisoning. Suppose a pull request from an untrusted fork, or a buggy branch, can write to the same cache that main reads from. It can upload an artifact under a hash that a trusted build will later compute. The trusted build gets a cache hit, restores the artifact, and ships it without rebuilding anything.

The official integrations keep caching inside Nx's own execution path. They still cannot protect you from storage credentials that are too broad. The next section deals with that.

How to set up a safe S3 cache

Start with a dedicated bucket that holds nothing but cache artifacts, with S3 Block Public Access enabled at both the account and bucket level. Keeping the cache separate from release artifacts or logs means its permissions can be strict without breaking anything else, and blocking public access ensures a single misconfigured ACL cannot expose compiled code.

Split access by trust level rather than by team. Main-branch and post-merge pipelines get an IAM role that allows s3:GetObject, s3:PutObject and s3:ListBucket, scoped to the bucket ARN and its object prefix. Internal pull request pipelines get a read-only role with only s3:GetObject and s3:ListBucket. Pull requests from forks get no cache credentials at all. Those builds run code you have not reviewed, and in many CI setups they should not see repository secrets anyway.

A shared writable cache is a security boundary. If an untrusted job can write an entry under a hash that a trusted build later reads, the trusted build will restore that output and ship it without rebuilding.

Use separate prefixes such as ci/main/ and ci/release/ so the role policies can target them individually. Add a lifecycle rule that expires objects after a fixed period, for example 14 days. Entries from old commits are rarely reused, and expiry keeps storage costs bounded. It also limits how long a poisoned entry could survive if one ever slipped through.

Copy#!/usr/bin/env bash
set -euo pipefail
BUCKET=example-nx-cache   # dedicated bucket: cache artifacts only

# Block public access on the bucket (enable it at the account level too).
aws s3api put-public-access-block --bucket "$BUCKET" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Expire entries after 14 days to bound storage and the lifetime of any poisoned entry.
cat > lifecycle.json <<'EOF'
{"Rules":[{"ID":"expire-nx-cache","Status":"Enabled","Filter":{"Prefix":"ci/"},"Expiration":{"Days":14}}]}
EOF
aws s3api put-bucket-lifecycle-configuration --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json

# Writer role: assumed ONLY by main-branch and post-merge pipelines.
cat > writer.json <<EOF
{"Version":"2012-10-17","Statement":[
 {"Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::${BUCKET}",
  "Condition":{"StringLike":{"s3:prefix":"ci/main/*"}}},
 {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"arn:aws:s3:::${BUCKET}/ci/main/*"}]}
EOF
aws iam put-role-policy --role-name nx-cache-writer-main \
  --policy-name nx-cache-rw --policy-document file://writer.json

# Reader role: internal PR pipelines can restore but never write (no s3:PutObject).
# Fork PRs get no role at all, so they cannot touch the cache.
cat > reader.json <<EOF
{"Version":"2012-10-17","Statement":[
 {"Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::${BUCKET}",
  "Condition":{"StringLike":{"s3:prefix":"ci/main/*"}}},
 {"Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::${BUCKET}/ci/main/*"}]}
EOF
aws iam put-role-policy --role-name nx-cache-reader-pr \
  --policy-name nx-cache-ro --policy-document file://reader.json

AWS CLI setup for a dedicated Nx cache bucket. It blocks public access, adds a 14-day expiry rule, gives a read-write policy scoped to the ci/main/ prefix to trusted main-branch pipelines, and gives a read-only policy to internal pull request pipelines.

Cache correctness depends on hashing inputs

A cache is only as trustworthy as its keys. Before running a target, Nx computes a hash from the task's declared inputs: the project's source files, the outputs of dependency projects (when the input uses the ^ prefix), the relevant parts of the lockfile, and the target's configuration. The inputs array on a target, often built from reusable namedInputs in nx.json such as default and production, controls which files count. Two more input types cover things outside the file tree. { "env": "API_BASE_URL" } adds an environment variable's value to the hash. { "runtime": "node --version" } runs a command and hashes its output.

Most false cache hits come from those last two categories. If a build bakes API_BASE_URL into a bundle and the variable is not declared, a staging build can restore a production artifact with the same hash. Undeclared tool versions cause the same problem. A Node, Python or compiler upgrade on one CI image changes the output while the hash stays the same. Declare every variable the build reads, and add a runtime input for each toolchain whose version affects output.

To debug, run nx show project <name> --web or --json and check the resolved inputs for each target. This shows what Nx hashes, which may differ from what you assumed. If a cache miss is unexpected, look for overly broad file globs. If a cache hit is wrong, look for an input that is missing.

Fix wrong declarations first, because their failures are silent. A slow pipeline is visible and gets fixed. A false cache hit ships an incorrect artifact with a green check mark.

Copy# Declare everything the build reads that is not a source file,
# so Nx includes it in the task hash.
cat > nx.json <<'EOF'
{
  "namedInputs": {
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "production": ["default", "!{projectRoot}/**/*.spec.ts"],
    "sharedGlobals": []
  },
  "targetDefaults": {
    "build": {
      "inputs": [
        "production",
        "^production",
        { "env": "API_BASE_URL" },
        { "runtime": "node --version" }
      ]
    }
  }
}
EOF

# Inputs explained:
#   "production"   = this project's non-test source files
#   "^production"  = the same files from dependency projects
#   env            = staging and prod builds get different hashes
#   runtime        = a Node upgrade on a CI image changes the hash

# Check what Nx resolved for the build target.
# Compare the result with what you expected Nx to hash.
nx show project my-app --json | jq '.targets.build.inputs'

# The hash should change when the env var changes.
# If the second run reports a cache hit, an input is missing.
API_BASE_URL= nx build my-app
API_BASE_URL= nx build my-app

This writes an nx.json in which the build target hashes production sources, the outputs of dependency projects, API_BASE_URL and the Node version, then uses nx show project and two builds with different environments to confirm that Nx hashes those inputs.

Replacing DTE with deterministic sharding across CI agents

With correct inputs in place, the cache becomes the coordination layer, and you can replace distributed task execution with a simpler model: decide the work up front, split it across a fixed number of agents, and let the cache absorb overlap.

Compute once, partition deterministically

Run nx show projects --affected in a single setup job and write the list to an artifact. Every agent in an N-way CI matrix reads that same list, takes its slice by index, and runs nx run-many -t build test --projects=<slice> against the shared remote cache. Computing the list once matters: if each agent ran the affected calculation itself, small differences in fetched git history could give agents different lists, and projects would be skipped or run twice.

Choosing a partition strategy

• Round-robin over a sorted list is trivial and stable. It balances project count, not duration, so one slow integration suite can leave an agent running long after the others finish.
• Historical duration (greedy assignment of the longest tasks to the least-loaded agent) gives the best wall-clock time. It needs stored timings, and new projects need a default estimate.
• Target type (one agent for lint, others for tests) is easy to reason about but rarely balanced.

Cross-shard dependencies

If shard 2 tests a library whose build is assigned to shard 1, both agents may build it at the same time on a cold cache. Neither gets a hit, so you pay for the build twice. On a warm cache this rarely happens. On cache misses from main-branch merges, it repeats across every shard. A shared upstream stage that runs build for all affected projects before the matrix starts removes the duplicate work, at the cost of one sequential step.

Copy#!/usr/bin/env bash
# shard.sh: deterministic round-robin sharding over a single affected list.
# CI sketch:
#   setup job:   ./shard.sh list origin/main HEAD   -> upload affected.txt as artifact
#   matrix job:  index: [0, 1, 2, 3], total: 4
#                download affected.txt, then ./shard.sh run $index $total
set -euo pipefail

case "${1:-}" in
  list)
    # Compute affected projects ONCE so every agent works from an identical list.
    # Sorting makes the order, and therefore the slices, stable between agents.
    npx nx show projects --affected --base="$2" --head="$3" | sort > affected.txt
    echo "affected projects: $(wc -l < affected.txt)"
    ;;
  run)
    index="$2"   # 0-based shard index from the CI matrix
    total="$3"   # number of shards in the matrix
    # Round-robin: line N goes to shard (N-1) % total.
    # This balances project count, not duration.
    slice=$(awk -v i="$index" -v n="$total" '(NR-1) % n == i' affected.txt | paste -sd, -)
    if [ -z "$slice" ]; then
      echo "shard $index/$total: nothing to run"
      exit 0
    fi
    echo "shard $index/$total: $slice"
    # Every agent uses the same remote cache. Dependencies built by another shard
    # are cache hits only if that shard finished first. On a cold cache, run a
    # separate upstream 'build' stage before the matrix to avoid duplicate builds.
    npx nx run-many -t build test --projects="$slice"
    ;;
  *)
    echo "usage: $0 list <base> <head> | run <index> <total>" >&2
    exit 1
    ;;
esac

A setup job computes and sorts the affected project list once, and each CI matrix agent then takes a round-robin slice by index and runs nx run-many against the shared remote cache.

Impact on cost and engineering time: an illustrative estimate

Illustrative estimate (assumed numbers, not a benchmark)

Assume 200 affected tasks with a 60% cache hit rate, leaving 80 to run: 70 short tasks at 0.5 min and 10 heavy tasks at 5 min. That is 85 min of work. Each agent adds 3 min of fixed overhead for checkout, install and cache restore. Assume one task runs at a time per agent.

• Agent-minutes billed = Σ(task durations) + N × overhead
• Static wall time = overhead + max over shards of Σ(durations in that shard)
• Ideal dynamic wall time ≈ overhead + max(total ÷ N, longest task)

In this example, static sharding and dynamic scheduling bill the same agent-minutes. The difference is wall time: about 7 minutes per run, lost because one shard was overloaded while the other three sat finished. That gap grows as task durations become more uneven.

For a cache hit to pay off, transfer has to be cheaper than recompute: (artifact size ÷ bandwidth) + request latency must be less than the task's run time. A large build output that takes minutes to rebuild is worth fetching. A 5-second lint task with a sizeable output may be faster to rerun. Storage costs grow with retention, and egress charges apply when agents run outside the bucket's region or cloud. The S3 cost is mostly set by large outputs, so set a retention policy on them.

The remaining cost is engineering time, and it recurs. Someone has to:

• review bucket policies
• rotate credentials
• tune lifecycle rules
• monitor hit rates and error spikes

Budget for this explicitly rather than treating it as zero.

When to use this approach, when to avoid it, and alternatives

This setup fits a workspace of moderate size with a shallow dependency graph and a team that can own the cache infrastructure. Shallow graphs matter because static sharding works best when most tasks can run in any order. Each agent can then take its slice without waiting on another agent's output.

Reconsider it when any of these matter to you:

• Deep dependency chains. Shards stall on upstream builds, or rebuild them redundantly.
• Heavy task-duration imbalance. One slow shard sets the pipeline's wall-clock time.
• Flaky-task retries. These need retry logic in your CI scripts.
• Automatic artifact distribution between agents. You have to rebuild this yourself.

DTE handles each of these for you. Without it, every one becomes CI script code you maintain, or a capability you give up.

Operational checklist:

• Track hit rate per pipeline, not globally. An aggregate figure can hide one pipeline that always misses.
• Measure upload and download time against task time. Caching a task that runs in two seconds can make it slower overall.
• Fail open when the backend is unreachable. A cache outage should make builds slower, not cause them to fail.
• Keep a cache-busting procedure ready. Document it and test it, so that a poisoned entry can be purged in minutes.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx self hosted remote cache, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.