What is External Secrets Operator and how does it sync AWS Secrets Manager into Kubernetes?

External Secrets Operator (ESO) is a Kubernetes controller that copies secret values from an external store into native Kubernetes Secrets. The sync runs in one direction only. AWS Secrets Manager (or SSM Parameter Store) stays the system of record, and the Kubernetes Secret is a derived copy. If you edit the Secret in the cluster, ESO overwrites your change on its next refresh.

Two custom resources define the sync:

  • SecretStore specifies the provider (Secrets Manager or Parameter Store), the region, and how to authenticate. On EKS, authentication usually goes through IRSA, which lets the controller assume an IAM role tied to a Kubernetes service account.
  • ExternalSecret names the remote secret and keys to read, the SecretStore to use, the target Kubernetes Secret to create, and how often to refresh.

The controller watches ExternalSecret resources and reads from AWS using the store's credentials. It then creates or updates the target Secret. Pods consume that Secret as environment variables or mounted files, the same way they consume any other Secret.

Secrets Manager → ESO controller (IRSA role) → ExternalSecret + SecretStore → Kubernetes Secret → Pod

Teams use ESO for three main reasons:

  • One source of truth. Rotation, auditing, and access policy stay in AWS, where they are already managed.
  • No secret values in Git. Manifests reference secret names, not values, so you can review and commit them freely.
  • No application changes. Deployments keep using secretKeyRef or volume mounts. They need no AWS SDK, no sidecar, and no knowledge of where the value came from.
+---------------------------+        +--------------------------+
|   AWS Secrets Manager     |        |  IAM role (IRSA)         |
|   (system of record)      |        |  trusts ESO service acct |
+---------------------------+        +--------------------------+
              |                                   |
              | secret values                     | assume role
              v                                   v
+-------------------------------------------------------------+
|  ESO controller pod  (service account annotated with role)  |
+-------------------------------------------------------------+
         ^                     ^                      |
         | watches             | auth config          | creates/updates
         |                     |                      v
+-----------------+   +-----------------+   +--------------------+
| ExternalSecret  |-->| SecretStore     |   | Kubernetes Secret  |
| remote key,     |   | provider, region|   | (derived copy)     |
| target, refresh |   | auth: IRSA      |   +--------------------+
+-----------------+   +-----------------+             |
                                                      | env vars / volume
                                                      v
                                            +--------------------+
                                            | Pod                |
                                            +--------------------+

The ESO controller assumes an IAM role through IRSA, reads values from AWS Secrets Manager as directed by each ExternalSecret and its SecretStore, and writes a derived Kubernetes Secret that Pods consume as environment variables or volumes.

Who should use External Secrets Operator, and when should you avoid it?

Who it fits

ESO suits platform teams running EKS with many services and many secrets, where Secrets Manager is already the source of truth and workloads should consume ordinary Kubernetes Secrets. The more services you have, the more you gain from one controller handling AWS access, so individual apps don't each need it.

When to avoid it

With a handful of secrets on a small team, the costs can outweigh the benefit. You take on another controller, its CRDs, upgrades and supply-chain surface. Skip it if policy requires secret values to stay out of etcd, because ESO's whole job is to write them there.

Costs you accept

  • A second copy in etcd. Synced values are only as protected as your etcd encryption and RBAC.
  • A silent deployment dependency. If ESO stalls, existing Secrets go stale without errors. New Secrets never appear, and pods fail to start.
  • Ownership and deletion semantics. By default, ESO owns the Secret it creates, so deleting the ExternalSecret can delete the Secret your pods depend on. Decide these policies explicitly.

Alternatives

Approach Where values live Env var support Rotation behavior Operational overhead
External Secrets Operator Secrets Manager plus etcd Yes Refreshed on interval; pods must reload Controller and CRDs to run
Secrets Store CSI Driver (AWS provider) Secrets Manager plus node tmpfs Only via optional Secret sync Mounted files can update in place Driver DaemonSet per node
Native SDK calls Secrets Manager plus process memory No App re-fetches on its own schedule Code and IAM per service
Static Kubernetes Secrets etcd only Yes Manual Low tooling, high toil

ESO is a reasonable default, not the only correct choice.

How do you install External Secrets Operator and authenticate with IRSA?

Add the project's Helm chart repository under the name external-secrets, then install the chart into its own namespace with helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace. Omitting --version installs whatever the repository currently publishes. In production, record the version you tested and pin it in your deployment pipeline. The chart installs the CRDs by default.

Run two checks before going further:

  • CRDs registered: kubectl get crds | grep external-secrets.io should list externalsecrets, secretstores and clustersecretstores, among others.
  • Pods running: kubectl get pods -n external-secrets should show the controller, the webhook and the cert-controller, all Running.

Wiring up IRSA

Your cluster needs an IAM OIDC provider associated with its issuer URL. Create an IAM role whose trust policy has these elements:

  • Principal: a Federated principal set to arn:aws:iam::<ACCOUNT_ID>:oidc-provider/<OIDC_ISSUER>.
  • Action: sts:AssumeRoleWithWebIdentity.
  • Condition: a StringEquals block with two keys:
    • <OIDC_ISSUER>:sub set to system:serviceaccount:external-secrets:external-secrets
    • <OIDC_ISSUER>:aud set to sts.amazonaws.com

The sub condition is the important one. Without it, any service account in the cluster could assume the role.

Next, annotate the controller's service account with eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>. Set this through the chart's serviceAccount.annotations value rather than with kubectl annotate, so a later Helm upgrade doesn't remove it. Restart the controller afterwards so its pods pick up the projected token.

Why IRSA instead of static access keys

Storing access keys in a Kubernetes Secret would work, but IRSA is better on three counts:

  • Short-lived credentials: the pod receives temporary STS credentials that expire on their own.
  • Nothing long-lived to leak: no key sits in etcd, in backups or in Git.
  • Clear identity: CloudTrail attributes each call to one role and one service account, not to an IAM user shared across tools.
#!/usr/bin/env bash
set -euo pipefail
ACCOUNT_ID=111122223333
CLUSTER=demo-cluster
ROLE_NAME=eso-secrets-reader
ESO_CHART_REPO="<project-chart-repo-url>"   # the project's published Helm repo

# IAM ARNs and condition keys use the issuer without the https:// prefix
OIDC_ISSUER=$(aws eks describe-cluster --name "$CLUSTER" \
  --query "cluster.identity.oidc.issuer" --output text | sed 's|^')

# Trust policy: only the ESO service account, via STS, may assume this role
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER}"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {"StringEquals": {
      "${OIDC_ISSUER}:sub": "system:serviceaccount:external-secrets:external-secrets",
      "${OIDC_ISSUER}:aud": "sts.amazonaws.com"
    }}
  }]
}
EOF
aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document file://trust.json
# Attach a separate permissions policy granting read access to the secrets ESO needs

# Keep the annotation in Helm values so a later upgrade does not drop it
printf 'serviceAccount:\n  annotations:\n    eks.amazonaws.com/role-arn: arn:aws:iam::%s:role/%s\n' \
  "$ACCOUNT_ID" "$ROLE_NAME" > eso-values.yaml

helm repo add external-secrets "$ESO_CHART_REPO" && helm repo update
# No --version: installs the latest published chart; pin the tested version in production
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace -f eso-values.yaml

kubectl get crds | grep external-secrets.io      # externalsecrets, secretstores, clustersecretstores...
kubectl get pods -n external-secrets             # controller, webhook, cert-controller: Running

Creates an IRSA role whose trust policy is scoped to the ESO service account, installs the unpinned Helm chart with the role-arn annotation set through chart values, and verifies the CRDs and pods.

How do you write a least-privilege IAM policy and choose between SecretStore and ClusterSecretStore?

Grant the role only two Secrets Manager actions: secretsmanager:GetSecretValue and secretsmanager:DescribeSecret. Scope the Resource to exact ARNs, or to a name prefix such as arn:aws:secretsmanager:<region>:<account>:secret:prod/payments/*. The trailing wildcard is needed either way, because Secrets Manager appends a random six-character suffix to every secret ARN. Without it, a policy that names the secret exactly stops matching.

If the secrets are encrypted with a customer-managed KMS key, add kms:Decrypt on that key's ARN. Adding a kms:ViaService condition for Secrets Manager stops the role from using the key directly. The AWS-managed key does not need this statement.

The store kind decides which Kubernetes namespaces can use that role:

  • SecretStore: namespace-scoped. An ExternalSecret can reference it only from the same namespace. With IRSA, it points at a service account in that namespace, so each team's namespace gets its own role and prefix.
  • ClusterSecretStore: cluster-wide. Any namespace where someone can create an ExternalSecret can read every secret the store's role can reach. In that case, Kubernetes RBAC on ExternalSecret becomes your only access control. You can restrict which namespaces may reference the store, but those rules live in the cluster, not in IAM.
SecretStore (per namespace) ClusterSecretStore
Scope One namespace All namespaces (unless restricted)
IAM roles to manage One per team or namespace One, or a few
Blast radius of a misused ExternalSecret That team's prefix Everything the role can read
Where isolation is enforced IAM Kubernetes RBAC and store conditions
Good fit Multi-team clusters Platform-owned, shared secrets
# Option A: namespace-scoped store. Only ExternalSecrets in 'payments' can use it.
# The 'eso-payments' SA is annotated with a role limited to prod/payments/*.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-payments   # SA in this namespace (IRSA role annotation)
---
# Option B: cluster-wide store. Every namespace allowed below can read
# everything the platform role can reach, so restrict it explicitly.
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-platform-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-platform
            namespace: external-secrets  # must be set for cluster-scoped stores
  # Isolation here is enforced by Kubernetes, not IAM.
  conditions:
    - namespaceSelector:
        matchLabels:
          secrets.example.com/platform-shared: "true"

A namespace-scoped SecretStore bound to a team's IRSA service account, next to a ClusterSecretStore that uses a single platform role and limits which namespaces can reference it.

How do you author an ExternalSecret and verify the sync?

Start with refreshInterval, which sets how often the controller re-reads the remote value. Every refresh is an API call billed by Secrets Manager, so one hour suits most secrets, and a shorter interval is worth it only where you need faster rotation pickup. Under target, set name to the Kubernetes Secret you want and choose a creationPolicy:

  • Owner (the default) adds an owner reference, so deleting the ExternalSecret garbage-collects the Secret.
  • Orphan leaves the Secret in place, which is safer during migrations.
  • Merge writes into an existing Secret.

Separately, deletionPolicy controls what happens when the remote secret disappears. Retain keeps the last value, and Delete removes the Kubernetes Secret.

Use data entries to map individual fields. Each entry needs a secretKey, a remoteRef.key (the secret name or ARN) and a remoteRef.property (the JSON field). Use dataFrom with extract to copy every top-level key at once. This is convenient, but any field added upstream will land in the pod.

To verify the sync, run kubectl describe externalsecret. Look for a Ready condition set to True with reason SecretSynced, and read the events for provider errors. Then decode the generated Secret and confirm it holds exactly the keys you expect.

Problem Symptom Fix
AccessDenied Ready=False, AccessDeniedException in events Correct the resource ARN in the policy, and add kms:Decrypt if the secret uses a customer-managed key
Missing secret ResourceNotFoundException Check remoteRef.key spelling and the ARN suffix
Wrong region ResourceNotFoundException even though the name is correct Set the region in the store to match where the secret lives
Wrong property Error that the key was not found in the JSON Match the field name exactly, because matching is case-sensitive

A failed refresh keeps serving the last value, so a sync failure gives no visible sign until rotation or a new deployment breaks. Scrape the controller's Prometheus metrics and alert on any ExternalSecret that is not Ready.

# ExternalSecret that maps individual JSON fields from one AWS Secrets Manager secret.
# Assumes a ClusterSecretStore named "aws-secrets-manager" already authenticates via IRSA.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  # Re-read the remote value hourly. Each refresh is a billed Secrets Manager call.
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    # Name of the Kubernetes Secret the controller will create.
    name: orders-db-credentials
    # Owner: deleting this ExternalSecret garbage-collects the Secret.
    # Switch to Orphan during migrations so the Secret survives.
    creationPolicy: Owner
    # Retain: if the remote secret disappears, keep the last synced value.
    deletionPolicy: Retain
  data:
    # Remote secret value is JSON, e.g. {"username":"...","password":"...","host":"..."}
    - secretKey: DB_USER
      remoteRef:
        key: prod/orders/db        # secret name or full ARN
        property: username         # JSON field; a typo here fails the sync
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/orders/db
        property: password
  # Alternative: copy every top-level JSON key instead of listing them.
  # Any field added upstream will then appear in the Secret and the pod.
  # dataFrom:
  #   - extract:
  #       key: prod/orders/db

An ExternalSecret that sets refreshInterval, target name, creationPolicy and deletionPolicy, maps two JSON fields with remoteRef key and property, and shows dataFrom extract as a commented alternative.

What does secret rotation actually mean once a secret reaches a pod?

Rotation is a chain of independent delays. Secrets Manager writes a new version, but ESO only notices on the ExternalSecret's next refreshInterval tick, when it updates the Kubernetes Secret. What happens next depends on how the pod consumes that Secret:

  • Environment variables are fixed when the container starts. The pod keeps the old value until it restarts.
  • Volume mounts are refreshed by the kubelet on its periodic sync, typically within a minute or two. Mounts that use subPath are never updated. The file changing on disk does nothing unless the application re-reads it.

There are three ways to close the gap:

  • Mount the Secret as a volume and have the app reload the file when it changes or when authentication fails.
  • Trigger a rollout restart from your rotation pipeline: kubectl rollout restart deployment/payments-api -n payments.
  • Run a reloader-style controller that watches Secrets and restarts the workloads that reference them.

None of these is instant, so the rotation itself must tolerate overlap. Old and new credentials both have to stay valid until every pod has picked up the new value. If the old value is revoked immediately, any pod still holding it fails. Dual-user database rotation exists for exactly this reason.

Shortening refreshInterval reduces the lag but multiplies API calls. Secrets Manager bills per call and throttles bursts.

Illustrative estimate. Assume 200 ExternalSecrets, each making one fetch per refresh:

  • At a 1-hour interval: 200 × 24 × 30 = 144,000 calls per month.
  • At a 1-minute interval: 8,640,000 calls per month.

The interval only needs to be short relative to your overlap window, not as short as possible.

Hardening the Kubernetes side and keeping ESO cheap to run

Syncing secrets into the cluster means the Kubernetes Secret becomes the weakest copy, so three controls are required.

Encrypt etcd with KMS. On EKS, check whether envelope encryption of Secrets is enabled, and use a customer-managed KMS key if you need to control key policy and audit decrypt calls. Without it, anyone holding an etcd snapshot holds every synced credential.

Restrict get, list and watch on Secrets. list and watch return full secret values, not just names. Watch for broad bindings: the built-in edit ClusterRole includes Secrets access. Give app developers a namespaced Role that grants what they need on Deployments, Pods, logs and ExternalSecrets, and leave secrets out of its resources entirely. RBAC is additive, so leaving it out is enough. Remember that anyone who can create a Pod in the namespace can still mount a Secret, so namespace boundaries still matter.

Limit who creates ExternalSecrets and which stores they can reference. An ExternalSecret that points at a broad ClusterSecretStore can pull anything that store's IAM role can read. Scope ClusterSecretStores with namespace conditions, or use per-namespace SecretStores.

These controls are what save time. ExternalSecret manifests contain only references, so they go through normal Git review. Nobody copies secrets by hand with kubectl, and rotation happens once, in Secrets Manager.

The ongoing costs are real:

  • Controller upgrades, which need testing.
  • CRD API version changes, which force manifest migrations.
  • IAM role sprawl, which grows as you add namespaces.
  • Sync-status alerting, which you need because a failed refresh is silent until a credential expires.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to external secrets operator aws, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.