Secrets Management in Kubernetes: Vault vs SOPS vs Cloud Managers

The three problems behind secrets management in Kubernetes

Most "secrets management" debates stall because people answer different questions with the same word. It helps to split the problem into three layers.

Source of truth is where the authoritative value lives and who can change it. This could be an encrypted file in Git, a Vault cluster, or a cloud provider's secret store.

Delivery path is how the value reaches the process: an environment variable, a file mounted into the pod, or an API call the application makes at runtime. Each path behaves differently when the value changes. Environment variables are fixed when the process starts. Mounted files can be updated in place. API calls return whatever the store holds at that moment.

Lifecycle covers rotation, revocation and audit. It answers who read the secret, when it changes, and how quickly a leaked value can be cut off.

The gaps in this table explain why teams combine tools. External Secrets Operator stores nothing. SOPS delivers nothing at runtime. A common production setup uses Vault or a cloud manager as the source of truth and External Secrets Operator to deliver values into the cluster.

What native Kubernetes Secrets do and do not protect

Before choosing an external store, it helps to know what the built-in object actually protects. A Kubernetes Secret stores values in its data field as base64. That is an encoding for binary-safe transport, not encryption: anyone who can read the object can decode it with a single command. The companion code example defines a Secret holding a database password and a Role that grants get on that one Secret by name.

Encryption at rest is narrower than it sounds. Configuring the API server to encrypt Secrets in etcd protects against someone who obtains the etcd data files, a disk snapshot or an etcd backup. It does nothing against API reads. The API server decrypts transparently, so any user or ServiceAccount allowed to read Secrets receives plaintext.

RBAC is the real boundary. Grant get on named Secrets through resourceNames. Avoid granting list and watch, because both return full Secret contents, not only names. A Role that allows list on Secrets exposes every Secret in the namespace.

Pod creation is secret access. Anyone who can create a pod in a namespace can mount any Secret in that namespace as a volume or environment variable. They can also run the pod as any ServiceAccount there and inherit its permissions. Controlling who can create pods, Deployments and Jobs matters as much as which store holds the secret. An external manager whose credentials are reachable from a loosely governed namespace adds little protection.

CopyapiVersion: v1
kind: Secret
metadata:
  name: orders-db
  namespace: orders
type: Opaque
data:
  username: YXBwX3VzZXI=                  # app_user
  password: cGxhY2Vob2xkZXItcGFzc3dvcmQ=  # placeholder-password
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-orders-db-secret
  namespace: orders
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["orders-db"]
    verbs: ["get"]

A Secret whose base64 values decode trivially, paired with a Role that grants get on that single Secret by name and deliberately omits list and watch.

How does it work? The common flow behind every approach

The tools differ in where they store values, but every external secrets setup runs the same four steps. Knowing them makes the tool-specific details easier to compare.

1. Prove identity. Something has to show the store who is asking. In Kubernetes this is usually the pod's ServiceAccount token, which the store checks against the cluster or against a cloud identity provider that trusts the cluster's OIDC issuer. For SOPS, the identity is whoever holds the decryption key or has permission to use the KMS key.
2. Authorize. The store maps that identity to a policy: a Vault role and its policies, an IAM role, or the set of keys listed in a SOPS file's metadata. The policy decides which paths or files the caller can read.
3. Fetch or decrypt. The caller reads the value from the store's API, or decrypts the ciphertext locally. This is the first point where plaintext exists outside the store.
4. Deliver and refresh. The value reaches the process as an environment variable, a file, or an in-memory result of an API call. Whatever component did the delivery is also responsible for picking up changes later, either by polling, by renewing a lease, or not at all.

Step 1 is the bootstrap problem: some credential must exist before any secret can be fetched. Steps 3 and 4 decide how many readable copies exist and how quickly rotation takes effect. The following sections show how SOPS, Vault and cloud secret managers implement these steps, and the walkthrough of one database password traces where plaintext ends up in each.

How SOPS, Vault and cloud secret managers work

SOPS

SOPS encrypts the values in a YAML or JSON file and leaves the keys readable. A file can show password: ENC[AES256_GCM,data:...] next to a sops: metadata block that records which KMS, age or PGP keys can decrypt it. Because the ciphertext is safe to commit, reviewers can see which keys changed in a diff without seeing any values. Decryption still has to happen somewhere, usually in CI or a GitOps controller. Whatever holds the decryption key becomes the trust boundary, and anyone who can run arbitrary steps in that pipeline can read every secret it decrypts.

HashiCorp Vault

With Vault's Kubernetes auth method, a pod presents its ServiceAccount token. Vault validates the token against the cluster and maps the ServiceAccount name and namespace to a role, and that role carries policies. Every read goes to an audit log. Dynamic secrets engines can create a short-lived database user for each lease, which means a leaked credential expires instead of lasting indefinitely. Applications receive secrets in one of three ways: an injected agent sidecar that writes files, the Secrets Store CSI driver, or direct API calls.

Cloud secret managers

AWS Secrets Manager and similar services control access with IAM, record audit events through the provider's logging, and run rotation through provider hooks, such as a Lambda function in AWS. Workload identity removes static access keys. A ServiceAccount annotated with eks.amazonaws.com/role-arn lets pods exchange their projected token for temporary credentials scoped to that one IAM role.

CopyapiVersion: v1
kind: Secret
metadata:
    name: orders-db
    namespace: orders
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:3kQx9w==,iv:Placeholder1=,tag:Placeholder2==,type:str]
    password: ENC[AES256_GCM,data:Zm9vYmFyYmF6,iv:Placeholder3=,tag:Placeholder4==,type:str]
sops:
    # Whoever can use one of these keys can decrypt the file.
    kms:
        - arn: arn:aws:kms:us-east-1:111122223333:key/EXAMPLE-KEY-ID
          enc: AQICAHh...placeholder...
    age:
        - recipient: age1exampleplaceholderrecipientkey
    # A MAC over the values detects tampering with the ciphertext.
    mac: ENC[AES256_GCM,data:placeholder,iv:placeholder,tag:placeholder,type:str]
    # (other metadata fields omitted)
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders-api
  namespace: orders
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/orders-api-secrets-read

A SOPS-encrypted Secret manifest whose keys stay readable while its values and MAC are ciphertext, followed by a ServiceAccount annotated with eks.amazonaws.com/role-arn so pods get temporary IAM credentials instead of static keys.

Implementing delivery with External Secrets Operator

External Secrets Operator (ESO) can use that same mechanism. Instead of each application calling the store, the operator authenticates once, reads values from Vault, AWS Secrets Manager or another backend, and writes them into ordinary Kubernetes Secrets. Applications keep mounting or reading environment variables from a Secret and do not need a store-specific client library.

The setup uses two resources. A SecretStore says where and how to authenticate, for example the Secrets Manager provider, a region, and a reference to a ServiceAccount whose IAM role grants read access. An ExternalSecret says what to fetch. It names the store, sets a refreshInterval, gives the target Secret name, and maps each remote key or JSON property to a key in that Secret. The example that follows shows this pair. Field names and API versions have changed across ESO releases, so check them against the CRDs you install.

This pattern has three costs:

• The value is back in etcd. The protections and gaps of native Secrets described earlier apply again, including encryption at rest and who can get Secrets in the namespace.
• The operator becomes a high-value target. Its role can often read many paths. Scope each SecretStore narrowly, and prefer namespaced stores over one cluster-wide store with broad access.
• Rotation is delayed. A new value reaches the cluster only on the next refresh. Pods see the change even later, and the rotation section below covers that delay.

CopyapiVersion: external-secrets.io/v1
kind: SecretStore              # namespaced store: limits blast radius to this namespace
metadata:
  name: aws-sm-payments
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-payments-reader   # IAM role scoped to payments/* paths only
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: payments
spec:
  refreshInterval: 1h          # rotation in the store reaches the cluster up to 1h later
  secretStoreRef:
    name: aws-sm-payments
    kind: SecretStore
  target:
    name: payments-db-credentials   # ordinary Secret, stored in etcd
    creationPolicy: Owner           # ESO owns and updates this Secret
  data:
    - secretKey: DB_USERNAME        # key in the Kubernetes Secret
      remoteRef:
        key: payments/prod/db       # name of the secret in Secrets Manager
        property: username          # JSON field inside that secret
    - secretKey: DB_PASSWORD
      remoteRef:
        key: payments/prod/db
        property: password

A namespaced SecretStore that authenticates to AWS Secrets Manager through a scoped ServiceAccount, and an ExternalSecret that maps two JSON fields into a native Secret refreshed hourly.

Following one database password through each approach

The refresh delay matters for rotation, which is covered in the next section. Here, the question is narrower: where does one database password exist in readable form, and what unlocks it?

Path 1: Git → SOPS → Secret. The repository holds only ciphertext. The deploying component, either a CI job or a GitOps controller, decrypts the file using an age private key or permission to call a KMS key. Plaintext then exists in that component's memory, in the Kubernetes Secret stored in etcd, and in the pod's environment or mounted file. The unlocking credential is the decryption key held by the deployer.

Path 2: cloud store → ESO → Secret. The password lives in the cloud secret manager, encrypted under the provider's keys. ESO reads it using a cloud IAM role bound to its service account. Plaintext appears in ESO's memory, then in etcd, then in the pod. This means two separate systems hold readable copies, so cluster RBAC on Secrets matters as much as the IAM policy.

Path 3: Vault → agent → file. The agent authenticates with the pod's service account token, and Vault validates that token against the Kubernetes API. The agent writes the password to an in-memory volume shared with the application. Nothing is written to etcd. Plaintext exists only inside Vault and inside that pod.

The bootstrap secret

Each path needs one credential to obtain all the others. For SOPS, that credential is the decryption key, which someone has to install by hand. For ESO, it is workload identity federation, which avoids a static key only if the cloud trusts the cluster's OIDC issuer. Vault relies on the service account token, but Vault also has to be unsealed first, and that is a separate bootstrap problem.

CopyPATH 1: Git -> SOPS -> Secret          [P] = plaintext readable here
+----------+   +----------------+   +-------------+   +-----------+
| Git repo |-->| CI / GitOps    |-->| K8s Secret  |-->| Pod env / |
| cipher   |   | decrypt    [P] |   | in etcd [P] |   | file  [P] |
+----------+   +----------------+   +-------------+   +-----------+
                 ^ unlock: age private key or KMS key permission
PATH 2: cloud store -> ESO -> Secret
+-------------+   +-------------+   +-------------+   +-----------+
| Cloud store |-->| ESO         |-->| K8s Secret  |-->| Pod env / |
| (encrypted) |   | memory  [P] |   | in etcd [P] |   | file  [P] |
+-------------+   +-------------+   +-------------+   +-----------+
                    ^ unlock: cloud IAM role bound to ESO's SA
PATH 3: Vault -> agent -> mounted file   (no copy in etcd)
+-------------+   +-------------+   +-----------------------------+
| Vault   [P] |-->| Vault agent |-->| in-memory volume -> app [P] |
| (unsealed)  |   | in pod  [P] |   | (same pod)                  |
+-------------+   +-------------+   +-----------------------------+
                    ^ unlock: pod SA token, checked via K8s API
BOOTSTRAP: the one credential that obtains all the others
+--------------+  +------------------------+  +----------------------+
| SOPS: decrypt|  | ESO: workload identity |  | Vault: SA token, AND |
| key, hand-   |  | needs cloud to trust   |  | Vault unseal (its    |
| installed    |  | cluster OIDC issuer    |  | own bootstrap)       |
+--------------+  +------------------------+  +----------------------+

One database password traced through SOPS, ESO, and Vault Agent, marking each hop where plaintext exists, the identity or key that unlocks it, and the bootstrap credential each path still depends on.

How rotation reaches a running process

Rotating a value in the store is the easy part. The process holding the old value is what decides whether rotation works.

Environment variables are copied into the process when the container starts, and the kubelet never changes them afterwards. External Secrets Operator can update the Secret object, but every pod still holds the old password until it restarts. Env vars also leak easily. Child processes inherit them, crash dumps and core files capture them, debug endpoints that print the environment expose them, and they often appear in logs when someone prints the process environment at startup.

Mounted Secret volumes avoid most of that. The kubelet refreshes them after a delay set by its sync period and cache, so updates are eventual rather than instant. A volume mounted with subPath is never refreshed. Files only help if the application rereads them, by watching the file or rereading it on each new connection, and if it drains connection pools that authenticated with the old credential.

Dynamic credentials add lease expiry. Something must renew the lease and request a new credential before it expires, either the application through a client library or a sidecar such as Vault Agent that rewrites a file.

What are the alternatives?

SOPS, Vault and cloud secret managers are not the only options. Several alternatives are worth considering, either on their own or alongside these tools.

Native Secrets with KMS-backed encryption at rest. For a single cluster with few secrets, plain Kubernetes Secrets plus API server encryption using a KMS provider, tight RBAC with resourceNames, and strict control over who can create pods may be enough. You give up central audit of reads outside the cluster, cross-cluster sharing and automated rotation, but you add no new system to operate.

Secrets Store CSI driver without syncing. Instead of copying values into Kubernetes Secrets, the CSI driver mounts them from Vault or a cloud provider directly into the pod's filesystem at pod start. This keeps the value out of etcd unless you enable its optional sync to Secret objects. The trade-off is that the pod depends on the external store being reachable when it starts.

Direct API calls from the application. The application authenticates with workload identity and reads secrets through the store's SDK at runtime. This gives the freshest values and no extra copies in the cluster, at the cost of a store-specific dependency in every service and the need for caching and retry logic so a store outage does not take the application down.

Encrypt-to-cluster controllers. Some GitOps setups use a controller that holds a private key inside the cluster. Developers encrypt values with the matching public key and commit the result, and the controller decrypts it into a Secret. This resembles SOPS, but the decryption key never leaves the cluster, which also ties the encrypted files to that cluster's key.

Removing the secret entirely. The strongest alternative is to need no static secret at all. Many cloud services accept workload identity directly, so a pod can call object storage or a queue with temporary credentials from its ServiceAccount instead of a stored access key. Some managed databases support IAM-based authentication with short-lived tokens. Where this is available, there is nothing to store, rotate or leak.

Choosing an approach: fit, when to avoid each, and cost

The development cost is the integration work: wiring authentication, writing policies, and changing applications to reload credentials. The return is engineering time you no longer spend on manual rotation, an audit trail that answers "who read this?", and fewer long-lived static keys left in CI variables and laptops.

Illustrative estimate (assumed numbers, not a benchmark). Annual hours saved = secrets × rotations per year × manual minutes per rotation ÷ 60. Assume 40 secrets rotated 4 times a year at 30 minutes each: 40 × 4 × 30 ÷ 60 = 80 hours. If a self-hosted Vault takes an assumed 6 hours a month to operate, that is 72 hours a year, and the time savings alone barely cover it. A managed service or SOPS would come out ahead on time unless you also value dynamic credentials.

Count combinations as a cost too. Running SOPS for bootstrap secrets alongside ESO for application secrets gives you two auth paths, two RBAC models and two places for a misconfiguration to leak data.

How can it save engineering time?

The estimate above counts only scheduled rotation. Most of the time a well-built secrets setup saves comes from removing manual, error-prone steps that recur across the whole team.

Rotation without redeploys. When applications read secrets from mounted files and reconnect on change, a rotation in the store reaches running pods without anyone editing manifests, re-encrypting files or rolling Deployments. With environment variables, every rotation also needs a coordinated restart, so moving to file-based delivery removes that step.

Faster incident response. When a credential leaks, the slow part is usually finding every place it is used and replacing it. With a central store and a single delivery path, you change it once and the sync or lease mechanism propagates it. With dynamic secrets, you can revoke the lease and the credential stops working immediately, instead of waiting for a manual change on the database.

Answering audit questions from logs. "Who read the production database password last month?" is a log query when Vault audit devices or provider audit logs are enabled. Without them, it turns into interviews and guesswork across CI variables, laptops and chat history.

Simpler onboarding and offboarding. When services authenticate with workload identity and people access secrets through policies, a new engineer or service gets access by being added to a role, and a departing one loses it by being removed. Nobody has to hand out copies of a shared key or rotate everything after someone leaves.

Less custom glue. ESO or the CSI driver replaces per-team scripts that fetch secrets in CI, template them into manifests and push them to clusters. Those scripts tend to break silently and need someone who remembers how they work.

Fewer outages from expired or mismatched credentials. Automated rotation with a known refresh path removes a common failure: a credential changed in one place and not another, discovered when a service starts failing authentication.

These savings only appear if the integration is finished. A store that still requires manual restarts, or that runs next to an older path nobody removed, adds work instead of saving it. Measure the current cost of rotation, incident handling and access requests with your own numbers before deciding how much tooling the savings justify.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to secrets management kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.