Solving Secret Sprawl in Multi-Account Kubernetes with External Secrets Operator

Infrastructure automation in Kubernetes is largely a solved problem-until secret management enters the picture. As environments scale, distributing and rotating secrets across multiple isolated clusters quickly becomes a major operational challenge.

Our team recently faced this exact problem while designing a scalable Kubernetes platform for a client running on EKS. The core issue wasn’t specific to one cloud provider or even to cloud infrastructure itself. The same challenges appear in Azure, Google Cloud, multi-cloud setups, on-prem environments, and even local development workflows using tools like KIND or Minikube.

The common denominator is isolation. Each environment-development, staging, and production-lives in its own account, namespace, or cluster. While this separation is essential for security and blast-radius control, it makes shared secret management difficult and error-prone.

This article explains how we solved multi-account Kubernetes secret synchronization using External Secrets Operator (ESO) with Bitwarden Secrets Manager, enabling centralized control with automated distribution.

The Challenge: Secret Sprawl Across Isolated Environments

Our client operates two applications with heavy reliance on third-party integrations. Problems surfaced immediately when automating new environment provisioning:

Shared Credentials in Non-Production

In development and staging, the applications relied on the same sandbox credentials for external services. Each environment needed identical values, but stored separately.

Fragmented Secret Storage

Each Kubernetes cluster lived in its own account. Using a native secrets service per account meant that rotating a single third-party API key required manual updates everywhere.

Manual Rotation and Operational Risk

The lack of centralized control led to rotation fatigue and increased risk. The requirement was clear:
update a secret once and have it propagate automatically to every cluster.

The key architectural decision was to separate secret storage from secret consumption.

The Solution: External Secrets Operator as the Integration Layer

To bridge external secret stores with Kubernetes-native secrets, we selected External Secrets Operator (ESO). ESO synchronizes secrets from an external backend into Kubernetes Secrets using declarative configuration.

For backend storage, we chose Bitwarden Secrets Manager. This decision was pragmatic: the client already used Bitwarden organization-wide, allowing us to reuse existing access controls and governance.

While Bitwarden was the selected provider, this architecture is provider-agnostic. The same pattern works with alternatives such as HashiCorp Vault. The important principle remains the same:

Store secrets centrally, and let ESO synchronize them into every Kubernetes cluster automatically.

Implementation Guide

The following steps outline the exact configuration used in our Kubernetes environment. While all resources were automated using Terraform, direct commands are shown here for clarity.
(For full automation code, refer to the linked GitHub repository.)

Prerequisites:

• A Kubernetes cluster (EKS, AKS, GKE, or equivalent)
• kubectl and helm installed locally
• A Bitwarden Secrets Manager account

Step 1: Secure Communication Between ESO and the Bitwarden SDK

The ESO Bitwarden integration requires secure HTTPS communication. To manage certificates dynamically inside the cluster, we first install Cert Manager.

helm repo add jetstack https://charts.jetstack.io 
helm repo update 

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

Create a Self-Signed ClusterIssuer

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: bitwarden-bootstrap-issuer
spec:
  selfSigned: {}
EOF

Create the External Secrets Namespace and Root CA Certificate

kubectl create namespace external-secrets

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: bitwarden-bootstrap-certificate
  namespace: external-secrets
spec:
  commonName: bitwarden-tls-ca
  isCA: true
  secretName: bitwarden-ca-certs
  issuerRef:
    name: bitwarden-bootstrap-issuer
    kind: ClusterIssuer
EOF

Create the Issuer and TLS Certificate for the SDK Server

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: bitwarden-certificate-issuer
  namespace: external-secrets
spec:
  ca:
    secretName: bitwarden-ca-certs
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: bitwarden-tls-certs
  namespace: external-secrets
spec:
  secretName: bitwarden-tls-certs
  dnsNames:
    - bitwarden-sdk-server.external-secrets.svc.cluster.local
    - external-secrets-bitwarden-sdk-server.external-secrets.svc.cluster.local
    - localhost
  issuerRef:
    name: bitwarden-certificate-issuer
    kind: Issuer
EOF

Step 2: Install External Secrets Operator with Bitwarden Support

ESO must be installed with Bitwarden SDK support enabled, which deploys the internal service used to communicate with the Bitwarden API.

helm repo add external-secrets https://charts.external-secrets.io 

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --set installCRDs=true \
  --set "bitwarden-sdk-server.enabled=true"

Step 3: Authentication

Generate a Machine Account Access Token in the Bitwarden portal. This token provides read-only access to the project containing your secrets.

Store the token as a Kubernetes Secret:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: bitwarden-access-token
  namespace: external-secrets
type: Opaque
stringData:
  token: <YOUR_BITWARDEN_ACCESS_TOKEN>
EOF

Note: The token can also be granted write access if you plan to use ESO’s PushSecret API to create secrets in Bitwarden.

Step 4: Create a ClusterSecretStore

A ClusterSecretStore allows all namespaces to reference a single global backend without duplicating authentication configuration.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: bitwarden-global-store
spec:
  provider:
    bitwardensecretsmanager:
      apiURL: https://vault.bitwarden.eu./api
      identityURL: https://vault.bitwarden.eu./identity
      auth:
        secretRef:
          credentials:
            key: token
            name: bitwarden-access-token
            namespace: external-secrets
      bitwardenServerSDKURL: https://bitwarden-sdk-server.external-secrets.svc.cluster.local:9998
      caProvider:
        type: Secret
        name: bitwarden-ca-certs
        key: ca.crt
        namespace: external-secrets
      organizationID: <YOUR_ORGANIZATION_ID>
      projectID: <YOUR_PROJECT_ID>
EOF

Step 5: Synchronize Secrets with ExternalSecret

The ExternalSecret resource defines how secrets are fetched and materialized as Kubernetes Secrets.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-payment-creds
  namespace: app-backend
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: bitwarden-global-store
    kind: ClusterSecretStore
  target:
    name: payment-creds
    creationPolicy: Owner
  data:
  - secretKey: api_key
    remoteRef:
      key: "<YOUR_NAME_OR_ID_SECRET_HERE"
EOF

The Result: Centralized Secret Management at Scale

This approach fully decouples secret lifecycle management from infrastructure provisioning.

When a new environment is created-whether in an existing account or a brand-new one-the only requirement is installing ESO and referencing the existing ClusterSecretStore. Secrets are fetched automatically.

When a secret is rotated, it’s updated once in Bitwarden and propagated to all clusters within minutes.

Whether managing two clusters or two hundred, the outcome is the same: fewer deployment bottlenecks, fewer human errors, and a significantly cleaner operational model. With ESO acting as a universal bridge, secret management becomes invisible infrastructure-secure, centralized, and automatic.

Read more of our blog post here.