Fetching Secrets in Kubernetes with the AWS Secrets Manager Agent Sidecar

Managing secrets in Kubernetes can sound straightforward-until you actually implement it at scale. Common approaches include mounting Kubernetes secrets as environment variables, using an external secrets operator, or calling cloud SDKs directly from application code. Each option introduces trade-offs around caching, auditing, security boundaries, and application coupling.

In this post, we walk through an alternative pattern: running the AWS Secrets Manager Agent as a sidecar container. With this approach, your application fetches secrets via a local HTTP call-no AWS SDK required-while benefiting from built-in caching, SSRF protection, and clean separation of concerns.

We’ll demonstrate the setup using a small Go service, but this sidecar pattern works with any language or runtime.

Why Use a Sidecar for Secret Management?

The Secrets Manager Agent runs as a separate container in the same pod. Your application communicates with it over localhost:2773, which provides several advantages:

The trade-offs are an extra container per pod and a local HTTP call. In practice, both introduce negligible overhead.

Architecture Overview

AWS Secrets Manager Agent

On pod startup, the agent generates a random SSRF token, writes it to a shared in-memory volume, and exports it as an environment variable.
The application container reads the same token from the shared volume and includes it in every request to the agent.

Step 1: Containerizing the AWS Secrets Manager Agent

AWS provides the agent as a Rust project on GitHub. We build it from source and pin it to a release tag for reproducibility.

# Build stage - compile the Rust binary
FROM rust:1.82-bookworm AS builder
RUN apt-get update && apt-get install -y build-essential && rm -rf /var/lib/apt/lists/*

WORKDIR /build
# Clone the agent repo (pin to a release tag for reproducibility)
RUN git clone --branch v2.0.0 --depth 1 https://github.com/aws/aws-secretsmanager-agent.git .
RUN cargo build --release

# Runtime stage - minimal image with just the binary
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /build/target/release/aws_secretsmanager_agent ./secrets-manager-agent
COPY startup.sh .
COPY config.toml .
RUN chmod +x startup.sh secrets-manager-agent
ENTRYPOINT ["./startup.sh"]

Agent Configuration (config.toml)

log_level = "INFO"
log_to_file = false
http_port = 2773
ttl_seconds = 300
cache_size = 1000

The ttl_seconds = 300 setting means secrets are cached for five minutes. After that, the agent automatically fetches fresh values-no pod restart required.

Startup Script and SSRF Token Handling

#!/bin/bash
set -e

# Generate a random SSRF token, write to shared volume and export as env var
TOKEN=$(head -c 32 /dev/urandom | base64 | tr -d '=+/')
echo -n "$TOKEN" > /shared/awssmatoken
chmod 444 /shared/awssmatoken
export AWS_TOKEN="$TOKEN"

# Start the agent with logging to stdout
exec ./secrets-manager-agent --config config.toml 2>&1

The token is ephemeral-generated on every pod start, stored only in memory, and destroyed with the pod. There’s nothing to rotate or persist.

Build and push to ECR (arm64 in this example for Graviton nodes):

docker buildx build --platform linux/arm64 \ -t <account>.dkr.ecr.us-east-1.amazonaws.com/aws-secrets-manager-agent:lates\--push ./secrets-manager-agent

Step 2: The Go Application – Calling the Sidecar

The application does not use the AWS SDK. Instead, it performs an HTTP GET against the local agent and passes the SSRF token in a request header.

func sidecarHandler(w http.ResponseWriter, r *http.Request) {
   secretName := os.Getenv("secret")
   if secretName == "" {
       writeJSON(w, http.StatusOK, map[string]string{"message": "no secret env var set"})
       return
   }

   agentURL := os.Getenv("SECRETS_MANAGER_AGENT_URL")
   if agentURL == "" {
       agentURL = "http://localhost:2773"
   }

   tokenBytes, err := os.ReadFile("/shared/awssmatoken")
   if err != nil {
       writeJSON(w, http.StatusInternalServerError,
           map[string]string{"error": "failed to read SSRF token: " + err.Error()})
       return
   }

   req, err := http.NewRequestWithContext(
       r.Context(),
       http.MethodGet,
       agentURL+"/secretsmanager/get?secretId="+secretName,
       nil,
   )
   if err != nil {
       writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
       return
   }

   req.Header.Set("X-Aws-Parameters-Secrets-Token", string(tokenBytes))

   resp, err := http.DefaultClient.Do(req)
   if err != nil {
       writeJSON(w, http.StatusBadGateway, map[string]string{
           "error":  "failed to reach secrets manager agent sidecar",
           "detail": err.Error(),
       })
       return
   }
   defer resp.Body.Close()

   var secretResp struct {
       SecretString string `json:"SecretString"`
   }

   if err := json.NewDecoder(resp.Body).Decode(&secretResp); err != nil {
       writeJSON(w, http.StatusInternalServerError,
           map[string]string{"error": "failed to decode secret response"})
       return
   }

   var secretData map[string]string
   if err := json.Unmarshal([]byte(secretResp.SecretString), &secretData); err != nil {
       writeJSON(w, http.StatusOK, map[string]string{"secret_value": secretResp.SecretString})
       return
   }

   writeJSON(w, http.StatusOK, secretData)
}

Key Implementation Details:

Step 3: Kubernetes Deployment with a Sidecar Container

This deployment runs two containers in a single pod, sharing an in-memory volume for the SSRF token.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-demo
  template:
    metadata:
      labels:
        app: go-demo
    spec:
      serviceAccountName: go-demo
      containers:
        - name: go-demo
          image: <account>.dkr.ecr.<region>.amazonaws.com/go-demo:latest
          ports:
            - containerPort: 8080
          env:
            - name: secret
              value: "demo-secret"
          volumeMounts:
            - name: ssrf-token
              mountPath: /shared
              readOnly: true
        - name: secrets-manager-agent
          image: <account>.dkr.ecr.<region>.amazonaws.com/aws-secrets-manager-agent:latest
          ports:
            - containerPort: 2773
          volumeMounts:
            - name: ssrf-token
              mountPath: /shared
      volumes:
        - name: ssrf-token
          emptyDir:
            medium: Memory

Important Notes:

Avoid mounting the shared volume at /var/run. Kubernetes needs that path for the service account token, and overriding it will break pods-especially with distroless images.

SSRF Token Validation Behavior

The agent validates the SSRF token against one of these environment variables:

If none are set, the agent fails to start.

That’s why the startup script both writes the token to disk and exports it as AWS_TOKEN.

How the SSRF Token Flow Works

  1. Pod starts and the sidecar runs startup.sh
  2. A random 32-byte token is generated
  3. The token is written to /shared/awssmatoken and exported as AWS_TOKEN
  4. The agent validates incoming requests against the token
  5. The application reads the token and sends it as an HTTP header
  6. The agent verifies the token and returns the secret

The token is ephemeral, scoped to the pod lifetime, and never stored externally.

Summary

Using the AWS Secrets Manager Agent as a Kubernetes sidecar keeps application code clean and focused. Instead of embedding cloud SDKs, managing credentials, and implementing caching logic, the application makes a simple local HTTP call.

The agent handles:

Key Takeaways:

This sidecar-based approach provides a secure, maintainable, and portable way to manage secrets in Kubernetes environments.

Check out more of our blog posts here.