Kubernetes ConfigMaps & Secrets: Patterns, Pitfalls, and Production-Ready Practices

You know the basics. Now let’s talk about what actually breaks in production.

Most tutorials show you how to create a ConfigMap. Fewer tell you what happens when your app doesn’t pick up the updated value — or why your Secret is leaking into your logs without you noticing.

This post is for engineers who’ve used ConfigMaps and Secrets before and want to understand the real-world patterns that make config management reliable, auditable, and safe.

Quick Recap — What We’re Working With

ConfigMap → non-sensitive configuration (env vars, feature flags, app settings) Secret → sensitive data (passwords, tokens, API keys) stored as base64-encoded values

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "50"
  FEATURE_NEW_UI: "true"

# Secret
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
type: Opaque
data:
  DB_PASSWORD: cGFzc3dvcmQxMjM=   # base64 encoded
  API_KEY: c3VwZXJzZWNyZXRrZXk=

Pattern 1: Env Vars vs. Volume Mounts — Choose Deliberately

Most teams default to environment variables. It works, but it has a critical limitation: env vars are set at pod startup and never updated. If you change a ConfigMap, pods won’t see the new value until they restart.

# Env var approach — static, requires pod restart on change
envFrom:
  - configMapRef:
      name: app-config
  - secretRef:
      name: app-secrets

Volume mounts, on the other hand, update automatically (with a small delay, typically 60s based on kubelet sync period):

# Volume mount approach — dynamic updates without pod restart
volumes:
  - name: config-volume
    configMap:
      name: app-config

containers:
  - name: app
    volumeMounts:
      - name: config-volume
        mountPath: /etc/config
        readOnly: true

Your app reads /etc/config/LOG_LEVEL at runtime instead of from $LOG_LEVEL. If your app supports file-based config reload (like most Java Spring apps with @RefreshScope, or apps watching inotify events), this is a much better pattern for config that changes frequently.

Rule of thumb:

• Feature flags, log levels, tunable params → volume mount
• DB connection strings, API keys → env var (less churn, restart acceptable)

Pattern 2: Don’t Put Secrets in ConfigMaps (Obviously) — But Also Watch Your Logs

This seems obvious, but it shows up in real codebases more than you’d think:

# ❌ Never do this
data:
  DATABASE_URL: "postgres://admin:password123@db-host:5432/mydb"

The subtler version of this mistake is logging your environment on startup. Many apps (Spring Boot, Rails, Node) log their full environment at boot. If your Secret is injected as an env var, it ends up in your log aggregator in plaintext.

Fix: Either suppress env logging in your app config, or use volume-mounted Secrets and have your app read only the specific keys it needs — never log the whole config object.

# Mount only what you need, not the whole secret
volumes:
  - name: db-creds
    secret:
      secretName: app-secrets
      items:
        - key: DB_PASSWORD
          path: db_password   # Only this key is mounted

Pattern 3: Namespace Isolation Is Your First Line of Defence

Secrets are namespaced. A pod in namespace-a cannot read a Secret in namespace-b — by design. Use this.

# Each environment gets its own namespace and its own secrets
namespaces:
  - dev
  - staging
  - production

# Verify — this should return nothing from another namespace
kubectl get secrets -n production --as=system:serviceaccount:dev:default

Pair this with RBAC to lock down which service accounts can read Secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["app-secrets"]   # Only this specific secret

Notice resourceNames — don’t give broad get on all Secrets. Be explicit.

Pattern 4: The Immutable ConfigMap Pattern

Kubernetes 1.21+ supports immutable ConfigMaps and Secrets:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v3   # Version in the name
immutable: true
data:
  LOG_LEVEL: "info"

Why bother? Two reasons:

1. Performance — the kubelet stops watching immutable ConfigMaps, which reduces API server load at scale
2. Safety — accidental kubectl edit won’t silently change running app behaviour

The trade-off: you can’t update it. You create a new versioned ConfigMap (app-config-v4) and update your Deployment to reference it. This forces a controlled rollout and gives you a natural audit trail.

This pattern is especially useful for things like feature flag snapshots tied to a specific release.

Pattern 5: Secrets Encryption at Rest — Don’t Skip This

By default in most Kubernetes distributions, Secrets are stored unencrypted in etcd. Base64 is encoding, not encryption. Anyone with etcd access can read your Secrets.

Check if encryption is enabled:

kubectl get apiserver -o yaml | grep -A5 encryptionConfig

On EKS, enable envelope encryption via KMS:

aws eks create-cluster \
  --name my-cluster \
  --encryption-config '[{
    "resources": ["secrets"],
    "provider": {
      "keyArn": "arn:aws:kms:us-east-1:123456789:key/your-key-id"
    }
  }]'

For teams already running clusters, consider moving to External Secrets Operator (ESO) with AWS Secrets Manager or HashiCorp Vault as the source of truth. Kubernetes Secrets become thin wrappers that sync from the real store:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-secrets
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/app/db
        property: password

This is the production-grade pattern. Rotation, auditing, and access control all live in the secrets manager — not in Kubernetes.

Common Gotchas

1. ConfigMap update doesn’t restart your pods Changing a ConfigMap doesn’t trigger a rollout. Use kubectl rollout restart deployment/app or add an annotation that changes on config update:

kubectl annotate deployment app \
  config/checksum="$(kubectl get cm app-config -o json | sha256sum)"

Or use Reloader (open source) to automatically restart pods when their ConfigMap/Secret changes.

2. Secret size limit Kubernetes Secrets have a 1MB limit per object. If you’re storing certificates, large config blobs, or JWKs — you’ll hit this. Use a volume-mounted file from an external store instead.

3. base64 with newlines When manually creating Secrets, echo “mypassword” | base64 includes a trailing newline. Use echo -n “mypassword” | base64. That trailing newline has broken more DB connections than I’d like to admit.

# Wrong
echo "mypassword" | base64       # bXlwYXNzd29yZAo=  ← has \n

# Correct
echo -n "mypassword" | base64    # bXlwYXNzd29yZA==

Summary

Concern Recommendation Config that changes often Volume mount, not env var Sensitive values Never in ConfigMap, watch your startup logs Cross-namespace access Namespace isolation + scoped RBAC Config drift prevention Immutable ConfigMaps with versioned names Secrets at rest KMS encryption or External Secrets Operator Auto-restart on config change Reloader or checksum annotation

ConfigMaps and Secrets look simple on the surface. The complexity comes from the edges — stale config after updates, secrets leaking through logs, etcd exposure, and config drift across environments. Getting these patterns right early saves a lot of painful debugging later.

If this helped, follow me on Medium for more real-world Kubernetes and Platform Engineering content. Have a pattern I missed? Drop it in the comments.

Originally published on Medium.