Secrets Management in Cloud-Native Applications

The Secret Leakage Epidemic

Database passwords in .env files. API keys in git commits. Certificates in Docker images.

Secrets leak to GitHub, get indexed by search engines, exploited by attackers.

Secrets management keeps credentials secure, rotated, and auditable.

What Is a Secret?

• Database passwords
• API keys (Stripe, Twilio, AWS)
• SSL certificates
• OAuth tokens
• Private encryption keys
• SSH keys

Anything that grants access should be treated as a secret.

The Secret Lifecycle

1. Generate: Create new credential
2. Store: Vault or secrets manager
3. Distribute: Application retrieves when needed
4. Rotate: Replace with new credential periodically
5. Revoke: Delete when no longer needed
6. Audit: Log all access

Most organizations fail at step 4 (rotation) or 6 (audit).

Storing Secrets Securely

Never commit secrets to git.

• Rotating them requires changing all copies in history
• Difficult to extract later
• Attracts attackers scanning GitHub

Use a secrets manager:

• HashiCorp Vault (self-hosted)
• AWS Secrets Manager (managed)
• GCP Secret Manager (managed)
• Azure Key Vault (managed)

HashiCorp Vault

Open-source secrets manager. Encrypt and manage secrets.

Architecture:

Application needs database password
  ↓
App authenticates to Vault (with JWT, AWS IAM, etc.)
  ↓
Vault issues temporary token
  ↓
App requests secret with token
  ↓
Vault logs access (audit trail)
  ↓
Vault returns database password
  ↓
App connects to database
  ↓
Vault can auto-rotate password without notifying app

Setup:

# Start Vault server
vault server -config=config.hcl

# Store a secret
vault kv put secret/database/prod   username="admin"   password="secure_password_here"

# Retrieve a secret
vault kv get secret/database/prod
# Output:
# username = admin
# password = secure_password_here

Advantages:

• Full control (self-hosted)
• Strong encryption (AES-GCM)
• Audit logs (who accessed what, when)
• Dynamic secrets (generate temp credentials per use)
• No need to distribute secrets manually

Disadvantages:

• Operational complexity (Vault cluster management)
• Another system to monitor and back up

AWS Secrets Manager

Managed secrets service. AWS handles infrastructure.

import boto3

client = boto3.client('secretsmanager')

# Store a secret
client.create_secret(
  Name='prod/database/password',
  SecretString='secure_password'
)

# Retrieve a secret
response = client.get_secret_value(SecretId='prod/database/password')
password = response['SecretString']

Advantages:

• Zero infrastructure overhead
• Easy integration with AWS services
• Audit logs (CloudTrail)
• Automatic rotation (integrate with Lambda)

Disadvantages:

• Vendor lock-in (AWS-specific)
• Cost: $0.40 per secret per month + API calls

Secret Rotation

Rotate secrets regularly. Compromised secret is active for only limited time.

Rotation frequency:

• Critical secrets (database root password): monthly
• API keys: quarterly
• Certificates: annually (before expiration)

Process:

1. Generate new secret
2. Test with new secret
3. Update application to use new secret
4. Keep old secret active for backward compatibility (1 week)
5. Revoke old secret

Database password rotation with Vault:

Vault generates new password
  ↓
Vault connects to database, changes password
  ↓
Vault returns new password to app
  ↓
App uses new password from that point forward
  ↓
Vault keeps old password in case rollback needed
  ↓
After 7 days, revoke old password

Zero downtime. App never knows about rotation.

Application Integration

Best practice: environment variables Don't hardcode secrets. Load from vault at startup.

import os
import hvac

vault_token = os.getenv('VAULT_TOKEN')
vault_addr = os.getenv('VAULT_ADDR')

client = hvac.Client(url=vault_addr, token=vault_token)
secret = client.secrets.kv.v2.read_secret_version(path='database/prod')

DATABASE_PASSWORD = secret['data']['data']['password']

Kubernetes native: Sealed Secrets or External Secrets

# Encrypted secret stored in git
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-secret
spec:
  encryptedData:
    password: AgBx8L2...

Kubernetes decrypts at runtime.

Secrets Checklist

• No secrets in git or container images
• Secrets stored in vault (self-hosted or managed)
• Access via API (app requests secret at runtime)
• Strong encryption (AES-256 minimum)
• Audit logs (log all access)
• Rotation policy (monthly for critical secrets)
• Access control (only authorized services can access)
• Monitoring (alert on access anomalies)
• Disaster recovery (backup vault, test restore)
• Zero-trust: apps authenticate to vault