- name
- secrets-management
- description
- Use when designing secret storage, rotation, or credential management systems. Covers HashiCorp Vault patterns, AWS Secrets Manager, Azure Key Vault, secret rotation, and zero-knowledge architectures.
- allowed-tools
- Read, Glob, Grep
Secrets Management
Comprehensive guide to managing secrets, credentials, and sensitive configuration - from storage to rotation to access control.
When to Use This Skill
- Designing secret storage architecture
- Implementing secret rotation
- Integrating applications with secret stores
- Managing API keys, passwords, certificates
- Understanding Vault, AWS Secrets Manager, Azure Key Vault
- Zero-knowledge and envelope encryption patterns
Secrets Management Fundamentals
What Are Secrets?
Types of Secrets:
Credentials:
├── Database passwords
├── API keys
├── OAuth client secrets
├── SSH private keys
└── Service account tokens
Certificates:
├── TLS certificates and private keys
├── Code signing certificates
├── Client certificates
└── CA certificates
Encryption Keys:
├── Data encryption keys (DEK)
├── Key encryption keys (KEK)
├── HMAC keys
└── Signing keys
Sensitive Configuration:
├── Connection strings
├── License keys
├── Webhook URLs with tokens
└── Third-party credentialsSecrets Lifecycle
Secret Lifecycle:
1. Generation
└── Create with sufficient entropy
└── Use cryptographic randomness
└── Appropriate key length
2. Storage
└── Encrypt at rest
└── Access control
└── Audit logging
3. Distribution
└── Secure transport (TLS)
└── Just-in-time access
└── Minimize copies
4. Usage
└── Memory protection
└── Minimize exposure window
└── Clear after use
5. Rotation
└── Regular schedule
└── Zero-downtime rotation
└── Update all consumers
6. Revocation
└── Immediate effect
└── Propagate to all systems
└── Audit trail
7. Destruction
└── Secure deletion
└── Verify removal
└── Clear backupsArchitecture Patterns
Centralized Secret Store
Centralized Architecture:
┌─────────────────────────────────────────────────┐
│ Secret Store │
│ ┌─────────────────────────────────────────┐ │
│ │ • Encrypted storage │ │
│ │ • Access control │ │
│ │ • Audit logging │ │
│ │ • Rotation management │ │
│ │ • High availability │ │
│ └─────────────────────────────────────────┘ │
└─────────────────────┬───────────────────────────┘
│
┌─────────────┼─────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ App A │ │ App B │ │ App C │
│ │ │ │ │ │
│Fetches │ │Fetches │ │Fetches │
│secrets │ │secrets │ │secrets │
│on start │ │on demand│ │cached │
└─────────┘ └─────────┘ └─────────┘
Benefits:
+ Single source of truth
+ Centralized audit
+ Consistent policies
+ Easier rotation
Challenges:
- Single point of failure
- Network dependency
- Latency for secret accessEnvelope Encryption
Envelope Encryption:
┌─────────────────────────────────────────────────┐
│ Key Hierarchy │
│ │
│ ┌─────────────┐ │
│ │ Master Key │ (Never leaves KMS) │
│ │ (KEK) │ │
│ └──────┬──────┘ │
│ │ Encrypts │
│ ▼ │
│ ┌─────────────┐ │
│ │ Data Key │ (Wrapped/encrypted) │
│ │ (DEK) │ │
│ └──────┬──────┘ │
│ │ Encrypts │
│ ▼ │
│ ┌─────────────┐ │
│ │ Data │ (Your secrets/data) │
│ │ │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────┘
Flow:
1. Generate DEK locally
2. Encrypt data with DEK
3. Send DEK to KMS for wrapping with KEK
4. Store: encrypted data + wrapped DEK
5. To decrypt: unwrap DEK with KMS, decrypt data
Benefits:
- Master key never exposed
- Can rotate DEK without re-encrypting all data
- Distributed encryption (KMS not a bottleneck)Zero-Knowledge Architecture
Zero-Knowledge Secret Access:
┌──────────────┐
│ Client │
│ │
│ Has: secret │
│ key derived │
│ from password│
└──────┬───────┘
│
┌──────────────────────▼───────────────────────┐
│ Server │
│ │
│ Stores: encrypted secrets │
│ Cannot decrypt (doesn't have key) │
│ │
│ User's secrets = Encrypt(data, userDerivedKey)│
└───────────────────────────────────────────────┘
Properties:
- Server cannot read secrets even if compromised
- User password never transmitted
- Key derived client-side using KDF
- Server only sees encrypted blobs
Trade-offs:
+ Maximum privacy
+ Server breach doesn't expose secrets
- Can't recover if user forgets password
- Can't audit what's stored
- Server can't validate secretsHashiCorp Vault
Vault Architecture
Vault Components:
┌─────────────────────────────────────────────────────┐
│ Vault │
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ API Layer │ │
│ │ (HTTP/HTTPS interface for all operations) │ │
│ └─────────────────────┬────────────────────────┘ │
│ │ │
│ ┌─────────────────────▼────────────────────────┐ │
│ │ Auth Methods │ │
│ │ Token │ OIDC │ LDAP │ K8s │ AWS │ Azure │ │
│ └─────────────────────┬────────────────────────┘ │
│ │ │
│ ┌─────────────────────▼────────────────────────┐ │
│ │ Secrets Engines │ │
│ │ KV │ Database │ PKI │ Transit │ AWS │ SSH │ │
│ └─────────────────────┬────────────────────────┘ │
│ │ │
│ ┌─────────────────────▼────────────────────────┐ │
│ │ Storage Backend │ │
│ │ Consul │ Raft │ S3 │ DynamoDB │ etcd │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘Vault Secrets Engines
Common Secrets Engines:
1. KV (Key-Value)
- Static secrets storage
- Versioning support
- Simplest engine
2. Database
- Dynamic database credentials
- Auto-rotation
- Short-lived credentials
vault write database/roles/my-role \
db_name=my-database \
creation_statements="CREATE USER..." \
default_ttl="1h" \
max_ttl="24h"
3. PKI
- Certificate authority
- Issue short-lived certificates
- Auto-renewal
4. Transit
- Encryption as a service
- Key management
- Sign/verify operations
5. AWS/Azure/GCP
- Dynamic cloud credentials
- IAM role assumption
- Temporary accessVault Authentication
Auth Method Selection:
Kubernetes:
- Best for: Pods in Kubernetes
- Identity: Service account token
- Config: Bound namespace/SA
vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes:443" \
[email protected]
AppRole:
- Best for: CI/CD, applications
- Identity: Role ID + Secret ID
- Config: Bound CIDR, metadata
vault auth enable approle
vault write auth/approle/role/my-app \
token_ttl=1h \
token_max_ttl=4h \
secret_id_num_uses=1
OIDC:
- Best for: Human users
- Identity: JWT from IdP
- Config: OIDC provider connectionCloud Secret Managers
AWS Secrets Manager
AWS Secrets Manager Features:
Capabilities:
├── Automatic rotation (Lambda-based)
├── Cross-region replication
├── IAM integration
├── CloudTrail auditing
└── Resource policies
Rotation Flow:
┌─────────────────────────────────────────────────┐
│ 1. createSecret - Create new secret version │
│ 2. setSecret - Update resource (RDS, etc) │
│ 3. testSecret - Verify new secret works │
│ 4. finishSecret - Mark rotation complete │
└─────────────────────────────────────────────────┘
Access Pattern:
aws secretsmanager get-secret-value \
--secret-id my-secret \
--version-stage AWSCURRENT
SDK Integration:
client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='my-secret')
secret = json.loads(response['SecretString'])Azure Key Vault
Azure Key Vault Features:
Object Types:
├── Secrets - Generic key/value
├── Keys - Cryptographic keys (HSM-backed)
└── Certificates - X.509 certificates
Access Control:
- RBAC (recommended)
- Access policies (legacy)
- Managed identities for Azure resources
Soft Delete + Purge Protection:
- Deleted secrets retained for recovery period
- Purge protection prevents permanent deletion
- Required for compliance scenarios
Integration:
// .NET example with DefaultAzureCredential
var client = new SecretClient(
new Uri("https://my-vault.vault.azure.net/"),
new DefaultAzureCredential());
KeyVaultSecret secret = await client.GetSecretAsync("my-secret");GCP Secret Manager
GCP Secret Manager Features:
Capabilities:
├── Automatic replication (regional/global)
├── IAM integration
├── Audit logging
├── Secret versioning
└── Expiration support
Access Control:
- roles/secretmanager.secretAccessor
- roles/secretmanager.admin
- IAM conditions for fine-grained access
Workload Identity:
- GKE pods use service accounts
- No credential files needed
- Automatic token refresh
Access Pattern:
gcloud secrets versions access latest --secret=my-secret
# Python
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
response = client.access_secret_version(name=secret_version_name)
secret = response.payload.data.decode('UTF-8')Secret Rotation
Zero-Downtime Rotation
Dual-Version Strategy:
Phase 1: Create New Version
┌─────────────────────────────────┐
│ Secret Store │
│ ├── Version 1 (current) ✓ │
│ └── Version 2 (pending) NEW │
└─────────────────────────────────┘
Phase 2: Update Consumers
┌─────────────────────────────────┐
│ Apps accept: Version 1 OR 2 │
│ Resource updated to Version 2 │
└─────────────────────────────────┘
Phase 3: Verify New Version
┌─────────────────────────────────┐
│ Test connections with Version 2 │
│ Monitor for errors │
└─────────────────────────────────┘
Phase 4: Deprecate Old Version
┌─────────────────────────────────┐
│ Secret Store │
│ ├── Version 1 (deprecated) │
│ └── Version 2 (current) ✓ │
└─────────────────────────────────┘
Phase 5: Remove Old Version
┌─────────────────────────────────┐
│ Secret Store │
│ └── Version 2 (current) ✓ │
└─────────────────────────────────┘Database Credential Rotation
Database Rotation Pattern:
Approach 1: Dual Users
┌─────────────────────────────────────────────────┐
│ Database has TWO users for the app: │
│ - app_user_a (current) │
│ - app_user_b (standby) │
│ │
│ Rotation: │
│ 1. Generate new password for app_user_b │
│ 2. Update secret to point to app_user_b │
│ 3. Apps pick up new credentials │
│ 4. Change app_user_a password (now standby) │
└─────────────────────────────────────────────────┘
Approach 2: Dynamic Credentials (Vault)
┌─────────────────────────────────────────────────┐
│ Each credential request creates new user: │
│ - v-app-abc123-ttl-1h (expires in 1 hour) │
│ │
│ Benefits: │
│ - No rotation needed │
│ - Automatic cleanup │
│ - Per-request credentials │
│ │
│ Vault handles: │
│ - Creating users │
│ - Setting expiration │
│ - Revoking expired credentials │
└─────────────────────────────────────────────────┘Application Integration
Sidecar Pattern
Sidecar Secret Injection:
┌─────────────────────────────────────────────────┐
│ Pod │
│ ┌─────────────────┐ ┌─────────────────────┐ │
│ │ Vault Agent │ │ App │ │
│ │ (Sidecar) │ │ │ │
│ │ │ │ │ │
│ │ - Authenticates│ │ Reads secrets from │ │
│ │ - Fetches │──► /vault/secrets/ │ │
│ │ - Renders │ │ (shared volume) │ │
│ │ - Refreshes │ │ │ │
│ └─────────────────┘ └─────────────────────┘ │
│ │ │ │
│ ▼ │ │
│ [Shared Volume]◄────────────┘ │
│ /vault/secrets/ │
│ └── db-password │
│ └── api-key │
└─────────────────────────────────────────────────┘CSI Driver Pattern
Secrets Store CSI Driver:
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
volumeMounts:
- name: secrets
mountPath: "/mnt/secrets"
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-database"
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-database
spec:
provider: vault
parameters:
vaultAddress: "https://vault:8200"
roleName: "db-role"
objects: |
- objectName: "db-password"
secretPath: "secret/data/db"
secretKey: "password"Best Practices
Security Best Practices:
1. Never Store Secrets In:
❌ Source code
❌ Environment variables (visible in logs)
❌ Container images
❌ Kubernetes ConfigMaps
❌ Plain text files
2. Access Control:
□ Least privilege access
□ Service-specific credentials
□ Time-limited access where possible
□ Regular access reviews
3. Audit and Monitoring:
□ Log all secret access
□ Alert on unusual patterns
□ Regular audit reviews
□ Compliance reporting
4. Rotation:
□ Automate all rotations
□ Short credential lifetimes
□ Test rotation procedures
□ Document emergency rotation
5. High Availability:
□ Secret store must be HA
□ Cache secrets for availability
□ Graceful degradation plan
□ Recovery procedures documentedRelated Skills
zero-trust-architecture- Overall security architectureapi-security- API authentication and authorizationmtls-service-mesh- Certificate management for servicescontainer-orchestration- Kubernetes secrets integration