Database Credential Rotation
What you'll learn: How to rotate database passwords seamlessly without breaking active connections or causing downtime.
The Challenge
Database credentials are among the most sensitive secrets in your infrastructure. Unlike API keys that applications fetch on-demand, database connections are often long-lived and pooled for performance. Simply changing a password would break every active connection, causing outages. You need a rotation strategy that keeps the database available while credentials change.
Core Technique: Dual-User Pattern
The dual-user pattern is the standard approach for zero-downtime database rotation:
- Create a second database user with identical permissions to the current user
- Update your application configuration to use the new credentials (stored in your secret store)
- Deploy the configuration change – new connections use the new user, old connections continue with the old user
- Wait for old connections to drain from the connection pool (typically minutes)
- Revoke or delete the old user once no active connections remain
Think of it like changing guards at a gate: the new guard arrives before the old one leaves, ensuring continuous coverage.
Connection Pool Management
Connection pools complicate rotation because they hold connections open. Key considerations:
- Max lifetime settings: Configure pools to recycle connections periodically (e.g., every 30 minutes)
- Graceful shutdown: Signal pools to close connections cleanly during rotation
- Health checks: Monitor for authentication failures that indicate incomplete rotation
- Rollback capability: Keep old credentials accessible briefly in case rollback is needed
Some systems support dual-credential pools where the application tries the new credential first, falling back to the old one during the transition window.
Key Takeaway: Database credential rotation requires dual-user patterns and careful connection pool management to avoid service interruption during password changes.