Hardcoding database passwords in configuration files or CI/CD variables creates a massive security debt. If a developer leaves the company or a build server is compromised, those static credentials remain valid until someone manually changes them—a task that often breaks applications. Static secrets are the primary fuel for lateral movement during a security breach.
You can solve this by implementing HashiCorp Vault Dynamic Secrets. Instead of storing a pre-existing password, Vault connects to your database and generates a unique, short-lived set of credentials on the fly for every requesting application. When the application is done or the lease expires, Vault automatically deletes the user from the database. This tutorial covers the implementation of dynamic secrets for HashiCorp Vault v1.15+ to achieve a zero-trust security posture.
TL;DR — Use the Vault Database Secrets Engine to generate ephemeral SQL users with time-to-live (TTL) bounds. This eliminates the risk of stale credential leaks and automates the entire rotation lifecycle.
Understanding the Dynamic Secrets Concept
In a traditional setup, your application retrieves a username and password from an environment variable. These credentials often have "admin" or "owner" level permissions and no expiration date. If your logs accidentally leak these strings, your database is exposed indefinitely. Because rotating these passwords manually causes downtime, most teams simply never do it.
HashiCorp Vault changes the flow. Your application authenticates with Vault (using an IAM role, Kubernetes Service Account, or AppRole) and asks for database access. Vault, acting as a privileged broker, executes a CREATE USER command on the database, assigns specific grants, and returns the new username and password to the app. This credential is tied to a Lease. When the lease expires, Vault executes a DROP USER command, ensuring the credential has a finite lifespan.
When to Transition to Dynamic Rotation
You should adopt dynamic database credentials if you operate in a distributed microservices environment where tracking "who has access to what" becomes impossible. It is a mandatory requirement for compliance frameworks like SOC2, PCI-DSS, and HIPAA, which demand regular rotation of sensitive keys and audit logs of every access request.
Another critical scenario is ephemeral infrastructure. If you are running jobs on Kubernetes or AWS Lambda, those processes only exist for minutes. Giving an ephemeral process a permanent password is a security mismatch. Dynamic secrets align the credential's lifecycle with the compute lifecycle. If your application scales horizontally, each instance gets its own unique identity in the database logs, making it much easier to debug which specific pod is running a slow query or locking a table.
Step-by-Step Implementation Guide
This guide uses PostgreSQL as the target database, but the logic applies to MySQL, MongoDB, MSSQL, and Oracle. Ensure you have a running Vault server and administrative access to your database.
Step 1: Enable the Database Secrets Engine
First, you must enable the engine path in Vault. This is a one-time setup per environment.
# Enable the database engine at a specific path
vault secrets enable -path=db-prod databaseStep 2: Configure Vault Connection to the Database
Vault needs a "root" credential to manage other users. This root user should have the permissions to create and drop roles. In this example, we use the postgresql-database-plugin.
vault write db-prod/config/my-postgres-db \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly-app" \
connection_url="postgresql://{{username}}:{{password}}@localhost:5432/postgres?sslmode=disable" \
username="vault_admin" \
password="supersecretpassword"Step 3: Create a Dynamic Role
This is where you define the SQL template Vault will execute. Use the {{name}}, {{password}}, and {{expiration}} tokens which Vault populates automatically.
vault write db-prod/roles/readonly-app \
db_name=my-postgres-db \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"Step 4: Request Credentials
Now, any authorized entity can request credentials. Notice that every call returns a different set of credentials.
# Fetch the dynamic credentials
vault read db-prod/creds/readonly-app
# Example Output:
# username: v-token-readonly-app-7b8x...
# password: hvs.mt9...
# lease_id: db-prod/creds/readonly-app/7b8x...
# lease_duration: 1hCommon Pitfalls and How to Fix Them
To fix connection pool issues, ensure your application logic handles Lease Renewal. If you are using the Vault SDK, you should periodically call the renew endpoint for your lease_id. Alternatively, use Vault Agent as a sidecar. The Agent handles the renewal and writes the fresh credentials to a shared file, which your application can watch for changes.
Another issue is "Lease Churn." If you have thousands of microservices requesting new credentials every minute, your database's system catalog (e.g., pg_authid in Postgres) will bloat with metadata from deleted users. Always set a reasonable default_ttl (e.g., 1 hour to 12 hours) to balance security with database performance. Monitor your database's internal tables to ensure dropped users are being vacuumed correctly.
Expert Tips for Production Hardening
When running this in production, never use the "postgres" or "sa" superuser as the Vault root credential. Instead, create a dedicated vault_manager user with limited CREATEROLE permissions. This limits the "blast radius" if the Vault server itself is compromised. You can further restrict this user's network access to only allow connections from the Vault cluster IP addresses.
Use Vault Sentinel (for Enterprise) or standard ACL policies to enforce TTL limits. You don't want a developer accidentally creating a role with a 10-year TTL, which effectively turns a dynamic secret back into a static one. Additionally, always enable Vault's audit logging to a secure, external destination like Splunk or AWS CloudWatch. This allows you to trace every database action back to a specific Vault token and, by extension, a specific human or service identity.
📌 Key Takeaways
- Dynamic secrets provide unique, time-bound credentials for every request.
- Lease durations (TTL) must be synchronized with application connection pool settings.
- Vault requires a management user in the database to automate
CREATE/DROPactions. - Centralizing secrets in Vault provides a single audit trail for compliance.
Frequently Asked Questions
Q. How does Vault rotate the actual root database password?
A. Vault can manage its own configuration password using the /database/rotate-root/:name endpoint. When triggered, Vault generates a new random password, updates the database, and updates its internal configuration. This ensures even the master management password is rotated regularly without human intervention.
Q. What happens if the Vault server goes down?
A. Existing leases remain valid until their TTL expires, so your applications won't immediately disconnect. However, new pods won't be able to start, and existing ones cannot renew leases. For high availability, always run Vault in a clustered mode with at least three nodes across different availability zones.
Q. Does Vault support NoSQL databases like MongoDB?
A. Yes, Vault supports a wide array of databases including MongoDB, Cassandra, and Couchbase. The implementation is similar: you enable the engine, provide a management connection, and define a role that specifies the JSON-based creation and revocation logic required by that specific NoSQL engine.
Post a Comment