Manage secrets with Vault inside a Kubernetes cluster
Typically usernames and passwords to resources are statically tied to a service account. These passwords rarely change and are usually difficult to rotate in an application stack. Sometimes, we're not even sure how many components are utilizing that service account which makes rotate even more difficult and teams end up not changing due to fear of downtime and errors.
Ideally we want a solution which allows us to rotate credentials dynamically and do so in a secure well-thought out way.
The main motivation of this project is to allow dynamic secrets to be requested from a MySQL database and enable a pod inside a Kubernetes cluster to consume those dynamic passwords. The secrets should be tied to a lease so they expire after a pre-defined ttl and the secrets should be rotated before a max ttl is met.
The implementation should be done so that the pod does not have to understand a specific secret generation tool (e.g. Hashicorp Vault). The application only needs to understand how to read from a file as well as get notified when that file changes.
This project uses Vault as it's secret distibution tool with the MySQL Secret Backend enabled. It's deployed via a custom ThirdPartyResource
and kubernetes controller which implements the Vault API. Credentials are exposed to pods via simple Kubernetes secrets. The application in the pod is only responsible for refreshing it's application state when those credentials are rotated.
kubectl create -f deployments/mysql.yaml
)kubectl create -f deployments/vault.yaml
)kubectl exec -it <vaultPodName> /bin/dumb-init /bin/sh
)
setup-vault.sh
args
section in deployment yaml (kubectl logs <vaultPodName>
)kubectl create -f deployments/secret-manager.yaml
kubectl create -f sample-app/deployments/sample-app.yaml
)
db-readonly-credentials
, db-full-credentials
, && foo-secret
Special thanks goes out to Kelsey Hightower for the base ideas of this project: (https://github.com/kelseyhightower/kube-cert-manager)
Built by UPMC Enterprises in Pittsburgh, PA. http://enterprises.upmc.com/