Kubernetes Secrets Management

Like many computing systems, your Kubernetes environment requires the use of sensitive information. Sensitive values (such as passwords, SSH keys, and security tokens) are referred to collectively as secrets. In this article, we will explain how to store and manage the secrets in your Kubernetes environment, and we will examine the capabilities of Kubernetes and other secrets management systems. In addition, we will explore use cases where you might prefer one approach over another and also introduce some of the best practices that you should implement within your organization.

An Introduction to Kubernetes Secrets

Kubernetes provides users with the ability to create secrets objects and provide them securely to pods within the cluster. Let’s consider a scenario where we need to provide access credentials for a data store to pods within our cluster. We’ll need to create and store the secrets within the cluster, and then make them available to the applicable pods.

You can create secrets manually or use the kubectl create secret command to create them. The create secret command can create secrets from string literals, or files containing the values. Using the file approach eliminates the need to escape special characters from the operating system. If we have our database credentials loaded into files named database_user.txt and database_password.txt, then we could load them into the cluster’s secret store with the following command.


$ kubectl create secret generic database-credentials --from-file=./database_user.txt --from-file=./database_password.txt

This command creates a secret named database-credentials in the secret store. The system should respond with a secret “database-credentials” created response.

Once you have created the secret object, you can add it to the container as volume attached to the pod, or you can load the values into environment variables when initializing a new container in the pod. Let’s look at an example configuration that includes the secrets as a volume on the pod:


apiVersion: v1
kind: Pod
metadata:
  name: apppod
spec:
  containers:
  - name: apppod
    image: application
  volumes:
  - name: secrets
    secret:
        secretName: database-credentials

Assuming that you used the original file names to load the database credentials into the secrets object and attached the secrets into a volume named secrets, the credentials could now be accessed at /etc/secrets/database_user and /etc/secrets/database_password.

Essential Considerations With Kubernetes Secrets

While using a Kubernetes secret object is more secure than including credentials within the image, there are some critical characteristics of secrets that you should keep in mind. From a positive perspective, the use of a secret object simplifies administration if credentials need to be updated. An updated secret is available to all of the containers using it as soon as it is updated.

Additionally, because secrets are created independently of the pods, they are less likely to be exposed during the process of creating and configuring the pods. Administrators can also enable encryption at rest if the pod is running on a recent version of Kubernetes in order to protect the data further.

But there are also some potential pitfalls that you should be aware of when using a secret object. Below, I’ll list some of the primary concerns and risks that you should keep in mind when deciding whether to use Kubernetes Secrets to hold your sensitive information. This is by no means an exhaustive list, and you should consult the official Kubernetes documentation on Secrets for a full and current list of potential risks and best practices.

First of all, secrets in a cluster may be available to all users of the cluster, so be careful about uploading personal credentials if you’re working in a shared environment. Secondly, if you load the secret into environmental variables, you should ensure that these logging statements don’t include the variables, as this exposes the secret in plain text in the logs. Finally, when creating secrets, you might use base64 encoded data or input files with the plaintext values. You should ensure that you delete these resources following the creation of the secrets so that they don’t inadvertently end up in source control.

Using A Non-Native Solution for Secrets Management

Kubernetes Secrets is an adequate solution for most secrets. Still, there are more robust security systems available from providers with a long pedigree of secrets management and focused investment in their solutions. The integration of one of these systems might be overkill for a small application footprint. Still, for larger organizations as well as organizations that support a hybrid of Kubernetes and other platforms, they may provide a more secure option.

Solutions like Vault from HashiCorp and CyberArk offer highly secure systems for secrets management and provide integrations with Kubernetes as well as all of the primary cloud providers. These systems usually include secrets management strategies, auditing, and role-based access control, and they might even integrate seamlessly with your corporate data center.

HashiCorp recently announced additional features that will allow Vault to integrate easily with a Kubernetes environment. CyberArk offers similar features with its flagship secrets management product, and they also offer an open-source product, Conjur Open Source, which is container-native and can integrate effortlessly with your Kubernetes environment.
In addition to these options, there are other offerings from other providers of security management systems. Before deciding on the right solution for your organization, you should be sure to examine the available options carefully so that you can select the best solution for your security needs as well as your budget.

Learning More

If you would like to learn more about Kubernetes Secrets, the official Kubernetes documentation provides a thorough look at Secrets and how to implement them. Similar information is also available in the documentation on how to Distribute Credentials Securely Using Secrets.

To learn more about vendors that provide external secret repository solutions, click on any of the links below.

HashiCorp Vault
CyberArk
Conjur Open Source

Author
Recent Posts

Platform9

Platform9 is the only container management and orchestration solution that supports the complete lifecycle of the cloud-native journey on infrastructure anywhere. Our unique management plane simplifies operations, including observability & monitoring, increases uptime, minimizes costs, and scales as you scale. Built on open foundations, our platform enables enterprises to keep up with the ever-changing cloud-native ecosystem. Our cloud-native wizards provide Always-on Assurance™ every step of the way to ensure a smooth, simplified journey. Innovative enterprises like Juniper, Kingfisher Plc, Mavenir, Redfin, and Cloudera achieve 4x faster time-to-market, up to 90% reduction in operational costs, and 99.99% uptime.Platform9 is a better way to go cloud native, paving the way for an open distributed cloud