Creating, Managing, and Securing Kubernetes Secrets

Introduction

Deploying applications on Kubernetes, a platform widely adopted or under evaluation by most organizations, often involves securely managing secrets. Although Kubernetes provides native mechanisms for handling secrets, these features have inherent limitations. As a result, implementing additional security measures is critical to protect sensitive data within Kubernetes environments.

This blog details everything you need to know about modern Kubernetes secrets management and how to secure them.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Kubernetes Secrets

In Kubernetes, a Secret is a mechanism for storing sensitive data, including login credentials, tokens, and certificates for authentication.

For instance, if you need to configure a container in Kubernetes to pull an image from a private, password-protected container registry, you can create a secret containing the required access credentials.

Similarly, Secrets can store SSH keys to connect to remote servers. Secrets aren’t exclusive to Kubernetes; any software system that handles sensitive information may use them. However, Kubernetes offers its approach to managing secrets, which we will explore below.

Types of Kubernetes Secrets

Kubernetes supports various types of Secrets, including:

Opaque: A versatile, generic type of Secret that can store user-defined data.
SSH-auth: Stores credentials for SSH authentication.
TLS: Holds Secrets required for securing a TLS client or server.
Service Account: Contains tokens for authenticating service accounts, which are non-human accounts used to automate processes within Kubernetes.

Opaque is the most used type of Secret because it stores virtually any data. However, it’s best to select the predefined Secret type that Kubernetes provides when dealing with specific use cases, such as SSH authentication. Doing so minimizes configuration efforts and ensures optimal usage.

How to Use Kubernetes Secrets?

Kubernetes provides the ability to define and manage Secrets using kubectl, the primary command-line tool for interacting with Kubernetes. Below, we’ll outline the fundamental steps for working with Secrets in Kubernetes.

Creating Kubernetes Secrets

There are several methods for creating secrets in Kubernetes. Kubernetes resources can consume these secrets differently, depending on your specific use case and preferences.

Using kubectl

We can create a secret directly from literal key-value pairs using the kubectl create secret command. This method is useful for creating simple secrets containing a small number of key-value pairs.

   kubectl create secret generic reg-pass \
   --from-literal=username=admin \
   --from-literal=password='some-password'

kubectl create secret generic reg-pass \

--from-literal=username=admin \

--from-literal=password='some-password'

kub

2. Using a configuration file

Another way to create a Kubernetes Secret is through a YAML configuration file. First, encode your sensitive data in base64 format, then create a YAML file with the following structure.

kub2

Apply the YAML file using kubectl apply -f my-secret.yaml

3. Using the Kustomize tool

Kustomize is a standalone tool for customizing Kubernetes objects through a kustomization.yaml file. We can create a secret generator with it like this:

kub3

To create the Secret, apply the directory that contains the kustomization file:

kubectl apply -k <directory-path>

Are Kubernetes Secrets Truly Secure?

Kubernetes’ built-in features for managing Secrets are more secure than storing sensitive information in plain text. However, they have limitations and are not entirely foolproof.

Without these features, developers might resort to insecure practices, such as embedding Secrets directly in YAML manifests or storing SSH keys in plain text files on Kubernetes nodes. These approaches expose sensitive information to anyone with access to the files. By storing Secrets in Etcd, Kubernetes helps mitigate this risk by centralizing and protecting sensitive data, reducing the chances of unauthorized access. However, additional security measures are essential for comprehensive protection.

That said, relying on Kubernetes alone to protect your secrets is not ideal for multiple reasons:

Kubernetes does not automatically scan for or warn about insecure Secrets in configuration files or manifests. Administrators are responsible for managing this risk.

By default, anyone with admin-level access to the cluster can decode Secrets.

Users with root access to Kubernetes control-plane nodes may directly access Secrets by viewing Etcd, bypassing Kubernetes entirely.

Sensitive data stored in Secrets can inadvertently leak through shell history files, such as .bash_history, which are typically generated automatically on Linux systems.

In short, while managing secrets using kubectl is better than storing them in plain text, it still leaves the door open to potential attacks.

Kubernetes secrets best practices

To mitigate the risk of insecure secrets management in Kubernetes, admins should adopt strategies for keeping Kubernetes secrets safe from unauthorized access. Best practices include the following.

Scan for Kubernetes secrets

Since Kubernetes doesn’t automatically detect insecure secrets, administrators should deploy a secret scanning solution. This should cover configuration files, container images, application source code, and other relevant resources to identify and mitigate unencrypted or exposed secrets.

Enable Etcd encryption

By default, Kubernetes stores secrets in Etcd in unencrypted form. To change that, you must explicitly enable Etcd encryption for data at rest. This ensures that someone who manages to access Etcd (such as by logging in as root on the node that hosts Etcd) won’t be able to view the secrets unless they have a decryption key.

Use RBAC to protect secrets

Kubernetes’ built-in Role-Based Access Control (RBAC) framework allows administrators to restrict access to secrets by permitting only specific users to perform actions like “get” or “list” secrets. This ensures that not everyone with kubectl access to the cluster can freely view or modify sensitive secrets.

Consider an external secrets manager

Integrating an external secrets manager like AWS Secrets Manager with Kubernetes enhances the security and flexibility of secret management beyond what is offered by native etcd-based storage. AWS Secrets Manager provides secure storage, automatic rotation, AWS IAM-based access control, encryption using AWS KMS, and audit logging through AWS CloudTrail. When combined with tools like the Secrets Store CSI Driver or External Secrets Operator, secrets are securely retrieved at runtime and mounted as volumes inside pods without ever being stored in etcd. This approach ensures secrets are ephemeral, centrally managed, and aligned with best practices for secure cloud-native deployments.

kub4

This architecture illustrates a Kubernetes workload securely retrieving secrets from AWS Secrets Manager at runtime. Using tools like the Secrets Store CSI Driver, secrets are mounted as volumes in pods without being stored in Kubernetes etcd, enhancing security and compliance.

Conclusion

Securing Kubernetes Secrets is crucial to prevent unauthorized access and data breaches. While Kubernetes provides built-in capabilities, they have limitations, making it necessary to implement additional security measures. Best practices like enabling Etcd encryption, using RBAC for access control, deploying secret scanning solutions, and leveraging external secrets managers can significantly enhance security. A proactive approach to managing secrets ensures the overall security and integrity of Kubernetes workloads.

Drop a query if you have any questions regarding Kubernetes Secrets and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

FAQs

1. Why should I encrypt Kubernetes Secrets in Etcd?

ANS: – By default, Etcd stores Secrets in plaintext, making them vulnerable to unauthorized access. Enabling Etcd encryption ensures Secrets are protected at rest, reducing security risks.

2. How does RBAC improve Kubernetes Secrets security?

ANS: – RBAC restricts access to Secrets by allowing only authorized users to retrieve or modify them, preventing unauthorized users from accessing sensitive data.

WRITTEN BY Sidda Sonali

Sidda Sonali is a Research Intern at CloudThat. She is keenly interested in learning advanced technologies and gaining insights into emerging and upcoming cloud services. Sonali actively seeks opportunities to learn about new cloud innovations and best practices.