Overview of AWS EKS Security Best Practices

Overview

Kubernetes security in EKS is the responsibility of both AWS and the client. This shared responsibility model divides the security aspect between AWS Security and Client-side Security. Amazon Elastic Kubernetes Service is a managed service we can use to run Kubernetes on AWS Kubernetes control plane or nodes. Kubernetes is an open-source system for automating containerized applications’ deployment, scaling, and management.

Amazon EKS integrates with various AWS services like Amazon Elastic Container Registry, Elastic Load Balancing, IAM, and VPC. It saves the users from the hassle of technical errors related to deployment, scalability, and management. With a focus on utilizing the containerized application’s core functionality, Amazon EKS automatically controls the Kubernetes control panel.

Amazon EKS Workflow

Amazon EKS works by starting and managing the Kubernetes control plane and worker nodes for us. It consists of two major components: a cluster of ‘worker nodes’ running on our containers and the control plane, which manages when and where our containers are getting started in our cluster while monitoring their status.

Without Amazon EKS, we must run both the Kubernetes control plane and the cluster of worker nodes ourselves. With Amazon EKS, we start our worker nodes using a single command in the EKS console. AWS handles provisioning, scaling, and managing the Kubernetes control plane in a highly available and secure configuration.

AWS EKS Security

Security is a shared model between the cloud service provider and the user. AWS is responsible for the security of the cloud, while a user is responsible for security in the cloud.

Managing Authorization, Authentication, and Access Control for AWS EKS

a) Identity and Access Management for Amazon EKS:

We have various strategies to authenticate requests to the kube-apiserver service, e.g., Bearer Tokens, X.509 certificates, OIDC, etc. EKS currently has native support for webhook token authentication and service account tokens.

Recommendations:

1. Avoid the use of service account tokens for authentication
2. Always use least privilege access to AWS Resources
3. Use IAM roles when multiple users need identical access to the cluster
4. Employ least privileged access when creating RoleBindings and ClusterRoleBindings
5. Make EKS Cluster EndPoint Private
6. Create an EKS cluster with a dedicated IAM role
7. Use tools like eksctl to make changes to aws-auth ConfigMap
8. Regular audit access to cluster
9. Alternative Approaches to Authentication and Access Management

b) Pod Security:

Node Authorization: All worker nodes use an authorization technique known as node authorization. It allows nodes to perform the following action:

Read:

1. services
2. endpoints
3. nodes
4. pods
5. secrets, configmap, persistent volumes

Write:

1. node and node status
2. pod and pod status

Auth-related operations:

1. read/write access to the CertificateSigningRequest API
2. Create TokenReview and SubjectAccessReview

Recommendations:

1. Use multiple Pod Security Admission (PSA) modes for a better user experience
2. Restrict the containers that can run as a privilege
3. Do not run processes in containers as root
4. Never run Docker in Docker or mount the socket in the container
5. Restrict the use of hostPath or if hostPath is necessary, restrict which prefixes can be used and configure the volume as read-only
6. Set requests and limits for each container to avoid resource contention and DoS attacks
7. Do not allow privileged escalation
8. Disable service discovery
9. Configure your images with a read-only root file system

c) Infrastructure security in Amazon EKS:

Security of container image is as essential as safeguarding the infrastructure that runs Amazon EKS.

Recommendations:

1. Use an OS optimized for running containers
2. Treat your infrastructure as immutable and automate the replacement of your worker nodes
3. Periodically run kube-bench to verify compliance with CIS benchmarks for Kubernetes
4. Minimize access to worker nodes
5. Deploy workers onto private subnets
6. Run Amazon Inspector to assess hosts for exposure, vulnerabilities, and deviations from best practices.

d) Integrating EKS Cluster with AWS KMS:

1. Use of AWS KMS for encryption of Kubernetes secrets
2. Auditing the use of Kubernetes secrets
3. Rotate secrets periodically
4. Use of separate namespace to isolate secrets from different applications
5. Use of volume mounts instead of environment variables
6. use of the external secret provider
7. Amazon EKS Connector Considerations: The Amazon EKS Connector is an open-source component that runs on our Kubernetes cluster. It creates additional security responsibility.

AWS Responsibility:

• AWS maintains, builds, and delivers Amazon EKS Connector, an open-source component that runs on a customer’s Kubernetes cluster and communicates with AWS.
• Maintaining transport and application layer communication security between the connected Kubernetes cluster and AWS services.

Customer Responsibility:

• Kubernetes cluster-specific security, specifically along the following lines:
  • Secrets must be appropriately encrypted and protected.
  • Lock down access to the eks-connector
• Configuring role-based access control (RBAC) permissions to manage user access from AWS.
• Maintaining the hardware, software, and infrastructure that supports the connected Kubernetes cluster.
• Securing their AWS accounts.

Companies using Amazon EKS are HSBC, GoDaddy, Delivery Hero, SuperAwesome, Mercari, and many more.

AWS Services for Monitoring and Security of Amazon EKS Resources

1. Identity and Access Management for Amazon EKS
2. Logging and monitoring in Amazon EKS
3. Compliance validation for Amazon EKS
4. Resilience in Amazon EKS
5. Infrastructure security in Amazon EKS
6. Configuration and vulnerability analysis in Amazon EKS
7. Pod Security Policy
8. Data Encryption and AWS Secrets Manager secrets with Kubernetes

Logging and Monitoring

Amazon EKS control plane provides audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs in our account. With the help of these logs, we can securely run our cluster. We have options to select the exact log types that we need.

Amazon EKS is integrated with AWS CloudTrail. It captures all API calls for Amazon EKS as events.

1. Compliance Validation for Amazon EKS: Compliance is a shared responsibility between AWS and the consumers of its services. AWS is responsible for the security of the cloud, and users are responsible for security in the cloud.

For example, at Fargate, AWS is responsible for managing the physical security of the data center, hardware, virtual infrastructure (Amazon EC2), and container runtime (Docker). Fargate users are responsible for protecting the container image and its applications.

Compliance Program:

Compliance Status changes over time. 

Resilience in Amazon EKS: Amazon EKS runs and scales the Kubernetes control plane across multiple Availability Zones to ensure high availability. It automatically scales control plane instances based on load, detects, and replaces unhealthy control plane instances, and automatically patches the control plane. Once we initiate a version update, Amazon EKS updates our control plane for us, maintaining the high availability of the control plane during the update.

This control plane consists of at least two API server instances and three etcd instances running in three Availability Zones within an AWS Region.

1. Amazon EKS Configuration and Vulnerability Analysis: Security is essential for configuring and maintaining Kubernetes clusters and applications. The Center for Internet Security Kubernetes Benchmark guides us through the security configuration of Amazon EKS nodes.
2. Applies to Amazon EC2 nodes (both managed and self-managed) where we are responsible for security configurations of Kubernetes components.
3. Provides a standard, community-approved way to ensure that we have configured our Kubernetes cluster and nodes securely when using Amazon EKS.
4. It consists of four sections: control plane logging configuration, node security configurations, policies, and managed services.
5. Supports all the Kubernetes versions currently available in Amazon EKS and can be run using kube-bench, a standard open-source tool for checking configuration using the CIS benchmark on Kubernetes clusters.
6. Data Encryption and AWS Secret Manager secrets with Kubernetes:

We have three different storage options from AWS that we can use with Kubernetes: EBS, EFS, and FSx for Lustre. All three-offer encryption at rest using a service-managed key or a customer master key (CMK).

Encryption Recommendation:

1. Encrypt data at rest
2. Rotate customer master key periodically
3. Use of EFS access point to simplify access to shared datasets

With the help of AWS Secrets and Configuration Provider, we can store and manage our secrets in Secrets Manager and then retrieve them through our workloads running on Amazon EKS. We can use IAM roles and policies to limit access to our secrets to specific Kubernetes pods in a cluster. The ASCP retrieves the pod identity and exchanges the identity for an IAM role. ASCP assumes the IAM role of the pod, and then it can retrieve secrets from Secrets Manager that are authorized for that role.

Amazon EKS Use Cases

1. Hybrid Deployment: With the help of Elastic Kubernetes Service, we can manage Kubernetes applications and clusters across hybrid environments. We can run Kubernetes on AWS and in our data centers.
2. Batch Processing: We can run parallel or sequential batch workloads on EKS clusters. It allows us to plan, schedule, and run batch computing workloads over the entire range of AWS compute services using the EKS. These services and features include Fargate, Amazon EC2, and Spot Instances.
3. Machine Learning: AWS deep learning containers use Kubeflow to model the machine learning workflows and effectively execute distributed training jobs with the latest GPU-powered instances.
4. Web Applications: With the help of EKS, we can develop applications that run in a highly available configuration over multiple Availability Zones and automatically scale in and out based on requirements. Increases the performance, reliability, scale, and availability of AWS. It integrates with AWS security and networking services (VPC) and Load Balancers for load distribution of web applications.

Conclusion

With the rise of cloud computing, Amazon EKS is an essential tool for running Kubernetes container applications. It is a powerful resource that can help us to automate Kubernetes resources. However, automation doesn’t provide better security. We need to implement security manually. We can deploy worker nodes in private subnets, use AWS KMS to manage secrets, continuous monitoring, encryption of data, etc.

About CloudThat