As we discussed in our Hands-on Introduction to Kubernetes, the shift to microservice architecture led to the need for an orchestration tool that can be used for the containers’ lifecycle management. Since its release in 2015 – Kubernetes quickly became sort of a standard for such orchestration.
Along with its adoption growth, the security concerns around it have also increased according to a Red Hat Survey from May this year. The results of it point out that:
- 93% of the survey respondents have experienced at least one security incident in their Kubernetes environments in the last 12 months, 53% of which due to detected misconfiguration
- 55% of the respondents have delayed or slowed down application deployment due to a security concern
In this article, I will go through some of the best security practices to keep in mind during and after the setup of a Kubernetes cluster.
Kubernetes Security Best Practices
Let’s begin with the hosts that form the cluster itself. All access to these hosts must be secured. For example, you should set up SSH Key authentication and disable root access and password authentication. You can set up a firewall and also any other measures you need to take to secure the infrastructure that hosts Kubernetes. Of course, if that is compromised – everything is compromised.
However, our focus in this blog is more on Kubernetes-related security.
1. Securing the Kubernetes API server
At the center of all operations within a Kubernetes cluster lies the Kube-apiserver. We interact with it through the kube control utility or by accessing the API directly and by doing so we can perform almost any operation on the cluster. So that is the first line of defense – controlling access to the API server itself.
We need to make two types of decisions: Who can access the Kube-apiserver and what can they do?
Your time is limited, so don’t waste it living someone else’s life. Don’t be trapped by dogma – which is living with the results of other people’s
Steve Jobs
2. Authentication
There are different ways to authenticate to the API server. Kubernetes does not manage user accounts natively, so it relies on an external source like a file with user details, certificates, or third-party identity services:
- Using files with user details, like usernames and passwords or tokens – This authentication mechanism is insecure and NOT recommended, since it stores the details in plain text.
- Certificates – TLS encryption is a must for communication with the API server and it is not recommended to skip the usage of certificates in general. Actually, Kubernetes expects that all API communication in the cluster is encrypted by default and as a best practice you should use TLS encryption not only for the API server but for the communication between all different components within the cluster as well.
- External authentication providers – Of course, you can use third-party authentication providers like LDAP or Kerberos for example, which is also considered a security strategy.
- Audit logging – Kubernetes includes a built-in logging system, which allows you to track access to the API server and other Kubernetes resources. This can be useful for detecting and responding to security incidents and identifying misconfigurations or suspicious behavior.
Get Started with Kubernetes?
Figure out your way into the world of container orchestration with Kubernetes in our comprehensive introduction guide for beginners!
3. Authorization
- Node authorization – This is a special-purpose authorization mode that specifically authorizes API requests made by kubelets.
- Attribute-based access control (ABAC) – This authorization mode associates a user or a group of users with a set of permissions. However, every time you need to add or make a change in the security, you must edit the corresponding policy file manually and restart the Kube-apiserver. As such, the attribute-based access control configurations are difficult to manage.
- Role-based access control (RBAC) – This mode provides a more standard approach to managing access within the Kubernetes cluster, by creating a role object via a role definition file and defining the rules for the specified resources and the allowed actions for these resources. The next step is to link a user to that role, with another object called roll binding.
4. Network Policies
By default, Kubernetes is configured with an “all-allow rule” that allows traffic from any pod to any other pod or service within the cluster.
As a best security practice, it is advised to create a network policy for each pod and limit its’ inbound and outbound traffic to the required one only. One of the common approaches is to have a few global policies that apply to all pods, and single-pod-specific policies that define all the ingress and egress rules for the necessary ports of the particular pods.
Another common practice is the definition of default deny network policy, which ensures the traffic is denied in case no other policy is defined specifically for the allowance of this traffic.
5. Service mesh
While the Network Policies operate on Layer 3 and Layer 4, the service mesh can help with the security of Layer 7 – the application layer.
The implementations also differ as the network policies are implemented at the Kernel level while the service mesh is implemented at the user space level, interacting with the network using standard sockets.
The most popular choice as a Kubernetes-native service mesh is the Istio. You might check out our introduction to Istio Traffic Management, as well as our guide for Installing Istio on Amazon EKS.
6. Base images
The base image is the foundation of your containers and applications running on Kubernetes, therefore they should be sourced from a trusted and secure repository. You should avoid using untrusted or unverified base images, as they may contain vulnerabilities or malicious code.
Typically, it is recommended to use private registries instead of public ones, as a place to store the images that have already been approved for usage.
Another recommendation is to always scan your images for security issues, vulnerabilities, or bad practices. Image scanning tools gather Common Vulnerabilities and Exposures (CVEs) information from multiple feeds, which can be integrated into your CI/CD pipeline as well.
A good strategy related to scanning is to always tag your images and avoid using the latest tag.
Another best practice when choosing the right image goes by the saying: “The lighter the image, the better.” With lighter images, you will achieve faster builds and scans in your workflow due to the reduced size of the image and also reduce potential vulnerabilities by using fewer dependencies.
Therefore you should consider using alpine images as base images.
7. Keep Kubernetes up to date
It is well known that keeping your applications up to date is essential for optimizing security and performance. Just like most applications, Kubernetes is constantly evolving with new features and security updates, so you should always keep your infrastructure up to date.
8. Use namespaces and resource limits/quotas
There are multiple benefits of using Kubernetes namespaces. For example:
- It helps isolate different environments, so issues with your staging environment would not affect your production for example.
Also in case, you have a large team of people, the isolation of different projects or microservices by using namespaces would help you to easily manage the access to different objects, which would prevent activity in one of the namespaces to affect any other namespaces.
- Using namespaces optimizes the performance of your cluster because the Kubernetes API server will have a lesser amount of objects to search for when you perform specific operations.
- From a security point of view, by using namespaces you can apply the resource quota objects that can limit the resources used by each of your namespaces which will prevent denial of service attacks or other resource exhaustion attacks affect your whole cluster.
You can also control each pod’s CPU and memory resource usage by using the resource request and resource limit options.
9. Secure etcd and secret data
By default the secrets used for storing the most sensitive data (like credentials, secret tokens, and private keys for example) are unencrypted. They are only base64 encoded, therefore not secured, since anyone who has permission to view the secrets can simply decode the contents of the secrets and see the sensitive data in plain text.
Secrets and all other Kubernetes configuration data are stored in the Etcd key-value store. Every single update gets saved in Etcd and also any change directly in the Etcd leads to a change in the cluster. This means if an attacker can get access to Etcd, they can bypass the API server and therefore get unlimited access to the cluster.
That is why you should consider enabling encryption at rest and also secure the Etcd store behind a firewall and allow only the API server to access it with proper authentication.
How to maintain security in Kubernetes?
With the increased container orchestrator adoption from large enterprises to SMBs, the requirement to protect any infrastructure that runs container workloads has also increased. While Kubernetes offers comprehensive security features and settings, security doesn’t come prebuilt in. These controls and additional measures must be properly implemented to ensure the containers are running safely on the cluster.
Kubernetes security starts with designing a safe architecture. Tools can then monitor the build process along the way and uncover any vulnerabilities as well as maintain continuous visibility.
To make sure your containers are running as safely as possible in your Kubernetes clusters, contact ITGix and our experts in Kubernetes security observability will help you achieve the level of efficiency you look for.