Kubernetes Multi-Tenancy Best Practices
In a simpler world, all Kubernetes deployments would host just a single tenant — meaning only one workload or application runs on the entire Kubernetes environment.
In the real world, it’s common to have multi-tenant Kubernetes deployments. In a multi-tenant deployment, multiple workloads share the same underlying Kubernetes infrastructure.
Multi-tenant Kubernetes creates some special challenges when it comes to resource sharing and security. Let’s walk through what those problems are and some strategies for addressing them.
What is multi-tenant Kubernetes?
Multi-tenant Kubernetes is a Kubernetes deployment where multiple applications or workloads run side-by-side.
Multi-tenancy is a common architecture for organizations that have multiple applications running in the same environment, or where different teams (like developers and IT Ops) share the same Kubernetes environment.
You can think of multi-tenant Kubernetes as being akin to an apartment building, whereas single-tenant Kubernetes is like a single-family house.
Challenges of Kubernetes multi-tenancy
We can take that metaphor further to help explain the challenges of multi-tenant Kubernetes environments.
In an apartment building, you have to ensure that each apartment is sufficiently isolated from the others in order to give each tenant appropriate privacy. You can’t have one tenant walking through another tenant’s apartment to get to the bathroom, for example.
Similarly, when it comes to resource sharing, you must ensure that each tenant has access to the resources they need. Your building won’t function very well if one apartment’s water shuts off when another tenant is taking a shower. Of course, none of these things would be an issue in a single-family home, where there is no sharing of resources.
Multi-tenancy Kubernetes poses similar challenges. Each workload must be isolated so that a security vulnerability or breach in one workload doesn’t spill over into another. Likewise, each workload must have fair access to the compute, networking, and other resources provided by Kubernetes. (I say “fair access” because you might not necessarily want each workload to have the same level of access as others, but you do want each workload to be able to access what it needs to do its job.)
Solving Kubernetes multi-tenancy challenges
How can you address these challenges and ensure that each Kubernetes workload has the proper level of isolation and resource availability? The following are some best practices:
Namespace isolation
A basic best practice for handling multiple tenants is to assign each tenant a separate namespace. Kubernetes was designed for this approach. Most of the isolation features that it provides expect you to have a separate namespace for each entity that you want to isolate.
Keep in mind, too, that in some cases it may be desirable to assign multiple namespaces to the same group within your Kubernetes deployment. For example, the same team of developers might need multiple namespaces for hosting different builds of their application.
Adding namespaces is relatively easy (it takes just a simple kubectl create namespace your-namespace command), and it’s always better to have the ability to separate workloads in a granular way using namespaces than to try to cram different workloads with different needs into the same namespace.
Block traffic between namespaces
By default, most Kubernetes deployments allow network communication between namespaces. If you need to support multiple tenants, you’ll want to change this in order to add isolation to each namespace.
You can do this using Network Policies. Here’s an example Network Policy file that will block traffic from external namespaces:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: block-external-namepsace-traffic
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
After creating the file, apply it with a command like:
kubectl apply -f networkpolicy.yaml -n=your-namespace
Resource Quotas
When you want to ensure that all Kubernetes tenants have fair access to the resources that they need, Resource Quotas are the solution to use. As the name of this feature implies, it lets you set quotas on how much CPU, storage, memory, and other resources can be consumed by all pods within a namespace.
For example, consider this Resource Quota file:
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu: "1"
limits.cpu: "2"
It can be applied with a command like:
kubectl apply -f resourcequota.yaml -n=your-namespace
When applied, the policy will limit all containers within a given namespace to 1 CPU request and 2 total CPUs.
Although you could enforce the same resource quota for all namespaces, you are also totally free to set different quotas for different namespaces. Thus, you can give some tenants access to more resources if desired. Doing so could be helpful for ensuring that production workloads have more resources than tenants that exist only for testing, for example, or if you want to allow some users to pay more for access to additional resources.
Secure your nodes
A final multi-tenancy best practice to keep in mind is the importance of making sure that your master and worker nodes are secure at the level of the host operating system.
Node security doesn’t reinforce namespace isolation in a direct way; however, since an attacker who is able to compromise a node on the operating system level can potentially use that breach to take control of any workloads that depend on the node, node security is important to keep in mind. (It would be important in a single-tenant environment too, but it’s even more important when you have multiple workloads, which makes the security stakes higher.)
Multi-tenancy – all the way
An important aspect of multi-tenancy is having multi-tenancy at a layer above kubernetes cluster – so that your DevOps and developers can have one or more clusters belonging to different users or teams of users within your organization. This concept isn’t built into Kubernetes itself. Platform9 supports this by adding a layer of multi-tenancy on top of Kubernetes via the concept of ‘regions’ and ‘tenants’. A region in Platform9 maps to a geographical location. A tenant can belong to multiple regions. A group of users can be given access to one or more tenants. Once in a tenant, the group of users can create one or more clusters, that will be isolated and accessible only to the users within that tenant. This provides separation of concerns across different teams and departments.
Recommended Readings
Conclusion
Ensuring the security and reliability of all of the tenants in a multi-tenant Kubernetes deployment takes some work, but Kubernetes provides several features to help you do it. By leveraging tools like namespaces, Network Policies, and Resource Quotas, you can achieve the right level of isolation and resource sharing whether your Kubernetes deployment hosts a handful of applications or several thousand.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- Platform9 Introduces Elastic Machine Pool (EMP) at KubeCon 2023 to Optimize Costs on AWS EKS - November 15, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023