Platform9 Blog

Kubernetes Multi-Tenancy Best Practices

In a simpler world, all Kubernetes deployments would host just a single tenant — meaning only one workload or application runs on the entire Kubernetes environment.

In the real world, it’s common to have multi-tenant Kubernetes deployments. In a multi-tenant deployment, multiple workloads share the same underlying Kubernetes infrastructure.

Multi-tenant Kubernetes creates some special challenges when it comes to resource sharing and security. Let’s walk through what those problems are and some strategies for addressing them.

What is multi-tenant Kubernetes?

Multi-tenant Kubernetes is a Kubernetes deployment where multiple applications or workloads run side-by-side.

Multi-tenancy is a common architecture for organizations that have multiple applications running in the same environment, or where different teams (like developers and IT Ops) share the same Kubernetes environment.

You can think of multi-tenant Kubernetes as being akin to an apartment building, whereas single-tenant Kubernetes is like a single-family house.

Challenges of Kubernetes multi-tenancy

We can take that metaphor further to help explain the challenges of multi-tenant Kubernetes environments.

In an apartment building, you have to ensure that each apartment is sufficiently isolated from the others in order to give each tenant appropriate privacy. You can’t have one tenant walking through another tenant’s apartment to get to the bathroom, for example.

Similarly, when it comes to resource sharing, you must ensure that each tenant has access to the resources they need. Your building won’t function very well if one apartment’s water shuts off when another tenant is taking a shower. Of course, none of these things would be an issue in a single-family home, where there is no sharing of resources.

Multi-tenancy Kubnernetes poses similar challenges. Each workload must be isolated so that a security vulnerability or breach in one workload doesn’t spill over into another. Likewise, each workload must have fair access to the compute, networking, and other resources provided by Kubernetes. (I say “fair access” because you might not necessarily want each workload to have the same level of access as others, but you do want each workload to be able to access what it needs to do its job.)

Solving Kubernetes multi-tenancy challenges

How can you address these challenges and ensure that each Kubernetes workload has the proper level of isolation and resource availability? The following are some best practices:

Namespace isolation

A basic best practice for handling multiple tenants is to assign each tenant a separate namespace. Kubernetes was designed for this approach. Most of the isolation features that it provides expect you to have a separate namespace for each entity that you want to isolate.

Keep in mind, too, that in some cases it may be desirable to assign multiple namespaces to the same group within your Kubernetes deployment. For example, the same team of developers might need multiple namespaces for hosting different builds of their application.

Adding namespaces is relatively easy (it takes just a simple kubectl create namespace your-namespace command), and it’s always better to have the ability to separate workloads in a granular way using namespaces than to try to cram different workloads with different needs into the same namespace.

Block traffic between namespaces

By default, most Kubernetes deployments allow network communication between namespaces. If you need to support multiple tenants, you’ll want to change this in order to add isolation to each namespace.

You can do this using Network Policies. Here’s an example Network Policy file that will block traffic from external namespaces:



kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: block-external-namepsace-traffic
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}

After creating the file, apply it with a command like:



kubectl apply -f networkpolicy.yaml -n=your-namespace

Resource Quotas

When you want to ensure that all Kubernetes tenants have fair access to the resources that they need, Resource Quotas are the solution to use. As the name of this feature implies, it lets you set quotas on how much CPU, storage, memory, and other resources can be consumed by all pods within a namespace.

For example, consider this Resource Quota file:



apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
spec:
  hard:
    requests.cpu: "1"
    limits.cpu: "2"

It can be applied with a command like:



kubectl apply -f resourcequota.yaml  -n=your-namespace

When applied, the policy will limit all containers within a given namespace to 1 CPU request and 2 total CPUs.

Although you could enforce the same resource quota for all namespaces, you are also totally free to set different quotas for different namespaces. Thus, you can give some tenants access to more resources if desired. Doing so could be helpful for ensuring that production workloads have more resources than tenants that exist only for testing, for example, or if you want to allow some users to pay more for access to additional resources.

Secure your nodes

A final multi-tenancy best practice to keep in mind is the importance of making sure that your master and worker nodes are secure at the level of the host operating system.

Node security doesn’t reinforce namespace isolation in a direct way; however, since an attacker who is able to compromise a node on the operating system level can potentially use that breach to take control of any workloads that depend on the node, node security is important to keep in mind. (It would be important in a single-tenant environment too, but it’s even more important when you have multiple workloads, which makes the security stakes higher.)

Conclusion

Ensuring the security and reliability of all of the tenants in a multi-tenant Kubernetes deployment takes some work, but Kubernetes provides several features to help you do it. By leveraging tools like namespaces, Network Policies, and Resource Quotas, you can achieve the right level of isolation and resource sharing whether your Kubernetes deployment hosts a handful of applications or several thousand.

Platform9

Platform9 delivers a SaaS-managed hybrid cloud solution that turns existing infrastructure into a cloud, instantly. We help enterprises drive digital transformation by enabling them to manage VMs, Containers and Serverless Functions on ANY infrastructure — on-premises, in public clouds, or at the edge – with a self-service, simple and unified experience. Customers such as Cadence, Autodesk, Splunk, EBSCO, Bitly, LogMeIn, and Aruba see upwards of 300 percent improvement in IT efficiency, 33 percent faster time to market, and 50-80 percent improvement in data center utilization and cost reduction. The company is headquartered in Sunnyvale, CA, and is backed by Redpoint Ventures, Menlo Ventures, Canvas Ventures, and HPE.
Platform9

You may also enjoy

Kubernetes in the Enterprise – Chapter 7: What You Need to Know About Platform9 Managed Kubernetes

By Vamsi Chemitiganti

[Video] KubeVirt – Beyond Containers: Coming full circle back to VMs!

By Josh Hurt

The browser you are using is outdated. For the best experience please download or update your browser to one of the following:

Enterprise Kubernetes Guide

Kubernetes

Gorilla Guide to Enterprise Kubernetes

Download Now

You have Successfully Subscribed!