Where to Install Kubernetes? Bare-Metal vs. VMs. vs. Cloud
One of the things that makes Kubernetes so powerful is that it can run virtually anywhere: On bare metal servers and virtual machines, on-premises or in the public cloud. (These are overlapping categories to an extent, because an on-premises cluster could be hosted using bare-metal servers or virtual machines.)
Yet the fact that Kubernetes can run in all of these places doesn’t mean that they’re completely interchangeable. The type of infrastructure you host Kubernetes on, and the exact way you host it (for example, using a managed Kubernetes service in the cloud instead of installing it directly on cloud-based virtual machines) has an important impact on performance, scalability and integrations.
This article discusses those factors and provides guidance for deciding where to host your Kubernetes deployments.
Bare-metal Kubernetes
Building a Kubernetes cluster out of bare-metal servers may not seem like the most obvious thing to do. It requires substantial work to provision each server before you can install Kubelet and the rest of your Kubernetes stack on them. Thus, this approach is probably the most labor-intensive method of creating a Kubernetes host infrastructure.
On the other hand, bare-metal Kubernetes offers the benefit of near bare-metal performance. I say “near” because there is some resource overhead associated even with containers; in other words, some system resources are consumed by running Kubernetes, your container runtime, your storage layer and whatever else you use to build out your stack.
Still, bare-metal Kubernetes nodes have much less overhead compared to nodes that are virtualized using hypervisors, which all other methods of installing Kubernetes are (with the exception of hosting Kubernetes on dedicated bare-metal servers in the cloud, which you could do, but it would be pretty expensive and probably not worth it).
Virtual machines
If you’re setting up Kubernetes on-premises, using virtual machines is the more efficient way to go about it. You can use imaging tools to provision the servers with virtual machine images quickly and easily, then install Kubernetes on them.
Virtual machines also give you the flexibility to choose to host one or multiple nodes on a single physical server. With bare-metal servers, you would only be able to host one node per server. More nodes on a single server could increase overall cluster availability by adding more nodes to your cluster, even if you have relatively few physical servers. (The caveat, of course, is that failure of a single bare-metal server could bring down multiple nodes, so this setup isn’t exactly what you could call high-availability Kubernetes.)
The major drawback of virtual machines is performance. You waste system resources running a hypervisor for each virtual machine and, therefore, have fewer resources available to devote to your actual cluster. Still, I think most admins would see this as the recommended approach in most situations.
On-premises Kubernetes vs. the cloud
Another common choice to make when planning a Kubernetes deployment is whether to host on-premises or in the cloud. This is a more complicated decision than it may appear, because there are multiple variables involved.
As noted above, on-premises infrastructure could involve bare-metal servers or virtual machines. (So could the cloud, technically, although it’s rare to use a bare-metal server in the cloud.) Plus, there are multiple ways to set up and run Kubernetes in the cloud. You could install it using a tool like kOps, or an installer tied to a specific distribution, like OpenShift. Or, you could avoid that hassle altogether by using a managed Kubernetes server, where the cluster is set up automatically for you.
Without picking through all of those nuances, suffice it to say that the general lessons about on-premises vs. cloud apply when it comes to thinking about where to run Kubernetes. On-premises, you get more control; but there is also more effort required to set up your cluster, and the cluster is less scalable (because the number of nodes you can use is limited by the size of your on-premises infrastructure). In the cloud, setup is generally easier and faster (with managed Kubernetes, it’s as simple as running a few commands), you have virtually unlimited scalability, but your installation and configuration options are limited to those supported by your cloud vendor or managed Kubernetes service.
It’s worth noting, too, that it’s easier to set up a high-availability cluster in the cloud. You can set up multiple master nodes and host them in different availability zones in order to build a highly fault-tolerant cluster.
Conclusion
There’s no wrong way to set up Kubernetes infrastructure. The approach you take should reflect your priorities and preferences. If you want a turnkey Kubernetes service, a managed, cloud-based option will get you closest to that. If, on the other hand, you don’t mind the trouble of setting up a cluster yourself, on-premises Kubernetes using bare-metal servers will give you the best bang for your buck performance-wise (with the tradeoff of less scalability). Other approaches fall somewhere between these two extremes.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023
- Getting to know Nate Conger: A candid conversation - June 12, 2023