Kubernetes on Bare Metal: Why and How
Running Kubernetes on bare metal offers a variety of advantages: It can increase performance, lower costs and more. Although bare metal isn’t always the best deployment approach for Kubernetes, it’s certainly worth considering if you have the infrastructure to support it.
Below, we walk through considerations for running Kubernetes on bare metal, then explain how to plan a bare-metal Kubernetes cluster that delivers the best cost and performance results.
Kubernetes Deployment Options: Bare Metal vs. VMs
Kubernetes on bare metal refers to a cluster whose nodes are bare-metal servers rather than virtual machines.
Most cloud vendors’ managed Kubernetes services support only VM-based infrastructure. Currently, only Elastic Kubernetes Service, or EKS, supports bare-metal nodes. (Google GKE can be extended to bare metal environments in users’ own data centers via Anthos, but GKE can’t run on bare metal in the Google Cloud itself.)
If you deploy Kubernetes on-premises, however, or if you build a self-managed cluster directly on a cloud vendors’ IaaS service rather than using the cloud’s managed Kubernetes service, you can opt to configure your nodes on bare-metal servers instead of VMs. That gives you Kubernetes on bare metal.
(Just in case it’s not clear: obviously, all Kubernetes nodes are ultimately running on bare-metal servers. The difference between a bare-metal node and a VM-based node is that in the case of bare metal, there is no hypervisor separating the node from the underlying server. A VM-based node is running on top of a physical server.)
Advantages of Kubernetes on Bare Metal
There are two main reasons why you’d want to run Kubernetes on bare metal: cost and performance.
Cost
Your total cost of ownership for Kubernetes may be lower when you build a cluster with bare-metal nodes, for several reasons:
- You don’t have to pay for virtualization software.
- You don’t have to pay for both a host OS on a bare-metal server and then another host OS for each VM running on top of it.
- Maintenance and management are simpler because you don’t have to deal with a virtualization layer. That translates to lower labor costs.
- There is no hypervisor overhead, which means that all of your server’s resources can be devoted to running your actual workloads. This results in lower infrastructure costs.
The cost savings achieved by avoiding VMs for Kubernetes can be significant. Ericsson estimates that Kubernetes can cost up to 30 percent more when running on VMs as compared to bare metal, although the company notes that that figure varies depending on your workload type and cluster configuration.
Management
From a management perspective, bare-metal Kubernetes provides more control and can simplify administration in several ways:
- Network configuration: By removing a layer of virtualized infrastructure, bare-metal Kubernetes simplifies networking setup.
- Troubleshooting: The reduced complexity of bare-metal infrastructure also simplifies troubleshooting. It’s easier to pinpoint the source of problems when you don’t have to monitor a layer of virtualized infrastructure on top of the bare metal.
- Automation and deployment: It can be simpler to automate services and deploy software on bare-metal clusters because, again, the virtualized infrastructure layer is not part of the equation.
Performance
From a performance perspective, too, bare-metal Kubernetes can deliver greater benefits than you may think.
CenturyLink found that network latency is three times lower for bare metal Kubernetes. And containers running on bare metal perform 25-30 percent better, according to a Stratoscale study (which benchmarked the performance of standalone Docker containers, not containers running in a Kubernetes cluster).
These differences in performance between bare metal and VM-based environments may be surprising given that, as noted above, hypervisor overhead only amounts to around 10 percent of total infrastructure resources.
However, you have to remember that hypervisor overhead is not the only source of performance hits in a virtualized environment. You’re also “paying” for the resource consumption of guest operating system environments, which will spend some memory and CPU on system processes even when the OS is idle. Noisy-neighbors issues in multi-tenant VM-based environments can impact performance, too: if one node experiences a spike in resource consumption, it could harm the performance of other VMs hosted on the same server. If your VMs are managed by some kind of orchestrator running on the host server, that will consume resources as well.
When you deploy bare-metal servers as part of an edge architecture, you can double-down on performance by taking advantage of both the efficiency of bare metal and the low latency of servers running at the edge.
And perhaps the biggest factor that may impact the performance of VM-based Kubernetes as compared to bare metal Kubernetes is applications’ dependence on access to bare-metal resources. If the applications you are hosting in Kubernetes can benefit from direct access to hardware devices like GPUs, running on bare metal may deliver a very substantial performance benefit.
Finally, it’s worth noting that virtualization adds another layer to your software stack that could cause performance issues if it fails. A hypervisor that crashes and brings down a node will deprive your Kubernetes cluster of the resources provided by that node, which may in turn degrade the performance of applications running in Kubernetes.
Challenges of Bare-Metal Kubernetes
There are two chief drawbacks to bare-metal Kubernetes clusters: management and resiliency against node failure.
Management
Generally speaking, VMs are easier to manage than bare-metal servers, and modern IT staff are more accustomed to using tools for virtualized, cloud-based environments than for bare metal. Using scripts or VM orchestration tools, you can quickly spin up a large number of VMs using prebuilt images. You can also use images to create backups of your VMs and to restore a failed VM. Most virtualization platforms offer snapshotting features, too, that allow you to store the state of a VM from different points in time. They also usually provide automated failover tools that will automatically start a replacement VM if one fails.
Much of this functionality is technically available for bare-metal servers, too, but it’s considerably more complicated to implement. You can create images of a Linux server and use them to provision another server if you want. You could also write a custom Bash script that would automatically start a backup server if your primary server fails to provide some sort of automated failover functionality. However, developing and maintaining tooling like this would require a lot of effort. It’s not out-of-the-box functionality in the way that it is when working with VM platforms.
Bear in mind that some VM platforms are more mature from a management and orchestration perspective than others. Open source platforms like KVM provide fewer out-of-the-box tools for managing and orchestrating VMs than do commercial platforms like VMware and Hyper-V. So, if you use KVM, the difference in the management burden for VMs versus bare metal is not as notable.
Configuration
Beyond the relative simplicity of managing VM images, it’s also worth noting that VMs tend to be more flexible when it comes to configuration. Configuring networking, storage, and other resources may be more complicated if you are dealing with bare metal servers that have unusual interfaces or are not well supported. In contrast, you can rely on mainstream virtualization platforms to be compatible with any type of Kubernetes configuration you need to throw at them. Many also offer the choice of multiple types of virtual network interfaces, which adds even more flexibility.
How to Decide between VMs and Bare Metal
To summarize, the key factors to consider when deciding between Kubernetes on bare metal and Kubernetes on VMs include:
- Cost: If your budget is particularly tight, and/or if your virtualization platform of choice is particularly expensive, bare metal may be best.
- Performance: Do you need to eke out every bit of potential performance from your apps, or can you accept a hit of up to 30 percent if you use VMs?
- Hardware acceleration: If any of your apps require direct access to hardware, bare metal is a no-brainer.
- Management: How able and willing are you to handle the added management burden that comes with bare-metal servers?
- Resiliency: How many node failures can your cluster tolerate? If you can’t handle that many, you may want to choose VMs to spread out your risk.
Best Practices for Kubernetes on Bare Metal
If you opt for bare-metal nodes, the following strategies can help you get the most value out of them:
- Choose smaller nodes: Smaller is generally better when it comes to bare metal node size. For the resiliency reasons described above, it’s preferable to have a larger number of lower-power nodes than to have a smaller number of high-end nodes.
- Choose standardized hardware: To avoid hardware support issues, aim to acquire standard, mainstream servers. Avoid obscure vendors, and stay clear of hardware that is so cutting-edge that it may not be well supported yet.
- Consider the cloud: If you lack the resources to manage bare-metal servers on-premises, you can deploy bare-metal server instances in a public cloud. This approach will reduce much of the management burden because you don’t have to deal with maintaining the physical hardware.
- Keep the OS consistent: Your servers will be easier to manage if each one runs the same version of the same OS.
- Use a bare-metal management tool: A management solution designed for bare metal simplifies operations and reduces risk.
Headache-Free Kubernetes on Bare Metal with Platform9
If the prospect of tediously setting up and managing each of your bare-metal nodes seems daunting, consider Platform9’s managed bare metal offering. Platform9 provides a bare metal controller that manages your servers automatically, as well as a SaaS management plane for administering your environment.
With Platform9’s managed offering for bare metal, you can turn bare-metal servers into a Kubernetes cluster in minutes. Enjoy the cost and performance benefits of bare-metal Kubernetes without the management headache.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023
- Getting to know Nate Conger: A candid conversation - June 12, 2023