The Seamless Upgrade for Kubernetes’ First Major Security Hole, CVE-2018-1002105
CVE-2018-1002105, a critical Kubernetes security vulnerability affecting multiple versions of the popular container orchestration system was recently discovered and fixed by the Kubernetes open source community.
What is the CVE-2018-1002105 Security Bug?
In summary, when exploited the vulnerability would allow a lesser privileged user to perform escalated privilege operations against Kubernetes. This could be misused in many ways, for example, to retrieve pod details like secrets or perform destructive operations within a pod.
The Kubernetes API service acts as a proxy to certain other Kubernetes services like kubelet, metrics-server, and others. In order to enable this, the API service performs authentication, authorization, and routing. The Kubernetes implementation prior to the security patch would enable clients to reuse proxy connections to reach backend services other than the one the initial connection was established for.
A client can open a TCP connection to the API server, complete authentication, authorization and reach a backend service behind the Kubernetes API. Once the connection is established to the backend service, it is opaque to the Kubernetes API server. A client could potentially keep this connection open, even after its intended usage, and craft a request to reach a different backend service. Given that the Kubernetes API server already had a permitted but opaque session, the client could access the different backend service even when if they did not have the privileges to do so.
For example, a self-service user with privileges to a Kubernetes namespace say “dev”, could connect to a service in that namespace via the Kubernetes API server, and they could use the established connection to access kubelet, or other services in a different namespace (elevating their privilege).
For more on the CVE-2018-1002105 bug see:
What do you need to do?
This is a critical bug. All Kubernetes users must upgrade NOW. The community has released patched version of Kubernetes v1.10.11, v1.11.5, v1.12.3, and v1.13.0-rc.1. While it is a painful process – it’s absolutely necessary.
Bite the bullet and upgrade. Now.
We know it’s a pain — having to go over all of your clusters across your environments to patch them with the fix. You need to get the latest version, validate that there are no upgrade issues or regressions, then apply it to each cluster. If you don’t have support for automated rolling updates across all nodes, you’d also need to prepare for cluster downtime.
Yes – it is a pain. But it doesn’t have to be.
Enter a SaaS-managed approach to Kubernetes Solutions
In the case of security vulnerabilities like this, one of the key factors is the time to roll out the fix. The longer it takes, the higher the chance of your Kubernetes cluster getting compromised.
Many organizations are leveraging Kubernetes for their container environments. However, Kubernetes is notoriously difficult to deploy and operate at scale – particularly for enterprises managing both on-premises and public cloud infrastructure.
When you need to deploy and upgrade to Kubernetes across large-scale environments QUICKLY, in order to address a critical security bug – things become even more difficult, and risky.
Let’s look at the options available for an administrator to setup Kubernetes and upgrade it. They can be broadly classified as:
- DIY: A dedicated team of engineers is responsible for building the Kubernetes clusters using open source or in-house developed practices or automation on the end user’s infrastructure – which can be on-premises data centers or public cloud environments. As we know, those DIY skills are tough to come by and maintain- as the community continues to evolve and the technical complexity of Kubernetes grows.
- Hosted: A Kubernetes service provided by public cloud vendors, on their own infrastructure – such as AWS’ EKS, Google Container Engine.
- Kubernetes Management Solutions: This is a solution that allows organizations to deliver a Kubernetes service – either on their internal environments or on public cloud infrastructure. Kubernetes management solutions are divided into two types:
- Unmanaged Distributions: The traditional way where the customer buys a product, installs it and manages the product on their own – including providing SLA for both internal stakeholders consumption (such as developers) as well as for Production applications.
- Managed Services: Where the Kubernetes service is provided as a remotely-managed one. This means the organization can benefit from a Kubernetes-as-a-Service that can be deployed anywhere, and the Kubernetes infrastructure is managed automatically by the service provider, without requiring management overhead or internal resources from the organization. This includes zero-touch upgrades — which significantly accelerate your time to mitigation in cases such as having to quickly deploy security patches, and greatly simplifies IT operations.
Platform9 Managed Kubernetes is the industry’s only enterprise-grade, SaaS-managed Kubernetes solution that is infrastructure agnostic, working across any public cloud or on-premises infrastructure. This fully-managed service eliminates the operational complexity in scaling Kubernetes for enterprise workloads by delivering it as a Service — with deployments, monitoring, upgrades, fault tolerance and troubleshooting — all handled automatically, and backed by a 24x7x365 SLA. Platform9 allows enterprises to run Kubernetes instantly, anywhere, reduce IT operational overhead and accelerate adoption and time to value with containers.
With a SaaS-managed solution, we can roll out an effective fix with a quick turnaround time, saving the user the pain of having to patch and manually upgrade their Kubernetes environment on their own.
With a turn around time of 3 days, our engineers incorporated the patch to our Managed Kubernetes solution, validated it, and rolled it out to our customers.
As of earlier this week, our customers are able to secure their Kubernetes clusters without any overhead on their end – and without the headache. All the magic happens behind the scenes.
At Platform9 we’ve always argued that “Fully managed” is the standard operating model for cloud environments. That has been the secret sauce for the public cloud leaders for years, and our vision is to enable this seamless management experience for cloud resource at scale- on any environment – whether on-premises or in public clouds. SaaS-managed is the future!
- The Seamless Upgrade for Kubernetes’ First Major Security Hole, CVE-2018-1002105 - December 6, 2018
- Kubernetes Helm Installation - January 24, 2018
- A Recipe For Continuous Integration Using Kubernetes - August 31, 2017