Kubernetes vs Mesos + Marathon
In this updated blog post we’ll compare Kubernetes with the Mesos + Marathon container orchestration solution. We’ll walk you through a high-level discussion of Kubernetes and Mesos with Marathon, look at their features, and cover their pros and cons.
Overview of Kubernetes
According to the Kubernetes website, “Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.” Kubernetes was built by Google based on their experience running containers in production using an internal cluster management system called Borg (sometimes referred to as Omega). The architecture for Kubernetes, which relies on this experience, is shown below:
As you can see from the figure above, there are a number of components associated with a Kubernetes cluster. The master node places container workloads in user pods on worker nodes or itself. The other components include:
- etcd: This component stores configuration data which can be accessed by the Kubernetes master’s API Server by simple HTTP or JSON API.
- API Server: This component is the management hub for the Kubernetes master node. It facilitates communication between the various components, thereby maintaining cluster health.
- Controller Manager: This component ensures that the clusters’ desired state matches the current state by scaling workloads up and down.
- Scheduler: This component places the workload on the appropriate node – in this case all workloads will be placed locally on your host.
- Kubelet: This component receives pod specifications from the API Server and manages pods running in the host.
The following list provides some other common terms associated with Kubernetes:
- Pods: Kubernetes deploys and schedules containers in groups called pods. Containers in a pod run on the same node and share resources such as filesystems, kernel namespaces, and an IP address.
- Deployments: These building blocks can be used to create and manage a group of pods. Deployments can be used with a service tier for scaling horizontally or ensuring availability.
- Services: These are endpoints that can be addressed by name and can be connected to pods using label selectors. The service will automatically round-robin requests between pods. Kubernetes will set up a DNS server for the cluster that watches for new services and allows them to be addressed by name. Services are the “external face” of your container workloads.
- Labels: These are key-value pairs attached to objects. They can be used to search and update multiple objects as a single set.
Overview of Mesos + Marathon
Mesos is a distributed kernel that aims to provide dynamic allocation of resources in your datacenter. Imagine that you manage the IT department of a mid-size business. You need to have workloads running on 100 nodes during the day but on 25 after hours. Mesos can redistribute workloads so that the other 75 nodes can be powered-off when they are not used. Mesos can also provide resource sharing. In the event that one of your nodes fails, workloads can be distributed among other nodes.
Mesos comes with a number of frameworks, application stacks that use its resource sharing capabilities. Each framework consists of a scheduler and a executor. Marathon is a framework (or meta framework) that can launch applications and other frameworks. Marathon can also serve as a container orchestration platform which can provide scaling and self-healing for containerized workloads. The figure below shows the architecture of Mesos + Marathon.
There are a number of different components in Mesos and Marathon. The following list provides some common terms:
- Mesos Master: This type of node enables the sharing of resources across frameworks such as Marathon for container orchestration, Spark for large-scale data processing, and Cassandra for NoSQL databases.
- Mesos Slave: This type of node runs agents that report available resources to the master.
- Framework: A framework registers with the Mesos master so that the master can be offered tasks to run on slave nodes.
- Zookeeper: This component provides a highly available database that can the cluster can keep state, i.e. the active master at any given time.
- Marathon Scheduler: This component receives offers from the Mesos master. Offers from the Mesos master list slave nodes’ available CPU and memory.
- Docker Executor: This component receives tasks from the Marathon scheduler and launches containers on slave nodes.
Mesosphere DCOS
Mesosphere Enterprise DC/OS leverages the Mesos distributed systems kernel and builds on it with container and big data management, providing installation, user interfaces, management and monitoring tools, and other features. The diagram below shows a high-level architecture of DCOS.
Source: DCOS Documentation – Architecture
As shown above, DCOS is comprised of package management, container orchestration (derived from Marathon), cluster management (derived from Mesos), and other components. Further details on Mesosphere DCOS can be found in DCOS documentation.
Kubernetes vs. Mesos + Marathon
 |  |
|---|---|
Application Definition | |
| |
High Availability | |
| |
Auto-scaling for the Application | |
Auto-scaling using resource metrics is available through community-supported components only. | |
| |
Health Checks | |
| |
| Two storage APIs: The first provides abstractions for individual storage backends (e.g. NFS, AWS EBS, Ceph, Flocker). The second provides an abstraction for a storage resource request (e.g. 8 Gb), which can be fulfilled with different storage backends. Modifying the storage resource used by the Docker daemon on a cluster node requires temporarily removing the node from the cluster. Kubernetes offers several types of persistent volumes with block or file support. Examples include iSCSI, NFS, FC, Amazon Web Services, Google Cloud Platform, and Microsoft Azure. The emptyDir volume is non-persistent and can used to read and write files with a container. | Local persistent volumes (beta) are supported for stateful applications such as MySQL. When needed, tasks can be restarted on the same node using the same volume. The use of external storage, such as Amazon EBS, is also in beta. At the present time, applications that use external volumes can only be scaled to a single instance because a volume can only attach to a single task at a time. |
Networking | |
| The networking model is a flat network, enabling all pods to communicate with one another. Network policies specify how pods communicate with each other. The flat network is typically implemented as an overlay. The model requires two CIDRs: one from which pods get an IP address, the other for services. | Networking can be configured in host mode or bridge mode. In host mode, the host ports are used by containers. This can lead to port conflicts on any given host. In bridge mode, the container ports are bridged to host ports using port mapping. Host ports can be dynamically assigned at time of deployment. |
| |
| Services can be found using environment variables or DNS. Kubelet adds a set of environment variables when a pod is run. Kubelet supports simple {SVCNAME_SERVICE_HOST} and {SVCNAME_SERVICE_PORT} variables, as well as Docker links compatible variables. DNS Server is available as an addon. For each Kubernetes service, the DNS Server creates a set of DNS records. With DNS enabled in the entire cluster, pods will be able to use service names that automatically resolve. | Services can be discovered via “named VIPs,” which are DNS records that are are associated with IPs and ports. Services are automatically assigned DNS records by Mesos-DNS. An optional named VIP can be created; requests via the VIP are load-balanced. |
Performance and Scalability | |
| With the release of 1.6, Kubernetes scales to 5,000-node clusters. Multiple clusters can be used to scale beyond this limit. | Mesos’ 2 tier architecture (with Marathon) makes is very scalable. According to Digital Ocean, Mesos and Marathon clusters have been scaled to 10,000 nodes. |
Application Scalability Constructs
Synopsis
 | |
|---|---|
Advantages of Kubernetes Over Mesos + Marathon | |
|
|
| |
|
|
Common Features | |
| |
Takeaways
A look at the mindshare of Kubernetes vs. Mesos + Marathon shows Kubernetes leading with over 70% on all metrics: news articles, web searches, publications, and Github.
Kubernetes offers significant advantages over Mesos + Marathon for three reasons:
- Much wider adoption by the DevOps and containers community
- Better scheduling options for pods, useful for complex application stacks
- Based on over a decade of experience at Google
However, Kubernetes has been known to be difficult to deploy and manage. Platform9’s Managed Kubernetes product can fill this gap by letting organizations focus on deploying microservices on Kubernetes, instead of managing and upgrading a highly available Kubernetes deployment themselves. Further details on these and other deployment models for Kubernetes can be found in The Ultimate Guide to Deploy Kubernetes.
In a follow-up blog, we’ll compare Kubernetes with Amazon ECS.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- Platform9 Introduces Elastic Machine Pool (EMP) at KubeCon 2023 to Optimize Costs on AWS EKS - November 15, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023