Menu
Home»Blog»Kubernetes vs Amazon ECS

Kubernetes vs Amazon ECS

Kubernetes Competitive

In this updated blog post we’ll compare Kubernetes with Amazon ECS (EC2 Container Service). We’ll walk you through high-level discussions of Kubernetes and Amazon ECS, and then compare these two competing solutions. For completeness, we provide an overview of Kubernetes. If you have already read the previous posts which compare Kubernetes to Docker Swarm or Mesos + Marathon, you can skip this section.

Overview of Kubernetes

According to the Kubernetes website, “Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.” Kubernetes was built by Google based on their experience running containers in production using an internal cluster management system called Borg (sometimes referred to as Omega). The architecture for Kubernetes, which relies on this experience, is shown below:

Recommended Readings

As you can see from the figure above, there are a number of components associated with a Kubernetes cluster. The master node places container workloads in user pods on worker nodes or itself. The other components include:

  • etcd: This component stores configuration data which can be accessed by the Kubernetes master’s API Server by simple HTTP or JSON API.
  • API Server: This component is the management hub for the Kubernetes master node. It facilitates communication between the various components, thereby maintaining cluster health.
  • Controller Manager: This component ensures that the clusters’ desired state matches the current state by scaling workloads up and down.
  • Scheduler: This component places the workload on the appropriate node.
  • Kubelet: This component receives pod specifications from the API Server and manages pods running in the host.

The following list provides some other common terms associated with Kubernetes:

  • Pods: Kubernetes deploys and schedules containers in groups called pods. Containers in a pod run on the same node and share resources such as filesystems, kernel namespaces, and an IP address.
  • Deployments: These building blocks can be used to create and manage a group of pods. Deployments can be used with a service tier for scaling horizontally or ensuring availability.
  • Services: These are endpoints that can be addressed by name and can be connected to pods using label selectors. The service will automatically round-robin requests between pods. Kubernetes will set up a DNS server for the cluster that watches for new services and allows them to be addressed by name. Services are the “external face” of your container workloads.
  • Labels: These are key-value pairs attached to objects. They can be used to search and update multiple objects as a single set.

Overview of Amazon ECS

Amazon ECS is the Docker-compatible container orchestration solution from Amazon Web Services. It allows you to run containerized applications on EC2 instances and scale both of them. The following diagram shows the high-level architecture of ECS.

As shown above, ECS Clusters consist of tasks which run in Docker containers, and container instances, among many other components. Here are some AWS services commonly used with ECS:

  • Elastic Load Balancer: This component can route traffic to containers. Two kinds of load balancing are available: application and classic.
  • Elastic Block Store: This service provides persistent block storage for ECS tasks (workloads running in containers).
  • CloudWatch: This service collects metrics from ECS. Based on CloudWatch metrics, ECS services can be scaled up or down.
  • Virtual Private Cloud: An ECS cluster runs within a VPC. A VPC can have one or more subnets.
  • CloudTrail: This service can log ECS API calls. Details captured include type of request made to Amazon ECS, source IP address, user details, etc.

ECS, which is provided by Amazon as a service, is composed of multiple built-in components which enable administrators to create clusters, tasks and services:

  • State Engine: A container environment can consist of many EC2 container instances and containers. With hundreds or thousands of containers, it is necessary to keep track of the availability of instances to serve new requests based on CPU, memory, load balancing, and other characteristics. The state engine is designed to keep track of available hosts, running containers, and other functions of a cluster manager.
  • Schedulers: These components use information from the state engine to place containers in the optimal EC2 container instances. The batch job scheduler is used for tasks that run for a short period of time. The service scheduler is used for long running apps. It can automatically schedule new tasks to an ELB.
  • Cluster: This is a logical placement boundary for a set of EC2 container instances within an AWS region. A cluster can span multiple availability zones (AZs), and can be scaled up/down dynamically. A dev/test environment may have 2 clusters: 1 each for production and test.
  • Tasks: A task is a unit of work. Task definitions, written in JSON, specify containers that should be co-located (on an EC2 container instance). Though tasks usually consist of a single container, they can also contain multiple containers.
  • Services: This component specifies how many tasks should be running across a given cluster. You can interact with services using their API, and use the service scheduler for task placement.

Note that ECS only manages ECS container workloads – resulting in vendor lock-in. There’s no support to run containers on infrastructure outside of EC2, including physical infrastructure or other clouds such as Google Cloud Platform and Microsoft Azure. The advantage, of course, is the ability to work with all the other AWS services like Elastic Load Balancers, CloudTrail, CloudWatch etc.

Further details about Amazon ECS can be found in AWS ECS Documentation.

Kubernetes vs. Amazon ECS

 

Application Definition

Applications can be deployed using a combination of pods, deployments, and services. A pod is a group of co-located containers and is the atomic unit of a deployment. A deployment can have replicas across multiple nodes. A service is the “external face” of container workloads and integrates with DNS to round-robin incoming requests. Load balancing of incoming requests is supported.
Applications can be deployed as tasks, which are Docker containers running on EC2 instances (aka container instances). Task definitions specify the container image, CPU, memory and persistent storage in a JSON template. Clusters comprise of one or more tasks that use these task definitions. Schedulers automatically place containers across compute nodes in a cluster, which can also span multiple AZs. Services can be created by specifying number of tasks and an Elastic Load Balancer.
 

  Application Scalability Constructs

Each application tier is defined as a pod and can be scaled when managed by a deployment, which is specified declaratively, e.g., in YAML. The scaling can be manual or automated. Pods are most useful for running co-located and co-administered helper applications, like log and checkpoint backup agents, proxies and adapters, though they can also be used to run vertically integrated application stacks such as LAMP (Apache, MySQL, PHP) or ELK/Elastic (Elasticsearch, Logstash, Kibana).
Applications can be defined using task definitions written in JSON. Tasks are instantiations of task definitions and can be scaled up or down manually. The built-in scheduler will automatically distribute tasks across ECS compute nodes. For a vertically integrated stack, task definitions can specify one tier which exposes an http endpoint. This endpoint can in-turn be used by another tier, or exposed to the user.
 

High Availability

Deployments allow pods to be distributed among nodes to provide HA, thereby tolerating infrastructure or application failures. Load-balanced services detect unhealthy pods and remove them. High availability of Kubernetes is supported. Multiple master nodes and worker nodes can be load balanced for requests from kubectl and clients. etcd can be clustered and API Servers can be replicated.
Schedulers place tasks, which are comprised of 1 or more containers, on EC2 container instances. Tasks can be increased or decreased manually to scale. Elastic Load Balancers can distribute traffic among healthy containers. ECS control plane high availability is taken care of by Amazon. Requests can be load-balanced to multiple tasks using ELB.
 

          Load Balancing       

Pods are exposed through a service, which can be used as a load-balancer within the cluster. Typically, an ingress resource is used for load balancing.
ELB provides a CNAME that can be used within the cluster. This CNAME serves as a front-facing resolvable FQDN for multiple tasks. Two kinds of service load balancers with ELB: application or classic.
 

Auto-scaling for the Application

Auto-scaling using a simple number-of-pods target is defined declaratively using deployments. Auto-scaling using resource metrics is also supported. Resource metrics range from CPU and memory utilization to requests or packets-per-second, and even custom metrics.
CloudWatch alarms can be used to auto-scale ECS services up or down based on CPU, memory, and custom metrics.
 

  Rolling Application Upgrades and Rollback

A deployment  supports both “rolling-update” and “recreate” strategies. Rolling updates can specify maximum number of pods.
Rolling updates are supported using “minimumHealthyPercent” and “maximumPercent” parameters. The same parameters can be adjusted to do blue-green updates, which adds a whole new batch of containers in parallel with the existing set.
 

Health Checks

Health checks are of two kinds: liveness (is app responsive) and readiness (is app responsive, but busy preparing and not yet able to serve).
ECS provides health checks using CloudWatch. 
 

          Block Storage         

Two storage APIs:
The first provides abstractions for individual storage backends (e.g. NFS, AWS EBS, Ceph, Flocker).
The second provides an abstraction for a storage resource request (e.g. 8 Gb), which can be fulfilled with different storage backends. Modifying the storage resource used by the Docker daemon on a cluster node requires temporarily removing the node from the cluster.
Kubernetes offers several types of persistent volumes with block or file support. Examples include iSCSI, NFS, FC, Amazon Web Services, Google Cloud Platform, and Microsoft Azure.
The emptyDir volume is non-persistent and can used to read and write files with a container.		Storage support is limited to Amazon Web Services. EBS volumes can be specified by using ECS task definitions (JSON files) and connected to container instances. Task definitions have a “containerDefinitions” section which can be used to enable “mountPoints.”
Multiple containers on a container instance can be connected to an EBS storage volume. (Reference: Amazon Web Services Docs)
 

Networking

The networking model is a flat network, enabling all pods to communicate with one another. Network policies specify how pods communicate with each other. The flat network is typically implemented as an overlay.
The model requires two CIDRs: one from which pods get an IP address, the other for services.		ECS is supported in a VPC, which can include multiple subnets in multiple AZs. Communication within a subnet cannot be restricted using AWS tools.
 

        Service Discovery       

Services can be found using environment variables or DNS.
Kubelet adds a set of environment variables when a pod is run. Kubelet supports simple “{SVCNAME_SERVICE_HOST}” and “{SVCNAME_SERVICE_PORT}” variables, as well as Docker links compatible variables.
DNS Server is available as an addon. For each Kubernetes service, the DNS Server creates a set of DNS records. With DNS enabled in the entire cluster, pods will be able to use service names that automatically resolve.		Services can be found using an ELB and a CNAME. A single ELB can be used per service. Route53 private hosted zones can be used to ensure that the ELB CNAMEs are only resolvable within your VPC.
 

Performance and Scalability

With the release of 1.6, Kubernetes scales to 5,000-node clusters. Multiple clusters can be used to scale beyond this limit.ECS has been scaled-up to over a 1000 container nodes without noticeable performance degradation. (Deep Dive on Microservices and Amazon ECS, skip to 11:27)

Synopsis

 

Advantages of Kubernetes Over Amazon ECS

  • Can be deployed on-premises, private clouds, and public clouds. (anywhere you can run x86 servers, or even on your laptops)
  • Wide variety of storage options, including on-premises SANs and public clouds.
  • Based on extensive experience running Linux containers at Google. Deployed at scale more often among organizations. Kubernetes is also backed by enterprise offerings from both Google (GKE) and RedHat (OpenShift).
  • Largest community among container orchestration tools. Over 50,000 commits and 1200 contributors.
  • Vendor lock-in. Containers can only be deployed on Amazon, and ECS can only manage containers that it has created.
  • External storage is limited to Amazon, including Amazon EBS.
  • Validated within Amazon. ECS is not publicly available for deployment outside Amazon.
  • Much of ECS code is not publicly available. Parts of ECS, including Blox, a framework that helps customers build custom schedulers, are open source. 200 commits and 15 contributors, many from Amazon.
 

  Disadvantages of Kubernetes

  • Lack of single vendor control can complicate a prospective customer’s purchasing decision. Community includes Google, Red Hat, and over 2000 authors. (Source: CNCF)
  • Do-it-yourself installation can be complex for Kubernetes. Deployment tools for Kubernetes include kubeadm, kops, kargo, and others. Further details in Kubernetes Deployment Guide.
  • Single vendor control may allow for accountability with bug fixes and better coordination with feature development.
  • ECS does not require installation on servers. ECS CLI installation is simple.
 

Common Features

  • Networking features such as load balancing and DNS.
  • Logging and Monitoring. External tools for Kubernetes include Elasticsearch/Kibana (ELK), sysdig, cAdvisor, Heapster/Grafana/InfluxDB (Reference: Logging and Monitoring for Kubernetes. For ECS, partner ecosystem includes external tools by Datadog and Sysdig Cloud, in addition to CloudWatch and CloudTrail (Reference: Datadog for ECS, Sysdig for ECS).
  • Autoscaling supported natively.
  • Use of separate set of tools for management. Kubernetes actions can be performed through the kubectl CLI and Kubernetes Dashboard. ECS can be managed using the AWS console and CLI.

Takeaways

Though both container orchestration solutions are validated by notable names, Kubernetes appears to be significantly more popular online, in the media, and among developers.

Above all, Kubernetes eclipses ECS through its ability to deploy on any x86 server (or even a laptop). Unlike ECS, Kubernetes is not restricted to the Amazon cloud. In fact, Kubernetes can be deployed on Amazon in a traditional cloud or hybrid cloud model, and serve as an alternative to those interested in containers using Amazon.

Customers looking to leverage Kubernetes’ capabilities across clouds and on-premises can use products such as Platform9 Managed Kubernetes. Such offerings can eliminate lock-in and let you focus on deploying containerized applications instead of managing an enterprise Kubernetes deployment on your own. Further details on Platform9 Managed Kubernetes and other deployment models, including Minikube, kubeadm and public clouds, can be found in The Ultimate Guide to Deploy Kubernetes.

This post concludes this blog series about Kubernetes vs Swarm, Kubernetes vs Mesos, and Kubernetes vs Amazon ECS. We’re looking forward to putting out an updated comparison ebook soon. 

Platform9
Latest posts by Platform9 (see all)

You may also enjoy

Mastering the operational model challenge for distributed AI/ML infrastructure

By Kamesh Pemmaraju

Tackling Kubernetes Underutilization: Cutting EKS Costs by 50%

By Kamesh Pemmaraju

Categorized within: Kubernetes Competitive Tags: , , ,

The browser you are using is outdated. For the best experience please download or update your browser to one of the following: