AWS Node Groups

Node Groups

A Node Group in PMK is an entity that represents a group of nodes. Node Groups provide you a way to group and segregate your worker nodes for different purposes. PMK support two types of Node Groups:

Machine Deployment
Machine Pool

Node Group Benefits

Node Groups automate the provisioning and lifecycle management of nodes in AWS CAPI Kubernetes clusters.
You can add multiple Node Groups of any type to the Cluster while creating it or to an existing cluster as well.
You can scale up, scale down, delete, update, and enable/disable auto-scaling for a Node Group individually.
Node updates and terminations automatically drain nodes to ensure that your applications stay available.

AWS Node Group Feature Matrix

You can decide the type of Node Group to use, based on the following.

Feature	Machine Deployment	AWS Machine Pool
CRUD Operations	✅	✅
CAPI Cluster Auto-Scaler Support	✅
Availability Zone Selection		✅
Mixed Instances Support		✅
Spot Instance Support	✅	✅
Bulk Labels and Taints	✅	✅
Nodes Rolling Updates	✅	✅

Node Groups Configuration Options

Machine Deployment

Field	Description
Name	User-defined name for the Machine Deployment. The default name is recommended and always present for this.
Node Count	Provide the number of nodes to be created and associated with the node group.
Instance Type	Choose the type of EC2 instance to be used for nodes belonging to the node group.
Operating System	Choose the Operating system to be nodes belonging to the node group. Platform9 recommends using AMIs that have the component required to use encrypted cloud-init boot script i.e. `Use AWS Secrets Manager: True`. Using an AMI that does not use Secrets Manager will inject the not encrypted cloud-init boot script and can expose any secrets that are part of the boot script.
Custom AMI ID	Enter an AMI ID of your choice. Note: Only Cluster API conformant AMIs can be used to create CAPI clusters. Read https://image-builder.sigs.k8s.io/capi/providers/aws.html to create your own custom AMIs. When using custom AMIs, the secret manager is set to false.
Make Nodes Public	Enabling this will put the nodes belonging to the node group in public subnets of the VPC and assign an elastic IP for the nodes so that the nodes are publicly accessible and users can ssh into them.
Enable Autoscaling	Enable this option will annotate the Node Group for the Cluster-AutoScaler to manage the auto-scaling of this node group. Following annotations are added on the Machine Deployment Type Node Group resource: cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: < MinimumNodeCount ' cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: '< MinimumNodeCount >'
Min Number of Workers	Choose the minimum number of nodes the node group can have.
Max Number of Workers	Choose the maximum number of nodes the node group can have.
Enable Spot Instances	Enables Spot Instances to be used for the nodes of the node group. Read more here: https://aws.amazon.com/ec2/spot/.
Spot Instance Maximum Price	Enter the price for the spot instance. Empty means max price which is recommended to ensure that the spot instance is assigned. Read more here on spot instance pricing: https://aws.amazon.com/ec2/spot/pricing/.
Strategy	The deployment strategy to use to replace existing machines(nodes). MachineDeployment type Node Groups only support Rolling Update type.
Max Surge - Type	Choose the type for the maximum number of machines that can be scheduled above the desired number of machines. Number and Percentage are supported options.
Max Surge - Value	Choose the maximum number of machines that can be scheduled above the desired number of machines. Value can be an absolute number (ex: 5) or a percentage of desired machines (ex: number (ex: 5) or a percentage of desired machines (ex: 10%). Value cannot be 0 if MaxUnavailable is 0. Defaults to 1. The absolute number is calculated from the percentage by rounding up. For example: when this is set to 30%, the new MachineSet can be scaled up immediately when the rolling update starts, such that the total number of old and new machines does not exceed 130% of desired machines. Once old machines have been killed, new MachineSet can be scaled up further, ensuring that the total number of machines running at any time during the update is at most 130% of desired machines.'
Max Unavailable - Type	Choose the type for the maximum number of machines that can be unavailable during the node updates. Number and Percentage are supported options.
Max Unavailable - value	Choose the maximum number of machines that can be unavailable during the node update. Value can be an absolute number (ex: 5) or a percentage of desired machines (ex: number (ex: 5) or a percentage of desired machines (ex: 10%). Value cannot be 0 if MaxSurge is 0. Defaults to 0. Absolute number is calculated from the percentage by rounding down. For example: when this is set to 30%, the old MachineSet can be scaled down to 70% of desired machines immediately when the rolling update starts. Once new machines are ready, the old MachineSet can be scaled down further, followed by scaling up the new MachineSet, ensuring that the total number of machines available at all times during the update is at least 70% of desired machines.
Labels	Add labels to the node Groups. Labels added will be applied on creation to each node belonging to this node group.
Taints	Add taints to the node Groups. Taints added will be applied on creation to each node belonging to this node group.

Machine Pool

Field	Description
Name	User-defined name for the Machine Pool. The default name is recommended and always present for this.
Availability Zone	Choose the AvailabilityZones in which the Node Instances can run in. Machine Pools have a dedicated AWS Auto Scaling Group which determines the placement of the nodes on the basis of the AZ selection.
Node Count	Provide the number of nodes to be created and associated with the node group.
Instance Type	Choose the type of EC2 instance to be used for nodes belonging to the node group.
Operating System	Choose the Operating system to be nodes belonging to the node group. Platform9 recommends using AMIs that have the component required to use encrypted cloud-init boot script i.e. `Use AWS Secrets Manager: True`. Using an AMI that does not use Secrets Manager will inject the not encrypted cloud-init boot script and can expose any secrets that are part of the boot script.
Custom AMI ID	Enter an AMI ID of your choice. Note: Only Cluster API conformant AMIs can be used to create CAPI clusters. Read https://image-builder.sigs.k8s.io/capi/providers/aws.html to create your own custom AMIs. When using custom AMIs, the secret manager is set to false.
Make Nodes Public	Enabling this will put the nodes belonging to the node group in public subnets of the VPC and assign an elastic IP for the nodes so that the nodes are publicly accessible and users can ssh into them.
Min Number of Workers	Choose the minimum number of nodes the node group can have.
Max Number of Workers	Choose the maximum number of nodes the node group can have.
Enable Mixed Instances	Enables Enable Mixed Instances to create a policy that describes how multiple instances( On-Demand vs Spot ) types will be used by the ASG. If not set all nodes of this node group will be of On-Demand type. Read more here: https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html
Nodes On Demand Base Capacity	Choose the minimum number of nodes that will be of On-Demand type. Defaults to 0, which means none of the nodes will be On-Demand type and all spot instances will be created for this node group.
Nodes on Demand % Above Base	Choose the number of nodes that will be of On-Demand type above the base capacity. Defaults to 100, which means all the nodes above base capacity will be Demand type. Example: If Node count = 10, Nodes On Demand Base Capacity = 5, Nodes on Demand % Above Base = 20. The group will have 6( 5 + 1 ) nodes of On-Demand type and 4 will be Spot Instance Type.
Spot Instance Allocation Strategy	Choose the Spot Allocation Strategy between lowest-price and capacity-optimized. lowest-price: Amazon EC2 Auto Scaling allocates your Spot Instances from the N number of pools per Availability Zone that you specify and from the Spot Instance pools with the lowest price in each Availability Zone. For example, if you specify four instance types and four Availability Zones, your Auto Scaling group has access to as many as 16 Spot pools (four in each Availability Zone). If you specify two Spot pools (N=2) for the allocation strategy, your Auto Scaling group can draw on the two cheapest pools per Availability Zone to fulfill your Spot capacity. capacity-optimized: Amazon EC2 Auto Scaling launches instances using Spot pools that are optimally chosen based on the available Spot capacity, which helps you reduce the possibility of a Spot interruption. Deploying in this way helps you make the most efficient use of spare EC2 capacity.
Strategy	The strategy to use to replace existing machines(nodes). Machine Pool type Node Groups only support Rolling Update type. The only valid value is Rolling. A rolling update is an update that is applied to all instances in an Auto Scaling group until all instances have been updated.
Minimum Healthy Nodes - Type	Choose the type for setting the amount of capacity in ASG that must remain healthy during an instance refresh. The only supported type is Percentage.
Minimum Healthy Nodes Percentage- Value	Choose the amount of capacity as a percentage in ASG that must remain healthy during an instance refresh. The default is 90. For example: When this is set to 90% and the Node Count is set to 10, the AGS will make sure that there are 9 instances always available in this node group.
Labels	Add labels to the node Groups. Labels added will be applied on creation to each node belonging to this node group.
Taints	Add taints to the node Groups. Taints added will be applied on creation to each node belonging to this node group.

Last updated by Madhura Maskasky on Jan 17, 2023

Was this page helpful?