Scale Master Nodes in AWS Clusters

Problem

How to scale/replace master nodes in a Multi-Master AWS PMK cluster due to one of the following reasons: maintenance, OS crash, or hardware failure or AMI update?

Environment

• Platform9 Managed Kubernetes - All Versions
• AWS Clusters

Procedure

• Ensure the etcd quorum of the cluster by checking the etcd cluster's health:

$ sudo docker cp etcd:/usr/local/bin/etcdctl /opt/pf9/pf9-kube/bin

$ export PATH=$PATH:/opt/pf9/pf9-kube/bin

$ etcdctl member list  -w=table 

​

$ etcdctl endpoint health --cluster  -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

​

$ etcdctl endpoint status --cluster  -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

• Currently, Platform9 UI does not support scaling up/down the master nodes for an AWS cluster. But this can be achieved using Qbert APIs.

# curl --request POST \

 --url https://<FQDN>/qbert/v4/<project_uuid>/clusters/<cluster_uuid>/detach \

 --header "X-Auth-Token: {X-Auth-Token}" \

 --header "Content-Type: application/json" \

 --data '[

  {

    "uuid": "<master_node_UUID>",

    "isMaster": "1"

  }

]'

• Once the node is detached from the cluster, delete the same master node from the AWS EC2 console so that AWS auto-scales a new node and attaches it to the cluster.
• The progress can be seen by clicking on Infrastructure --> Clusters --> Node Health --> Select the newly added node --> Click on "View All Steps".
• The final step is to verify the status of nodes using the command kubectl get nodes -w