Scale Master Nodes in AWS Clusters

Problem

How to scale/replace master nodes in a Multi-Master AWS PMK cluster due to one of the following reasons: maintenance, OS crash, or hardware failure or AMI update?

Environment

• Platform9 Managed Kubernetes - All Versions

• AWS Clusters

Procedure

• Ensure the etcd quorum of the cluster by checking the etcd cluster's health:

Docker

$ sudo docker cp etcd:/usr/local/bin/etcdctl /opt/pf9/pf9-kube/bin
$ export PATH=$PATH:/opt/pf9/pf9-kube/bin
$ etcdctl member list  -w=table 

$ etcdctl endpoint health --cluster  -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

$ etcdctl endpoint status --cluster  -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"
{% endtab %}
{% tab language="javascript" title="Containerd" %}
# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key member list  -w=table

# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint health --cluster  -w=table

• Currently, Platform9 UI does not support scaling up/down the master nodes for an AWS cluster. But this can be achieved using Qbert APIs.

Info

To get X-Auth-Token use the following steps.

Example

# curl --request POST \
 --url https://<FQDN>/qbert/v4/<project_uuid>/clusters/<cluster_uuid>/detach \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --header "Content-Type: application/json" \
 --data '[
  {
    "uuid": "<master_node_UUID>",
    "isMaster": "1"
  }
]'

• Once the node is detached from the cluster, delete the same master node from the AWS EC2 console so that AWS auto-scales a new node and attaches it to the cluster.

• The progress can be seen by clicking on Infrastructure --> Clusters --> Node Health --> Select the newly added node --> Click on "View All Steps".

• The final step is to verify the status of nodes using the command kubectl get nodes -w