How to Restore Cluster from ETCD Backup on New Master Nodes in Case of Loss of All Existing Master N

Problem

How to restore etcd backup on new master nodes in case of loss of all existing master nodes in the cluster?

Environment

• Platform9 Managed Kubernetes - All Versions

• Docker or Containerd

• ETCD

Procedure

IMPORTANT

Make sure to have access to the etcd backup file. For working cluster the default path for the backup storage path is /etc/pf9/etcd-backup. Also the worker nodes should be not affected and should be in running state.

1. Use Force Remove in the Management Plane UI to deauthorize and remove the old master nodes from Management Plane. Note: For PMK v5.3 and below please contact Platform9 Support to Force Remove the nodes from Management Plane.

2. Create new master nodes with the same IP address and hostname as that of the removed master nodes.

3. Onboard the nodes to Management Plane using either the Platform9 CLI or through downloading and installing the Platform9 HostAgent manually.

4. Once nodes comes online, first add the same active master(leader) to the cluster using below API.

IMPORTANT

Get TOKEN, PROJECT_ID by following steps mentioned in Keystone Identity. And the NODE_UUID and CLUSTER_ID can be found from the Management Plane UI by enabling UUID in columns.

API

# curl -k -X POST -H "X-Auth-Token: $TOKEN" -H "Content-type: application/json" -d '[
 {
     "uuid": "<NODE_UUID>",
     "isMaster": 1
 }]' "https://<DU-FQDN>/qbert/v3/<project_id>/clusters/<cluster_id>/attach"

Example

# TOKEN=gAAAAABhnQ4ykxkfDvEd5QyhZaqTvXjiysB2xt9hY7QgjpSi_mWdCUMEgbS0w-5DAOVLRTtZeHRrCql9-aWfOFidF-1mjx0yUeIuKBel2wWQ91VsOoPK4Knz9KU-59u73vqhWaD_RibU0-eReaCx7Et1OsJ6aybFcL9ug-Kp_BHTpOxrlqGIeCI

# curl -k -X POST -H "X-Auth-Token: $TOKEN" -H "Content-type: application/json" -d '[
{
    "uuid": "992e976b-0527-4359-bc28-00c697b368eb",
    "isMaster": 1
}]' "https://ajohn.platform9.net/qbert/v3/9dc12eee0b794b9a8e8f8dc90a88f7ec/clusters/601a4510-262d-4551-bf05-395667708957/attach"

1. Once the master node is attached, the cluster looks healthy with a single master node.

Example v1.19

# kubectl get nodes
NAME             STATUS   ROLES    AGE     VERSION
10.128.145.204   Ready    master   4m36s   v1.19.6
10.128.145.248   Ready    worker   4m15s   v1.19.6

Example v1.20

# kubectl get nodes
NAME             STATUS   ROLES    AGE     VERSION
10.128.146.162   Ready    master   6m10s   v1.20.11

1. Copy the etcd backup to the Master node and copy etcdctl binary.

Docker

# docker cp etcd:/usr/local/bin/etcdctl /opt/pf9/pf9-kube/bin
# export PATH=$PATH:/opt/pf9/pf9-kube/bin

1. Stop the PMK stack and move the current etcd directory to some other path on the Master node.

Command

# systemctl stop pf9-{hostagent,nodeletd}
# /opt/pf9/nodelet/nodeletd phases stop
# mkdir /tmp/etcd_dir_backup
# mv /var/opt/pf9/kube/etcd/* /tmp/etcd_dir_backup

1. Perform etcd DB restore from the master node.

Docker

# ETCDCTL_API=3 etcdctl snapshot restore <ETCD_BACKUP.db> --data-dir /var/opt/pf9/kube/etcd/data --initial-advertise-peer-urls="https://<MASTER_IP>:2380" --initial-cluster="<MASTER_NODE_UUID>=https://<MASTER_IP>:2380" --name="<MASTER_NODE_UUID>"
{% endtab %}
{% tab language="bash" title="Containerd" %}
# /opt/pf9/pf9-kube/bin/etcdctl snapshot restore /path/to/snapshshot_file --data-dir /var/opt/pf9/kube/etcd/data --initial-advertise-peer-urls="https://<MASTER_IP>:2380" --initial-cluster="<MASTER_NODE_UUID>=https://<MASTER_IP>:2380" --name="<MASTER_NODE_UUID>"

Example

# ETCDCTL_API=3 etcdctl snapshot restore /home/centos/etcd_backup.db --data-dir /var/opt/pf9/kube/etcd/data --initial-advertise-peer-urls="https://10.128.145.204:2380" --initial-cluster="992e976b-0527-4359-bc28-00c697b368eb=https://10.128.145.204:2380" --name="992e976b-0527-4359-bc28-00c697b368eb"

2021-11-23 16:08:28.229042 I | etcdserver/membership: added member 343efe1ee1440e81 [https://10.128.145.204:2380] to cluster 90c381d304967556

1. Once restore is complete, start PMK stack.

Command

# systemctl start pf9-hostagent

1. Check the etcd cluster health status post-restoration.

• For cluster running pf9-kube v1.19

Docker

# /opt/pf9/pf9-kube/bin/etcdctl --ca-file /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.key cluster-health

# /opt/pf9/pf9-kube/bin/etcdctl --ca-file /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.key member list

Sample

# /opt/pf9/pf9-kube/bin/etcdctl --ca-file /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.key cluster-health
member 343efe1ee1440e81 is healthy: got healthy result from https://10.128.145.204:4001
cluster is healthy

# /opt/pf9/pf9-kube/bin/etcdctl --ca-file /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key-file /etc/pf9/kube.d/certs/etcdctl/etcd/request.key member list
343efe1ee1440e81: name=992e976b-0527-4359-bc28-00c697b368eb peerURLs=https://10.128.145.204:2380 clientURLs=https://10.128.145.204:4001 isLeader=true

• For cluster running pf9-kube v1.20

Docker

# ETCDCTL_API=3 /opt/pf9/pf9-kube/bin/etcdctl member list 
# ETCDCTL_API=3 /opt/pf9/pf9-kube/bin/etcdctl endpoint health --write-out=table
# ETCDCTL_API=3 /opt/pf9/pf9-kube/bin/etcdctl endpoint status --cluster --write-out=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"
{% endtab %}
{% tab language="bash" title="Containerd" %}
# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key member list  -w=table

# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint health --cluster  -w=table

Example

# ETCDCTL_API=3 /opt/pf9/pf9-kube/bin/etcdctl member list 
b60fda6661d77461 started  70ee9439-06e8-4ad3-8ae2-b2fd5360cbd6  https://10.128.146.162:2380 https://10.128.146.162:4001       false 


# ETCDCTL_API=3 /opt/pf9/pf9-kube/bin/etcdctl endpoint health --write-out=table 
+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://10.128.146.162:4001 |   true | 22.107927ms |       |
+-----------------------------+--------+-------------+-------+

1. Delete the stale master nodes which are in NotReady state using kubectl.

Example

# kubectl get nodes
NAME             STATUS     ROLES    AGE    VERSION
10.128.144.242   NotReady   master   136m   v1.19.6
10.128.145.204   Ready      master   136m   v1.19.6
10.128.145.248   Ready      worker   136m   v1.19.6
10.128.145.253   NotReady   master   136m   v1.19.6

# kubectl delete node 10.128.144.242 10.128.145.253
node "10.128.144.242" deleted
node "10.128.145.253" deleted

1. Scale the master nodes one by one from the Management Plane UI and verify the etcd cluster health using Step 10.

Note

Restoring from the etcd backup is a complicated process, if not familiar with it we recommend involving Platform9 Support for any assistance before initiating the restore operation.