How to Find ETCD Leader and Check ETCD Health

Problem

How to find ETCD Leader and check ETCD Health in a Platform9 Managed Kubernetes Cluster?

Environment

• Platform9 Managed Kubernetes - All Versions
• ETCD
• Docker and Containerd

Procedure

1. SSH into any one of the Master nodes and become a root user.
2. For docker based container runtime: Run the following command to copy the etcdctl binary from the etcd container running on the node and update the PATH variable.

Docker

docker cp etcd:/usr/local/bin/etcdctl /opt/pf9/pf9-kube/bin 

export PATH=$PATH:/opt/pf9/pf9-kube/bin

1. Run the etcdctl command to find out the ETCD Leader and look for IS LEADER column

# /opt/pf9/pf9-kube/bin/etcdctl  endpoint status -w=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 5ad09a7031a6a983 |  3.4.14 |   13 MB |     false |      false |        83 |    6676368 |            6676368 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
{% endtab %}
{% tab language="bash" title="Containerd" %}
# sudo /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint status --cluster  -w=table

1. Run the etcdctl command to check ETCD Health of Member/Endpoints.

Docker

# /opt/pf9/pf9-kube/bin/etcdctl endpoint health --cluster  -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"
+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://10.128.146.244:4001 |   true | 18.427771ms |       |
| https://10.128.147.235:4001 |   true | 18.958459ms |       |
|  https://10.128.146.71:4001 |   true | 20.086052ms |       |
+-----------------------------+--------+-------------+-------+
{% endtab %}
{% tab language="bash" title="Containerd" %}
# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint health --cluster  -w=table

Additional Information

Kubernetes Docs - Configure/Upgrade ETCD