How to Check ETCD Cluster Health?

Problem

How to find ETCD leader and check ETCD cluster health.

Environment

• Self-Hosted Private Cloud Director - All Versions.

Procedure

Info

The etcdctl binary and certificates/keys are present in the masters nodes, so run all the below commands from any of the master nodes.

1. To list the endpoint health:

Command

$ /opt/pf9/pf9-kube/bin/etcdctl \
--endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>:4001 \
--cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \
--cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \
--key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \
endpoint health -w=table

+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://<Master1_IP>:4001  |   true | 10.833037ms |       |
| https://<Master2_IP>:4001  |   true | 10.999511ms |       |
| https://<Master3_IP>:4001  |   true | 11.424566ms |       |
+----------------------------+--------+-------------+-------+

1. To list the endpoint health status and ETCD leader details:

Command

$ /opt/pf9/pf9-kube/bin/etcdctl \
--endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>::4001 \
--cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \
--cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \
--key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \
endpoint status -w=table

+---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+-----------------------------+
|         ENDPOINT          |        ID    | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://[Master1_IP]:4001 | [Member1_ID] |  3.4.14 |  168 MB |     false |      false |      1124 |  238839485 |          238839485 |        |
| https://[Master2_IP]:4001 | [Member2_ID] |  3.4.14 |  168 MB |      true |      false |      1124 |  238839495 |          238839495 |        |
| https://[Master3_IP]:4001 | [Member3_ID] |  3.4.14 |  168 MB |     false |      false |      1124 |  238839495 |          238839575 |        |
+---------------------------+--------------+---------+---------+-----------+------------+-----------+------------+-----------------------------+

1. To list member list:

Command

$ /opt/pf9/pf9-kube/bin/etcdctl \
--endpoints=https://<Master1_IP>:4001,https://<Master2_IP>:4001,https://<Master3_IP>:4001 \
--cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt \
--cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt \
--key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key \
member list -w=table

+--------------+---------+----------------+---------------------------+---------------------------+------------+
|    ID        | STATUS  |  NAME          |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+--------------+---------+----------------+---------------------------+---------------------------+------------+
| [Member1_ID] | started | [Master1-UUID] | https://[Master1_IP]:4001 | https://[Master1_IP]:4001 |      false |
| [Member2_ID] | started | [Master1-UUID] | https://[Master2_IP]:4001 | https://[Master2_IP]:4001 |      false |
| [Member3_ID] | started | [Master1-UUID] | https://[Master3_IP]:4001 | https://[Master3_IP]:4001 |      false |
+--------------+---------+----------------+---------------------------+---------------------------+------------+