Unable to Detach Node From the Cluster Using UI After It is Removed from ETCD Member List

Problem

• The detach operation fails for a node that has been removed from etcd member list.
• This node was removed from etcd cluster as it was listed as an unstarted member in the cluster. It was ensured that the quorum of the cluster is not lost while removing it from etcd.
• Following error observed in the qbert logs during the detach operation

[ERROR] api - TypeError: Cannot read property 'message' of undefined

• The node deauthorized operation is not permitted because this node is already part of a cluster in qbert and the Deauthorize operation is disabled in UI for this node.

Environment

• Platform9 Managed Kubernetes - v5.3 and Higher
• ETCD

Cause

When the detach operation is triggered on a Master node it looks for the etcd member-id of the node in order to remove it from the etcd member list. If the member id is not found it throws the error described in the problem statement section.

Resolution

1. Update the _clusterId=NULL_ in the nodes table in qbert database for the node in question.

# mysql qbert -e "UPDATE nodes SET clusterId=NULL where uuid='[node uuid]';"

2. Once this is done the node will no longer be part of the cluster and the deauthorize option will be enabled in the UI.
3. Deauthorize the node from UI and it will be removed from the cluster.