Nodelet Does Not Remove The Node From Kubernetes Cluster When Node Is Removed From Qbert Cluster

Problem

• The kubectl command is still listing the master node that has been scaled down from UI.

Environment

• Platform9 Managed Kubernetes - v5.4 and above

Cause

• The task of removing the scaled down node from the kubernetes cluster is handled by the all-none-scripts.sh phase script which runs when ROLE=”none” and KUBE_SERVICE_STATE=false is set in the config.
• However there is a possibility that the certs are removed prior to this which causes node removal to be skipped because the node cannot reach the other master nodes or master VIP.

Resolution

• A internal JIRA ticket PMK-5585 has been raised to fix this issue.
• The workaround for this issue is, post node scale down wait for the node and cluster to get converged in the Platform9 UI, and then delete the node from the cluster using below mentioned command.

# kubectl delete node <scaled_down_node>