Worker Node Fails to Join the Cluster

Problem

• A Kubernetes cluster stays in a scaling-up / provisioning state for an extended period.
• In Private Cloud Director, under Kubernetes > Infrastructure > Clusters, opening the affected cluster and going to its Capacity and Health tab shows a Node Groups entry where Ready Replicas is lower than Desired Replicas, and the corresponding worker row in the Nodes table shows a K8s Node status Unknown (or blank) even though its Machine Status already shows Running.
• kubectl get nodes does not list the new worker node either, even though it has had enough time to boot and join.

Environment

• Private Cloud Director
• Component - Kubernetes

Cause

The new worker node booted, but the step that registers it with the cluster (kubeadm join) failed, and nothing retried it automatically. Once this happens, the worker node stays in Unknown state — it never finishes joining, and the cluster keeps waiting for a node that will never appear on its own.

A common trigger is the cluster's control plane not yet being fully ready when the new worker tried to join (for example, right after the cluster was created or scaled). The join step contacts the control plane's API server, and if that call times out, the join fails outright rather than retrying.

Detecting it

1. In Private Cloud Director dashboard, go to Kubernetes > Infrastructure > Clusters and open the affected cluster, then switch to its Capacity and Health tab. In the Nodes table, find the worker row whose K8s Node status is Unknown — that's the stuck node.
2. SSH into that worker VM (on an isolated/virtual network this typically requires going through a jumphost) and check the node's boot and kubelet status:

sudo cloud-init status
sudo tail -n 80 /var/log/cloud-init-output.log
sudo journalctl -u kubelet | tail

• cloud-init status reporting error, together with a kubeadm join / preflight error in the kubelet log (for example, a timeout fetching cluster config, or a "kubelet config file ... no such file or directory" error), confirms the join failed on this node.

1. Check that the cluster's control plane is reachable from the worker, using the same endpoint the node would use to join:

curl -vk --max-time 5 https://<control-plane-endpoint>:443/healthz

• HTTP 200 ok means the control plane is healthy right now, so the earlier failure was transient — it's safe to retry the join (see Resolution below).
• If this still fails, the control plane itself is not reachable — that's a separate connectivity/DNS/certificate issue. Resolve that first; retrying the join will not help until this succeeds.

Resolution

The steps below assume the control plane is confirmed reachable and healthy (previous section, step 3).

1. Confirm the VM is in a clean, failed-join state before retrying — no leftover Kubernetes config means it's safe to simply re-run the join, no reset needed:

sudo test -f /etc/kubernetes/kubelet.conf && echo EXISTS || echo MISSING
sudo test -f /etc/kubernetes/pki/ca.crt && echo EXISTS || echo MISSING

Both commands should print MISSING.

1. If so, the simplest fix is to let Private Cloud Director recreate the node instead of fixing the VM in place: go to Kubernetes > Infrastructure > Clusters, open the cluster, switch to the Capacity and Health tab, select the affected row in the Nodes table, and delete it. The node group will bring up a fresh replacement automatically — this is the recommended option when the exact VM does not need to be preserved.
2. Alternative: re-run the join on the existing VM instead, using the same configuration it was originally given (reusing it avoids having to regenerate tokens/certificates):

sudo cat /var/lib/cloud/instances/<instance-id>/scripts/runcmd   # find the exact command used
sudo kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml

1. Verify the node joins and becomes Ready within a couple of minutes — either from the Nodes table in the Capacity and Health tab, or:

kubectl get nodes -o wide