> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/kubernetes/worker-node-stuck-due-to-kubeadm-join-failure.md). # Worker Node Fails to Join the Cluster ## Problem * A Kubernetes cluster stays in a scaling-up / provisioning state for an extended period. * In Private Cloud Director, under **Kubernetes > Infrastructure > Clusters**, opening the affected cluster and going to its **Capacity and Health** tab shows a **Node Groups** entry where **Ready Replicas** is lower than **Desired Replicas**, and the corresponding worker row in the **Nodes** table shows a **K8s Node status** `Unknown` (or blank) even though its **Machine Status** already shows `Running`. * `kubectl get nodes` does not list the new worker node either, even though it has had enough time to boot and join. ## Environment * Private Cloud Director * Component - Kubernetes ## Cause The new worker node booted, but the step that registers it with the cluster (`kubeadm join`) failed, and nothing retried it automatically. Once this happens, the worker node stays in `Unknown` state — it never finishes joining, and the cluster keeps waiting for a node that will never appear on its own. A common trigger is the cluster's control plane not yet being fully ready when the new worker tried to join (for example, right after the cluster was created or scaled). The join step contacts the control plane's API server, and if that call times out, the join fails outright rather than retrying. ## Detecting it 1. In Private Cloud Director dashboard, go to **Kubernetes > Infrastructure > Clusters** and open the affected cluster, then switch to its **Capacity and Health** tab. In the **Nodes** table, find the worker row whose **K8s Node status** is `Unknown` — that's the stuck node. 2. SSH into that worker VM (on an isolated/virtual network this typically requires going through a jumphost) and check the node's boot and kubelet status: ```bash sudo cloud-init status sudo tail -n 80 /var/log/cloud-init-output.log sudo journalctl -u kubelet | tail ``` * `cloud-init status` reporting `error`, together with a `kubeadm join` / preflight error in the kubelet log (for example, a timeout fetching cluster config, or a "kubelet config file ... no such file or directory" error), confirms the join failed on this node. 3. Check that the cluster's control plane is reachable from the worker, using the same endpoint the node would use to join: ```bash curl -vk --max-time 5 https://:443/healthz ``` * `HTTP 200 ok` means the control plane is healthy right now, so the earlier failure was transient — it's safe to retry the join (see Resolution below). * If this still fails, the control plane itself is not reachable — that's a separate connectivity/DNS/certificate issue. Resolve that first; retrying the join will not help until this succeeds. ## Resolution The steps below assume the control plane is confirmed reachable and healthy (previous section, step 3). 1. Confirm the VM is in a clean, failed-join state before retrying — no leftover Kubernetes config means it's safe to simply re-run the join, no reset needed: ```bash sudo test -f /etc/kubernetes/kubelet.conf && echo EXISTS || echo MISSING sudo test -f /etc/kubernetes/pki/ca.crt && echo EXISTS || echo MISSING ``` Both commands should print `MISSING`. 2. If so, the simplest fix is to let Private Cloud Director recreate the node instead of fixing the VM in place: go to **Kubernetes > Infrastructure > Clusters**, open the cluster, switch to the **Capacity and Health** tab, select the affected row in the **Nodes** table, and delete it. The node group will bring up a fresh replacement automatically — this is the recommended option when the exact VM does not need to be preserved. 3. Alternative: re-run the join on the existing VM instead, using the same configuration it was originally given (reusing it avoids having to regenerate tokens/certificates): ```bash sudo cat /var/lib/cloud/instances//scripts/runcmd # find the exact command used sudo kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml ``` 4. Verify the node joins and becomes `Ready` within a couple of minutes — either from the **Nodes** table in the **Capacity and Health** tab, or: ```bash kubectl get nodes -o wide ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd/kubernetes/worker-node-stuck-due-to-kubeadm-join-failure.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.