> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmk/solutions/cluster-upgrade-failures-due-to-certificate-issues.md).

# Nodelet Phases Stuck At Master Node Due to CA Certificate Issue, Which In Turn Affected All worker n

## Problem

Identified multiple issues while performing a Kubernetes cluster upgrade from **v1.24 to v1.25.**

* While upgrading the cluster, the nodelet phase got stuck at the etcd phase on the master node where the upgrade was started.

{% tabs %}
{% tab title="ETCD Logs" %}

```javascript
{"log":"{\"level\":\"warn\",\"ts\":\"2024-04-24T08:16:49.171922Z\",\"caller\":\"embed/config_logging.go:169\",\"msg\":\"rejected connection\",\"remote-addr\":\"10.96.8.51:58162\",\"server-name\":\"\",\"error\":\"tls: failed to verify certificate: x509: certificate signed by unknown authority\"}
```

{% endtab %}
{% endtabs %}

To address the issue, a PMK stack restart was performed on all master nodes. However, as an after-effect, all worker nodes transitioned to the NotReady state following the master node upgrade/CA chain rotation after the stack restart.

{% tabs %}
{% tab title="Kubelet Logs" %}

```javascript
E0424 09:18:50.382151 1746680 kubelet.go:2424] "Error getting node" err="node \"kube-837943-zone1-worker28\" not found"
E0424 09:18:53.106797 1746680 kubelet_node_status.go:92] "Unable to register node with API server" err="Unauthorized" node="kube-837943-zone1-worker28"
```

{% endtab %}
{% endtabs %}

* The existing kubeconfig with the previous CA certificate becomes invalid after the cluster is upgraded.

## Environment

* Platform9 Managed Kubernetes - v5.9
* Kubernetes Version 1.24+

## Cause

* The issue stemmed from the pf9-kube code, which failed to utilize the entire CA chain after certificate rotation for generating certs. This oversight wasn't detected during testing, primarily due to a missing step of restarting the management plane service(Qbert) pod after initiating the certificate rotation.
* After Cluster CA rotation, the Old CA Cert is not available in the CA chain.

## Resolution

* The issue has been resolved in **PMK version 5.10 with Kubernetes version 1.25 or later**

## Workaround

* To unblock restart the nodelet phases on the unaffected master nodes first, and then restart the phases on the affected master node.

{% tabs %}
{% tab title="Phases restart" %}

```javascript
# systemctl stop pf9-hostagent pf9-nodeletd 
# /opt/pf9/nodelet/nodeletd phases stop 
# systemctl start pf9-hostagent
```

{% endtab %}
{% endtabs %}

* To recover from the NotReady state, proceed with the upgrades of the worker nodes. After the upgrade, the nodes that were previously NotReady will transition to a Ready state, as they have been upgraded to the required version


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/cluster-upgrade-failures-due-to-certificate-issues.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.