# Kubernetes Node in NotReady State After Reboot for Containerd Runtime Cluster

## Problem

* A Kubernetes (master or worker) node that has been rebooted (e.g. due to a maintenance activity) is showing as `NotReady`.

{% tabs %}
{% tab title="Node Status" %}

```bash
$ kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       NotReady   master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3
```

{% endtab %}
{% endtabs %}

* A description of the node similarly reports `KubeletNotReady` due to the CNI plugin being uninitialized.

{% tabs %}
{% tab title="Node Description" %}

```bash
$ kubectl describe node master1
.....
Conditions:
Ready                False   Mon, 13 Jun 2022 21:42:05 +0000   Mon, 13 Jun 2022 21:32:01 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Managed Kubernetes - v5.4 and Higher
* Kubernetes - All 1.21 versions except v1.21.3-pmk.183
* Runtime - *Containerd*

## Cause

Due to an [upstream issue](https://github.com/containerd/containerd/issues/6490) in containerd, the CNI config is not reloaded when the directory is deleted and recreated during the Platform9 Kubernetes stack initialization.

## Resolution

* This issue is now fixed on the **pf9-kube-1.21.3-pmk.183. and above** releases.

## Workaround

1. Verify that the CNI configuration directory referenced by *containerd* is not empty.

For *Flannel* based clusters the directory should contain the following files:

{% tabs %}
{% tab title="Javascript" %}

```javascript
# ls -l /etc/cni/net.d/
total 4
-rw-r--r-- 1 root root 362 Jun 16 14:45 10-flannel.conflist
```

{% endtab %}
{% endtabs %}

For *Calico* based clusters the directory should contain the following files:

{% tabs %}
{% tab title="Bash" %}

```bash
# ls -l /etc/cni/net.d/
total 8
-rw-r--r-- 1 root root  730 Jun 16 14:16 10-calico.conflist
-rw------- 1 root root 3094 Jun 16 14:16 calico-kubeconfig
```

{% endtab %}
{% endtabs %}

2. Restart containerd service on the affected node.

{% hint style="info" %}
**Info**

Restarting *containerd* will not affect restart any running containers
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
# sudo systemctl restart containerd
```

{% endtab %}
{% endtabs %}

{% hint style="success" %}
**Success**

The Kubernetes nodes should now all show as `Ready` .
{% endhint %}

3. Verify the status of the node after restarting containerd

{% tabs %}
{% tab title="Node Status" %}

```bash
# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       Ready      master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3
```

{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/containerd-runtime-node-in-notready-state-after-reboot.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.