Kubernetes Node in NotReady State After Reboot for Containerd Runtime Cluster

Problem

• A Kubernetes (master or worker) node that has been rebooted (e.g. due to a maintenance activity) is showing as NotReady.

Node Status

$ kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       NotReady   master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3

• A description of the node similarly reports KubeletNotReady due to the CNI plugin being uninitialized.

Node Description

$ kubectl describe node master1
.....
Conditions:
Ready                False   Mon, 13 Jun 2022 21:42:05 +0000   Mon, 13 Jun 2022 21:32:01 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Environment

• Platform9 Managed Kubernetes - v5.4 and Higher

• Kubernetes - All 1.21 versions except v1.21.3-pmk.183

• Runtime - Containerd

Cause

Due to an upstream issue in containerd, the CNI config is not reloaded when the directory is deleted and recreated during the Platform9 Kubernetes stack initialization.

Resolution

• This issue is now fixed on the pf9-kube-1.21.3-pmk.183. and above releases.

Workaround

1. Verify that the CNI configuration directory referenced by containerd is not empty.

For Flannel based clusters the directory should contain the following files:

Javascript

# ls -l /etc/cni/net.d/
total 4
-rw-r--r-- 1 root root 362 Jun 16 14:45 10-flannel.conflist

For Calico based clusters the directory should contain the following files:

Bash

# ls -l /etc/cni/net.d/
total 8
-rw-r--r-- 1 root root  730 Jun 16 14:16 10-calico.conflist
-rw------- 1 root root 3094 Jun 16 14:16 calico-kubeconfig

1. Restart containerd service on the affected node.

Info

Restarting containerd will not affect restart any running containers

Bash

# sudo systemctl restart containerd

Success

The Kubernetes nodes should now all show as Ready .

1. Verify the status of the node after restarting containerd

Node Status

# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       Ready      master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3