Kubernetes Master Node in NotReady State With Message "cni plugin not initialized"

Problem

• A Kubernetes master node is showing as NotReady and the describe output for the node is showing "cni not initialized".

Node Status

$ kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       NotReady   master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3

• A description of the node similarly reports KubeletNotReady due to the CNI plugin being uninitialized.

Node Description

$ kubectl describe node master1
.....
Conditions:
Ready                False   Mon, 13 Jun 2022 21:42:05 +0000   Mon, 13 Jun 2022 21:32:01 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Environment

• Platform9 Managed Kubernetes - v5.4 and Higher
• Kubernetes - All v1.21
• Runtime - Containerd
• Container Network Interface - Calico

Cause

Due to an bug in the Platform9 Managed Kubernetes Stack the CNI config is not reloaded when a partial restart of the stack takes place.

Resolution

This issue is now fixed on the pf9-kube- 1.23.8-pmk.218 and above releases.

Workaround

1. Verify that the CNI configuration directory referenced by containerd is not empty on the affected node.

For Calico based clusters the directory should contain the following files:

Bash

# ls -l /etc/cni/net.d/
total 8
-rw-r--r-- 1 root root  730 Jun 16 14:16 10-calico.conflist
-rw------- 1 root root 3094 Jun 16 14:16 calico-kubeconfig

1. If the files mentioned above are missing, restart calico-node pod on the affected node.

Bash

# kubectl get pods -o wide -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE    IP               NODE             NOMINATED NODE   READINESS GATES
calico-node-8brm4                         1/1     Running   0          3d3h   10.128.142.117   master1   <none>           <none>
calico-node-xwgn4                         1/1     Running   0          3d3h   10.128.141.109   master2   <none>           <none>
calico-node-bgmi3                         1/1     Running   0          3d3h   10.128.141.109   master3   <none>           <none>


# kubectl delete pod calico-node-8brm4 -n kube-system

Success

The Kubernetes nodes should now all show as Ready .

1. Verify the status of the node after restarting calico-node pod.

Node Status

# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1       Ready      master   34d   v1.21.3
master2       Ready      master   34d   v1.21.3
master3       Ready      master   34d   v1.21.3