Kubernetes Node in NotReady State After Reboot for Containerd Runtime Cluster
Problem
- A Kubernetes (master or worker) node that has been rebooted (e.g. due to a maintenance activity) is showing as
NotReady
.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady master 34d v1.21.3
master2 Ready master 34d v1.21.3
master3 Ready master 34d v1.21.3
- A description of the node similarly reports
KubeletNotReady
due to the CNI plugin being uninitialized.
$ kubectl describe node master1
.....
Conditions:
Ready False Mon, 13 Jun 2022 21:42:05 +0000 Mon, 13 Jun 2022 21:32:01 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Environment
- Platform9 Managed Kubernetes - v5.4 and Higher
- Kubernetes - All 1.21 versions except v1.21.3-pmk.183
- Runtime - Containerd
Cause
Due to an upstream issue in containerd, the CNI config is not reloaded when the directory is deleted and recreated during the Platform9 Kubernetes stack initialization.
Resolution
- This issue is now fixed on the pf9-kube-1.21.3-pmk.183. and above releases.
Workaround
- Verify that the CNI configuration directory referenced by containerd is not empty.
For Flannel based clusters the directory should contain the following files:
# ls -l /etc/cni/net.d/
total 4
-rw-r--r-- 1 root root 362 Jun 16 14:45 10-flannel.conflist
For Calico based clusters the directory should contain the following files:
# ls -l /etc/cni/net.d/
total 8
-rw-r--r-- 1 root root 730 Jun 16 14:16 10-calico.conflist
-rw------- 1 root root 3094 Jun 16 14:16 calico-kubeconfig
- Restart containerd service on the affected node.
Restarting containerd will not affect restart any running containers
# sudo systemctl restart containerd
The Kubernetes nodes should now all show as Ready
.
- Verify the status of the node after restarting containerd
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 34d v1.21.3
master2 Ready master 34d v1.21.3
master3 Ready master 34d v1.21.3
Was this page helpful?