Kubernetes Dashboard Pod Not Running

Problem

Kubernetes dashboard pod in crashLoopBackOff state:

Javascript

Warning  FailedCreatePodSandBox  29m (x119337 over 2d15h)    kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9fb6f3cafb17ba48554856e9cd9742b3215f566e3f6d8cf91d458b8271c734b3" network for pod "kubernetes-dashboard-bd8dd74bd-lv9g7": networkPlugin cni failed to set up pod "kubernetes-dashboard-bd8dd74bd-lv9g7_kubernetes-dashboard" network: stat /etc/cni/net.d/calico-kubeconfig: no such file or directory

Normal   SandboxChanged          4m26s (x120137 over 2d15h)  kubelet  Pod sandbox changed, it will be killed and re-created.

Also, the clusteraddon associated with the kubernetes dashboard will be in installAddonError state:

Kubernetes Dashboard

/ # kubectl describe clusteraddon 3xxxxxxxxxxxe6267b7d-kubernetes-dashboard
Name:         3xxxxxxb7d-kubernetes-dashboard
Namespace:    default
Labels:       sunpike.pf9.io/cluster=3xxxxxxxxxxxx7b7d
              type=kubernetes-dashboard
...
Spec:
  Cluster ID:  31xxxxxxxxxxxxxxxx7d
  Override:
  Type:     kubernetes-dashboard
  Version:  2.0.3
  Watch:    true
Status:
  Last Checked:  <nil>
  Message:       could not update object (/v1, Kind=Secret) kubernetes-dashboard/kubernetes-dashboard-key-holder: Operation cannot be fulfilled on secrets "kubernetes-dashboard-key-holder": the object has been modified; please apply your changes to the latest version and try again
  Phase:         InstallAddonError
Events:          <none>

Environment

• Platform9 Managed Kubernetes - v5.5 and Higher

Cause

The kubernetes dashboard pod is in crashLoopBackOff state due to the missing /etc/cni/net.d/calico-kubeconfig file in the node.

Resolution

Restart PMK stack on the affected node, this should re-populate the file /etc/cni/net.d/calico-kubeconfig, and recreate the pods and related resources.

The commands shared below will restart PMK stack in the affected node:

Nodeletd Phases restart

sudo systemctl stop pf9-hostagent pf9-nodeletd 
sudo /opt/pf9/nodelet/nodeletd phases stop
sudo /opt/pf9/nodelet/nodeletd phases start
sudo systemctl start pf9-hostagent pf9-nodeletd