Pod Creation Failing with Multus pod Error

Problem

The ip-reconciler pods fail to start, resulting in unauthorized errors due to expired Multus service account tokens on older worker nodes and the inability to create new pods in this node.

Logs during a pod creation

Warning  FailedCreatePodSandBox  5s                kubelet                  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a381d9238e3a43834dbf74faad8951dbce14c11f57211c3a978d23359534566e": plugin type="multus" name="multus-cni-network" failed (add): Multus: [default/task-pv-pod]: error getting pod: Unauthorized

Environment

• Platform9 Managed Kubernetes 5.9 and Higher.

• Multus 0.3.1.

Cause

The Multus service account token on the nodes has expired, which caused the unauthorized errors.

New worker nodes receive a valid token upon joining the cluster, so the new nodes are unaffected.

Workaround

To resolve the issue, restart the Multus pod on the affected nodes. This forces Kubernetes to assign a new token.

Steps to Fix the Issue:

On an affected old worker node, check the Multus token expiry:

Master node

# cat /etc/cni/net.d/multus.d/multus.kubeconfig | grep token
# echo "<TOKEN_VALUE>" | cut -d "." -f2 | base64 -d | jq
# date -d @<exp_value>

If the token is expired, proceed with restarting the Multus pods. This should renew the Multus token.

Additional Information

Multus bug reported: https://github.com/k8snetworkplumbingwg/multus-cni/issues/852

There is no fix-version shared for this issue from Multus community. Once the fix is available Platform9 team will add the same in the upcoming PMK release. Hence, the ETA is not available at the moment.