Calico Kubeconfig Expires Causing Unauthorized Errors And Pods Creation is Failing

Problem

During a cluster upgrade operation, pods that were drained became stuck in the ContainerCreating state on other worker nodes. Upon describing the pods, the following messages were observed:

Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "2e89d4554a181d4fb6277e83b0ee5433f915c6ebbcfce78a04d405eed3d135aa" network for pod "xxxxx-6dbdb4d58f-f6nwr": networkPlugin cni failed to set up pod "xxxxx-6dbdb4d58f-f6nwr_mm-to-core" network: error getting ClusterInformation: connection is unauthorized: Unauthorized, failed to clean up sandbox container "2e89d4554a181d4fb6277e83b0ee5433f915c6ebbcfce78a04d405eed3d135aa" network for pod "xxxxxx-6dbdb4d58f-f6nwr": networkPlugin cni failed to teardown pod "xxxxxx-6dbdb4d58f-f6nwr_mm-to-core" network: error getting ClusterInformation: connection is unauthorized: Unauthorized

Environment

• Platform9 Managed Kubernetes - v5.6
• Kubernetes version 1.21

Cause

Calico Kubeconfig at /etc/cni/net.d path having an expired token. Token is having validity of 1 year. Here is the upstream bug for this issue.

Resolution

• An internal Jira PMK-5816 has been created and Platform9 Engineering team delivered the fix in version PMK 5.9.1
• As a workaround, perform a rollout restart of the calico node daemon set, which will refresh the expired token if the pods are running from more than 1 year.

# kubectl rollout restart daemonset/calico-node -n kube-system

Additional Information

Upstream fix has been introduced with calico v3.24.