Pause Container Deletion Stuck Due To "WorkloadEndpoint not found" Error In CNI Plugin

Problem

Pause containers are stuck in exited state while the corresponding WorkloadEndpoint has already been deleted. Kubelet repeatedly tries to clean up the resources, however fails with the following errors:

Error deleting <container-name>/<container-ID> from network multus/multus-cni-network: delegateDel: error invoking ConflistDel - "k8s-pod-network": conflistDel: error in getting result from DelNetworkList: WorkloadEndpoint.crd.projectcalico.org "WorkloadEndpoint(<workloadendpoint-name>)" not found

Environment

• Platform9 Managed Kubernetes - v5.1 and Higher
• Calico CNI v3.18.2 and below versions.
• Container Runtime - Docker

Cause

This is a known upstream issue with Calico CNI

Resolution

The latest version of Calico which contains the fix is only available on pf9-kube-1.22 and higher pf9-kube releases which is available with PMK v5.6.

Please reach out to Platform9 Support to know more about the issue.

Additional Information

Use the below commands to manually clean all the exited containers.

# docker rm $(docker ps -a -f status=exited -q)