Kubelet logs UnmountVolume.NewUnmounter Failure logs On vol_data.json Is Deleted

Problem

Kubelet continuously logs below logs on failing to unmount a volume from a pod and filling up on disk usage.

Kubelet logs

Error reconciler.go:193] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_UUID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : UnmountVolume.NewUnmounter failed for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_ID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json]: open /var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume \"filestore\" (UniqueName: \"kubernetes.io/csi/csi.trident.netapp.io^pvc-<PVC_ID>\") pod \"<POD_ID>\" (UID: \"<POD_ID>\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json]: open /var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/pvc-<PVC_ID>/vol_data.json: no such file or directory"

Environment

• Platform9 Managed Kubernetes - v5.3 and Higher

Cause

This is a known upstream bug https://github.com/kubernetes/kubernetes/issues/101911 and https://github.com/kubernetes/kubernetes/pull/102576

Workaround

• Find the path of the pod mentioned in logs

Example

# ll /var/lib/kubelet/pods/<POD_ID>/
drwxr-x---. 3 root root 37 May 20 14:53 plugins
drwxr-x---. 5 root root 87 May 20 14:53 volumes

• Check if the above pod running in the cluster

Command

# kubectl get pods  -o custom-columns=PodName:.metadata.name,PodUID:.metadata.uid -A | egrep "<POD_ID>"
#

• Remove the content of those pod directories and restart pf9-kubelet

# rmd -Rf  /var/lib/kubelet/pods/<POD_ID>
# systemctl restart pf9-kubelet