Pod Deployment is Failing Due to Stale Mount Paths of Orphaned Pods On The Node
Problem
- Pods are getting stuck in the
ContianerCreatingstate post deployment and below mentioned error is recorded at the same time in kubelet logs.
E0422 18:03:55.666613 44260 kubelet_volumes.go:225] orphaned pod "ca7bcbd3-35f9-4497-a7f5-2e901728c3a3" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/ca7bcbd3-35f9-4497-a7f5-2e901728c3a3/volumes/kubernetes.io~portworx-volume/pvc-50a9d02c-57cd-4141-96d6-d6cdf0182625: operation not permitted : There were a total of 2 errors similar to this. Turn up verbosity to see them.Environment
- Platform9 Managed Kubernetes - All Versions
- Kubelet
Answer
- The logs explain that there is failure in executing the
rmdir()kernel call on the stale mount paths left on the node by the orphaned pods. - This is a routine login in the
kubelet.volume.gocode to clean up the stale mount paths left on the node by orphaned pods.
- Sometimes these volume paths have a left over file named
vol_data.jsonand due to this thermdir()kernel call on this mount path fails.
sudo ls -la /var/lib/kubelet/pods/1d4bfc07-3469-4eaa-992f-6d23c17f3aee/volumes/kubernetes.io~csi/pvc-13c81b28-4038-40d5-b6e8-4194e1d7be0etotal 12drwxr-x--- 2 root root 4096 Oct 7 12:37 .drwxr-x--- 3 root root 4096 Oct 2 19:37 ..-rw-r--r-- 1 root root 270 Oct 7 12:37 vol_data.jsonWas this page helpful?