Ctrlk

Pod Deployment is Failing Due to Stale Mount Paths of Orphaned Pods On The Node

Problem

  • Pods are getting stuck in the ContianerCreating state post deployment and below mentioned error is recorded at the same time in kubelet logs.

E0422 18:03:55.666613   44260 kubelet_volumes.go:225] orphaned pod "ca7bcbd3-35f9-4497-a7f5-2e901728c3a3" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/ca7bcbd3-35f9-4497-a7f5-2e901728c3a3/volumes/kubernetes.io~portworx-volume/pvc-50a9d02c-57cd-4141-96d6-d6cdf0182625: operation not permitted : There were a total of 2 errors similar to this. Turn up verbosity to see them.

Environment

  • Platform9 Managed Kubernetes - All Versions

  • Kubelet

Answer

  • The logs explain that there is failure in executing the rmdir() kernel call on the stale mount paths left on the node by the orphaned pods.

  • This is a routine login in the kubelet.volume.go code to clean up the stale mount paths left on the node by orphaned pods.

  • Sometimes these volume paths have a left over file named vol_data.json and due to this the rmdir() kernel call on this mount path fails.

PreviousPersistent Volume is in a "released" State and Failing to Mount Within the PodNextPods Stuck in ContainerCreating with Error: "no space left on device"

Last updated