Pod Stuck in Terminating State Due to Inability to Clean Volume subPath Mount

Problem

A pod is stuck in a terminating state as the configmap mounted as a volume fails to unmount when trying to clean the subPath mount for it.

• Pod Status

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE podname 0/1 Terminating 0 3h3m 10.200.42.191 10.3.9.109 [none]

• Pod Description

Name: podnameVolumes:config:Type:     ConfigMap (a volume populated by a ConfigMap)Name:     podnameOptional: false

• Kubelet Error

Error: "error cleaning subPath mounts for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b45800b2-1b3f-11ea-928b-005056bcb9b4-config\") pod \"b45800b2-1b3f-11ea-928b-005056bcb9b4\" (UID: \"b45800b2-1b3f-11ea-928b-005056bcb9b4\") : error processing /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem:error cleaning subpath mount /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: remove /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: device or resource busy"

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

In this case, the /var/lib/kubelet directory has a symlink associated to it.

$ ls -l /var/lib/kubelet lrwxrwxrwx. 1 root root XX XXX X YY:YY /var/lib/kubelet -> /mnt/docker-images/kubelet

Resolution

Temporary Quick Fix

1. Manually unmount the volume.

# umount /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7

2. Manually force delete the Pod.

$ kubectl delete pod --grace-period=0 --force [pod_name]

Long Term Solution

This involves making a change to the existing setup. On every host where the symlink persists, perform the following operations.

1. Stop pf9-hostagent service.

$ systemctl stop pf9-hostagent

2. Stop pf9-kube service.

$ systemctl stop pf9-kube

3. Drop the symlink for /var/lib/kubelet.

$ rm -rf /var/lib/kubelet

4. Start pf9-hostagent service.

$ systemctl start pf9-hostagent

Additional Information

Note: The bug Issue 65879 talks about setting the Kubelet flag --root-dir to the value referenced by symlink from /var/lib/kubelet to resolve the issue. If the --root-dir value has to be set, it would have to be hard coded as --root-dir isn't an option available for dynamic Kubelet config.