Pod Stuck in Pending State As Container Fails to Mount Persistent NFS Volumes

Problem

A pod is stuck in a pending state as the container fails to mount the NFS volume. We observe the below error while running kubectl describe command for the affected pod.

None

$ kubectl describe pod <pod-name>
...
Events:  Type     Reason       Age                  From                     Message
         ----     ------       ----                 ----                     -------  
Warning  FailedMount  26m (x15 over 60m)   kubelet, XX.XX.XX.XX  Unable to attach or mount volumes: unmounted volumes=[volume-name], unattached volumes=[volume-name [name]-svc-token-f7227]: timed out waiting for the condition  Warning  FailedMount  111s (x40 over 60m)  kubelet, XX.XX.XX.XX  (combined from similar events): MountVolume.SetUp failed for volume "pvc-uuid" : mount failed: exit status 32Mounting command: systemd-runMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id] --scope -- mount -t nfs XX.XX.XX.XX:/path/to/nfs/storage /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id]Output: Running scope as unit: run-[id].scopemount: /var/lib/kubelet/pods/[id]/volumes/kubernetes.io~nfs/pvc-[id]: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.[type] helper program

Environment

• Platform9 Managed Kubernetes - All Versions

• Persistent Volumes

• NFS

Cause

The package nfs-common is not available on the node which is required to mount the Persistent Volume shared using NFS.

Resolution

Install the package nfs-common on the node on which the affected pod is running. Once the package is installed the pod will be able to mount the volume on the next retry.