> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmk/solutions/pod-stuck-in-terminating-state-due-to-inability-to-clean-volume.md).

# Pod Stuck in Terminating State Due to Inability to Clean Volume subPath Mount

## Problem

A pod is stuck in a **terminating** state as the *configmap* mounted as a volume fails to unmount when trying to clean the subPath mount for it.

* Pod Status

{% tabs %}
{% tab title="None" %}

```none
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE podname 0/1 Terminating 0 3h3m 10.200.42.191 10.3.9.109 [none]
```

{% endtab %}
{% endtabs %}

* Pod Description

{% tabs %}
{% tab title="None" %}

```none
Name: podnameVolumes:config:Type:     ConfigMap (a volume populated by a ConfigMap)Name:     podnameOptional: false
```

{% endtab %}
{% endtabs %}

* Kubelet Error

{% tabs %}
{% tab title="None" %}

```none
Error: "error cleaning subPath mounts for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b45800b2-1b3f-11ea-928b-005056bcb9b4-config\") pod \"b45800b2-1b3f-11ea-928b-005056bcb9b4\" (UID: \"b45800b2-1b3f-11ea-928b-005056bcb9b4\") : error processing /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem:error cleaning subpath mount /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: remove /var/lib/kubelet/pods/b45800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7: device or resource busy"
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Managed Kubernetes - All Versions

## Cause

In this case, the */var/lib/kubelet* directory has a symlink associated to it.

{% tabs %}
{% tab title="None" %}

```none
$ ls -l /var/lib/kubelet lrwxrwxrwx. 1 root root XX XXX X YY:YY /var/lib/kubelet -> /mnt/docker-images/kubelet
```

{% endtab %}
{% endtabs %}

## Resolution

Temporary Quick Fix

1. Manually unmount the volume.

{% tabs %}
{% tab title="None" %}

```none
# umount /var/lib/kubelet/pods/b35800b2-1b3f-11ea-928b-005056bcb9b4/volume-subpaths/config/init-filesystem/7
```

{% endtab %}
{% endtabs %}

2. Manually force delete the Pod.

{% tabs %}
{% tab title="None" %}

```none
$ kubectl delete pod --grace-period=0 --force [pod_name]
```

{% endtab %}
{% endtabs %}

Long Term Solution

This involves making a change to the existing setup. On every host where the symlink persists, perform the following operations.

1. Stop ***pf9-hostagent*** service.

{% tabs %}
{% tab title="None" %}

```none
$ systemctl stop pf9-hostagent
```

{% endtab %}
{% endtabs %}

2. Stop ***pf9-kube*** service.

{% tabs %}
{% tab title="None" %}

```none
$ systemctl stop pf9-kube
```

{% endtab %}
{% endtabs %}

3. Drop the symlink for */var/lib/kubelet.*

{% tabs %}
{% tab title="None" %}

```none
$ rm -rf /var/lib/kubelet
```

{% endtab %}
{% endtabs %}

4. Start ***pf9-hostagent*** service.

{% tabs %}
{% tab title="None" %}

```none
$ systemctl start pf9-hostagent
```

{% endtab %}
{% endtabs %}

## Additional Information

**Note:** The bug [Issue 65879](https://github.com/kubernetes/kubernetes/issues/65879#issuecomment-404881242) talks about setting the Kubelet flag **`--root-dir`** to the value referenced by symlink from */var/lib/kubelet* to resolve the issue. If the **`--root-dir`** value has to be set, it would have to be hard coded as **`--root-dir`** isn't an option available for dynamic Kubelet config.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/pod-stuck-in-terminating-state-due-to-inability-to-clean-volume.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.