# Container keeps running on node after Kubernetes pod deletion

## Problem

* Container keeps running on the node even after you deleted the pod from the Kubernetes cluster.
* Kubelet logs show that an API call was made to delete the pod but the container associated with that pod keeps running on the node.
* Message logs show it did try to kill the container using SIGKILL but failed to delete the container.

{% tabs %}
{% tab title="On node - Messages log" %}

```javascript
2022-10-05T18:42:49-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:49.951251921-07:00" level=info msg="Container d651b04e617c170115e84462b52eb42e8068aae565f0af40777fc304162d1dd0 failed to exit within 2 seconds of signal 15 - using the force"
2022-10-05T18:42:59-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:59.963726137-07:00" level=info msg="Container d651b04e617c failed to exit within 10 seconds of kill - trying direct SIGKILL"
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="On node - Kubelet log" %}

```javascript
I1014 11:08:05.510428   45736 kubelet.go:1948] "SyncLoop DELETE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.510569   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"
containerID="docker://7862332cf8f03cff83cc706031102835ed3a84f0086b38f1c526cb49fce04e10" gracePeriod=2
I1014 11:08:05.517822   45736 kubelet.go:1942] "SyncLoop REMOVE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.517897   45736 kubelet_pods.go:1285] "Killing unwanted pod" podName="speaker-4pcxk"
I1014 11:08:05.517944   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Managed Kubernetes - v 5.4
* Docker and linux kernel versions are `19.03.11` and `5.4.0-123-generic` respectively.

{% tabs %}
{% tab title="Version output" %}

```javascript
# docker version
Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        ID
 Built:             Mon Jun  1 09:12:34 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       ID
  Built:            Mon Jun  1 09:11:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        ID
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.18.0
  GitCommit:        ID

  
#uname -a
Linux 5.4.0-123-generic #138 SMP Sun Aug 7 21:49:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
```

{% endtab %}
{% endtabs %}

## Cause

This is an upstream docker bug on docker 19.03 version, which kept container running in the background.

{% tabs %}
{% tab title="Upstream bug" %}

```javascript
https://github.com/moby/moby/issues/41579
```

{% endtab %}
{% endtabs %}

## Resolution

* Manual deletion of the container can be done on the node.
* Upgrade docker to possible bug fix version 20.10.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pmk/frequently-asked-questions/container-keeps-running-on-node-after-kubernetes-pod-deletion.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.