Container keeps running on node after Kubernetes pod deletion

Problem

• Container keeps running on the node even after you deleted the pod from the Kubernetes cluster.

• Kubelet logs show that an API call was made to delete the pod but the container associated with that pod keeps running on the node.

• Message logs show it did try to kill the container using SIGKILL but failed to delete the container.

On node - Messages log

2022-10-05T18:42:49-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:49.951251921-07:00" level=info msg="Container d651b04e617c170115e84462b52eb42e8068aae565f0af40777fc304162d1dd0 failed to exit within 2 seconds of signal 15 - using the force"
2022-10-05T18:42:59-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:59.963726137-07:00" level=info msg="Container d651b04e617c failed to exit within 10 seconds of kill - trying direct SIGKILL"

On node - Kubelet log

I1014 11:08:05.510428   45736 kubelet.go:1948] "SyncLoop DELETE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.510569   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"
containerID="docker://7862332cf8f03cff83cc706031102835ed3a84f0086b38f1c526cb49fce04e10" gracePeriod=2
I1014 11:08:05.517822   45736 kubelet.go:1942] "SyncLoop REMOVE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.517897   45736 kubelet_pods.go:1285] "Killing unwanted pod" podName="speaker-4pcxk"
I1014 11:08:05.517944   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"

Environment

• Platform9 Managed Kubernetes - v 5.4

• Docker and linux kernel versions are 19.03.11 and 5.4.0-123-generic respectively.

Version output

# docker version
Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        ID
 Built:             Mon Jun  1 09:12:34 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       ID
  Built:            Mon Jun  1 09:11:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        ID
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.18.0
  GitCommit:        ID

  
#uname -a
Linux 5.4.0-123-generic #138 SMP Sun Aug 7 21:49:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Cause

This is an upstream docker bug on docker 19.03 version, which kept container running in the background.

Upstream bug

https://github.com/moby/moby/issues/41579

Resolution

• Manual deletion of the container can be done on the node.

• Upgrade docker to possible bug fix version 20.10.