Container keeps running on node after Kubernetes pod deletion
Problem
- Container keeps running on the node even after you deleted the pod from the Kubernetes cluster.
- Kubelet logs show that an API call was made to delete the pod but the container associated with that pod keeps running on the node.
- Message logs show it did try to kill the container using SIGKILL but failed to delete the container.
2022-10-05T18:42:49-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:49.951251921-07:00" level=info msg="Container d651b04e617c170115e84462b52eb42e8068aae565f0af40777fc304162d1dd0 failed to exit within 2 seconds of signal 15 - using the force"2022-10-05T18:42:59-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:59.963726137-07:00" level=info msg="Container d651b04e617c failed to exit within 10 seconds of kill - trying direct SIGKILL" I1014 11:08:05.510428 45736 kubelet.go:1948] "SyncLoop DELETE" source="api" pods=[metallb-system/speaker-4pcxk]I1014 11:08:05.510569 45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"containerID="docker://7862332cf8f03cff83cc706031102835ed3a84f0086b38f1c526cb49fce04e10" gracePeriod=2I1014 11:08:05.517822 45736 kubelet.go:1942] "SyncLoop REMOVE" source="api" pods=[metallb-system/speaker-4pcxk]I1014 11:08:05.517897 45736 kubelet_pods.go:1285] "Killing unwanted pod" podName="speaker-4pcxk"I1014 11:08:05.517944 45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"Environment
- Platform9 Managed Kubernetes - v 5.4
- Docker and linux kernel versions are
19.03.11and5.4.0-123-genericrespectively.
x
# docker versionClient: Docker Engine - Community Version: 19.03.11 API version: 1.40 Go version: go1.13.10 Git commit: ID Built: Mon Jun 1 09:12:34 2020 OS/Arch: linux/amd64 Experimental: falseServer: Docker Engine - Community Engine: Version: 19.03.11 API version: 1.40 (minimum version 1.12) Go version: go1.13.10 Git commit: ID Built: Mon Jun 1 09:11:07 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: ID runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.18.0 GitCommit: ID #uname -aLinux 5.4.0-123-generic #138 SMP Sun Aug 7 21:49:28 PDT 2022 x86_64 x86_64 x86_64 GNU/LinuxCause
This is an upstream docker bug on docker 19.03 version, which kept container running in the background.
https://github.com/moby/moby/issues/41579Resolution
- Manual deletion of the container can be done on the node.
- Upgrade docker to possible bug fix version 20.10.
Was this page helpful?