Pod Stuck in Terminating State Due to PreStop Hook.
Problem
Pod stuck in terminating state, Even the --force
option in kubectl is failing to delete.
Reason:"FailedPreStopHook", Message:"Exec lifecycle hook ([/bin/bash -c service httpd stop; sleep 20]) for Container \"container1\" in Pod \"pod_name3(a8xxx-xxxx-xxx-xxx-xx5e9bb1)\" failed - error: command '/bin/bash -c service httpd stop; sleep 20' exited with 137: , message: \"\\x1b[60G[\\x1b[0;32m OK \\x1b[0;39m]\\rForcibly killing nodemanager which has gone rogue. Please check if any of the Applications are still hanging.
Environment
- Platform9 Managed Kubernetes - v5.4 and Higher
- Platform9 Edge Cloud - v5.3 and Higher
Cause
PreStop hook failure is the reason for the pod to be stuck in termination state, these errors can be seen in the kubelet log.
The preStop lifecycle hook in Kubernetes pods allows for performing a specific action before a pod is terminated. It gives applications running in a pod an opportunity to gracefully shut down and clean up any resources or connections before the pod is terminated.
When a pod receives a termination signal, such as when it is being scaled down or during a rolling update, Kubernetes initiates the termination process. Before the pod is actually terminated, Kubernetes executes the preStop hook, if defined, in each container of the pod:
The preStop hook within the pod:
# kubectl get pod pod_name3 -n ns1 -oyaml
apiVersion: v1
kind: Pod
metadata:
name: pod_name3
namespace: ns1
spec:
containers:
preStop:
exec:
command:
- /bin/bash
- -c
- service stop httpd; sleep 20
Solution
Possible solutions to stop pod being stuck in termination is to identify the reason why the preStop hook commands are failing, and fixing the issue associated with these commands should resolve the issue.
From the above example: Investigate why the httpd
service is failing to stop within the container1
of the pod pod_name3.
Additional Information
For more details, please refer: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks