Pod Stuck in Terminating State Due to PreStop Hook.

Problem

Pod stuck in terminating state, Even the --force option in kubectl is failing to delete.

Kubelet log

Reason:"FailedPreStopHook", Message:"Exec lifecycle hook ([/bin/bash -c service httpd stop; sleep 20]) for Container \"container1\" in Pod \"pod_name3(a8xxx-xxxx-xxx-xxx-xx5e9bb1)\" failed - error: command '/bin/bash -c service httpd stop; sleep 20' exited with 137: , message: \"\\x1b[60G[\\x1b[0;32m  OK  \\x1b[0;39m]\\rForcibly killing nodemanager which has gone rogue. Please check if any of the Applications are still hanging.

Environment

• Platform9 Managed Kubernetes - v5.4 and Higher

• Platform9 Edge Cloud - v5.3 and Higher

Cause

PreStop hook failure is the reason for the pod to be stuck in termination state, these errors can be seen in the kubelet log.

The preStop lifecycle hook in Kubernetes pods allows for performing a specific action before a pod is terminated. It gives applications running in a pod an opportunity to gracefully shut down and clean up any resources or connections before the pod is terminated.

When a pod receives a termination signal, such as when it is being scaled down or during a rolling update, Kubernetes initiates the termination process. Before the pod is actually terminated, Kubernetes executes the preStop hook, if defined, in each container of the pod:

The preStop hook within the pod:

Pod manifest

# kubectl get pod pod_name3 -n ns1 -oyaml
apiVersion: v1
kind: Pod
metadata:
  name: pod_name3
  namespace: ns1
spec:
  containers:  
    preStop:                                                              
        exec:
          command:
          - /bin/bash
          - -c
          - service stop httpd; sleep 20

Solution

Possible solutions to stop pod being stuck in termination is to identify the reason why the preStop hook commands are failing, and fixing the issue associated with these commands should resolve the issue.

From the above example: Investigate why the httpd service is failing to stop within the container1 of the pod pod_name3.

Additional Information

For more details, please refer: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks