Graceful Node ShutDown is Not Working.

Problem

  • Graceful NodeShutdown is not working for the PMK cluster nodes.

  • Pods on the cluster node gets stuck in the Terminating state after unexpected shutdown even after graceful shutdown is configured.

Environment

  • Platform9 Managed Kubernetes – All Versions

  • Cluster Node OS: RHEL7, CentOS7

  • Cluster Node systemd version earlier than systemd 251

Cause

  • There is an upstream issuearrow-up-right with the systemd where the systemd inhibitors are not honoured by the systemd.

  • Due to this the cluster nodes never shutdown gracefully, instead they shutdown immediately as soon as the underlying system shutdown event has occurred.

  • This further results in failure of pod/container graceful shutdown and the pods remain in the terminating state even after configuration of graceful node shutdown.

Resolution

  • Upgrade the systemd version on the affected nodes to systemd 251 or beyond that.

Last updated