How to Enable Graceful Node Shutdown?

Problem

• How to Enable Graceful Node Shutdown in a PMK cluster?

Environment

• Platform9 Managed Kubernetes - v5.2 Onward

• Kubernetes - v1.20 and Above

Procedure

• For enabling Graceful Node Shutdown feature, there are changes that needs to be made to the node's Kubelet configuration.

• For a cluster with Kubernetes version 1.20, it is required to explicitly enable the GracefulNodeShutdown feature under the featureGates (as it is a alpha feature). For Kubernetes v1.21 onward, it is enabled by default (as it is a beta feature).

• To enable the GracefulNodeShutdown under the featureGates, edit the required master/worker configmap . Along with the feature gate, there is a requirement to set values ShutdownGracePeriod & ShutdownGracePeriodCriticalPods (by default are set to zero).

Warning

In order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, first stop the pf9-hostagent & pf9-nodeletd services on ALL the worker/master nodes depending on which configmap is being edited.

None

sudo systemctl stop pf9-{hostagent,nodeletd}

• There are 2 different default-kubelet-config configmaps under the namespace kube-system, one for master nodes master-default-kubelet-config and other for worker nodes worker-default-kubelet-config.

• Edit appropriate configmap according to the requirement. e.g. in this case we are changing the worker configmap.

Enable GracefulNodeShutdown under the featureGates

# kubectl edit configmap worker-default-kubelet-config -n kube-system
configmap/worker-default-kubelet-config edited

Example

# kubectl get configmap worker-default-kubelet-config -n kube-system -o yaml
apiVersion: v1
data:
kubelet: |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
    featureGates:
      DynamicKubeletConfig: true
      GracefulNodeShutdown: true
    maxPods: 200
    shutdownGracePeriod: 30s
    shutdownGracePeriodCriticalPods: 10s

• Start the pf9-hostagent service on all the worker/master nodes where the service was stopped initially. This will eventually start the pf9-nodeletd service.

Command

sudo systemctl start pf9-hostagent

• Verify if the GracefulNodeShutdown feature has been enabled or not from the worker node post editing the configmap.

Example

# less /var/log/pf9/kubelet/kubelet.INFO | grep "feature gates"
I1208 22:01:52.214565   22305 feature_gate.go:243] feature gates: &{map[]}
I1208 22:01:52.217698   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.217779   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.220247   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.220355   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243430   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243550   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}

Verifying Enablement of GracefulNodeShutdown from Worker Node

[root@worker01 ~]# systemd-inhibit --list
Who: kubelet (UID 0/root, PID 22305/kubelet)
What: shutdown
Why: Kubelet needs time to handle node shutdown
Mode: delay

1 inhibitors listed.

Additional Information

Info

Graceful Node Shutdown

How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster

How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster