# How to Enable Graceful Node Shutdown?

## Problem

* How to Enable Graceful Node Shutdown in a PMK cluster?

## Environment

* Platform9 Managed Kubernetes - v5.2 Onward
* Kubernetes - v1.20 and Above

## Procedure

* For enabling Graceful Node Shutdown feature, there are changes that needs to be made to the node's Kubelet configuration.
* For a cluster with Kubernetes version 1.20, it is required to explicitly enable the `GracefulNodeShutdown` feature under the featureGates (as it is a alpha feature). For Kubernetes v1.21 onward, it is enabled by default (as it is a beta feature).
* To enable the `GracefulNodeShutdown` under the featureGates, edit the required master/worker `configmap` . Along with the feature gate, there is a requirement to set values `ShutdownGracePeriod` & `ShutdownGracePeriodCriticalPods` (by default are set to zero).

{% hint style="warning" %}
**Warning**

In order for this to happen safely without any intervention from other services which keeps track of the status of the *pf9-kubelet* service, first stop the *pf9-hostagent* & *pf9-nodeletd* services on **ALL** the worker/master nodes depending on which configmap is being edited.
{% endhint %}

{% tabs %}
{% tab title="None" %}

```none
sudo systemctl stop pf9-{hostagent,nodeletd}
```

{% endtab %}
{% endtabs %}

* There are 2 different `default-kubelet-config` configmaps under the namespace `kube-system`, one for master nodes `master-default-kubelet-config` and other for worker nodes `worker-default-kubelet-config.`
* Edit appropriate configmap according to the requirement. e.g. in this case we are changing the worker configmap.

{% tabs %}
{% tab title="Enable GracefulNodeShutdown under the featureGates" %}

```javascript
# kubectl edit configmap worker-default-kubelet-config -n kube-system
configmap/worker-default-kubelet-config edited
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example" %}

```none
# kubectl get configmap worker-default-kubelet-config -n kube-system -o yaml
apiVersion: v1
data:
kubelet: |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
    featureGates:
      DynamicKubeletConfig: true
      GracefulNodeShutdown: true
    maxPods: 200
    shutdownGracePeriod: 30s
    shutdownGracePeriodCriticalPods: 10s
```

{% endtab %}
{% endtabs %}

* Start the *pf9-hostagent* service on all the worker/master nodes where the service was stopped initially. This will eventually start the *pf9-nodeletd* service.

{% tabs %}
{% tab title="Command" %}

```none
sudo systemctl start pf9-hostagent
```

{% endtab %}
{% endtabs %}

* Verify if the GracefulNodeShutdown feature has been enabled or not from the worker node post editing the configmap.

{% tabs %}
{% tab title="Example" %}

```none
# less /var/log/pf9/kubelet/kubelet.INFO | grep "feature gates"
I1208 22:01:52.214565   22305 feature_gate.go:243] feature gates: &{map[]}
I1208 22:01:52.217698   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.217779   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true]}
I1208 22:01:52.220247   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.220355   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243430   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
I1208 22:01:52.243550   22305 feature_gate.go:243] feature gates: &{map[DynamicKubeletConfig:true GracefulNodeShutdown:true]}
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Verifying Enablement of GracefulNodeShutdown from Worker Node" %}

```javascript
[root@worker01 ~]# systemd-inhibit --list
Who: kubelet (UID 0/root, PID 22305/kubelet)
What: shutdown
Why: Kubelet needs time to handle node shutdown
Mode: delay

1 inhibitors listed.
```

{% endtab %}
{% endtabs %}

## Additional Information

{% hint style="info" %}
**Info**

[Graceful Node Shutdown](https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/)

[How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster](https://platform9.com/kb/kubernetes/change-configuration-for-kubelet-service)

[How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster](https://platform9.com/kb/kubernetes/single-node-change-config-kubelet-service)
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pmk/how-to/how-to-enable-graceful-node-shutdown.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.