How To Enable fail-on-swap=false Option as a Parameter Value in Kubelet on a Worker Node

Problem

How to Enable fail-on-swap=false option as a parameter value in Kubelet one a worker node.

Environment

• Platform9 Managed Kubernetes - All Versions

Procedure

Steps to enable swap on a worker node.

1. Start kubectl proxy in the background.

None

root@test-master1:~# kubectl proxy --port=8001 &
[1] 22409
root@test-master1:~# Starting to serve on 127.0.0.1:8001

1. Current Configmap associated with the worker node.

None

root@test-master1:~# kubectl get node 172.20.4.210 -o yaml
...
spec:
configSource:
  configMap:
    kubeletConfigKey: kubelet
name: worker-default-kubelet-config
namespace: kube-system
...

1. Download and unpackage the worker node configuration.

None

root@test-master1:~# curl -sSL "http://localhost:8001/api/v1/nodes/172.20.4.210/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet

1. Edit the config file named kubelet and change the failSwapOn parameter value to false and saved the file.

None

root@test-master1:~# cat kubelet | grep failSwapOn
  "failSwapOn": false,

1. Push the config file by creating a new ConfigMap. In this case, the configmap "new-worker-config-swapon" is created.

None

root@test-master1:~# kubectl create configmap new-worker-config-swapon -n kube-system --from-file=/root/kubelet
configmap/new-worker-config-swapon created

None

root@test-master1:~# kubectl get configmap -n kube-system | grep swap
new-worker-config-swapon            1      18s

1. Edit the worker node reference to point to the new ConfigMap.

None

root@test-master1:~# kubectl edit node 172.20.4.210
node/172.20.4.210 edited

None

root@test-master1:~# kubectl get node 172.20.4.210 -o yaml
...
spec:
configSource:
  configMap:
    kubeletConfigKey: kubelet
name: new-worker-config-swapon
namespace: kube-system
...

1. Post this you will observe that the pf9-kubelet service on the node will be restarted and a new checkpoint will be created in this case ca8caeaa-da7d-4d95-8fbb-6adac5795958 (Kubelet always refers to the newest created checkpoint when setting dynamic configuration). Note that initially when the service restarts you would see kubelet setting the swap value to "true" from its default configuration in kubelet logs but it will then load the newer configmap and set the value to false accordingly.

None

ubuntu@test-worker1:/var/log/pf9/kubelet$ less kubelet.INFO
I0111 21:56:30.148885  22192 fsstore.go:116] kubelet config controller: loading Kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/new-worker-config-swapon,
UID: ca8caeaa-da7d-4d95-8fbb-6adac5795958, ResourceVersion: 9898516

None

root@test-worker1:/var/opt/pf9/kube/kubelet-config/dynamic-config/store/checkpoints# ls -ltr
total 8
drwxr-xr-x 3 pf9 pf9group 4096 Dec 1 23:09 1ed02d48-73e5-448b-8b1e-163a73dac8b0
drwxr-xr-x 3 root root 4096 Jan 11 21:56 ca8caeaa-da7d-4d95-8fbb-6adac5795958

root@juniperlimit-worker1:/var/opt/pf9/kube/kubelet-config/dynamic-config/store/checkpoints/ca8caeaa-da7d-4d95-8fbb-6adac5795958/9898516# cat kubelet | grep -i swap
"failSwapOn": false,

1. In terms of memory allocation, the node's memory resource request/limit allocation will be honored but since the swap is not supported natively the way in which kubelet does the calculations might not be accurate leading to pod evictions due to memory.

Note: If the same change has to be made across all nodes, you can read more about it here.

Additional Information

• Enabling swap on a node in Kubernetes supported starting K8s v1.22. Reference: https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/#node-system-swap-support

• From Platform9's perspective, a feature request PMK8-I-148 has been created to look into ways and options in which we can expose an option to enable the fail-on-swap=false as a feature flag in the cluster creation wizard.