How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster

Problem

How to add additional flags or change values for parameters defined in the set Master/Worker Kubelet configuration file of an individual node by creating a new configuration file and applying the same that will persist through an upgrade on a Platform9 Managed Kubernetes cluster.

Environment

• Platform9 Managed Kubernetes - All Versions
• Kubernetes up to v1.23
• Kubelet

Procedure

• Choose a node to modify/reconfigure. As an example, we will consider node 10.128.226.33

$ kubectl get nodes

NAME            STATUS    ROLES   AGE  VERSION

10.128.226.20   NotReady  worker  86d  v1.13.10

10.128.226.33   Ready     worker  86d  v1.13.12

10.128.226.7    Ready     master  86d  v1.13.12

• Start kubectl proxy in the background.

$ kubectl proxy --port=8001 &

[1] 10708

$ Starting to serve on 127.0.0.1:8001

^C

• Download and unpack the choosen node's current configuration.

# curl -sSL "http://localhost:8001/api/v1/nodes/10.128.226.33/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_10.128.226.33

Example:

Limit Number of Pods on a Single Node [--max-pods int32 Default: 110]

• Edit the newly created configuration file kubelet_configz_10.128.226.33_and change _maxPods parameter as required and save the file.
• Create a new ConfigMap resource using the configuration file kubelet_configz_10.128.226.33.

# kubectl get configmap --all-namespaces

NAMESPACE    NAME                                      DATA  AGE

kube-system  10.128.226.33-node-config-new-fmc5kk7484  1      23s

• Once the node is edited with the new ConfigMap, the pf9-kubelet service on the node will be restarted. Once restarted, the pf9-kubelet service will start using the new configuration from the set ConfigMap. But in order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, stop the pf9-hostagent &pf9-nodeletd services first on the node.

sudo systemctl stop pf9-{hostagent,nodeletd}

• Now, edit the node reference to point to the new ConfigMap resource.

# kubectl edit node 10.128.226.33

node/10.128.226.33 edited

spec:

  configSource:

    configMap:

      kubeletConfigKey: kubelet

      name: 10.128.226.33-node-config-new-fmc5kk7484

      namespace: kube-system

• Start the pf9-hostagent_service on the master node. This will eventually start the _pf9-nodeletd service.

sudo systemctl start pf9-hostagent

• Retrieve the Node configuration and now it should be showing the updated capacity.

# kubectl get node 10.128.226.33 -o json | jq '.status.capacity'

{

  "cpu": "8",

  "ephemeral-storage": "20263528Ki",

  "hugepages-1Gi": "0",

  "hugepages-2Mi": "0",

  "memory": "16431692Ki",

  "pods": "100"

}