How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster

Problem

How to add additional flags or change values for parameters defined in the set Master/Worker Kubelet configuration file of an individual node by creating a new configuration file and applying the same that will persist through an upgrade on a Platform9 Managed Kubernetes cluster.

Environment

• Platform9 Managed Kubernetes - All Versions

• Kubernetes up to v1.23

• Kubelet

Warning

The Dynamic kubelet configuration feature has been completely removed from Kubernetes v1.24. Therefore, the procedure below only works for Kubernetes clusters up to v1.23.

Please contact Platform9 support, if you have a similar requirement for Kubernetes clusters v1.24 & higher.

Procedure

• Choose a node to modify/reconfigure. As an example, we will consider node 10.128.226.33

Example

$ kubectl get nodes
NAME            STATUS    ROLES   AGE  VERSION
10.128.226.20   NotReady  worker  86d  v1.13.10
10.128.226.33   Ready     worker  86d  v1.13.12
10.128.226.7    Ready     master  86d  v1.13.12

• Start kubectl proxy in the background.

Example

$ kubectl proxy --port=8001 &
[1] 10708
$ Starting to serve on 127.0.0.1:8001
^C

• Download and unpack the choosen node's current configuration.

Example

# curl -sSL "http://localhost:8001/api/v1/nodes/10.128.226.33/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_10.128.226.33

Example:

Limit Number of Pods on a Single Node [--max-pods int32 Default: 110]

• Edit the newly created configuration file _kubelet_configz_10.128.226.33_and change maxPods parameter as required and save the file.

• Create a new ConfigMap resource using the configuration file kubelet_configz_10.128.226.33.

Example

# kubectl get configmap --all-namespaces
NAMESPACE    NAME                                      DATA  AGE
kube-system  10.128.226.33-node-config-new-fmc5kk7484  1      23s

• Once the node is edited with the new ConfigMap, the pf9-kubelet service on the node will be restarted. Once restarted, the pf9-kubelet service will start using the new configuration from the set ConfigMap. But in order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, stop the pf9-hostagent &pf9-nodeletd services first on the node.

sudo systemctl stop pf9-{hostagent,nodeletd}

• Now, edit the node reference to point to the new ConfigMap resource.

Example

# kubectl edit node 10.128.226.33
node/10.128.226.33 edited

Spec Changes

spec:
  configSource:
    configMap:
      kubeletConfigKey: kubelet
      name: 10.128.226.33-node-config-new-fmc5kk7484
      namespace: kube-system

• Start the _pf9-hostagent_service on the master node. This will eventually start the pf9-nodeletd service.

Command

sudo systemctl start pf9-hostagent

• Retrieve the Node configuration and now it should be showing the updated capacity.

None

# kubectl get node 10.128.226.33 -o json | jq '.status.capacity'
{
  "cpu": "8",
  "ephemeral-storage": "20263528Ki",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "16431692Ki",
  "pods": "100"
}

Warning

If there are any issues, inspect the status of pf9-kubelet service on the node and further look at /var/log/pf9/kubelet/kubelet.INFO logs.

Info

If there is a need to make changes to the Kubelet configuration on ALL Master/Worker nodes, we can modify one or both of the set ConfigMaps. Reference: How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster