How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster

Problem

There is a need to change the configuration of the Kubelet service on all Worker or Master nodes and have the custom configuration(s) persist through cluster upgrades.

Environment

• Platform9 Managed Kubernetes - All Versions

• Kubernetes up to v1.23

• Kubelet

Warning

The Dynamic kubelet configuration feature has been completely removed from Kubernetes v1.24. Therefore, the procedure below only works for Kubernetes clusters up to v1.23.

Please contact Platform9 support if you have a similar requirement for Kubernetes clusters v1.24 & higher.

Procedure

In this example, we will edit node-status-update-frequency value on the Worker Nodes.

1. Having scoped to the target cluster, Identify the currently set Worker and/or Master Node ConfigMap that is used to deploy the Kubelet configuration on cluster nodes.

Set ConfigMap Output

$ kubectl get cm -n kube-system | grep -i "worker\|master"
master-default-kubelet-config 1 4d4h
worker-default-kubelet-config 1 4d4h

1. Back up the original configmap before making changes. In this example, worker-default-kubelet-config is backed up.

Backup Current ConfigMap

$ kubectl -n kube-system get cm worker-default-kubelet-config -o yaml > worker-default-kubelet-config-bkp.yaml

1. Stop pf9-hostagent and pf9-nodeletd on worker or masters.

Warning

In order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, first stop the pf9-hostagent & pf9-nodeletd services on ALL the worker or master nodes, depending on which configmap is being edited.

Command

# sudo systemctl stop pf9-{hostagent,nodeletd}

Example

1. Add the nodeStatusUpdateFrequency parameter to the worker configmap.

ConfigMap Edit

# kubectl -n kube-system edit cm worker-default-kubelet-config

ConfigMap Output Post Modification

# kubectl get configmap worker-default-kubelet-config -n kube-system -o yaml
apiVersion: v1
data:
  kubelet: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    address: 0.0.0.0
    authentication:
      anonymous:
        enabled: false
      webhook:
        enabled: true
      x509:
        clientCAFile: "/etc/pf9/kube.d/certs/kubelet/server/ca.crt"
    authorization:
      mode: AlwaysAllow
    clusterDNS:
    - "10.21.0.10"
    clusterDomain: "cluster.local"
    cpuManagerPolicy: "none"
    topologyManagerPolicy: "none"
    featureGates:
      DynamicKubeletConfig: true
    maxPods: 200
    nodeStatusUpdateFrequency: 4s
...

Once the change is made to the ConfigMap spec, the pf9-kubelet service will be restarted on ALL the worker nodes part of the cluster. The pf9-kubelet service will then start using the new configuration from the set ConfigMap on ALL the worker nodes part of the cluster.

Warning

The parameter name to be specified in the configuration spec may differ from its CLI counterpart. In this case, --node-status-update-frequency will have to be specified as nodeStatusUpdateFrequency.

Similar formatting style will have to be done for any other Kubelet parameter that needs to be modified using Dynamic Kubelet Configuration within PMK.

1. Start the pf9-hostagent service on all the worker/master nodes where the service was stopped (step #3). This will eventually start the pf9-nodeletd service.

Command

# sudo systemctl start pf9-hostagent

1. To verify the change has taken effect, check the status of the pf9-kubelet service, and kubelet logs, on one of the cluster's worker nodes.

Example Kubelet Service Status

# systemctl status pf9-kubelet
● pf9-kubelet.service - Platform9 Kubelet Agent
Loaded: loaded (/run/systemd/system/pf9-kubelet.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2021-12-14 20:43:40 UTC; 57min ago

Example Kubelet Log

# less kubelet.test02.root.log.INFO.20211214-204015.5780
Log file created at: 2021/12/14 20:40:15
Running on machine: mav-test02
Binary: Built with gc go1.15.15 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
...
I1214 20:40:15.262886 5780 flags.go:59] FLAG: --node-status-update-frequency="10s"
...
I1214 20:40:15.268040 5780 fsstore.go:59] kubelet config controller: initializing config checkpoints directory "/var/opt/pf9/kube/kubelet-config/dynamic-config/store"
I1214 20:40:15.269564 5780 fsstore.go:116] kubelet config controller: loading Kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/worker-default-kubelet-config, UID: 051ab801-f153-45b0-b09a-9e053777a939, ResourceVersion: 1580080

As expected, on restart of the service, first, the default value is loaded, i.e. --node-status-update-frequency="10s" and then the checkpoint referring to the latest configmap is sourced and loaded.

---

To check if the change got applied or not, you can opt to increase the logging verbosity level for the pf9-kubelet service. Reference: How To Enable Verbose Logging For Kubelet On a Node?

Now looking at the verbose logs, we see the node status is being updated every 4 seconds.

Example: /var/log/pf9/kubelet/kubelet.INFO

# less kubelet.INFO | grep -i "Updating node status"
I1214 20:43:44.315215 12939 kubelet_node_status.go:464] Updating node status
I1214 20:43:48.399309 12939 kubelet_node_status.go:464] Updating node status
I1214 20:43:52.486795 12939 kubelet_node_status.go:464] Updating node status
I1214 20:43:56.570719 12939 kubelet_node_status.go:464] Updating node status
I1214 20:44:00.643556 12939 kubelet_node_status.go:464] Updating node status
I1214 20:44:04.747021 12939 kubelet_node_status.go:464] Updating node status
I1214 20:44:08.819597 12939 kubelet_node_status.go:464] Updating node status

Additional Information

Info

Kubelet Options

If there is a need to make changes to the Kubelet configuration on a specific master/worker node, refer How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster