How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster
Problem
There is a need to change the configuration of the Kubelet service on all Worker or Master nodes and have the custom configuration(s) persist through cluster upgrades.
Environment
- Platform9 Managed Kubernetes - All Versions
- Kubernetes up to v1.23
- Kubelet
The Dynamic kubelet configuration feature has been completely removed from Kubernetes v1.24. Therefore, the procedure below only works for Kubernetes clusters up to v1.23.
Please contact Platform9 support if you have a similar requirement for Kubernetes clusters v1.24 & higher.
Procedure
In this example, we will edit node-status-update-frequency value on the Worker Nodes.
- Having scoped to the target cluster, Identify the currently set Worker and/or Master Node ConfigMap that is used to deploy the Kubelet configuration on cluster nodes.
$ kubectl get cm -n kube-system | grep -i "worker\|master"master-default-kubelet-config 1 4d4hworker-default-kubelet-config 1 4d4h- Back up the original configmap before making changes. In this example, worker-default-kubelet-config is backed up.
$ kubectl -n kube-system get cm worker-default-kubelet-config -o yaml > worker-default-kubelet-config-bkp.yaml- Stop pf9-hostagent and pf9-nodeletd on worker or masters.
In order for this to happen safely without any intervention from other services which keeps track of the status of the pf9-kubelet service, first stop the pf9-hostagent & pf9-nodeletd services on ALL the worker or master nodes, depending on which configmap is being edited.
# sudo systemctl stop pf9-{hostagent,nodeletd}Example
- Add the
nodeStatusUpdateFrequencyparameter to the worker configmap.
# kubectl -n kube-system edit cm worker-default-kubelet-config# kubectl get configmap worker-default-kubelet-config -n kube-system -o yamlapiVersion: v1data: kubelet: | apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 authentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: "/etc/pf9/kube.d/certs/kubelet/server/ca.crt" authorization: mode: AlwaysAllow clusterDNS: - "10.21.0.10" clusterDomain: "cluster.local" cpuManagerPolicy: "none" topologyManagerPolicy: "none" featureGates: DynamicKubeletConfig: true maxPods: 200 nodeStatusUpdateFrequency: 4s...Once the change is made to the ConfigMap spec, the pf9-kubelet service will be restarted on ALL the worker nodes part of the cluster. The pf9-kubelet service will then start using the new configuration from the set ConfigMap on ALL the worker nodes part of the cluster.
The parameter name to be specified in the configuration spec may differ from its CLI counterpart. In this case, --node-status-update-frequency will have to be specified as nodeStatusUpdateFrequency.
Similar formatting style will have to be done for any other Kubelet parameter that needs to be modified using Dynamic Kubelet Configuration within PMK.
- Start the pf9-hostagent service on all the worker/master nodes where the service was stopped (step #3). This will eventually start the pf9-nodeletd service.
# sudo systemctl start pf9-hostagent- To verify the change has taken effect, check the status of the pf9-kubelet service, and kubelet logs, on one of the cluster's worker nodes.
# systemctl status pf9-kubelet● pf9-kubelet.service - Platform9 Kubelet AgentLoaded: loaded (/run/systemd/system/pf9-kubelet.service; disabled; vendor preset: disabled)Active: active (running) since Tue 2021-12-14 20:43:40 UTC; 57min ago # less kubelet.test02.root.log.INFO.20211214-204015.5780Log file created at: 2021/12/14 20:40:15Running on machine: mav-test02Binary: Built with gc go1.15.15 for linux/amd64Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg...I1214 20:40:15.262886 5780 flags.go:59] FLAG: --node-status-update-frequency="10s"...I1214 20:40:15.268040 5780 fsstore.go:59] kubelet config controller: initializing config checkpoints directory "/var/opt/pf9/kube/kubelet-config/dynamic-config/store"I1214 20:40:15.269564 5780 fsstore.go:116] kubelet config controller: loading Kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/worker-default-kubelet-config, UID: 051ab801-f153-45b0-b09a-9e053777a939, ResourceVersion: 1580080As expected, on restart of the service, first, the default value is loaded, i.e. --node-status-update-frequency="10s" and then the checkpoint referring to the latest configmap is sourced and loaded.
To check if the change got applied or not, you can opt to increase the logging verbosity level for the pf9-kubelet service. Reference: How To Enable Verbose Logging For Kubelet On a Node?
Now looking at the verbose logs, we see the node status is being updated every 4 seconds.
# less kubelet.INFO | grep -i "Updating node status"I1214 20:43:44.315215 12939 kubelet_node_status.go:464] Updating node statusI1214 20:43:48.399309 12939 kubelet_node_status.go:464] Updating node statusI1214 20:43:52.486795 12939 kubelet_node_status.go:464] Updating node statusI1214 20:43:56.570719 12939 kubelet_node_status.go:464] Updating node statusI1214 20:44:00.643556 12939 kubelet_node_status.go:464] Updating node statusI1214 20:44:04.747021 12939 kubelet_node_status.go:464] Updating node statusI1214 20:44:08.819597 12939 kubelet_node_status.go:464] Updating node statusAdditional Information
If there is a need to make changes to the Kubelet configuration on a specific master/worker node, refer How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster