How To Disable and Re-enable Dynamic Kubelet Configuration
Problem
If you incorrectly configure the Master node with the wrong configuration you might lose the apiserver container on the node. Because of that kubectl will no longer work and you will not be able to revert your changes.
This article will walk you through the steps to disable the Dynamic Kubelet Config, edit the Node to use the correct config map, then re-enable Dynamic Kubelet Config.
Environment
- Platform9 Managed Kubernetes - All Versions
Procedure
- Disable Dynamic Kubelet Config
- Stop pf9-hostagent and pf9-nodelet
- Make the following 2 changes to the /opt/pf9/pf9-kube/utils.sh file
- On line 506, comment out or delete the
--dynamic-config-dir=${KUBELET_DYNAMIC_CONFIG_DIR}"line. Be sure to correctly end the list, the last few lines should look something this: (The parameters can be different in your utils.sh file, do not make any other edits except removing the above line)
- On line 506, comment out or delete the
504 --register-schedulable=false \
505 --pod-infra-container-image=${pause_img}"
- On line 401, change the following value to false
featureGates:
DynamicKubeletConfig: true
- Run /opt/pf9/nodelet/nodeletd phases stop --force
- Run systemctl start pf9-hostagent.
- Wait for all the phases to start running and ensure that kubectl is functional
- Correct the Master Configuration and Re-enabling Dynamic Kubelet Config
- Run kubectl edit node <Node Name>.
- Add the desired Config map to the node under the spec configSource section and save. Eg
spec:
configSource:
configMap:
kubeletConfigKey: kubelet
name: master-default-kubelet-config -> Change name here.
namespace: kube-system
- Stop pf9-hostagent and pf9-nodelet - Run /opt/pf9/nodelet/nodeletd phases stop --force
- Revert the changes made in first 2 points of Step 1. The parameters will look like this now
504 --register-schedulable=false \ #values on this line may be different in your file
505 --pod-infra-container-image=${pause_img} \ #values on this line may be different in your file
506 --dynamic-config-dir=${KUBELET_DYNAMIC_CONFIG_DIR}"
413 featureGates: #line numbers may vary
414 DynamicKubeletConfig: true #line numbers may vary
- Delete the 2 folders under /var/opt/pf9/kube/kubelet-config/dynamic-config/. This will prevent the Kubelet from picking up the previously checkpointed worker config. - Run systemctl start pf9-hostagent
- Once the pf9-nodelet service brings up the Kubernetes stack you will see that the Master node now has the updated configuration and the cluster is functioning as expected.