How To Disable and Re-enable Dynamic Kubelet Configuration

Problem

If you incorrectly configure the Master node with the wrong configuration you might lose the apiserver container on the node. Because of that kubectl will no longer work and you will not be able to revert your changes.

This article will walk you through the steps to disable the Dynamic Kubelet Config, edit the Node to use the correct config map, then re-enable Dynamic Kubelet Config.

Environment

• Platform9 Managed Kubernetes - All Versions

Procedure

• Disable Dynamic Kubelet Config

  • Stop pf9-hostagent and pf9-nodelet
  • Make the following 2 changes to the /opt/pf9/pf9-kube/utils.sh file

    • On line 506, comment out or delete the --dynamic-config-dir=${KUBELET_DYNAMIC_CONFIG_DIR}" line. Be sure to correctly end the list, the last few lines should look something this: (The parameters can be different in your utils.sh file, do not make any other edits except removing the above line)

504 --register-schedulable=false \

505 --pod-infra-container-image=${pause_img}"

    - On line 401, change the following value to false

featureGates:

DynamicKubeletConfig: true

- Run `/opt/pf9/nodelet/nodeletd phases stop --force`

- Run `systemctl start pf9-hostagent.`
- Wait for all the phases to start running and ensure that kubectl is functional

• Correct the Master Configuration and Re-enabling Dynamic Kubelet Config

  • Run kubectl edit node .
  • Add the desired Config map to the node under the spec configSource section and save. Eg

spec:

configSource:

configMap:

kubeletConfigKey: kubelet

name: master-default-kubelet-config -> Change name here.

namespace: kube-system

- Stop pf9-hostagent and pf9-nodelet
- Run /opt/pf9/nodelet/nodeletd phases stop --force

- Revert the changes made in first 2 points of Step 1. The parameters will look like this now 

504 --register-schedulable=false \ #values on this line may be different in your file

505 --pod-infra-container-image=${pause_img} \ #values on this line may be different in your file

506 --dynamic-config-dir=${KUBELET_DYNAMIC_CONFIG_DIR}"

413 featureGates: #line numbers may vary

414 DynamicKubeletConfig: true #line numbers may vary

- Delete the 2 folders under /var/opt/pf9/kube/kubelet-config/dynamic-config/. This will prevent the Kubelet from picking up the previously checkpointed worker config. - Run systemctl start pf9-hostagent

- Once the pf9-nodelet service brings up the Kubernetes stack you will see that the Master node now has the updated configuration and the cluster is functioning as expected.

Additional Information

• Dynamic Kubelet Configuration
• Reconfigure a Node's Kubelet in a Live Cluster