How To Modify/Add APIServer, ControllerManager & Scheduler Flags on an Existing Cluster
Problem
How To Modify/Add APIServer, ControllerManager & Scheduler Flags on an Existing Cluster?
As an example, we will be looking to modify the default pod eviction timeout "--pod-eviction-timeout" controlled by the kube-controller-manager.
Environment
- Platform9 Managed Kubernetes (PMK) - All Versions
Procedure
The default pod eviction timeout value is 5 minutes. This can be modified using the steps outlined on the master node part of the cluster.
The steps will remain same for any other flag that needs to be changed or added for kube-apiserver, kube-controller-manager & kube-scheduler.
Certain default flag values are already set in PMK's default configuration file. The path for the same is mentioned below.
Steps highlighted below are to be done directly on the master nodes that are part of the cluster and thus post cluster upgrade they will be wiped out.
If there is a need to persist the flag values post cluster upgrade on existing clusters, please contact Platform9 Support.
- Backup the original master.yaml file (PMK's default configuration file) to some other location on the node.
# cp /opt/pf9/pf9-kube/conf/masterconfig/base/centos/master.yaml /root/# cp /opt/pf9/pf9-kube/conf/masterconfig/base/ubuntu/master.yaml /root/- Stop the PMK Stack.
Running this command will drain all pods/containers running on the node.
# systemctl stop pf9-hostagent pf9-nodeletd# /opt/pf9/nodelet/nodeletd phases stop- Add/Modify the required flag and it's corresponding value in this case "--pod-eviction-timeout=7m0s" to the file
_/opt/pf9/pf9-kube/conf/masterconfig/base/centos/master.yaml_in the_kube-controller-manager_container's command section. The file path will be different for Ubuntu OS as mentioned previously.
# vi /opt/pf9/pf9-kube/conf/masterconfig/base/centos/master.yamlPost modification, the kube-controller-manager container section of master.yaml file should look as shown below.
containers: - name: "kube-controller-manager" image: "k8s.gcr.io/kube-controller-manager:__KUBERNETES_VERSION__" command: - "kube-controller-manager" - "--cloud-provider=__CLOUD_PROVIDER__" - "--leader-elect=true" - "--master=127.0.0.1:8080" - "--root-ca-file=/srv/kubernetes/certs/apiserver/ca.crt" - "--service-account-private-key-file=/srv/kubernetes/certs/apiserver/svcacct.key" - "--v=__DEBUG_LEVEL__" - "--horizontal-pod-autoscaler-use-rest-clients=true" - "--pod-eviction-timeout=7m0s"- Start the PMK Stack.
# systemctl start pf9-hostagent# /etc/init.d/pf9-kube startOn a multi-master cluster, ensure that the changes are made to one master node at a time else ETCD will lose quorum and the cluster will be unreachable.
Additional Information
- APIServer: Command Line Tools Reference - Kube-APIServer
- Controller Manager: Command Line Tools Reference - Kube-Controller-Manager
- Scheduler: Command Line Tools Reference - Kube-Scheduler