How to Re-deploy CoreDNS Pods Only to the Master Nodes.

Problem

Re-deploy CoreDNS pods only to the master nodes as part of performance improvement. Process to achieve this by editing Clusteraddon and by using taints.

Environment

• Platform9 Managed Kubernetes - 5.4.0 and Higher.

Procedure

To implement this via the taint and toleration method, the steps to be performed is to set watch flag to false in CoreDNS clusteraddon object and manually edit the CoreDNS deployment to add taints.

CoreDNS ClusterAddon

Edit watch flag to false in clusteraddon of CoreDNS:

# kubectl edit clusteraddons -n default 91bc678c-7da5-41d0-be51-c295cd0a6453-coredns

After editing:

CoreDNS ClusterAddon

# kubectl describe clusteraddons -n default 91bc678c-7da5-41d0-be51-c295cd0a6453-coredns
Name:         91bc678c-7da5-41d0-be51-c295cd0a6453-coredns
Namespace:    default
Labels:       sunpike.pf9.io/cluster=91bc678c-7da5-41d0-be51-c295cd0a6453
              type=coredns
Annotations:  <none>
API Version:  sunpike.platform9.com/v1alpha2
Kind:         ClusterAddon

Spec:
  Cluster ID:  91bc678c-7da5-41d0-be51-c295cd0a6453
  Override:
    Params:
      Name:   dnsMemoryLimit
      Value:  170Mi
      Name:   dnsDomain
      Value:  cluster.local
  Type:       coredns
  Version:    1.8.0
  Watch:      false   < ---------------------

Add toleration to CoreDNS deployment:

CoreDNS Deployment

nodeSelector:                                    
      node-role.kubernetes.io/master: ""            <--------- Add 
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: coredns
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule                            <--------- Add
        key: node-role.kubernetes.io/master           <--------- Add
        operator: Equal                               <--------- Add 
        value: "true"                                 <--------- Add

The CoreDNS pods will be re-deploy to master node/s only:

CoreDNS Pods

# k get pods -A -owide | grep core
kube-system            coredns-84fffcdbd6-52p4j                     1/1     Running       0          10s   10.20.73.133     10.128.147.114   <none>           <none>
kube-system            coredns-84fffcdbd6-shng9                     1/1     Running       0          10s   10.20.73.134     10.128.147.114   <none>           <none>
kube-system            coredns-88d66bcc7-rlxqr                      1/1     Terminating   0          35m   10.20.165.196    10.128.147.204   <none>           <none>

Additional Information

The changes made will not persists across cluster upgrade. The new version CoreDNS yaml post upgrade wipes out the taints and toleration.