CoreDNS Pod Replica Gets Continuously Scaled Down by the Deployment-Controller

Problem

• One replica of the CoreDNS pod is getting scaled down continuously by the Deployment-Controller and the Kube-DNS_____Autoscaler_ scaling up the replica at the same time.

Replica Set

NewReplicaSet: coredns-8dd49b79d (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 3m1s (x698 over 2d10h) deployment-controller Scaled down replica set coredns-8dd49b79d to 1

• Kube-DNS_____Autoscaler_ keep scaling the replica back up to 2

Kube-DNSAutoscaler

I0614 15:10:50.259106 1 k8sclient.go:272] Cluster status: SchedulableNodes[6], SchedulableCores[360] I0614 15:10:50.259140 1 k8sclient.go:273] Replicas are not as expected: updating replicas from 1 to 2

Environment

• Platform9 Managed Kubernetes - v5.6.4 and Higher.

Cause

• This is as expected from an AutoScaler perspective because **preventSinglePointFailure** is default set to true.

Autoscaler default value

{ "coresPerReplica": 256, "nodesPerReplica": 16, "min": 1, "max": 10, "preventSinglePointFailure": true, "includeUnschedulableNodes": true }

• The Addon-Operator show the log that it was updating the CoreDNS-Addon continuously.

Addon operator log

{"level":"info","msg":"Applying ConfigMap name: coredns (namespace: kube-system, api: v1)","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"No change from existing state (/v1, Kind=ConfigMap) kube-system/coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Applying Deployment name: coredns (namespace: kube-system, api: apps/v1)","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Update was successful","time":"2023-06-14T15:20:22Z"} ........... {"level":"info","msg":"Installed pkg: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns-1.8.6","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Updating finalizer for addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Adding finalizer for addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Updated addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Ignoring reconcile due to previous status update: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Watch: Addon 1c4f2564-f433-4177-93e2-8d56521b46be-coredns converged after triggering it","time":"2023-06-14T15:20:27Z"}

• Users may also see DNS resolution failures.

Resolution

• This issue has been fixed in PMK v5.9.1. Upgrade the environment to v5.9.1+ to fix the issue.