CoreDNS Pod Replica Gets Continuously Scaled Down by the Deployment-Controller
Problem
- One replica of the CoreDNS pod is getting scaled down continuously by the Deployment-Controller and the Kube-DNS__Autoscaler scaling up the replica at the same time.
NewReplicaSet: coredns-8dd49b79d (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 3m1s (x698 over 2d10h) deployment-controller Scaled down replica set coredns-8dd49b79d to 1- Kube-DNS__Autoscaler keep scaling the replica back up to 2
I0614 15:10:50.259106 1 k8sclient.go:272] Cluster status: SchedulableNodes[6], SchedulableCores[360] I0614 15:10:50.259140 1 k8sclient.go:273] Replicas are not as expected: updating replicas from 1 to 2Environment
- Platform9 Managed Kubernetes - v5.6.4 and Higher.
Cause
- This is as expected from an AutoScaler perspective because
**preventSinglePointFailure**is default set to true.
{ "coresPerReplica": 256, "nodesPerReplica": 16, "min": 1, "max": 10, "preventSinglePointFailure": true, "includeUnschedulableNodes": true }- The Addon-Operator show the log that it was updating the CoreDNS-Addon continuously.
{"level":"info","msg":"Applying ConfigMap name: coredns (namespace: kube-system, api: v1)","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"No change from existing state (/v1, Kind=ConfigMap) kube-system/coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Applying Deployment name: coredns (namespace: kube-system, api: apps/v1)","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Update was successful","time":"2023-06-14T15:20:22Z"} .. {"level":"info","msg":"Installed pkg: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns-1.8.6","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Updating finalizer for addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Adding finalizer for addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Updated addon: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Ignoring reconcile due to previous status update: 1c4f2564-f433-4177-93e2-8d56521b46be-coredns","time":"2023-06-14T15:20:22Z"} {"level":"info","msg":"Watch: Addon 1c4f2564-f433-4177-93e2-8d56521b46be-coredns converged after triggering it","time":"2023-06-14T15:20:27Z"}- Users may also see DNS resolution failures.
Resolution
- This issue has been fixed in PMK v5.9.1. Upgrade the environment to v5.9.1+ to fix the issue.
Was this page helpful?