Calico-kube-controllers pod is in CrashLoopBackOff state

Problem

• calico-kube-controllers pod is in CrashLoopBackOff state.

• The pod logs shows below error:

Javascript

2022-10-18 11:18:40.660 [INFO][1] main.go 92: Loaded configuration from environment config=&config.Config{LogLevel:"info", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", DatastoreType:"kubernetes"}
W1018 11:18:40.665042       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2022-10-18 11:18:40.665 [INFO][1] main.go 113: Ensuring Calico datastore is initialized
2022-10-18 11:18:41.667 [ERROR][1] client.go 261: Error getting cluster information config ClusterInformation="default" error=Get "https://10.21.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.21.0.1:443: connect: connection refused
2022-10-18 11:18:41.667 [FATAL][1] main.go 118: Failed to initialize Calico datastore error=Get "https://10.21.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.21.0.1:443: connect: connection refused

Environment

• Platform9 Managed Kubernetes - K8s v1.20.15

Cause

• The default route was misconfigured with wrong interface:

Javascript

# docker inspect 1c45574157a3ba | grep CALICO |grep CALICO_IPV4_DETECTION_METHOD
export CALICO_IPV4_DETECTION_METHOD="interface=bond1.100"

# ip route show |grep default
default via 10.69.63.1 dev bond0.663

Resolution

• Update the node's routing table for the interface which is being used by Calico.