Calico Node Pods in CrashLoopBackOff: "Unable to get Typha service endpoints from Kubernetes"

Problem

• The calico-node pod(s) are in CrashLoopBackOff state and/or constantly restarting.

Bash

NAME                                              READY   STATUS             RESTARTS   AGE
calico-kube-controllers-f6b98779-qfmhh            1/1     Running            1          101d
calico-node-69nk4                                 0/1     CrashLoopBackOff   131        7h37m
calico-node-g7pz7                                 0/1     CrashLoopBackOff   131        7h37m
calico-node-mngnc                                 0/1     Running            130        7h37m
calico-node-p4ktd                                 0/1     CrashLoopBackOff   129        7h37m
calico-node-wlv9t                                 0/1     CrashLoopBackOff   130        7h37m
calico-node-zl5ng                                 0/1     CrashLoopBackOff   129        7h37m
calico-typha-95b7898f5-g559h                      1/1     Running            0          8h
calico-typha-95b7898f5-lmxbw                      1/1     Running            0          8h
calico-typha-95b7898f5-zmn2x                      1/1     Running            0          8h

• The pod log (via kubectl logs) for the calico-node pod(s) shows the following error related to calico-typha .

Bash

[ERROR][68] confd/discovery.go 126: Unable to get Typha service endpoints from Kubernetes. error=endpoints "calico-typha" not found

• Similarly, the following error is recorded in the calico-typha pod(s).

Bash

[ERROR][7] lookup.go 65: Failed to get Typha endpoint from Kubernetes error=endpoints "calico-typha" not found
[WARNING][7] rebalance.go 55: Failed to get number of Typhas error=<nil> numTyphas=0 thread="k8s-poll"

• No calico-typha service (separate from the DaemonSet and Deployment objects) exists in the kube-system namespace.

Environment

• Platform9 Managed Kubernetes – v5.5

• Calico

Cause

The calico-typha service is missing from the kube-system namespace.

Resolution

1. Create a file calico-typha.yaml with the following content.

Bash

# Source: calico/templates/calico-typha.yaml
# This manifest creates a Service, which will be backed by Calico's Typha daemon.
# Typha sits in between Felix and the API server, reducing Calico's load on the API server.

apiVersion: v1
kind: Service
metadata:
  name: calico-typha
  namespace: kube-system
  labels:
    k8s-app: calico-typha
spec:
  ports:
    - port: 5473
      protocol: TCP
      targetPort: calico-typha
      name: calico-typha
  selector:
    k8s-app: calico-typha

1. Apply the YAML to the affected cluster.

Bash

kubectl apply -f calico-typha.yaml

Additional Information

• Calico – Install Typha