Flannel Connecting to Etcd Cluster Using AWS ELB Reports ERROR

Problem

Cluster running on AWS and have an ELB, the flannel container logs reports below errors:

flannel logs

E0503 08:10:22.781361 1 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured; error #0: unexpected EOF

E0503 08:15:08.904157 1 watch.go:171] Subnet watch failed: client: etcd cluster is unavailable or misconfigured; error #0: unexpected EOF

Environment

• Platform9 Managed Kubernetes - v5.3 and Higher
• Flannel CNI
• AWS ELB

Cause

• This is a known upstream issue with Flannel https://github.com/flannel-io/flannel/issues/508 and https://github.com/flannel-io/flannel/issues/528 where AWS ELB close inactive connection and flannel trying reconnect.
• There are NO impacts and these logs can be ignored.

Resolution

• Increase the ELD idle timeout from AWS console (under Load Balancing -> Loadbalancers -> Healthcheck -> Edit Health Check -> Response Timeout.)

Warning

Timeout value will be reset on any cluster update operation so should reapply post upgrade.