Applications Failing With Error "504 Gateway Timeout

Problem

Ingress controller logs shows a gateway timeout error.

time="YYYY-YYY-YYTXX:XX:XXZ" level=debug msg="'504 Gateway Timeout' caused by: dial tcp X.X.X.X:YY: i/o timeout"

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

• The flannel container is not able to connect to the etcd database to obtain network configuration because port 4001 is blocked by Security Groups on the master nodes.
• The following errors will be logged by the flannel container.

3261621 main.go:349] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp X.X.X.X:YYYY: i/o timeouttimed out

3266981 main.go:349] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp X.X.X.X:YYYY: i/o timeouttimed out

Resolution

Edit the Security Groups for the master node to allow traffic on port 4001.