No Connectivity to Docker Containers Within Instance

Problem

A virtual machine instance, running the Docker daemon, has containers which cannot communicate externally nor be reached (on their exposed ports – if any) despite such traffic being allowed within the associated security group or firewall of the network.

Example

Unable to find image 'jekyll/jekyll:3.8' locally3.8: Pulling from jekyll/jekyll9d48c3bd43c5: Pull complete9ce9598067e7: Pull complete278f4c997324: Pull completebfca09e5fd9a: Pull complete2612f15b9d22: Pull complete{322c093d5418: Pull completeDigest: sha256:9521c8aae4739fcbc7137ead19f91841b833d671542f13e91ca40280e88d6e34Status: Downloaded newer image for jekyll/jekyll:3.8Fetching gem metadata from http://rubygems.org/.Retrying dependency api due to error (2/4): Bundler::HTTPError Network error while fetching https://index.rubygems.org/api/v1/dependencies?gems=[...] (Net::OpenTimeout)

Environment

• Platform9 Managed OpenStack - All Versions
• Neutron
• OVS
• Docker

Cause

The Docker daemon is not aware of and does not account for the VXLAN packet encapsulation overhead which is present on an OpenStack network with OVS. As such, the bridge interface, used for Docker containers within the VM, is created with a default MTU of 1500 – resulting in any incoming and outgoing packets to be dropped once outside of the VM since they are over the MTU size of the external interface (with the added overhead).

Resolution

1. Modify (or create – if it does not exist) the Docker daemon configuration file at /etc/docker/daemon.json and explicitly set the correct MTU size (accounting for the packet overhead).

{ "mtu": 1450}

Note: The MTU will vary depending on whether jumbo frames are configured for your environment. If jumbo frames are enabled, you will need to set an MTU of 8950.