Outbound Connectivity Failing for Instances on Tenant Network

Problem

• Instances residing on a VXLAN-based tenant network without floating IPs are failing to send outbound traffic via SNAT.
• Instances with or without floating IPs are unable to resolve DNS on DHCP-enabled subnetswithin the aforementioned tenant network.
• If DHCP is enabled for the subnet, no ARP replies are observed for the DHCP host IP(s) specified in /etc/resolv.conf within the VMs (the DHCP hosts acting to masquerade DNS requests upstream).

# DNS_ADDRESS=<IP>

# ping $DNS_ADDRESS > /dev/null & tcpdump-lnevv any arp host $DNS_ADDRESS

Example:

# DNS_ADDRESS=10.0.0.11

# ping $DNS_ADDRESS > /dev/null & tcpdump -lnevv any arp host $DNS_ADDRES

...

00:26:02.055023 00:8c:fa:1c:4b:56 > 00:8c:fa:53:9d:a4, ethertype IPv4 (0x0800), length 92: (tos 0x0, ttl 64, id 60750, offset 0, flags [DF], proto UDP (17), length 78)

172.21.0.243.40650 > 172.21.0.241.4789: [no cksum] VXLAN, flags [I] (0x08), vni 10010

fa:16:3e:4d:5d:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.11 tell 10.0.0.10, length 28

Additionally, a connection timeout is observed attempting to resolve via hostnames via DNS via the qdhcp namespace (corresponding to the UUID of the network) residing on the respective DHCP agent host(s).

Example:

# ip netns exec qdhcp-3e1b8cd2-54f1-4867-b0c4-0d44ba607053 dig +short google.com @10.0.0.10

;; connection timed out; no servers could be reached

• If DHCP is not enabled for the subnet, ARP replies are similarly not observed for the SNAT port address.
• Connectivity to upstream DNS resolvers succeed and addresses are otherwise resolvable from the hypervisor (outside of any network namespace).

Environment

• Platform9 Managed OpenStack - All Versions
• Neutron
• OVS

Cause

• L2 routes, corresponding to hosts on which SNAT and/or DHCP ports (Neutron) reside, failed to be populated on the source host where the instance resides, thus, it is unknown where the traffic should be sent, resulting in a connection timeout.
• The routes failed to be populated as a result of RPC timeouts when flows were being populated in OVS via the neutron-ovs-agent.

Background

• The L2population mechanism driver creates the tunnel interfaces connecting each node.
• ofctl flow rules are then added in br-tun, specifying how to strip VLANs/add tunnel header, and over which tunnel interfaces to go.
• There is a table for unicast entries (remote MACs of VMs), as well as a flooding entry (broadcast packets like ARP, unknown unicast) that lists output ports as all tunnel interfaces.

Table 10 defines normal MAC learning behavior (how the hypervisors learn behind which remote hypervisor a VM's MAC lies). Table 20 is subsequently added to speed up this process for known VMs; analogous to a static MAC entry for non-tunnel.

For example, rows in table 20 have MACs of remote VMs and say which tunnel ID to set and which specific hypervisor tunnel to send on.

Table 22 defines flooding behavior. Without these, the switch does not know where to send flooded packets sent by the VM itself - such as ARP packets. For example below we have 2 tenant networks, and 2 other remote hypervisors (represented by ports 2 and 3). That's why there are 2 rules for VLAN to tunnel ID translation and saying to send it on all tunnel ports.

Scenario

• Instance A, residing on Host A(among Hosts A-F), is unable to reach platform9.com as its IP fails to be discovered via DNS. The connection to the DNS resolver address 10.0.0.11, specified in /etc/resolv.conf, times out. The DNS resolver address is associated with a Neutron port bound to host B-authorized as a DHCP agent.

A ping is initiated to 10.0.0.11 and, simultaneously, a capture of the network traffic is taken.

Based on the outer packet header (in the example above), we can tell that Host A (172.21.0.243) - the source host - is reaching out to Host C (172.21.0.241).

As ARP packets aren't natively flooded, but, rather, encapsulated in an IP unicast tunnel, a separate copy is generated to each remote host as per the entries in the flood table (within OVS).

Inspecting the flood entries within the flood table (22) for this network, identified by converting the tunnel ID of the network to hexadecimal) - in this case, 10010 being the tunnel ID - converted to hex, that is 0x271a - it can be observed that there there are only five (5) output ports present, each of which corresponds to a separate hypervisor.

# ovs-ofctl dump-flows br-tun | grep table=22 | grep 271a

cookie=0x480aaf2f6e79b7c4, duration=124819.246s, table=22, n_packets=95809, n_bytes=10830848, idle_age=603, hard_age=65534, priority=1,dl_vlan=1 actions=strip_vlan,load:0x271a->NXM_NX_TUN_ID[],output:7,output:11,output:12,output:13,output:14

In actuality, there should be six (6) output ports present to account for all of the hosts; the one which missing being that of Host B.

• If DHCP is not enabled for the subnet, but, external traffic is still failing for instances without a floating IP (via SNAT), the same scenario above is applicable but, the output port being that of the host on which the SNAT port resides.

Resolution

1. Restart the pf9-neutron-ovs-agent service.

# systemctl restart pf9-neutron-ovs-agent

2. Tail the /var/log/pf9/pf9-neutron-ovs.agent.log file.

# tail -f /var/log/pf9/pf9-neutron-ovs-agent.log

3. Determine the time which the agent initialized.

$ grep 'Agent initialized successfully' /var/log/pf9/pf9-neutron-ovs-agent.log

Example

20XX-XX-XX YY:YY:YY.466 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] Agent initialized successfully, now running...

4. Check for stale flow cleanup and determine the time of the first occurrence (since after the agent initialization).

$ grep 'stale' /var/log/pf9/pf9-neutron-ovs-agent.log

Example

20XX-XX-XX YY:YY:YY.237 4436 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] Cleaning stale br-int flows

5. Calculate the total time of all OVS_rpc_loop events since the agent was initialized.

# grep 'PF9_EVENT' /var/log/pf9/pf9-neutron-ovs-agent.log | grep 'rpc_loop'

Example

INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] PF9_EVENT: OVS_rpc_loop elapsed: 456.56 seconds

The following is a helpful one-liner to sum the time of all RPC loops.

$ grep 'PF9_EVENT' /var/log/pf9/pf9-neutron-ovs-agent.log | grep 'rpc_loop' | grep '2020-04-22 16*' | sed 's/^.*: //' | awk '{s+=$1} END {print "Total: " s}'

Note: Ensure you replace the timestamp (bolded) with the date/time based on agent initialization (identified in the prior step).

6. If the time between the agent initialization and the first stale flow cleanup in addition to the sum of all RPC loops exceeds 360 seconds - default value of agent_boot_time in ml2_conf.ini - the agent boot time may need to be increased accordingly.

/etc/neutron/plugins/ml2/ml2_conf.ini

[l2pop]

agent_boot_time = 360

7. To avoid any RPC timeouts/re-transmissions, which can cause a further backlog, increasing the rpc_response_timeout threshold may also be recommended, especially if further such behavior is exhibited after increasing the agent boot time accordingly.

   • In /opt/pf9/etc/pf9-neutron-ovs-agent/neutron.conf, check for the presence of rpc_response_timeout. If it set to a value higher than the default 60 seconds, it may not be necessary to increase it further.
   • Otherwise, in /opt/pf9/etc/pf9-neutron-ovs-agent/neutron_override.conf, set the parameter to a value higher than the default or whatever value is otherwise specified.

[DEFAULT]

rpc_response_timeout = 120

8. Restart the OVS agent.

# systemctl restart pf9-neutron-ovs-agent

Additional Information

• OpenStack Docs: Visualizing OpenStack Networking Service Traffic in the Cloud
• OpenStack Docs: Troubleshooting Open vSwitch