> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmo/solution/outbound-connectivity-failing-for-instances-on-tenant-network.md).

# Outbound Connectivity Failing for Instances on Tenant Network

## Problem

* Instances residing on a VXLAN-based tenant network without floating IPs are failing to send outbound traffic via [SNAT](https://docs.openstack.org/mitaka/networking-guide/intro-nat.html).
* Instances with or without floating IPs are unable to resolve **DNS** on DHCP-enabled subnetswithin the aforementioned tenant network.
* If **DHCP** is enabled for the subnet, no **ARP** replies are observed for the **DHCP** host IP(s) specified in */etc/resolv.conf* within the VMs (the DHCP hosts acting to masquerade DNS requests upstream).

{% tabs %}
{% tab title="None" %}

```none
# DNS_ADDRESS=<IP>
# ping $DNS_ADDRESS > /dev/null & tcpdump-lnevv any arp host $DNS_ADDRESS
```

{% endtab %}
{% endtabs %}

**Example:**

{% tabs %}
{% tab title="None" %}

```none
# DNS_ADDRESS=10.0.0.11
# ping $DNS_ADDRESS > /dev/null & tcpdump -lnevv any arp host $DNS_ADDRES
...
00:26:02.055023 00:8c:fa:1c:4b:56 > 00:8c:fa:53:9d:a4, ethertype IPv4 (0x0800), length 92: (tos 0x0, ttl 64, id 60750, offset 0, flags [DF], proto UDP (17), length 78)
172.21.0.243.40650 > 172.21.0.241.4789: [no cksum] VXLAN, flags [I] (0x08), vni 10010
fa:16:3e:4d:5d:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.11 tell 10.0.0.10, length 28
```

{% endtab %}
{% endtabs %}

Additionally, a connection timeout is observed attempting to resolve via hostnames via DNS via the **qdhcp** namespace (corresponding to the UUID of the network) residing on the respective DHCP agent host(s).

**Example:**

{% tabs %}
{% tab title="None" %}

```none
# ip netns exec qdhcp-3e1b8cd2-54f1-4867-b0c4-0d44ba607053 dig +short google.com @10.0.0.10
;; connection timed out; no servers could be reached
```

{% endtab %}
{% endtabs %}

* If **DHCP** is not enabled for the subnet, ARP replies are similarly not observed for the **SNAT** port address.
* Connectivity to upstream DNS resolvers succeed and addresses are otherwise resolvable from the hypervisor (outside of any network namespace).

## Environment

* Platform9 Managed OpenStack - All Versions
* Neutron
* OVS

## Cause

* L2 routes, corresponding to hosts on which SNAT and/or DHCP ports (Neutron) reside, failed to be populated on the source host where the instance resides, thus, it is unknown where the traffic should be sent, resulting in a connection timeout.
* The routes failed to be populated as a result of **RPC** timeouts when flows were being populated in OVS via the ***neutron-ovs-agent***.

### Background

* The [**L2population mechanism driver**](https://wiki.openstack.org/wiki/L2population) creates the tunnel interfaces connecting each node.
* **ofctl** flow rules are then added in **br-tun**, specifying how to strip VLANs/add tunnel header, and over which tunnel interfaces to go.
* There is a table for **unicast** entries (remote MACs of VMs), as well as a **flooding** entry (broadcast packets like **ARP**, unknown unicast) that lists output ports as all tunnel interfaces.

**Table 10** defines normal **MAC learning behavior** (how the hypervisors learn behind which remote hypervisor a VM's MAC lies). **Table 20** is subsequently added to speed up this process for known VMs; analogous to a **static MAC entry** for non-tunnel.

For example, rows in table 20 have MACs of remote VMs and say which tunnel ID to set and which specific hypervisor tunnel to send on.

**Table 22** defines **flooding** behavior. Without these, the switch does not know where to send flooded packets sent by the VM itself - such as ARP packets. For example below we have 2 tenant networks, and 2 other remote hypervisors (represented by ports 2 and 3). That's why there are 2 rules for VLAN to tunnel ID translation and saying to send it on all tunnel ports.

### Scenario

* **Instance A,** residing on **Host A**(among Hosts **A-F**), is unable to reach **platform9.com** as its IP fails to be discovered via DNS. The connection to the DNS resolver address **10.0.0.11**, specified in */etc/resolv.conf*, times out. The DNS resolver address is associated with a Neutron port bound to **host B**-authorized as a **DHCP agent**.

A ping is initiated to **10.0.0.11** and, simultaneously, a capture of the network traffic is taken.

Based on the outer packet header (in the example above), we can tell that **Host A** (172.21.0.243) - the **source** host - is reaching out to **Host C** (172.21.0.241).

As ARP packets aren't natively flooded, but, rather, **encapsulated** in an IP unicast tunnel, a **separate copy** is generated to each **remote host** as per the entries in the **flood table** (within OVS).

Inspecting the flood entries within the flood table (22) for this network, identified by converting the **tunnel ID** of the network to hexadecimal) - in this case, **10010** being the tunnel ID - converted to hex, that is **0x271a -** it can be observed that there there are only five (5) **output** ports present, each of which corresponds to a separate hypervisor.

{% tabs %}
{% tab title="None" %}

```none
# ovs-ofctl dump-flows br-tun | grep table=22 | grep 271a
cookie=0x480aaf2f6e79b7c4, duration=124819.246s, table=22, n_packets=95809, n_bytes=10830848, idle_age=603, hard_age=65534, priority=1,dl_vlan=1 actions=strip_vlan,load:0x271a->NXM_NX_TUN_ID[],output:7,output:11,output:12,output:13,output:14
```

{% endtab %}
{% endtabs %}

In actuality, there should be six (6) output ports present to account for all of the hosts; the one which missing being that of **Host B.**

* If DHCP is not enabled for the subnet, but, external traffic is still failing for instances **without a floating IP** (via **SNAT**), **the same scenario above is applicable** but, the output port being that of the host on which the SNAT port resides.

## Resolution

1. Restart the ***pf9-neutron-ovs-agent*** service.

{% tabs %}
{% tab title="None" %}

```none
# systemctl restart pf9-neutron-ovs-agent
```

{% endtab %}
{% endtabs %}

2. Tail the */var/log/pf9/pf9-neutron-ovs.agent.log* file.

{% tabs %}
{% tab title="None" %}

```none
# tail -f /var/log/pf9/pf9-neutron-ovs-agent.log
```

{% endtab %}
{% endtabs %}

3. Determine the time which the agent initialized.

{% tabs %}
{% tab title="None" %}

```none
$ grep 'Agent initialized successfully' /var/log/pf9/pf9-neutron-ovs-agent.log
```

{% endtab %}
{% endtabs %}

**Example**

{% tabs %}
{% tab title="None" %}

```none
20XX-XX-XX YY:YY:YY.466 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] Agent initialized successfully, now running...
```

{% endtab %}
{% endtabs %}

4. Check for stale flow cleanup and determine the time of the first occurrence (since after the agent initialization).

{% tabs %}
{% tab title="None" %}

```none
$ grep 'stale' /var/log/pf9/pf9-neutron-ovs-agent.log
```

{% endtab %}
{% endtabs %}

**Example**

{% tabs %}
{% tab title="None" %}

```none
20XX-XX-XX YY:YY:YY.237 4436 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] Cleaning stale br-int flows
```

{% endtab %}
{% endtabs %}

5. Calculate the total time of all **OVS\_rpc\_loop** events since the agent was initialized.

{% tabs %}
{% tab title="None" %}

```none
# grep 'PF9_EVENT' /var/log/pf9/pf9-neutron-ovs-agent.log | grep 'rpc_loop'
```

{% endtab %}
{% endtabs %}

**Example**

{% tabs %}
{% tab title="None" %}

```none
INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [...] PF9_EVENT: OVS_rpc_loop elapsed: 456.56 seconds
```

{% endtab %}
{% endtabs %}

The following is a helpful one-liner to sum the time of all RPC loops.

{% tabs %}
{% tab title="None" %}

```none
$ grep 'PF9_EVENT' /var/log/pf9/pf9-neutron-ovs-agent.log | grep 'rpc_loop' | grep '2020-04-22 16*' | sed 's/^.*: //' | awk '{s+=$1} END {print "Total: " s}'
```

{% endtab %}
{% endtabs %}

**Note**: Ensure you replace the timestamp (bolded) with the date/time based on agent initialization (identified in the prior step).

6. If the time between the agent initialization and the first stale flow cleanup in addition to the sum of all RPC loops exceeds **360 seconds** - default value of ***agent\_boot\_time*** in *ml2\_conf.ini* - the agent boot time may need to be increased accordingly.

*/etc/neutron/plugins/ml2/ml2\_conf.ini*

{% tabs %}
{% tab title="None" %}

```none
[l2pop]
agent_boot_time = 360
```

{% endtab %}
{% endtabs %}

7. To avoid any RPC timeouts/re-transmissions, which can cause a further backlog, increasing the ***rpc\_response\_timeout*** threshold may also be recommended, especially if further such behavior is exhibited after increasing the agent boot time accordingly.
   * In */opt/pf9/etc/pf9-neutron-ovs-agent/neutron.conf*, check for the presence of ***rpc\_response\_timeout***. If it set to a value higher than the default ***60 seconds***, it **may not** be necessary to increase it further.
   * Otherwise, in */opt/pf9/etc/pf9-neutron-ovs-agent/neutron\_override.conf*, set the parameter to a value **higher** than the default or whatever value is otherwise specified.

{% tabs %}
{% tab title="None" %}

```none
[DEFAULT]
rpc_response_timeout = 120
```

{% endtab %}
{% endtabs %}

8. Restart the OVS agent.

{% tabs %}
{% tab title="None" %}

```none
# systemctl restart pf9-neutron-ovs-agent
```

{% endtab %}
{% endtabs %}

## Additional Information

* [OpenStack Docs: Visualizing OpenStack Networking Service Traffic in the Cloud](https://docs.openstack.org/operations-guide/ops-network-troubleshooting.html#visualizing-openstack-networking-service-traffic-in-the-cloud)
* [OpenStack Docs: Troubleshooting Open vSwitch](https://docs.openstack.org/operations-guide/ops-network-troubleshooting.html#troubleshooting-open-vswitch)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://platform9.com/kb/pmo/solution/outbound-connectivity-failing-for-instances-on-tenant-network.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.