# Tenant Router Unreachable Due to Missing OVN Localnet Port on Provider Network ## Problem Tenant router gateway becomes unreachable from the external network even though the router and provider network configuration appear correct. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component - Networking ## Cause The issue occurs due to a race condition between Neutron worker processes during network and port creation. When the network and associated ports are created concurrently, different Neutron workers may process these requests before the logical switch state is fully synchronized in OVN. As a result, the required localnet port is not created in the OVN logical switch. Without this port, the logical network cannot connect to the physical provider network bridge, preventing ARP responses from the router gateway and causing external connectivity failure. This is a known bug in neutron. An upstream fix has been released in latest version. Please see [Neutron Commit 71ff8ee](https://github.com/openstack/neutron/commit/71ff8ee881a2c387ff318df92c61e7b8333e7014) for more information. ## Diagnostics 1. Verify that the router status is ACTIVE and get the \[NETWORK\_ID]: ``` $ openstack router show ``` 2. Verify provider network configuration: ``` $ openstack network show ``` {% code title="Example Output" %} ``` | provider:network_type | vlan | | provider:physical_network | [phprod] | | provider:segmentation_id | [266] | | router:external | External | | status | ACTIVE | ``` {% endcode %} * Physical network mapping should match the expected provider network * VLAN ID should match the configured external VLAN
3. Verify that OVS bridge mappings on the gateway host: Run the following command on the compute host where the router gateway port is scheduled. ``` $ sudo ovs-vsctl list open ``` {% code title="Example Output" %} ``` external_ids : {ovn-bridge-mappings="[phprod:br-phy1]"} ``` {% endcode %} The provider network, in this example (`phprod`), must be mapped to the correct physical bridge (`br-phy1`)
4. Verify that ARP requests reach the compute host: Run the following commands on the compute host where the router gateway port is scheduled. This helps confirm whether ARP requests from the external network are reaching the host. {% code title="Sample command:" %} ``` $ tcpdump -i -evvvn arp and host ``` {% endcode %} or on the provider bridge: ``` $ sudo tcpdump -eni br-phy1 vlan and arp ``` {% code title="Example:" %} ``` $ sudo tcpdump -eni br-phy1 vlan 266 and arp ``` {% endcode %} {% code title="Sample Output" %} ``` 14:40:10.100908 00:09:0f:09:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42 14:40:11.149943 00:09:0f:19:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42 ``` {% endcode %} In this example: * `100.77.1.40` → External firewall * `100.77.1.50` → Router gateway IP * VLAN `266` → Provider network VLAN The ARP request will typically appear as: ``` Request who-has tell ``` If ARP requests are visible but no ARP reply is observed, it indicates that even though traffic is successfully reaching the compute host via the physical network, OVN is not responding to the ARP request.
5. Verify OVN logical switch configuration: ``` $ ovn-nbctl show ``` Locate the logical switch corresponding to the network. {% code title="Example:" %} ``` switch ([neutron-dgsdtn-c819-4b6a-b9ff-3459fndf9d8cd]) ``` {% endcode %} The expected configuration should include a `localnet` port: ``` port type: localnet options: network_name= ``` If the `localnet` port is missing, the provider network cannot connect to the physical network, which prevents traffic from reaching the router gateway. {% hint style="info" %} *Note: Please check the **Additional Information** section for more information on how to run OVN commands depending on your environment* {% endhint %} 6. Compare with a working provider network Run the following command to inspect a working external network. ``` $ openstack network show ``` {% code title="Example:" %} ``` | provider:segmentation_id | [270] | ``` {% endcode %} If routers connected to this network work correctly while the failing network does not, the issue is isolated to the affected VLAN network. ## Resolution This issue is caused by a known Neutron race condition that has been addressed in the upstream project. An internal engineering tracker has been created for this issue: **PCD-5897** ## Workaround As an immediate workaround, recreate the affected provider network so that the OVN logical switch and localnet port are created correctly. 1. Delete the affected Network. Run the following command from the controller node: ``` $ openstack network delete ``` 2. Recreate the provider network: {% code title="Example command" %} ``` $ openstack network create \ --provider-network-type vlan \ --provider-physical-network [phprod] \ --provider-segment [266] \ --external \ ``` {% endcode %} 3. Recreate the router gateway: Attach the router to the recreated network: ``` $ openstack router set --external-gateway ``` 4. Verify OVN localnet port creation: Run: ``` $ ovn-nbctl show ``` Confirm the logical switch now contains: ``` port type: localnet ``` Once the localnet port is present, traffic will be correctly patched between `br-int` and `br-phy`, restoring router connectivity. ## Additional Information * For further questions/concerns regarding the bug, reach out to the [Platform9 Support Team](https://support.platform9.com/). * To run `ovn-*` commands on the hosts onboarded to PCD, execute below steps. 1. Create an environment file `ovs-alias.rc` as below: ``` EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}') export NBDB=tcp:${EXTERNAL_ID}:6641 export SBDB=tcp:${EXTERNAL_ID}:6642 alias ovn-sbctl="ovn-sbctl --db=$SBDB" alias ovn-nbctl="ovn-nbctl --db=$NBDB" alias ovn-trace="ovn-trace --db=$SBDB" ``` 2. Export the rc file and start using the ovn commands: ``` $ source ovs-alias.rc $ ovn-nbctl ls-list ``` {% hint style="info" %} **NOTE:** `ovn` commands can also be executed from inside the OVN North Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the steps below. {% endhint %} 1. Access the OVN North Bond Pod in the Management Cluster using the command below. ``` $ kubectl -n exec -it -- bash --kubeconfig ``` 2. Run the below command to validate: ``` $ ovn-trace neutron- 'inport=="" && eth.src== && ip4.src== && ip4.dst== && icmp' ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://platform9.com/kb/pcd/networking/tenant-router-unreachable-due-to-missing-ovn-localnet-port-on-provider-network.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.