Tenant Router Unreachable Due to Missing OVN Localnet Port on Provider Network

Problem

Tenant router gateway becomes unreachable from the external network even though the router and provider network configuration appear correct.

Environment

• Private Cloud Director Virtualization - v2025.10 and Higher

• Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher

• Component - Networking

Cause

The issue occurs due to a race condition between Neutron worker processes during network and port creation.

When the network and associated ports are created concurrently, different Neutron workers may process these requests before the logical switch state is fully synchronized in OVN. As a result, the required localnet port is not created in the OVN logical switch. Without this port, the logical network cannot connect to the physical provider network bridge, preventing ARP responses from the router gateway and causing external connectivity failure.

This is a known bug in neutron. An upstream fix has been released in latest version. Please see Neutron Commit 71ff8ee for more information.

Diagnostics

1. Verify that the router status is ACTIVE and get the [NETWORK_ID]:

$ openstack router show <ROUTER_ID>

1. Verify provider network configuration:

$ openstack network show <NETWORK_ID>

Example Output

| provider:network_type     | vlan     |
| provider:physical_network | [phprod] |
| provider:segmentation_id  | [266]    |
| router:external           | External |
| status                    | ACTIVE   |

• Physical network mapping should match the expected provider network

• VLAN ID should match the configured external VLAN

1. Verify that OVS bridge mappings on the gateway host:

Run the following command on the compute host where the router gateway port is scheduled.

$ sudo ovs-vsctl list open

Example Output

external_ids : {ovn-bridge-mappings="[phprod:br-phy1]"}

The provider network, in this example (phprod), must be mapped to the correct physical bridge (br-phy1)

1. Verify that ARP requests reach the compute host:

Run the following commands on the compute host where the router gateway port is scheduled. This helps confirm whether ARP requests from the external network are reaching the host.

Sample command:

$ tcpdump -i <PHYSICAL_INTERFACE> -evvvn arp and host <FIREWALL_IP>

or on the provider bridge:

$ sudo tcpdump -eni br-phy1 vlan <VLAN_ID> and arp

Example:

$ sudo tcpdump -eni br-phy1 vlan 266 and arp

Sample Output

14:40:10.100908 00:09:0f:09:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60:
vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42

14:40:11.149943 00:09:0f:19:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60:
vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42

In this example:

• 100.77.1.40 → External firewall

• 100.77.1.50 → Router gateway IP

• VLAN 266 → Provider network VLAN

The ARP request will typically appear as:

Request who-has <ROUTER_GATEWAY_IP> tell <FIREWALL_IP>

If ARP requests are visible but no ARP reply is observed, it indicates that even though traffic is successfully reaching the compute host via the physical network, OVN is not responding to the ARP request.

1. Verify OVN logical switch configuration:

$ ovn-nbctl show

Locate the logical switch corresponding to the network.

Example:

switch <SWITCH_ID> ([neutron-dgsdtn-c819-4b6a-b9ff-3459fndf9d8cd])

The expected configuration should include a localnet port:

port <PORT_ID>
type: localnet
options: network_name=<NETWORK_NAME>

If the localnet port is missing, the provider network cannot connect to the physical network, which prevents traffic from reaching the router gateway.

Note: Please check the Additional Information section for more information on how to run OVN commands depending on your environment

1. Compare with a working provider network

Run the following command to inspect a working external network.

$ openstack network show <WORKING_NETWORK_ID>

Example:

| provider:segmentation_id | [270] |

If routers connected to this network work correctly while the failing network does not, the issue is isolated to the affected VLAN network.

Resolution

This issue is caused by a known Neutron race condition that has been addressed in the upstream project. An internal engineering tracker has been created for this issue: PCD-5897

Workaround

As an immediate workaround, recreate the affected provider network so that the OVN logical switch and localnet port are created correctly.

1. Delete the affected Network.

Run the following command from the controller node:

$ openstack network delete <NETWORK_ID>

1. Recreate the provider network:

Example command

$ openstack network create \
--provider-network-type vlan \
--provider-physical-network [phprod] \
--provider-segment [266] \
--external \
<NETWORK_NAME>

1. Recreate the router gateway:

Attach the router to the recreated network:

$ openstack router set --external-gateway <NETWORK_ID> <ROUTER_ID>

1. Verify OVN localnet port creation:

Run:

$ ovn-nbctl show

Confirm the logical switch now contains:

port <port-id>
type: localnet

Once the localnet port is present, traffic will be correctly patched between br-int and br-phy, restoring router connectivity.

Additional Information

• For further questions/concerns regarding the bug, reach out to the Platform9 Support Team.

• To run ovn-* commands on the hosts onboarded to PCD, execute below steps.

1. Create an environment file ovs-alias.rc as below:

EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}')
export NBDB=tcp:${EXTERNAL_ID}:6641
export SBDB=tcp:${EXTERNAL_ID}:6642
alias ovn-sbctl="ovn-sbctl --db=$SBDB"
alias ovn-nbctl="ovn-nbctl --db=$NBDB"
alias ovn-trace="ovn-trace --db=$SBDB"

1. Export the rc file and start using the ovn commands:

$ source ovs-alias.rc
$ ovn-nbctl ls-list

NOTE: ovn commands can also be executed from inside the OVN North Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the steps below.

1. Access the OVN North Bond Pod in the Management Cluster using the command below.

$ kubectl -n <REGION_NAMESPACE> exec -it <OVN_NORTH_POD> -- bash --kubeconfig <PATH_TO_KUBECONFIG>

1. Run the below command to validate:

$ ovn-trace neutron-<SWITCH_UUID>  'inport=="<VM_NETWORK_PORT_ID>" && eth.src==<MAC_ADDR_OF_VM_PORT> && ip4.src==<SOURCE_IP> && ip4.dst==<Destination_IP> && icmp'