Troubleshoot Outside to VM (North-South) Connectivity

Problem

This guide provides instructions for troubleshooting network connectivity failures between a Virtual Machine (VM) and the External Network (Internet or Datacenter). This includes outbound traffic (SNAT) and inbound access via Floating IPs (DNAT). Traffic must traverse a Neutron Logical Router and a designated Gateway Chassis.

Environment

• Private Cloud Director Virtualization - v2025.4 and Higher

• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

• Component - Networking Service

Deep Dive: Architecture & Packet Flow

North-South traffic differs from East-West because it is not fully distributed. While internal routing happens on the compute node, the transition to the physical external network is pinned to a specific Gateway Node.

• Chassisredirect (The Border): A special OVN port type (type=chassisredirect) that centralizes the external gateway logic on a specific physical host to manage NAT and external ARP.

• The Gateway Node: The physical server (e.g., punhv0059) where the Logical Router's external leg is physically bound.

• NAT (SNAT/DNAT): The process of translating the VM’s internal IP (192.xx.xx.xx) to the Floating IP (10.xx.xx.xx).

How the Packet Flows

Source VM $\rightarrow$ Source Tap $\rightarrow$ br-int (Compute Node) $\rightarrow$ Logical Router (Distributed Leg) $\rightarrow$ Geneve Tunnel (UDP 6081) $\rightarrow$ br-int (Gateway Node) $\rightarrow$ chassis redirect Port $\rightarrow$ NAT Engine (SNAT Applied) $\rightarrow$ br-phy1 (External Bridge) $\rightarrow$ Physical NIC $\rightarrow$ Datacenter Switch

Prerequisites: Executing OVN Commands

Refer to the existing guide for SaaS Alias setup or Self-Hosted kubectl exec instructions to run ovn-sbctl and ovn-nbctl commands.

Procedure

1. Variable Discovery & Gateway Identification

Gather the required IDs and locate the physical "Exit Door" for the traffic(Gateway node).

# 1. Get the VM's Port ID 
$ openstack port list --server <VM_ID>


# 2. Get the Network ID of that Port 
$ openstack port show <VM_PORT_ID_FROM_1.1> -c network_id -f value


# 3. Get the Port ID of the Router Interface on that Network 
$ openstack port list --network <NETWORK_ID_FROM_1.2> --device-owner network:router_interface -c ID -f value
# DESIRED OUTPUT: Router Interface Port ID. If empty, Network is not attached to a router.

# 4. Find the Router ID (device_id) 
$ openstack port show <ROUTER_INTERFACE_PORT_ID_FROM_1.3> -c device_id -f value


# 5. Get the External Gateway Port ID for that Router
$ openstack port list --router <ROUTER_ID_FROM_1.4> --device-owner network:router_gateway -c ID -f value
#  If empty, Router has no external gateway.

# 6. Find the Datapath and Chassis UUID
# CORRECT SYNTAX: --columns must come before 'find'
$ ovn-sbctl --columns=datapath,chassis find port_binding logical_port="cr-lrp-<GW_PORT_ID>"
# OUTPUT: 
# datapath: 7f95b7af... (This is your <DATAPATH_UUID> for Step 3)
# chassis : 649d312b... (This is your <CHASSIS_UUID> for Step 2.7)

# 7. Identify the Physical Gateway Node Name. Look for the hostname in the output
$ ovn-sbctl list chassis 649d312b-321d-41f9-a716-8594c886ac98

• Analysis: * If 1.3 is empty: The Network is not attached to a router.

  • If 1.5 is empty: The Router has no external gateway. NAT is impossible.

  • If 1.7 is empty: The OVN database hasn't bound the router to a host.

• Logs to Check (If 1.7 fails): On the expected Gateway Node, check /var/log/ovn/ovn-controller.log and grep for cr-lrp-<GW_PORT_ID>. Look for claim failed or unrecognized port errors indicating the node is refusing to host the gateway.

2. Identify the Data Plane Interfaces

Log into the Gateway Node to verify the physical bridge mapping.

# 1. Verify Bridge Mappings
$ sudo ovs-vsctl get open . external_ids:ovn-bridge-mappings
# ANALYSIS: If empty or missing your provider net (e.g., pun-lab:br-phy1), OVS cannot reach the physical wire.

# 2. Confirm Physical NIC is a member of the bridge
$ sudo ovs-vsctl show
# ANALYSIS: If br-phy1 does not contain a physical port (e.g., bond0), traffic hits a dead end inside the server.

• Success: Output shows your provider net mapped to a bridge (e.g., pun-lab:br-phy1), and that bridge contains a physical port (e.g., bond0).

• Failure: If mappings are empty, or br-phy1 lacks a physical interface, traffic hits a dead end inside the server and cannot reach the datacenter switch.

• Logs to Check: tail -f /var/log/openvswitch/ovs-vswitchd.log on the Gateway Node. Look for errors related to adding physical ports or bridge initialization failures.

3. Logical Simulation (North-South Trace)

Goal: Verify if the OVN "Brain" processes the packet. Because a router has multiple ports, we must pick the Internal one for an outbound trace.

# 1. List all ports for this router to find the Internal LRP. use the <DATAPATH_UUID> obtained from 1.6 
$ ovn-sbctl --columns=logical_port,mac find port_binding datapath="<DATAPATH_UUID>"
# IDENTIFY: 
# - The "Internal LRP" has the VM's gateway IP (e.g., 192.168.2.1).
# - The "External LRP" has the public IP (e.g., 10.22.178.x).

# 2. Run the Trace (Use the Internal LRP as inport)
$ ovn-trace <DATAPATH_UUID> 'inport == "<INTERNAL_LRP_NAME>" && eth.src == <VM_MAC> && eth.dst == <INTERNAL_LRP_MAC> && ip4.src == <VM_IP> && ip4.dst == 8.8.8.8 && ip.ttl == 64 && icmp'
# SUCCESS: Trace ends with "output to localnet".
# FAILURE: Trace ends with "drop".

The output reveals the exact logical table where the packet was killed. Use this matrix:

• ls_out_acl / ls_in_acl (Security Group Drop): A Neutron Security Group is explicitly denying the traffic.

• lr_in_ip_routing (Routing Drop): The Logical Router has no route to the destination. Verify the External Gateway is set.

• lr_in_arp_resolve (ARP Drop): The router cannot resolve the MAC of the next-hop (physical switch). Check upstream ARP.

• ls_in_port_sec (Anti-Spoofing Drop): The VM is trying to use a MAC/IP that does not belong to its port.

• SUCCESS: Trace shows ct_snat(FLOATING_IP) and ends with output to localnet.

• Failure (Drop): Trace ends with drop. The logical configuration is blocking the packet (e.g., missing route, Security Group). Proceed to Step 6.

4. Physical Validation (Gateway Node)

Log into the Gateway Node to prove the packet hits the wire.

the Gateway node can be identified using step 1

# 1. Check arrival from VM via Tunnel
$ sudo tcpdump -ni genev_sys_6081 host <VM_INTERNAL_IP>
# ANALYSIS: If empty, the packet is being dropped on the COMPUTE node.

# 2. Check exit to physical wire (NATed)
$ sudo tcpdump -ni br-phy1 host <FLOATING_IP>
# ANALYSIS SUCCESS: You see "FLOATING_IP > 8.8.8.8". NAT is working.
# ANALYSIS FAILURE: You see "INTERNAL_IP > 8.8.8.8" (NAT failed) or nothing (OVS Drop).

• Success: Packets arrive via tunnel (4.1) and exit br-phy1 with the translated Floating IP (4.2). NAT is working perfectly.

• Failure (No Arrival): 4.1 is empty. Packets are dropping on the Compute Node (Check SG egress or MTU).

• Failure (No Exit): 4.1 shows traffic, but 4.2 is empty. OVS is dropping the packet internally on the Gateway. Proceed to Step 5.

5. Advanced Physical Trace (ofproto/trace)

If Phase 4.2 is empty, ask OVS why it is dropping the packet internally on the Gateway Node.To run a physical ofproto/trace, we need the Geneve tunnel metadata and the correct input port.

# 5.1 Get the Router VNI (tun_id)
$ ovn-sbctl --columns=tunnel_key find datapath_binding datapath="<DATAPATH_UUID_FROM_1.6>"
# Convert result to HEX (e.g., 88 -> 0x58). This is your <VNI_HEX>.

# 5.2 Get the Source Port Key (Internal LRP)
$ ovn-sbctl --columns=tunnel_key find port_binding logical_port="<INTERNAL_LRP_NAME_FROM_2.1>"
# This is your <SRC_KEY> (e.g., 3).

# 5.3 Get the Destination Port Key (Chassisredirect/Gateway Port)
$ ovn-sbctl --columns=tunnel_key find port_binding logical_port="cr-lrp-<GW_PORT_ID_FROM_1.5>"
# This is your <DST_KEY> (e.g., 2).

# 5.4 Find the Input ofport (Run on Gateway Node)
# First, find the IP of the host where the VM is living:
$ openstack server show <VM_ID> -c "OS-EXT-SRV-ATTR:host" -f value 
# Then get the IP for that host
# Now, find which tunnel port on the Gateway points to that IP:
$ sudo ovs-vsctl find interface type=geneve options:remote_ip="<COMPUTE_NODE_IP>" | grep ofport


# 5.5 Metadata Construction: 0x<SRC_KEY>000<DST_KEY> 

# 5.6 Finally run the trace
$ sudo ovs-appctl ofproto/trace br-int in_port=<OFPORT_FROM_5.4>,tun_id=<VNI_HEX_FROM_5.1>,tun_metadata0=0x<SRC_KEY_FROM_5.2>000<DST_KEY_FROM_5.3>,dl_src=<VM_MAC>,dl_dst=<INTERNAL_LRP_MAC_FROM_3.1>,ip,nw_src=<VM_IP>,nw_dst=8.8.8.8,nw_proto=1

• Success: Datapath actions show set_tunnel or output:<ID> to a patch port leading to br-phy1.

• Failure (Drop): Action is drop. Note the cookie=0x... value in the trace output and proceed to Step 6.

• Failure (Flooded): Action outputs to local tap interfaces or other tunnels. OVN is confused about where the physical exit is (usually a bridge mapping mismatch).

• Logs to Check: If OVS actions don't match OVN intent, check /var/log/pf9/ovn/ovn-controller.log on the Gateway Node for flow programming errors (ofctrl_put errors).

6. Cookie Decoding (Root Cause Analysis)

If Step 5.5 dropped the packet, map the relevant_cookie back to the logical rule.

# Run in OVN SB Pod
$ ovn-sbctl lflow-list | grep <COOKIE_WITHOUT_0X>

• ls_out_acl / ls_in_acl: A Security Group is explicitly denying the traffic.

• lr_in_ip_routing: The Router has no route to the destination. Verify the External Gateway is set.

• lr_in_arp_resolve: The Gateway cannot resolve the MAC of the physical switch. Check upstream ARP.

• ls_in_port_sec: The VM is spoofing its MAC/IP.

7. Clear Stale FDB Entries (Ghost Traffic)

If the Gateway Node was recently migrated or the Chassis binding changed, traffic may be sent to a "ghost" location.

# 1. Search the FDB for the Floating IP or Gateway MAC
$ ovn-sbctl list fdb | grep <FLOATING_IP_OR_MAC>

# 2. If the entry points to the wrong chassis, delete it
$ ovn-sbctl destroy fdb <FDB_UUID>

• Success: Traffic immediately resumes once the stale entry is cleared.

• Logs to Check: On the Gateway Node, tail -f /var/log/pf9/ovn/ovn-controller.log | grep pinctrl. This tracks the ARP learning and FDB updates natively from the physical wire.

8. Physical Killers (MTU & Offloads)

If traces pass but traffic fails, the physical NIC may be corrupting large Geneve-encapsulated packets.

# 1. Check MTU (External Bridge and Physical NIC must be 1558+ to support VM 1500)
$ ip link show br-phy1
$ ip link show <PHYSICAL_NIC>

# 2. Disable NIC Offloads (Run on Gateway and Compute Nodes)
$ sudo ethtool -K <PHYSICAL_NIC> tso off gso off gro off tx off rx off

• Analysis: If MTU is 1500 on the physical NIC, Geneve packets (which add 58 bytes of overhead) will be fragmented or dropped by the physical switch.

• Logs to Check: * Run dmesg -T | grep -i eth or check /var/log/syslog for hardware-level drops, CRC errors, or driver crashes related to the physical NIC (bond0).

  • Check /var/log/openvswitch/ovs-vswitchd.log for unreasonably large packet or fragmentation warnings.

Most Common Causes

• Missing External Gateway: The Logical Router was created but never assigned an external gateway port. NAT cannot occur without an "Exit Door."

• Asymmetric Routing (Gateway Migration): The Gateway Chassis migrated to a new node, but the physical datacenter switch is still sending return traffic to the old node's MAC address because it missed the Gratuitous ARP (GARP).

• SNAT/DNAT Rule Mismatch: The VM has a Floating IP, but the Logical Router’s NAT table is missing the corresponding entry. OVN will route the packet but will not translate the source IP, causing the physical firewall to drop it as "spoofed."

• Stale FDB (Ghost Traffic): OVN is still tunneling traffic to the previous Gateway Node after a failover event. This "blackholes" all external traffic until the FDB entry is cleared or times out.

• Provider Network Bridge Mapping: The physical bridge (e.g., br-phy1) on the Gateway Node is not mapped to the correct physical_network name in OVS external_ids.

• Physical MTU Mismatch: North-South traffic often fails for Large Packets (HTTP/Downloads) because the Geneve overhead (58 bytes) makes the packet exceed the 1500 MTU of the physical datacenter switches.

• Upstream MAC Filtering: The physical switch port connected to the Gateway Node is configured with "Port Security" or a "MAC Limit" that prevents it from learning the virtual MAC addresses of the Floating IPs.