Instance Unreachable on Separate Hypervisor – Stale Flow (OVS/VLAN)

Problem

• An instance provisioned on a VLAN, provider network is unable to reach another instance residing on a separate hypervisor.
• The OVS ovs-appctl utility shows that there is a "Nonexistent output port" for the flow matching the SRC MAC trying to reach the DST MAC via the port corresponding to the VETH PAIR connecting the Linux bridge which is used to facilitate traffic in and out of the VM and via OVS.

Example:

Bash

$ sudo ovs-appctl ofproto/trace br-int in_port=406,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62
Flow: in_port=406,vlan_tci=0x0000,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000

bridge("br-int")
----------------
 0. priority 0, cookie 0x9b52dfee970c9aa7
    goto_table:60
60. in_port=406,dl_src=fa:16:3e:38:bb:86, priority 9, cookie 0x9b52dfee970c9aa7
    set_field:0x3->reg6
    resubmit(,61)
61. reg6=0x3,dl_dst=fa:16:3e:7b:ee:62, priority 12, cookie 0x9b52dfee970c9aa7
    output:775
     >> Nonexistent output port

Final flow: reg6=0x3,in_port=406,vlan_tci=0x0000,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000
Megaflow: recirc_id=0,eth,in_port=406,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000
Datapath actions: drop

How do I gather the info needed?

The following specifications are needed to run the ovs-appctl ofproto/trace br-int command: in_port, dl_src, and dl_dst.

The in_port corresponds to the OVS port number for the "qvo" port/interface which is part of the aforementioned VETH pair.

The dl_src corresponds to the MAC address associated with the Neutron port of the source instance or VM.

The dl_dst corresponds to the MAC address associated with the Neutron port of the destination instance or VM.

How do I find the OVS port numbers?

The ovs-ofctl show command can be used to display the ports/numbers for a particular OVS bridge – in this case, "br-int" (see example below).

Example

Bash

$ sudo ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000da9f8fbfa643
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(int-br-pf9): addr:9e:e5:61:1a:36:c8
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(patch-tun): addr:f6:0f:11:a1:80:5f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
[...]
 406(qvo14947074-a3): addr:7e:76:9d:c3:e0:32
     config:     0
     state:      0
     current:    10GB-FD COPPER
     speed: 10000 Mbps now, 0 Mbps max

Environment

• Platform9 Managed OpenStack - All Versions
• Neutron
• OpenVSwitch

Cause

A stale flow exists in OVS – specifically, within the "integration bridge" or "br-int" – pertaining to the the destination's IP MAC address (determined via ARP).

Resolution

1. (Optional) Dump flows from the OVS integration bridge ("br-int").
2. Delete the flow.

Bash

$ sudo ovs-ofctl dump-flows br-int | grep 'dl_dst=fa:16:3e:7b:ee:62'
 cookie=0x9b52dfee970c9aa7, duration=3771175.459s, table=61, n_packets=1035, n_bytes=83218, idle_age=65534, hard_age=65534, priority=12,dl_vlan=3,dl_dst=fa:16:3e:7b:ee:62 actions=strip_vlan,output:775
 cookie=0x9b52dfee970c9aa7, duration=3771175.460s, table=61, n_packets=1129171, n_bytes=105985878, idle_age=0, hard_age=65534, priority=12,reg6=0x3,dl_dst=fa:16:3e:7b:ee:62 actions=output:775

Bash

ovs-ofctl --strict del-flows br-int "cookie=0x9b52dfee970c9aa7/-1,table=61,priority=12,dl_dst=fa:16:3e:7b:ee:62"

Tip

Note: Either the ovs-appctl ofproto/trace br-int command or ovs-ofctl dump-flows br-int can be used to gather the information needed to delete the flow.

Additional Information

• https://docs.openvswitch.org/en/latest/topics/design/
• https://www.openvswitch.org/support/dist-docs/ovs-ofctl.8.txt