Instance Unreachable on Separate Hypervisor – Stale Flow (OVS/VLAN)
Problem
- An instance provisioned on a VLAN, provider network is unable to reach another instance residing on a separate hypervisor.
- The OVS
ovs-appctlutility shows that there is a "Nonexistent output port" for the flow matching the SRC MAC trying to reach the DST MAC via the port corresponding to the VETH PAIR connecting the Linux bridge which is used to facilitate traffic in and out of the VM and via OVS.
Example:
$ sudo ovs-appctl ofproto/trace br-int in_port=406,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62Flow: in_port=406,vlan_tci=0x0000,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000bridge("br-int")---------------- 0. priority 0, cookie 0x9b52dfee970c9aa7 goto_table:6060. in_port=406,dl_src=fa:16:3e:38:bb:86, priority 9, cookie 0x9b52dfee970c9aa7 set_field:0x3->reg6 resubmit(,61)61. reg6=0x3,dl_dst=fa:16:3e:7b:ee:62, priority 12, cookie 0x9b52dfee970c9aa7 output:775 >> Nonexistent output portFinal flow: reg6=0x3,in_port=406,vlan_tci=0x0000,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000Megaflow: recirc_id=0,eth,in_port=406,dl_src=fa:16:3e:38:bb:86,dl_dst=fa:16:3e:7b:ee:62,dl_type=0x0000Datapath actions: dropThe following specifications are needed to run the ovs-appctl ofproto/trace br-int command: in_port, dl_src, and dl_dst.
The in_port corresponds to the OVS port number for the "qvo" port/interface which is part of the aforementioned VETH pair.
The dl_src corresponds to the MAC address associated with the Neutron port of the source instance or VM.
The dl_dst corresponds to the MAC address associated with the Neutron port of the destination instance or VM.
The ovs-ofctl show command can be used to display the ports/numbers for a particular OVS bridge – in this case, "br-int" (see example below).
Example
$ sudo ovs-ofctl show br-intOFPT_FEATURES_REPLY (xid=0x2): dpid:0000da9f8fbfa643n_tables:254, n_buffers:0capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IPactions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 1(int-br-pf9): addr:9e:e5:61:1a:36:c8 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 2(patch-tun): addr:f6:0f:11:a1:80:5f config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max[...] 406(qvo14947074-a3): addr:7e:76:9d:c3:e0:32 config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps maxEnvironment
- Platform9 Managed OpenStack - All Versions
- Neutron
- OpenVSwitch
Cause
A stale flow exists in OVS – specifically, within the "integration bridge" or "br-int" – pertaining to the the destination's IP MAC address (determined via ARP).
Resolution
- (Optional) Dump flows from the OVS integration bridge ("br-int").
- Delete the flow.
$ sudo ovs-ofctl dump-flows br-int | grep 'dl_dst=fa:16:3e:7b:ee:62' cookie=0x9b52dfee970c9aa7, duration=3771175.459s, table=61, n_packets=1035, n_bytes=83218, idle_age=65534, hard_age=65534, priority=12,dl_vlan=3,dl_dst=fa:16:3e:7b:ee:62 actions=strip_vlan,output:775 cookie=0x9b52dfee970c9aa7, duration=3771175.460s, table=61, n_packets=1129171, n_bytes=105985878, idle_age=0, hard_age=65534, priority=12,reg6=0x3,dl_dst=fa:16:3e:7b:ee:62 actions=output:775ovs-ofctl --strict del-flows br-int "cookie=0x9b52dfee970c9aa7/-1,table=61,priority=12,dl_dst=fa:16:3e:7b:ee:62"Note: Either the ovs-appctl ofproto/trace br-int command or ovs-ofctl dump-flows br-int can be used to gather the information needed to delete the flow.