Floating IP Associated With VLAN-Based Provider Network is Inaccessible

Problem

• A floating IP (FIP), associated with an instance provisioned on a VLAN-based provider network, is inaccessible on the configured protocols/ports defined in the associated security group(s) (SG).

• The FIP is otherwise still pingable via ICMP.

• Additionally, the ports (allowed by the SG) may still be reached via fixed IP (via a host/device which has routes for that VLAN).

• The inbound Neutron security group chain is present and has the expected rules defined.

• Traffic for the FIP is not observed on any virtual interfaces.

• IP associated with the returned SYN-ACK packets (such as when trying to establish an HTTP(s) connection on TCP 80/443) differs from the IP for which the initial SYN packet was sent to.

Example

TCP connection is attempted to be established with 108.xxx.xxx.35(FIP), but, the corresponding SYN-ACK packet is received from 10.25.219.13 (fixed IP of the instance), which is unexpected.

None

$ sudo tcpdump -nnei bond0 tcp and port 80 -c3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes

784798 64:51:06:f0:15:7c > 00:50:56:a4:8c:f4, ethertype 802.1Q (0x8100), length 78: vlan 700, p 0, ethertype IPv4, 10.25.0.3.47180 > 108.xxx.xxx.35.80: Flags [S], seq 854224565, win 29200, options [mss 1460,sackOK,TS val 640948412 ecr 0,nop,wscale 7], length 0
785277 00:50:56:a4:8c:f4 > 64:51:06:f0:15:7c, ethertype 802.1Q (0x8100), length 78: vlan 700, p 0, ethertype IPv4, 10.25.219.13.80 > 10.25.0.3.47180: Flags [S.], seq 1286869436, ack 854224566, win 28960, options [mss 1460,sackOK,TS val 39232845 ecr 640948412,nop,wscale 7], length 0
785308 64:51:06:f0:15:7c > 00:50:56:a4:8c:f4, ethertype 802.1Q (0x8100), length 58: vlan 700, p 0, ethertype IPv4, 10.25.0.3.47180 > 10.25.219.13.80: Flags [R], seq 854224566, win 0, length 0

3 packets captured
3 packets received by filter
0 packets dropped by kernel
12 packets dropped by interface

Environment

• Platform9 Managed OpenStack - All Versions

• Neutron

• OVS

Cause

An IP of a physical router is set as the gateway IP for the subnet associated with a physical, VLAN-based provider network (provisioned in Neutron). The same network is connected to a virtual router (for the purpose of routing both the internal and external VLAN networks together), ultimately resulting in a split-horizon issue.

Resolution

Option #1: Switch to VXLAN Tenant Network (Recommended)

In an isolated tenant network (VXLAN-based), no physical router would be present, and the VMs would rely on the virtual Neutron router to be connected to any other network, internal or external. The VMs would not otherwise be accessible via their fixed IP from outside of the network.

In this case, no split-horizon routing or conflict would exist and the floating IP would be reachable on the ports defined within the associated SG(s).

Option #2: Maintain VLAN Provider Network

For this approach, the same network architecture would be maintained; however, the VMs would by default rely on the virtual router for routing external traffic if not otherwise specified as a host route (advertised via DHCP, similar to the gateway IP) for which that traffic would be routed to the physica l router.

1. Modify the subnet and set the gateway IP to be that of the virtual router.

None

$ openstack subnet set --gateway [NEUTRON_ROUTER_IP] [SUBNET_UUID]

1. Populate static host routes on the subnet such that the internal CIDR(s) point to the gateway IP of the physical router; otherwise, the traffic will be routed externally.

None

$ openstack subnet set --host-route destination=[PHYSICAL_CIDR],gateway=[PHYSICAL_ROUTER_IP]

Additional Information

• Platform9 – Networking Concepts