Floating IP Associated With VLAN-Based Provider Network is Inaccessible
Problem
- A floating IP (FIP), associated with an instance provisioned on a VLAN-based provider network, is inaccessible on the configured protocols/ports defined in the associated security group(s) (SG).
- The FIP is otherwise still pingable via ICMP.
- Additionally, the ports (allowed by the SG) may still be reached via fixed IP (via a host/device which has routes for that VLAN).
- The inbound Neutron security group chain is present and has the expected rules defined.
- Traffic for the FIP is not observed on any virtual interfaces.
- IP associated with the returned SYN-ACK packets (such as when trying to establish an HTTP(s) connection on TCP 80/443) differs from the IP for which the initial SYN packet was sent to.
Example
TCP connection is attempted to be established with 108.xxx.xxx.35(FIP), but, the corresponding SYN-ACK packet is received from 10.25.219.13 (fixed IP of the instance), which is unexpected.
Environment
- Platform9 Managed OpenStack - All Versions
- Neutron
- OVS
Cause
An IP of a physical router is set as the gateway IP for the subnet associated with a physical, VLAN-based provider network (provisioned in Neutron). The same network is connected to a virtual router (for the purpose of routing both the internal and external VLAN networks together), ultimately resulting in a split-horizon issue.
Resolution
Option #1: Switch to VXLAN Tenant Network (Recommended)
In an isolated tenant network (VXLAN-based), no physical router would be present, and the VMs would rely on the virtual Neutron router to be connected to any other network, internal or external. The VMs would not otherwise be accessible via their fixed IP from outside of the network.
In this case, no split-horizon routing or conflict would exist and the floating IP would be reachable on the ports defined within the associated SG(s).
Option #2: Maintain VLAN Provider Network
For this approach, the same network architecture would be maintained; however, the VMs would by default rely on the virtual router for routing external traffic if not otherwise specified as a host route (advertised via DHCP, similar to the gateway IP) for which that traffic would be routed to the physica l router.
- Modify the subnet and set the gateway IP to be that of the virtual router.
- Populate static host routes on the subnet such that the internal CIDR(s) point to the gateway IP of the physical router; otherwise, the traffic will be routed externally.