Outbound Network Connectivity Failing for Instances Connected to a Virtual Router

Problem

< A list of observable facts that characterize the undesirable behavior, starting with the most obvious and customer-identified, but continuing with those identified during the diagnostic process. Error messages, Bug references (link to Launchpad), log file entries, and behavior different than expected are all candidate symptoms. >

Environment

• Platform9 Managed OpenStack - v4.0 and Higher

< The hardware, software, environmental conditions, and recent changes that are associated with this particular problem and resolution. Generally, our company’s product and/or product components are listed first. It’s OK for the environment to partially overlap with product category metadata. >

Cause

<The reason for the problem, if known.>

Resolution

Javascript

[[e-mail] chinmay(admin)]# openstack router show 3d3ec419-e0e5-45fa-a661-bfeb14261658 -c ha
+-------+-------+
| Field | Value |
+-------+-------+
| ha    | False |
+-------+-------+

There is no l3 agent hosting this router and since it has HA disabled it did not failover to any other node.

[[e-mail] chinmay(admin)]# neutron l3-agent-list-hosting-router 3d3ec419-e0e5-45fa-a661-bfeb14261658
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.

[[e-mail] chinmay(admin)]#
To correct this, added the router to another L3 agent.
[[e-mail] chinmay(admin)]# openstack network agent list | grep l3
| 57b9312f-1f60-42c6-9ed2-0e9f18d0d0b7 | L3 agent           | e66b666a-29fe-4a4a-ad8d-168bc9cb3e16 | nova              | :-)   | UP    | neutron-l3-agent          |
| 673e1117-8962-4e67-baa2-fa8c6b91f0b3 | L3 agent           | 48d1203d-8f7d-4d43-be67-da4850812c3f | nova              | :-)   | UP    | neutron-l3-agent          |
| 72bfdf6a-44b6-460f-8f05-0e2242a276d4 | L3 agent           | b4b96a75-d591-4dc6-8150-09b2c05f31ef | nova              | :-)   | UP    | neutron-l3-agent          |
| 7c43afd0-1bc4-4dba-8d31-72789e8b4b81 | L3 agent           | 3cf9bcfd-311e-45c9-b37b-2be594420f2c | nova              | :-)   | UP    | neutron-l3-agent          |
| 9705141c-1513-4407-a971-76459d7e7fcd | L3 agent           | 7c6e8ba0-c97e-4ab2-95d6-76418151c8e1 | nova              | :-)   | UP    | neutron-l3-agent          |
| a132f959-6686-45a8-8d18-d523848fe5c3 | L3 agent           | a93e5278-e6d1-4783-9682-c5c5fb156d34 | nova              | :-)   | UP    | neutron-l3-agent          |
| a86c04dd-741f-45bd-bca5-561b7a3ab4e1 | L3 agent           | 64e3ded1-3d18-4e8a-8cab-fc40e36d8374 | nova              | :-)   | UP    | neutron-l3-agent          |
| aba44282-5302-4d30-bb70-150045cd5a45 | L3 agent           | a1abc0ff-32a6-4f73-a63c-7879181a9a43 | nova              | :-)   | UP    | neutron-l3-agent          |
| af9eaf28-35f9-4e28-b844-0e2e00b6b5f3 | L3 agent           | ea111f2a-36a6-46cd-8d66-71d78e9b3ab6 | nova              | :-)   | UP    | neutron-l3-agent          |
| b8d160f0-0c81-490d-9991-632355655212 | L3 agent           | 62d417f8-f4fb-47e6-afa3-f2ec88ffd23c | nova              | :-)   | UP    | neutron-l3-agent          |
| be11fc91-f900-450c-ba68-7f65f4041ff7 | L3 agent           | f98d26b4-bf46-41ed-a099-4d416c1c0a0f | nova              | :-)   | UP    | neutron-l3-agent          |
| ccb847e6-81e1-4e71-ba79-f16980b7ccab | L3 agent           | 60bd112b-edeb-4adc-8313-05f52e367eaa | nova              | :-)   | UP    | neutron-l3-agent          |
| e2e49c76-b008-4954-9905-e3e5ea251db7 | L3 agent           | cbefd094-dffb-4400-ab72-68498690e718 | nova              | :-)   | UP    | neutron-l3-agent          |
| fb243470-304f-4ca8-9bd6-28426f3ae687 | L3 agent           | 869ad61a-35db-4814-a76b-327302d90bc7 | nova              | :-)   | UP    | neutron-l3-agent          |
| ffd613da-7b50-4bb6-ad3a-ad4edc0614ab | L3 agent           | 5043f58d-4d5b-41fe-9fb1-e308de27a2e9 | nova              | :-)   | UP    | neutron-l3-agent          |

  [[e-mail] chinmay(admin)]# neutron l3-agent-router-add 7c43afd0-1bc4-4dba-8d31-72789e8b4b81 3d3ec419-e0e5-45fa-a661-bfeb14261658
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Added router 3d3ec419-e0e5-45fa-a661-bfeb14261658 to L3 agent

The SNAT and the gateway port are active on the router now. This should have restored the network connectivity for the instances. If not, I would test restarting one instance.

[[e-mail] chinmay(admin)]# neutron l3-agent-list-hosting-router 3d3ec419-e0e5-45fa-a661-bfeb14261658
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------------------------+----------------+-------+----------+
| id                                   | host                                 | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------------------+----------------+-------+----------+
| 7c43afd0-1bc4-4dba-8d31-72789e8b4b81 | 3cf9bcfd-311e-45c9-b37b-2be594420f2c | True           | :-)   |          |
+--------------------------------------+--------------------------------------+----------------+-------+----------+
[[e-mail] chinmay(admin)]# openstack port list --router 3d3ec419-e0e5-45fa-a661-bfeb14261658
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                           | Status |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| 7b0b7565-931d-4e9b-bffc-bc657571ac55 |      | fa:16:3e:08:fb:b3 | ip_address='10.10.68.14', subnet_id='b0bc7439-9d43-4c7b-8d9c-753eae0cd761'   | ACTIVE |
| a6c7461a-50df-4bdb-90f1-6dc9f4f72d7e |      | fa:16:3e:22:24:90 | ip_address='10.10.68.1', subnet_id='b0bc7439-9d43-4c7b-8d9c-753eae0cd761'    | ACTIVE |
| bef02752-5a2d-4840-affa-06a48f28753b |      | fa:16:3e:99:69:12 | ip_address='85.159.144.23', subnet_id='c93011ee-e739-45e7-ab5e-5427a8ed6764' | ACTIVE |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

Additional Information

ZD case #1385911