VMs Inaccessible due to Underlying Hypervisors Frequently Going Offline
Problem
The VMs are inaccessible via SSH and console as the Hypervisor connectivity is flapping frequently.
$ ssh ubuntu@<VM_IP>Connection closed by [VM_IP] port 22Environment
- Private Cloud Director Virtualization - v2025.4 and Higher
- Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
- Component - Networking
Cause
- When the DHCP lease expires, the host interface gets reconfigured. During this process, interfaces that are attached to the bridge also receive IP addresses unexpectedly. Because of this,
ovn-controllerenters a failed state while processing its configuration updates. - This is a known issue and is reported internally with ID: PCD-2492.
Diagnostics
In the Neutron logs:
pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via [IP] dev br-phy1 metric 0 failed. 2:b'RTNETLINK answers: File exists\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 625, in handle_msg current_config = get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 262, in get_current_config config = app_db.get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/app_db.py", line 80, in get_current_config 'config': app.get_config(), File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app.py", line 311, in get_config raise ConfigOperationError() pf9app.exceptions.ConfigOperationErrorNetwork related files:
$ ls -lt /run/systemd/network/ -rw-r----- 1 root systemd-network 213 Jun 11 06:25 10-netplan-bond0.600.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.601.netdev -rw-r----- 1 root systemd-network 130 Jun 11 06:25 10-netplan-bond0.601.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.602.netdev -rw-r----- 1 root systemd-network 131 Jun 11 06:25 10-netplan-bond0.602.network -rw-r----- 1 root systemd-network 65 Jun 11 06:25 10-netplan-bond0.3001.netdev -rw-r----- 1 root systemd-network 155 Jun 11 06:25 10-netplan-bond0.3001.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.600.netdev -rw-r----- 1 root systemd-network 196 Jun 11 06:25 10-netplan-bond0.netdev -rw-r----- 1 root systemd-network 195 Jun 11 06:25 10-netplan-bond0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f0.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f1.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f1.networkHostagent Logs:
pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via <IP> dev br-phy12 metric 0 failed. 2:b'RTNETLINK answers: File exists\n'session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'}Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 625, in handle_msg current_config = get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 262, in get_current_config config = app_db.get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/app_db.py", line 80, in get_current_config 'config': app.get_config(), File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app.py", line 311, in get_config raise ConfigOperationError()pf9app.exceptions.ConfigOperationErrorResolution
The fix for the bug is available in the v2025.6 release.
Workaround
Check if the interface IP is assigned to the bridge and the default route is associated with the bridge.
- If yes, delete the default route via bondx.x.
- If not, reboot the host and check again.
x
$ ip rdefault via [VM_IP] dev br-phy12 default via [VM_IP] dev bond0.10 proto static1.1.1.1/24 dev br-phy12 proto kernel scope link src 1.1.1.251 1.1.1.2/24 dev bond0.10 proto kernel scope link src 1.1.1.251 1.1.1.3/24 dev br-phy15 proto kernel scope link src 1.1.1.251 1.1.1.4/24 dev bond0.50 proto kernel scope link src 1.1.1.251 $ sudo ip route del default via <VM_IP> dev bond0.10Additional Information
- For further questions/concerns regarding the bug, reach out to the Platform9 Support Team.
Was this page helpful?