VMs Inaccessible due to Underlying Hypervisors Frequently Going Offline

Problem

The VMs are inaccessible via SSH and console as the Hypervisor connectivity is flapping frequently.

$ ssh ubuntu@<VM_IP>
Connection closed by [VM_IP] port 22

Environment

• Private Cloud Director Virtualization - till v2025.4-497

• Self-Hosted Private Cloud Director Virtualization - till v2025.4-497

• Component - Networking

Cause

• When the DHCP lease expires, the host interface gets reconfigured. During this process, interfaces that are attached to the bridge also receive IP addresses unexpectedly. Because of this, ovn-controller enters a failed state while processing its configuration updates.

• This is a known issue and is reported internally with ID: PCD-2492.

Diagnostics

neutron logs:

pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via [IP] dev br-phy1 metric 0 failed. 2:b'RTNETLINK answers: File exists\\n' 
session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last):   
File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 625, in handle_msg     current_config = get_current_config()  
File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 262, in get_current_config     config = app_db.get_current_config()   
File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_app_db.py", line 80, in get_current_config     'config': app.get_config(),   
File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_pf9_app.py", line 311, in get_config     raise ConfigOperationError() pf9app.exceptions.ConfigOperationError

Network related files:

$ ls -lt /run/systemd/network/
-rw-r----- 1 root systemd-network 213 Jun 11 06:25 10-netplan-bond0.600.network 
-rw-r----- 1 root systemd-network  63 Jun 11 06:25 10-netplan-bond0.601.netdev 
-rw-r----- 1 root systemd-network 130 Jun 11 06:25 10-netplan-bond0.601.network 
-rw-r----- 1 root systemd-network  63 Jun 11 06:25 10-netplan-bond0.602.netdev 
-rw-r----- 1 root systemd-network 131 Jun 11 06:25 10-netplan-bond0.602.network 
-rw-r----- 1 root systemd-network  65 Jun 11 06:25 10-netplan-bond0.3001.netdev 
-rw-r----- 1 root systemd-network 155 Jun 11 06:25 10-netplan-bond0.3001.network 
-rw-r----- 1 root systemd-network  63 Jun 11 06:25 10-netplan-bond0.600.netdev 
-rw-r----- 1 root systemd-network 196 Jun 11 06:25 10-netplan-bond0.netdev 
-rw-r----- 1 root systemd-network 195 Jun 11 06:25 10-netplan-bond0.network 
-rw-r----- 1 root root             82 Jun 11 06:25 10-netplan-enp1s0f0.link 
-rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f0.network 
-rw-r----- 1 root root             82 Jun 11 06:25 10-netplan-enp1s0f1.link 
-rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f1.network

/var/log/pf9/hostagent.log

pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via <IP> dev br-phy12 metric 0 failed. 2:b'RTNETLINK answers: File exists\\n'
session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'}
Traceback (most recent call last):
  File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 625, in handle_msg
    current_config = get_current_config()
  File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 262, in get_current_config
    config = app_db.get_current_config()
  File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_app_db.py", line 80, in get_current_config
    'config': app.get_config(),
  File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_pf9_app.py", line 311, in get_config
    raise ConfigOperationError()
pf9app.exceptions.ConfigOperationError

Resolution

The fix for the bug is available in the v2025.6 and above releases.

Workaround

Check if the interface IP is assigned to the bridge and the default route is associated with the bridge.

• If yes, delete the default route via bondx.x.

• If not, reboot the host and check again.

$ ip r
default via [VM_IP] dev br-phy12 
default via [VM_IP] dev bond0.10 proto static
1.1.1.1_24 dev br-phy12 proto kernel scope link src 1.1.1.251 
1.1.1.2_24 dev bond0.10 proto kernel scope link src 1.1.1.251 
1.1.1.3_24 dev br-phy15 proto kernel scope link src 1.1.1.251 
1.1.1.4_24 dev bond0.50 proto kernel scope link src 1.1.1.251 

$ sudo ip route del default via <VM_IP> dev bond0.10

Additional Information

• For further questions/concerns regarding the bug, reach out to the Platform9 Support Team.