Network Agents Missing for Re-onboarded Host Due to Stale OVN Chassis Entry

Problem

• Network agents for a host are missing after re-onboarding
• Host does not appear correctly in:

  $ openstack network agent list

• OVN agents (ovn-controller / metadata agent) are not visible or not alive
• OVN controller logs show following error in /var/log/pf9/ovn/ovn-controller.log :

  transaction error: constraint violation
  multiple rows in "Encap" table with same (geneve, [IP])

• Host networking functionality is degraded due to missing OVN registration

Environment

• Private Cloud Director Virtualization - v2025.4 and Higher
• Private Cloud Director Kubernetes – v2025.4 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
• Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher
• Component - Networking

Cause

This issue occurs specifically on re-onboarded hosts where a stale OVN chassis entry remains in the OVN Southbound database. This typically happens when the host is not properly decommissioned prior to re-onboarding, resulting in residual OVN state persisting in the control plane. To prevent such issues, ensure the host is fully deauthorized and decommissioned by following the official Platform9 documentation. More details on how deauthorization and decommission work can be found in Demystifying Deauthorization and Decommissioning of PCD Hosts.

• During re-onboarding, the host attempts to register a new chassis
• The old chassis entry (from before decommissioning) still exists with the same Geneve IP
• This leads to a constraint violation in the OVN database
• As a result:

  • New chassis registration fails
  • OVN agents do not come up
  • Network agents appear missing in PCD.

Diagnostics

1. Check Network Agents

$ openstack network agent list

• Confirm that the re-onboarded host is missing or agents are not alive.

2. Check OVN Controller Logs (on the re-onboarded host)

$ cat /var/log/pf9/ovn/ovn-controller.log 

Look for:

|ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"Encap\" table to have identical values (geneve and \"[HOST_IP_ADDRESS]\") for index on columns \"type\" and \"ip\". First row, with UUID [UUID_1], was inserted by this transaction. Second row, with UUID [UUID_2], existed in the database before this transaction and was not modified by the transaction.","error":"constraint violation"}

Resolution

1. Identify the existing (stale) chassis entry:

Refer the Additional Information section to understand how to execute ovn-* commands.

$ ovn-sbctl show

Example Output (Before Cleanup):

#example output
Chassis "[OLD_CHASSIS_UUID]"
    hostname: "[OLD_HOST_ID]"
    Encap geneve
        ip: "[HOST_GENEVE_IP]"
        options: {csum="true"}

Observation:

• A chassis entry already exists for the Geneve IP
• This entry is stale and preventing new registration
• The expected (current) host is not present in the output

1. Delete the stale chassis entry:

   $ ovn-sbctl chassis-del <OLD_CHASSIS_UUID>

2. Restart OVN services on the host:

   $ systemctl restart ovn-controller
   $ systemctl restart pf9-neutron-ovn-metadata-agent

3. Verify new chassis registration:

   $ ovn-sbctl show
   #example output
   Chassis "[NEW_CHASSIS_UUID]"
       hostname: "[NEW_HOST-ID]"
       Encap geneve
           ip: "[HOST_GENEVE_IP]"
           options: {csum="true"}

4. Verify network agents are now visible:

   $ openstack network agent list

Validation

• Network agents for the host are visible in openstack network agent list
• Agents show Alive :-) and State UP
• No constraint violation errors in OVN logs

Additional Information

• To run ovn-* commands on the hosts onboarded to PCD, execute below steps.

1. Create an environment file ovs-alias.rc as below:

EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}')
export NBDB=tcp:${EXTERNAL_ID}:6641
export SBDB=tcp:${EXTERNAL_ID}:6642
alias ovn-sbctl="ovn-sbctl --db=$SBDB"
alias ovn-nbctl="ovn-nbctl --db=$NBDB"
alias ovn-trace="ovn-trace --db=$SBDB"

1. Export the rc file and start using the ovn commands:

$ source ovs-alias.rc
$ ovn-sbctl show

ovn commands can also be executed from inside the OVN South Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the steps below.

1. Access the OVN South Bond Pod in the Management Cluster using the command below.

$ kubectl -n <REGION_NAMESPACE> exec -it <OVN_SOUTH_POD> -- bash --kubeconfig <PATH_TO_KUBECONFIG>

• Always ensure that hosts are properly decommissioned before re-onboarding to avoid stale OVN entries. Refer to the official documentation.