Force Remove the Host Role From DB

Problem

Unable to remove a host role from a specific host via the Platform9 UI, pcdctl and curl command. The UI shows an RoleUpdateConflict error when attempting to deauthorize and decommission the host.

$  curl -s -X DELETE -H "X-Auth-Token: $token" https://<PCD_FQDN>/resmgr/v2/hosts/<HOST-UUID>/roles/<ROLE_NAME> -vvv
...
"message":"RoleUpdateConflict: Role cannot be updated in the current state: Cannot remove role [ROLE_NAME] from host [HOST_UUID] in the current state".

Environment

• Self-Hosted Private Cloud Director Virtualization

• Component: ResMgr

Cause

The host role remained in a not-applied state in the backend database, which blocked further updates or role removal actions through the UI. This inconsistency prevented the host from being deauthorized or decommissioned.

Diagnostics

These steps are applicable only for Self-hosted Private Cloud Director product

To check the role state:

1. Login to the resmgr DB:

$ NS=<NAMESPACE>

$ DB_PASS=$(kubectl exec -i -n "$NS" $(kubectl get po -n "$NS" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") -c resmgr -- /bin/bash -c "DB=\$(consul-dump-yaml --start-key customers/\$CUSTOMER_ID/regions/\$REGION_ID/db | awk '/dbserver:/ {print \$2}'); consul-dump-yaml --start-key customers/\$CUSTOMER_ID/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")

$ IP=$(kubectl get pods -n $NS  $(kubectl get po -n $NS -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") --template '{{.status.podIP}}')

$ kubectl exec -it -n "$NS" $(kubectl get po -n "$NS" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"

1. Check the role state:

MySQL [resmgr]> "SELECT * FROM host_role_map WHERE res_id = '<HOST_UUID>';"

+----------------+-------------+---------------+
| res_id         | rolename    | current_state |
+----------------+-------------+---------------+
| [HOST_UUID]    | [ROLE_NAME] |  not-applied  |
+----------------+-------------+---------------+

The not-applied state indicates that the role was not fully or correctly applied, resulting in a conflict when attempting UI-based role changes.

Resolution

Caution: The following steps involve direct database manipulation and should only be performed at most care.

1. Delete the role from the backend database:

MySQL [resmgr]> "DELETE FROM host_role_map WHERE res_id='<HOST_UUID>';"

1. Verify that the role mapping has been removed:

MySQL [resmgr]> "SELECT * FROM host_role_map WHERE res_id = '<HOST_UUID>';"

## Expected output: Empty set

Caution: The following steps only if need to delete the host as well.

1. Optionally, delete the host from the database if needed:

MySQL [resmgr]> "DELETE FROM hosts WHERE id='<HOST_UUID>';"

1. Purge PF9 packages (if host is being decommissioned):

$ sudo dpkg --purge --force-all pf9-hostagent
$ sudo dpkg --purge --force-all pf9-comms
$ sudo dpkg --purge --force-all pf9-neutron-ovn-controller
$ sudo dpkg --purge --force-all pf9-ostackhost

1. Clean up PF9 services and directories:

$ sudo systemctl stop 'pf9-*.service'
$ sudo systemctl disable 'pf9-*.service'
$ sudo systemctl daemon-reexec
$ sudo systemctl daemon-reload

$ rm -rf /etc/pf9 /opt/pf9 /var/log/pf9 /var/opt/pf9

Validation

• Confirm that the role mapping associated with the host has been removed

• The host is no longer listed in the UI under Infrastructure > Hosts.

• If re-onboarding, reauthorize the host and apply roles cleanly.

• Review logs (/var/log/pf9/hostagent.log, /var/log/pf9/comms/) to ensure role application is successful.