Delete Orphaned Allocations Using Nova-Manage

Issue

• When an instance is deleted/migrated/evacuated, sometimes its corresponding allocation on the source host is not deleted from placement, and they remain stale.

Resolution

• Query the placement API to list all the stale allocations. The command below will list allocations for instances that are either deleted or moved to other compute nodes. Run the below command from any one compute node.

Command

$ kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash
$ nova-manage placement audit --verbose

• Verify if the instance exists or is on another compute node.

Command

$ openstack server list --all-projects --host <SOURCE_HOST>
$ openstack server show -c OS-EXT-SRV-ATTR:host <INSTANCE_UUID>

• Identify the resource provider ID

Command

$ openstack resource provider list
//For Specific VM check
$ openstack resource provider allocation show <INSTANCE_UUID>

• After validating that the allocation is stale and the instance is deleted or moved to another compute node, delete the orphaned allocation using the command below.

Command

$  kubectl exec -it <nova_api_osapi-pod_name> -n <NS> -- bash
$  nova-manage placement audit --verbose --delete <INSTANCE_UUID>

• If there are multiple orphaned allocations, delete them altogether with the command below.

$ kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash
$ nova-manage placement audit --verbose --delete

• After running above we can run a heal allocations for placement to ensure that all placement entries are consistent

Command

$ kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash
$ nova-manage placement heal_allocations

• To validate that there are no more orphaned allocations, please run the command

Command

$ kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash
$ nova-manage placement audit --verbose

Root Cause

• When evacuating an instance, if the compute service on the source host is stopped, then the records in placement are not deleted automatically.

• As a result, ostackhost service cannot communicate with the nova-conductor service running on the control plane, so details are not shared there, and it results in nova-compute remaining in an impression that still holds the instance.