> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/gpu/how-to-enable-pci-resource-class-based-placement-reporting-in-nova-2.md).

# GPU VM Resize and Cold Migration Fail with ReshapeFailed

## Problem

GPU Passthrough VM resize and cold migration operations fail intermittently even when the destination compute host has sufficient physical GPU capacity. The Nova scheduler filters out all available GPU hosts, and the VM may enter ERROR state or remain stuck in a migration state.

## Environment

* Self-Hosted Private Cloud Director Virtualization — v2025.10-180 and above versions.
* Private Cloud Director Virtualization — v2025.10-180 and above versions.
* Components**:** GPU and Compute Service.

## Diagnostics

{% stepper %}
{% step %}
Check for **ReshapeFailed** on in `ostackhost.log` on the source compute host:

{% code title="/var/log/pf9/ostackhost.log" %}

```ruby
CRITICAL: nova.compute.manager  Resource provider data migration failed fatally during startup
for node [HOSTNAME].: nova.exception.ReshapeFailed: Resource provider inventory and
allocation data migration failed: {"errors": [{"status": 409, "title": "Conflict",
"detail": "Unable to create allocation for 'CUSTOM_PCI_[PRODUCT_ID]' on resource provider
'[RP_UUID]'. The requested amount would exceed the capacity."}]}

ERROR: oslo_service.backend.eventlet.service  Error starting thread: nova.exception.ReshapeFailed
```

{% endcode %}
{% endstep %}

{% step %}
Check if PF device is still reported as attached

{% code title="/var/log/pf9/ostackhost.log" %}

```bash
DEBUG nova.virt.libvirt.host  Cannot get MAC address of the PF [PCI_ADDRESS_1].It is probably attached to a guest already
DEBUG nova.virt.libvirt.host  Cannot get MAC address of the PF [PCI_ADDRESS_2].It is probably attached to a guest already
```

{% endcode %}
{% endstep %}

{% step %}
Check for Placement generation conflict

{% code title="/var/log/pf9/ostackhost.log" %}

```bash
resource provider generation conflict for provider [RP_UUID]: actual: [ACTUAL_GEN], given: [GIVEN_GEN]
```

{% endcode %}
{% endstep %}

{% step %}
Check scheduler filtering out all GPU hosts

{% code title="/var/log/pf9/ostackhost.log" %}

```bash
Filter PciPassthroughFilter returned 0 hosts
```

{% endcode %}
{% endstep %}

{% step %}
Check for instance stuck in error/delete state

{% code title="/var/log/pf9/ostackhost.log" %}

```bash
stats={'failed_builds': '[VALUE]', 'num_instances': '[VALUE]', 'num_vm_stopped': '[VALUE]',
       'num_vm_active': '[VALUE]', 'num_vm_error': '[VALUE]'}

vm_state='error', task_state='deleting'
```

{% endcode %}
{% endstep %}

{% step %}
Check for stale allocations in compute service logs

{% code title="/var/log/pf9/ostackhost.log" %}

```bash
Instance [INSTANCE_UUID] has allocations against this compute host but is not found in the database.
```

{% endcode %}
{% endstep %}
{% endstepper %}

## Cause

When a GPU VM resize or cold migration fails mid-workflow, Nova does not always clean up the Placement allocations tied to the failed operation. This results in:

* **Stale consumer allocations** remaining on PCI child resource providers (e.g., `CUSTOM_PCI_[VENDOR_ID]_[PRODUCT_ID]`) for instance UUIDs that no longer exist in the Nova database.
* **Stale PCI child resource providers** that are no longer valid but still registered in Placement, holding consumed inventory.
* On the next `nova-compute` startup, Nova attempts a Placement reshape (inventory + allocation migration). If stale allocations exceed the declared inventory capacity, Placement returns **HTTP 409 Conflict**, causing `ReshapeFailed` and preventing the compute service from starting.
* A stuck instance in `vm_state='error'`, `task_state='deleting'` can continue to hold GPU/PCI allocations in Placement indefinitely, blocking all subsequent scheduling for the affected resource class.

Contributing Factors:

* Failed resize/migration workflows (especially multi-GPU flavor changes) do not roll back Placement allocations atomically.
* Placement generation conflicts during concurrent updates can leave allocations in an inconsistent state.
* The `nova-manage placement audit` tool only detects orphans for VCPU/MEMORY — it may not catch all PCI child RP orphans.
* Compute service restart after a failed reshape loop can compound the stale state.

## Resolution

### **Option A** — Restart compute service to re-register PCI RPs cleanly

After stale allocations have been cleaned up, restart the compute service to allow Nova to re-register PCI child resource providers with correct inventory.

{% stepper %}
{% step %}
Restart the compute service

{% code title="Hypervisor" %}

```bash
sudo systemctl restart pf9-ostackhost
sudo systemctl status pf9-ostackhost
```

{% endcode %}
{% endstep %}

{% step %}
Verify no 409 errors on startup
{% endstep %}
{% endstepper %}

### **Option B** — Use `nova-manage` to clean orphaned allocations

{% hint style="warning" %}
Option B is applicable only for Self-Hosted PCD Customers. Refer <https://platform9.com/kb/pcd-ts/host/delete-orphaned-allocations-using-nova-manage>
{% endhint %}

**Note:** `nova-manage placement audit` may not detect all PCI child RP orphans. If the issue persists after this step, proceed to [#option-c-manual-placement-api-cleanup-for-stale-pci-child-resource-providers](#option-c-manual-placement-api-cleanup-for-stale-pci-child-resource-providers "mention").<br>

### **Option C** — Manual Placement API cleanup for stale PCI child resource providers

Use this method when `nova-manage` does not resolve the issue, or when the stale entry is a PCI child resource provider (not just a consumer allocation).

{% stepper %}
{% step %}
Get the Placement endpoint

```bash
PLACEMENT_URL=$(openstack endpoint list --service placement \
  --interface public -f value -c URL)
echo $PLACEMENT_URL
```

{% endstep %}

{% step %}
Get an auth token

```bash
export TOKEN=$(openstack token issue -f value -c id)
```

{% endstep %}

{% step %}
Inspect allocations on the stale PCI resource provider and get **CONSUMER\_UUID**

```bash
curl -k -H "X-Auth-Token: $TOKEN" \
  "$PLACEMENT_URL/resource_providers/<STALE_RP_UUID>/allocations"
```

{% endstep %}

{% step %}
For each consumer UUID in the output, verify it is stale

```bash
openstack server show <CONSUMER_UUID> --fit-width
```

> Expected output for a stale consumer: `No server with a name or ID...`
> {% endstep %}

{% step %}
Delete the stale consumer allocation

```bash
curl -k -X DELETE -H "X-Auth-Token: $TOKEN" \
  "$PLACEMENT_URL/allocations/<CONSUMER_UUID>"
```

{% endstep %}

{% step %}
Verify usage is now zero on the RP

```bash
openstack resource provider usage show <STALE_RP_UUID> --insecure
```

{% endstep %}

{% step %}
Delete the stale child resource provider

```bash
openstack resource provider delete <STALE_RP_UUID> --insecure
```

{% endstep %}
{% endstepper %}

After the cleanup, Nova will re-register the PCI child resource provider cleanly on the next `pf9-ostackhost` restart.

## Validation

After performing the cleanup steps, validate in the following order:

{% stepper %}
{% step %}
Verify compute services are up on all GPU hosts

```bash
openstack compute service list --service nova-compute
```

{% endstep %}

{% step %}
Verify PCI inventory is correctly reported

```bash
openstack resource provider list --name <COMPUTE_HOST>
openstack resource provider inventory list <COMPUTE_NODE_RESOURCEPROVIDER_UUID>
```

{% endstep %}

{% step %}
Check allocation candidates for a GPU flavor

```bash
openstack allocation candidate list \
  --resource VCPU=<VCPU_COUNT> \
  --resource MEMORY_MB=<MEMORY_MB> \
  --resource CUSTOM_PCI_<VENDOR_ID>_<PRODUCT_ID>=1
```

{% endstep %}

{% step %}
Test normal VM cold migration (non-GPU) first.
{% endstep %}

{% step %}
Test GPU VM cold migration.
{% endstep %}

{% step %}
Test GPU VM flavor resize.
{% endstep %}
{% endstepper %}

## Additional Information

* This is a known product gap tracked in PCD-6116. Currently, there is no automated reconciliation for stale PCI child resource providers. The manual cleanup steps described in this KB are the supported recovery method until the product fix is available.
* The product fix will enable automated or controller-level cleanup of stale Placement allocations and PCI child resource providers.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/gpu/how-to-enable-pci-resource-class-based-placement-reporting-in-nova-2.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.