> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/compute/vm-stuck-in-migrating-state-after-live-migration.md).

# VM Stuck in Migrating State After Live Migration

### Problem

A virtual machine remains in MIGRATING state in the PCD UI indefinitely after a live migration attempt. No VM operations can be initiated on the instance, and the state does not transition to ACTIVE or ERROR on its own. Refreshing the PCD UI does not resolve the condition.

### Environment

* Private Cloud Director Virtualization — v2025.10 and Higher
* Self-Hosted Private Cloud Director Virtualization — v2025.10-180 and Higher
* Component: Compute Service

## Cause

A live migration interacts with three layers that must stay in sync: the `libvirt` driver on the source and destination hypervisors, the Nova database (`instances` and `migrations` tables), and the OpenStack Placement service (which tracks VCPU and MEMORY\_MB allocations per resource provider). When any layer falls out of sync, the VM can become permanently stuck in `task_state = migrating`.

Two distinct failure modes produce this symptom:

* **Migration accepted but failed before starting (VM still on source)** — the destination compute service accepted the migration but failed before memory transfer began. The migration record is in `accepted` status. An orphaned Placement allocation exists under the migration UUID pointing to the destination RP.
* **Migration completed at hypervisor level but management plane desynced (VM on destination)** — the VM successfully transferred to the destination at the `libvirt` level, but the Nova conductor did not process the completion event. The migration record is stuck in `running` status. The Nova `instances` table still references the source host, and the Placement service holds a stale allocation under the migration UUID on the source RP while the instance UUID has zero allocation.

## Diagnostics

**1. Get the libvirt domain name for the VM:**

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c OS-EXT-SRV-ATTR:instance_name
```

{% endcode %}

{% code title="Sample Output" %}

```
+-----------------------------------------+-------------------+
| Field                                   | Value             |
+-----------------------------------------+-------------------+
| OS-EXT-SRV-ATTR:instance_name           | [INSTANCE_NAME]   |
+-----------------------------------------+-------------------+
```

{% endcode %}

**2. SSH into both the source and destination compute hosts and check whether the VM domain is running:**

{% code title="Compute Host" %}

```bash
$ sudo virsh list --all | grep <INSTANCE_NAME>
```

{% endcode %}

{% code title="Sample Output (VM found on destination)" %}

```
 42   [VM_UUID]   running
```

{% endcode %}

**3. Confirm the migration record status:**

{% hint style="info" %}
**Self-Hosted only:** The Nova DB query below requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf.
{% endhint %}

Retrieve the database password and MySQL pod IP, then open an interactive session:

{% code title="Control Plane Node" %}

```bash
$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"
```

{% endcode %}

{% code title="Nova DB" %}

```sql
USE nova;
SELECT id, uuid, status, source_compute, dest_compute, updated_at
FROM migrations
WHERE instance_uuid = '<VM_UUID>'
ORDER BY id DESC
LIMIT 5;
```

{% endcode %}

Note the `uuid` value from the output — referred to as `<MIGRATION_UUID>` throughout the Workaround steps.

Use the migration `status` and the `virsh` result to identify the correct method:

| Migration `status` | VM location (virsh) | Method                                                                                                                     |
| ------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `accepted`         | Source host only    | [Method 1 — Migration Failed Before Completing](#method-1--migration-failed-before-completing-vm-on-source)                |
| `running`          | Destination host    | [Method 2 — Migration Completed at Hypervisor Level](#method-2--migration-completed-at-hypervisor-level-vm-on-destination) |

If the checks above match, proceed to the [Workaround](#workaround).

## Workaround

Apply the method that matches the diagnostic results above.

### Method 1 — Migration Failed Before Completing (VM on Source)

The migration record is in `accepted` status and the VM is still running on the source host.

{% hint style="warning" %}
Before proceeding, confirm that the VM domain is running **only** on the source host using `virsh list`. Applying Method 1 to a VM that has already completed migration to the destination will corrupt Placement state.
{% endhint %}

{% hint style="info" %}
**Self-Hosted only:** Steps 1 and 3 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf.
{% endhint %}

{% stepper %}
{% step %}

#### Step 1 — Mark the Migration Record as Error

Retrieve database credentials and open a Nova DB session:

{% code title="Control Plane Node" %}

```bash
$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"
```

{% endcode %}

{% code title="Nova DB" %}

```sql
USE nova;
UPDATE migrations SET status = 'error', updated_at = NOW() WHERE uuid = '<MIGRATION_UUID>';
```

{% endcode %}

Verify:

{% code title="Nova DB" %}

```sql
SELECT id, uuid, status FROM migrations WHERE uuid = '<MIGRATION_UUID>';
```

{% endcode %}

{% code title="Sample Output" %}

```
+------+--------------+--------+
| id   | uuid         | status |
+------+--------------+--------+
| [ID] | [VM_UUID]    | error  |
+------+--------------+--------+
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 2 — Reset the VM Task State

{% code title="OpenStack CLI" %}

```bash
$ openstack server set --state active <VM_UUID>
```

{% endcode %}

Verify:

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host
```

{% endcode %}

{% code title="Sample Output" %}

```
+----------------------------+---------------+
| Field                      | Value         |
+----------------------------+---------------+
| OS-EXT-SRV-ATTR:host       | [SOURCE_HOST] |
| OS-EXT-STS:task_state      | None          |
| status                     | ACTIVE        |
+----------------------------+---------------+
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 3 — Delete the Orphaned Migration Placement Allocation

The migration UUID holds a provisional Placement allocation created when the migration was accepted. Remove this orphaned allocation:

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation delete <MIGRATION_UUID>
```

{% endcode %}

Verify the allocation is removed:

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation show <MIGRATION_UUID>
```

{% endcode %}

An empty table confirms the orphaned allocation has been cleared.
{% endstep %}

{% step %}

#### Step 4 — Verify Final State

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> | grep -E 'status|task_state|host'
```

{% endcode %}

The VM should show `status = ACTIVE`, `task_state = None`, and `host` pointing to the source compute node. The VM is now fully recovered and available for operations.
{% endstep %}
{% endstepper %}

### Method 2 — Migration Completed at Hypervisor Level (VM on Destination)

The migration record is stuck in `running` status. The VM is running on the destination hypervisor, but the Nova `instances` table still points to the source host and the Placement service holds a stale allocation for the migration UUID on the source RP while the instance UUID has zero allocation.

{% hint style="danger" %}
This procedure modifies the Nova database directly and performs surgery on Placement allocations. Take a Nova DB backup before proceeding. Performing steps out of order will result in ghost resource reservations on the source RP or an over-allocated destination RP.
{% endhint %}

{% hint style="info" %}
**Self-Hosted only:** This entire procedure requires direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf.
{% endhint %}

{% hint style="warning" %}
Fix the `instances` table and Placement allocations **before** marking the migration as `completed` (Steps 4–7 must complete before Step 8). Marking the migration complete first triggers a Nova cleanup path that may remove allocations incorrectly.
{% endhint %}

{% stepper %}
{% step %}

#### Step 1 — Verify VM Accessibility on the Destination

SSH into the destination compute host and confirm the VM domain is running:

{% code title="Compute Host" %}

```bash
$ sudo virsh list | grep <INSTANCE_NAME>
```

{% endcode %}

Also confirm the VM is reachable over the network from the assigned IP before proceeding.
{% endstep %}

{% step %}

#### Step 2 — Retrieve Database Credentials and Take a Nova Database Backup

Retrieve the database password and MySQL pod IP. The `$DB_PASS` and `$IP` shell variables set here are reused in Steps 4 and 8.

{% code title="Control Plane Node" %}

```bash
$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')
```

{% endcode %}

Take a backup of the relevant Nova tables:

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysqldump -u root -h $IP -p$DB_PASS nova instances migrations" \
    > nova_backup_$(date +%Y%m%d_%H%M%S).sql
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 3 — Collect Required Values

Collect all variable values needed for subsequent steps before making any changes.

**Get the destination RP UUID** (this value is also used as the `node` field in the Nova `instances` table):

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider list | grep <DEST_HOSTNAME>
```

{% endcode %}

{% code title="Sample Output" %}

```
+--------------------------------------+-----------------+------------+
| uuid                                 | name            | generation |
+--------------------------------------+-----------------+------------+
| [DEST_RP_UUID]                       | [DEST_HOSTNAME] | 42         |
+--------------------------------------+-----------------+------------+
```

{% endcode %}

**Get the project ID, user ID, and flavor resource requirements for the VM:**

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c project_id -c user_id -c flavor
```

{% endcode %}

Note the `vcpus` and `ram` (in MB) values from the flavor output — these are the `<VCPU_COUNT>` and `<MEMORY_MB>` values used in Step 6.

**Confirm the destination RP has sufficient remaining capacity:**

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider usage show <DEST_RP_UUID>
$ openstack resource provider inventory list <DEST_RP_UUID>
```

{% endcode %}

Verify the destination RP can accommodate the VM's VCPU and MEMORY\_MB requirements before proceeding.
{% endstep %}

{% step %}

#### Step 4 — Update the Instance Record in Nova DB

Update the `instances` table to point to the destination host. Use `<DEST_RP_UUID>` as the `node` value (the Nova `instances.node` field stores the compute node's RP UUID).

Connect to the Nova DB using the credentials retrieved in Step 2:

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"
```

{% endcode %}

{% code title="Nova DB" %}

```sql
USE nova;
UPDATE instances
SET
  host        = '<DEST_HOSTNAME>',
  node        = '<DEST_RP_UUID>',
  vm_state    = 'active',
  power_state = 1,
  task_state  = NULL,
  updated_at  = NOW()
WHERE uuid = '<VM_UUID>';
```

{% endcode %}

Verify:

{% code title="Nova DB" %}

```sql
SELECT uuid, host, node, vm_state, power_state, task_state
FROM instances
WHERE uuid = '<VM_UUID>';
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 5 — Reset the VM State via OpenStack API

{% code title="OpenStack CLI" %}

```bash
$ openstack server set --state active <VM_UUID>
```

{% endcode %}

Verify:

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host
```

{% endcode %}

{% code title="Sample Output" %}

```
+----------------------------+----------------+
| Field                      | Value          |
+----------------------------+----------------+
| OS-EXT-SRV-ATTR:host       | [DEST_HOSTNAME]|
| OS-EXT-STS:task_state      | None           |
| status                     | ACTIVE         |
+----------------------------+----------------+
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 6 — Set the Instance Placement Allocation on the Destination RP

Create the correct Placement allocation for the instance UUID on the destination RP. Include all resources tracked by Placement for the VM's flavor (at minimum VCPU and MEMORY\_MB; add DISK\_GB if local storage is tracked in the environment):

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation set \
  --allocation rp=<DEST_RP_UUID>,MEMORY_MB=<MEMORY_MB>,VCPU=<VCPU_COUNT> \
  --project <PROJECT_ID> \
  --user <USER_ID> \
  <VM_UUID>
```

{% endcode %}

Verify the instance allocation is now on the destination RP:

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation show <VM_UUID>
```

{% endcode %}

The output should show `<DEST_RP_UUID>` with the correct VCPU and MEMORY\_MB values.
{% endstep %}

{% step %}

#### Step 7 — Delete the Stale Migration Placement Allocation

Remove the migration consumer's orphaned allocation from the source RP:

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation delete <MIGRATION_UUID>
```

{% endcode %}

Verify the allocation is removed:

{% code title="OpenStack CLI" %}

```bash
$ openstack resource provider allocation show <MIGRATION_UUID>
```

{% endcode %}

An empty table confirms the stale source RP allocation has been cleared.
{% endstep %}

{% step %}

#### Step 8 — Mark the Migration Record as Completed

With the `instances` table and Placement state now consistent with the destination, mark the migration as `completed`.

Connect to the Nova DB using the credentials retrieved in Step 2:

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"
```

{% endcode %}

{% code title="Nova DB" %}

```sql
USE nova;
UPDATE migrations
SET status     = 'completed',
    updated_at = NOW()
WHERE uuid = '<MIGRATION_UUID>';
```

{% endcode %}

Verify:

{% code title="Nova DB" %}

```sql
SELECT id, uuid, status, dest_compute FROM migrations WHERE uuid = '<MIGRATION_UUID>';
```

{% endcode %}

{% code title="Sample Output" %}

```
+------+-----------+-----------+----------------+
| id   | uuid      | status    | dest_compute   |
+------+-----------+-----------+----------------+
| [ID] | [VM_UUID] | completed | [DEST_HOSTNAME]|
+------+-----------+-----------+----------------+
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 9 — Final Verification

Run all three checks to confirm a clean recovery:

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> | grep -E 'status|task_state|host|progress'
```

{% endcode %}

{% code title="Sample Output" %}

```
+---------------------------+----------------+
| Field                     | Value          |
+---------------------------+----------------+
| OS-EXT-SRV-ATTR:host      | [DEST_HOSTNAME]|
| OS-EXT-STS:task_state     | None           |
| progress                  | 0              |
| status                    | ACTIVE         |
+---------------------------+----------------+
```

{% endcode %}

{% code title="OpenStack CLI" %}

```bash
# Instance allocation — should show DEST_RP_UUID with correct VCPU and MEMORY_MB
$ openstack resource provider allocation show <VM_UUID>

# Migration allocation — should return an empty table
$ openstack resource provider allocation show <MIGRATION_UUID>
```

{% endcode %}

Expected results:

* `status = ACTIVE`, `task_state = None`, `host = [DEST_HOSTNAME]`, `progress = 0`
* Instance allocation references `[DEST_RP_UUID]` with correct resource values
* Migration allocation returns an empty table

If these steps prove insufficient to resolve the issue, reach out to the [Platform9 Support team](https://support.platform9.com/) for additional assistance.
{% endstep %}
{% endstepper %}

## Additional Information

**Related Articles:**

* [VM Migration Failure Which Results the VM in "ERROR" State](https://platform9.com/kb/pcd/compute/vm-migration-failure-which-results-the-vm-in-error-state)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/compute/vm-stuck-in-migrating-state-after-live-migration.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.