VM Stuck in Migrating State After Live Migration

Problem

A virtual machine remains in MIGRATING state in the PCD UI indefinitely after a live migration attempt. No VM operations can be initiated on the instance, and the state does not transition to ACTIVE or ERROR on its own. Refreshing the PCD UI does not resolve the condition.

Environment

• Private Cloud Director Virtualization — v2025.10 and Higher
• Self-Hosted Private Cloud Director Virtualization — v2025.10-180 and Higher
• Component: Compute Service

Cause

A live migration interacts with three layers that must stay in sync: the libvirt driver on the source and destination hypervisors, the Nova database (instances and migrations tables), and the OpenStack Placement service (which tracks VCPU and MEMORY_MB allocations per resource provider). When any layer falls out of sync, the VM can become permanently stuck in task_state = migrating.

Two distinct failure modes produce this symptom:

• Migration accepted but failed before starting (VM still on source) — the destination compute service accepted the migration but failed before memory transfer began. The migration record is in accepted status. An orphaned Placement allocation exists under the migration UUID pointing to the destination RP.
• Migration completed at hypervisor level but management plane desynced (VM on destination) — the VM successfully transferred to the destination at the libvirt level, but the Nova conductor did not process the completion event. The migration record is stuck in running status. The Nova instances table still references the source host, and the Placement service holds a stale allocation under the migration UUID on the source RP while the instance UUID has zero allocation.

Diagnostics

1. Get the libvirt domain name for the VM:

OpenStack CLI

$ openstack server show <VM_UUID> -c OS-EXT-SRV-ATTR:instance_name

Sample Output

+-----------------------------------------+-------------------+
| Field                                   | Value             |
+-----------------------------------------+-------------------+
| OS-EXT-SRV-ATTR:instance_name           | [INSTANCE_NAME]   |
+-----------------------------------------+-------------------+

2. SSH into both the source and destination compute hosts and check whether the VM domain is running:

Compute Host

$ sudo virsh list --all | grep <INSTANCE_NAME>

Sample Output (VM found on destination)

 42   [VM_UUID]   running

3. Confirm the migration record status:

Self-Hosted only: The Nova DB query below requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on their behalf.

Retrieve the database password and MySQL pod IP, then open an interactive session:

Control Plane Node

$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"

Nova DB

USE nova;
SELECT id, uuid, status, source_compute, dest_compute, updated_at
FROM migrations
WHERE instance_uuid = '<VM_UUID>'
ORDER BY id DESC
LIMIT 5;

Note the uuid value from the output — referred to as <MIGRATION_UUID> throughout the Workaround steps.

Use the migration status and the virsh result to identify the correct method:

Migration status	VM location (virsh)	Method
accepted	Source host only	Method 1 — Migration Failed Before Completing
running	Destination host	Method 2 — Migration Completed at Hypervisor Level

If the checks above match, proceed to the Workaround.

Workaround

Apply the method that matches the diagnostic results above.

Method 1 — Migration Failed Before Completing (VM on Source)

The migration record is in accepted status and the VM is still running on the source host.

Before proceeding, confirm that the VM domain is running only on the source host using virsh list. Applying Method 1 to a VM that has already completed migration to the destination will corrupt Placement state.

Self-Hosted only: Steps 1 and 3 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact Platform9 Support to have this completed on their behalf.

1

Step 1 — Mark the Migration Record as Error

Retrieve database credentials and open a Nova DB session:

Control Plane Node

$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')
$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"

Nova DB

USE nova;
UPDATE migrations SET status = 'error', updated_at = NOW() WHERE uuid = '<MIGRATION_UUID>';

Verify:

Nova DB

SELECT id, uuid, status FROM migrations WHERE uuid = '<MIGRATION_UUID>';

Sample Output

+------+--------------+--------+
| id   | uuid         | status |
+------+--------------+--------+
| [ID] | [VM_UUID]    | error  |
+------+--------------+--------+

2

Step 2 — Reset the VM Task State

OpenStack CLI

$ openstack server set --state active <VM_UUID>

Verify:

OpenStack CLI

$ openstack server show <VM_UUID> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host

Sample Output

+----------------------------+---------------+
| Field                      | Value         |
+----------------------------+---------------+
| OS-EXT-SRV-ATTR:host       | [SOURCE_HOST] |
| OS-EXT-STS:task_state      | None          |
| status                     | ACTIVE        |
+----------------------------+---------------+

3

Step 3 — Delete the Orphaned Migration Placement Allocation

The migration UUID holds a provisional Placement allocation created when the migration was accepted. Remove this orphaned allocation:

OpenStack CLI

$ openstack resource provider allocation delete <MIGRATION_UUID>

Verify the allocation is removed:

OpenStack CLI

$ openstack resource provider allocation show <MIGRATION_UUID>

An empty table confirms the orphaned allocation has been cleared.

4

Step 4 — Verify Final State

OpenStack CLI

$ openstack server show <VM_UUID> | grep -E 'status|task_state|host'

The VM should show status = ACTIVE, task_state = None, and host pointing to the source compute node. The VM is now fully recovered and available for operations.

Method 2 — Migration Completed at Hypervisor Level (VM on Destination)

The migration record is stuck in running status. The VM is running on the destination hypervisor, but the Nova instances table still points to the source host and the Placement service holds a stale allocation for the migration UUID on the source RP while the instance UUID has zero allocation.

This procedure modifies the Nova database directly and performs surgery on Placement allocations. Take a Nova DB backup before proceeding. Performing steps out of order will result in ghost resource reservations on the source RP or an over-allocated destination RP.

Self-Hosted only: This entire procedure requires direct access to the control plane. SaaS customers cannot perform these steps independently — contact Platform9 Support to have this completed on their behalf.

Fix the instances table and Placement allocations before marking the migration as completed (Steps 4–7 must complete before Step 8). Marking the migration complete first triggers a Nova cleanup path that may remove allocations incorrectly.

1

Step 1 — Verify VM Accessibility on the Destination

SSH into the destination compute host and confirm the VM domain is running:

Compute Host

$ sudo virsh list | grep <INSTANCE_NAME>

Also confirm the VM is reachable over the network from the assigned IP before proceeding.

2

Step 2 — Retrieve Database Credentials and Take a Nova Database Backup

Retrieve the database password and MySQL pod IP. The $DB_PASS and $IP shell variables set here are reused in Steps 4 and 8.

Control Plane Node

$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")
$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')

Take a backup of the relevant Nova tables:

Control Plane Node

$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysqldump -u root -h $IP -p$DB_PASS nova instances migrations" \
    > nova_backup_$(date +%Y%m%d_%H%M%S).sql

3

Step 3 — Collect Required Values

Collect all variable values needed for subsequent steps before making any changes.

Get the destination RP UUID (this value is also used as the node field in the Nova instances table):

OpenStack CLI

$ openstack resource provider list | grep <DEST_HOSTNAME>

Sample Output

+--------------------------------------+-----------------+------------+
| uuid                                 | name            | generation |
+--------------------------------------+-----------------+------------+
| [DEST_RP_UUID]                       | [DEST_HOSTNAME] | 42         |
+--------------------------------------+-----------------+------------+

Get the project ID, user ID, and flavor resource requirements for the VM:

OpenStack CLI

$ openstack server show <VM_UUID> -c project_id -c user_id -c flavor

Note the vcpus and ram (in MB) values from the flavor output — these are the <VCPU_COUNT> and <MEMORY_MB> values used in Step 6.

Confirm the destination RP has sufficient remaining capacity:

OpenStack CLI

$ openstack resource provider usage show <DEST_RP_UUID>
$ openstack resource provider inventory list <DEST_RP_UUID>

Verify the destination RP can accommodate the VM's VCPU and MEMORY_MB requirements before proceeding.

4

Step 4 — Update the Instance Record in Nova DB

Update the instances table to point to the destination host. Use <DEST_RP_UUID> as the node value (the Nova instances.node field stores the compute node's RP UUID).

Connect to the Nova DB using the credentials retrieved in Step 2:

Control Plane Node

$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"

Nova DB

USE nova;
UPDATE instances
SET
  host        = '<DEST_HOSTNAME>',
  node        = '<DEST_RP_UUID>',
  vm_state    = 'active',
  power_state = 1,
  task_state  = NULL,
  updated_at  = NOW()
WHERE uuid = '<VM_UUID>';

Verify:

Nova DB

SELECT uuid, host, node, vm_state, power_state, task_state
FROM instances
WHERE uuid = '<VM_UUID>';

5

Step 5 — Reset the VM State via OpenStack API

OpenStack CLI

$ openstack server set --state active <VM_UUID>

Verify:

OpenStack CLI

$ openstack server show <VM_UUID> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host

Sample Output

+----------------------------+----------------+
| Field                      | Value          |
+----------------------------+----------------+
| OS-EXT-SRV-ATTR:host       | [DEST_HOSTNAME]|
| OS-EXT-STS:task_state      | None           |
| status                     | ACTIVE         |
+----------------------------+----------------+

6

Step 6 — Set the Instance Placement Allocation on the Destination RP

Create the correct Placement allocation for the instance UUID on the destination RP. Include all resources tracked by Placement for the VM's flavor (at minimum VCPU and MEMORY_MB; add DISK_GB if local storage is tracked in the environment):

OpenStack CLI

$ openstack resource provider allocation set \
  --allocation rp=<DEST_RP_UUID>,MEMORY_MB=<MEMORY_MB>,VCPU=<VCPU_COUNT> \
  --project <PROJECT_ID> \
  --user <USER_ID> \
  <VM_UUID>

Verify the instance allocation is now on the destination RP:

OpenStack CLI

$ openstack resource provider allocation show <VM_UUID>

The output should show <DEST_RP_UUID> with the correct VCPU and MEMORY_MB values.

7

Step 7 — Delete the Stale Migration Placement Allocation

Remove the migration consumer's orphaned allocation from the source RP:

OpenStack CLI

$ openstack resource provider allocation delete <MIGRATION_UUID>

Verify the allocation is removed:

OpenStack CLI

$ openstack resource provider allocation show <MIGRATION_UUID>

An empty table confirms the stale source RP allocation has been cleared.

8

Step 8 — Mark the Migration Record as Completed

With the instances table and Placement state now consistent with the destination, mark the migration as completed.

Connect to the Nova DB using the credentials retrieved in Step 2:

Control Plane Node

$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS"

Nova DB

USE nova;
UPDATE migrations
SET status     = 'completed',
    updated_at = NOW()
WHERE uuid = '<MIGRATION_UUID>';

Verify:

Nova DB

SELECT id, uuid, status, dest_compute FROM migrations WHERE uuid = '<MIGRATION_UUID>';

Sample Output

+------+-----------+-----------+----------------+
| id   | uuid      | status    | dest_compute   |
+------+-----------+-----------+----------------+
| [ID] | [VM_UUID] | completed | [DEST_HOSTNAME]|
+------+-----------+-----------+----------------+

9

Step 9 — Final Verification

Run all three checks to confirm a clean recovery:

OpenStack CLI

$ openstack server show <VM_UUID> | grep -E 'status|task_state|host|progress'

Sample Output

+---------------------------+----------------+
| Field                     | Value          |
+---------------------------+----------------+
| OS-EXT-SRV-ATTR:host      | [DEST_HOSTNAME]|
| OS-EXT-STS:task_state     | None           |
| progress                  | 0              |
| status                    | ACTIVE         |
+---------------------------+----------------+

OpenStack CLI

# Instance allocation — should show DEST_RP_UUID with correct VCPU and MEMORY_MB
$ openstack resource provider allocation show <VM_UUID>

# Migration allocation — should return an empty table
$ openstack resource provider allocation show <MIGRATION_UUID>

Expected results:

• status = ACTIVE, task_state = None, host = [DEST_HOSTNAME], progress = 0
• Instance allocation references [DEST_RP_UUID] with correct resource values
• Migration allocation returns an empty table

If these steps prove insufficient to resolve the issue, reach out to the Platform9 Support team for additional assistance.

Additional Information

Related Articles:

• VM Migration Failure Which Results the VM in "ERROR" State