VM Fails to Start Due to Boot Volume Format Mismatch in Block Device Mapping

Problem

A virtual machine is active in the Private Cloud Director platform but fails to start or reboot. Every start and hard reboot attempt fails consistently. The VM console may display an "Instance not found" error.

The Compute Service logs on the hypervisor (/var/log/pf9/ostackhost.log) record the following error for every start attempt:

libvirt.libvirtError: internal error: process exited while connecting to monitor:
qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","driver":"qcow2",
"file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format

The VM reports as ACTIVE in the platform, but no corresponding libvirt domain exists on the hypervisor.

Environment

• Private Cloud Director Virtualization - v2026.1 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2026.1 and Higher
• Component: Compute Service
• NFS-backed boot volume (Tintri NFS or equivalent)

Cause

The VM's boot volume is hosted on NFS storage. Under certain conditions — most commonly during a live migration — the on-disk format of the volume file can change from qcow2 to raw. The format field inside the connection_info JSON column of the block_device_mapping table in the Compute Service database is not automatically updated when this format change occurs.

When the Compute Service subsequently attempts to start the VM, it reads the stale "format": "qcow2" metadata from the database and instructs QEMU to open the disk as qcow2 via libvirt. QEMU rejects the file because the actual on-disk format is raw, causing every start and reboot attempt to fail with Image is not in qcow2 format.

Diagnostics

The following checks confirm the VM is affected by this specific issue. Complete all steps before proceeding to Workaround.

1

Step 1 — Identify the Boot Volume and Hypervisor

Run the following command to retrieve the hypervisor hostname and boot volume UUID for the stuck VM.

OpenStack CLI

$ openstack server show <VM_UUID> -c OS-EXT-SRV-ATTR:hypervisor_hostname -c volumes_attached -c status

Sample Output

+--------------------------------------+---------------------------------------------------+
| Field                                | Value                                             |
+--------------------------------------+---------------------------------------------------+
| OS-EXT-SRV-ATTR:hypervisor_hostname  | [HYPERVISOR_HOSTNAME]                             |
| status                               | ACTIVE                                            |
| volumes_attached                     | delete_on_termination='True', id='[VOLUME_UUID]'  |
+--------------------------------------+---------------------------------------------------+

Note the [HYPERVISOR_HOSTNAME] and the volume id ([VOLUME_UUID]) — both are required in subsequent steps.

2

Step 2 — Confirm the Libvirt Domain Is Absent

SSH to the hypervisor identified in Step 1 and verify no libvirt domain exists for the VM.

Compute Host

$ virsh list --all | grep <VM_UUID>

If no output is returned, the libvirt domain is absent. The VM has no running or stopped guest on the hypervisor, even though the platform reports it as ACTIVE.

3

Step 3 — Confirm the Actual On-Disk Volume Format

On the same hypervisor, locate the boot volume file and inspect its format using qemu-img.

First, locate the full path to the volume file:

Compute Host

$ ls /opt/pf9/data/state/mnt/*/volume-<VOLUME_UUID>

Sample Output

/opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID]

Then run qemu-img info on the full path returned above:

Compute Host

$ qemu-img info /opt/pf9/data/state/mnt/<MOUNT_UUID>/volume-<VOLUME_UUID>

Sample Output

image: /opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID]
file format: raw
virtual size: [DISK_SIZE] GiB ([DISK_BYTES] bytes)
disk size: [USED_SIZE] GiB

file format: raw confirms the on-disk format does not match the qcow2 value stored in the database.

4

Step 4 — Confirm the Stale Format Metadata in the Database

Self-Hosted only: This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Query the block_device_mapping table in the Nova database to confirm the stale format entry.

Nova DB

MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format')
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;

Sample Output

+-----------------------------------------------+
| JSON_EXTRACT(connection_info,'$.data.format') |
+-----------------------------------------------+
| "qcow2"                                       |
+-----------------------------------------------+
1 row in set (0.002 sec)

A result of "qcow2" when the actual on-disk format is raw (confirmed in Step 3) confirms the database metadata mismatch causing the boot failure.

If all checks above match — domain absent in Step 2, file format: raw in Step 3, and "qcow2" in Step 4 — proceed to Workaround.

Resolution

No permanent code fix exists for this issue. Applying the Workaround below permanently corrects the stale format metadata in the Compute Service database and restores normal VM operation.

Workaround

The following procedure corrects the stale format value in the block_device_mapping table and restores the ability to start the VM.

Self-Hosted only (Steps 1–4): Steps 1–4 require direct access to the Compute Service database on the control plane. SaaS customers cannot perform these steps independently — contact Platform9 Support to have the database correction applied on your behalf. Steps 5–7 use the OpenStack CLI and can be run from any machine with the OpenStack RC file sourced.

1

Step 1 — Back Up the Current connection_info Row

Before making any changes, record the existing connection_info value so the original row can be restored if needed.

Nova DB

MySQL [nova]> SELECT connection_info
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;

Save the full output to a safe location before proceeding.

2

Step 2 — Verify a Single Boot Device Mapping Exists

Confirm there is only one block device mapping with boot_index = 0 for the VM before modifying any records.

Nova DB

MySQL [nova]> SELECT id, device_name, source_type, boot_index, volume_id, deleted
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;

Sample Output

+------+-------------+-------------+------------+--------------------+---------+
| id   | device_name | source_type | boot_index | volume_id          | deleted |
+------+-------------+-------------+------------+--------------------+---------+
| [ID] | /dev/vda    | image       |          0 | [VOLUME_UUID]      |       0 |
+------+-------------+-------------+------------+--------------------+---------+
1 row in set (0.001 sec)

The output must show exactly one row with boot_index = 0. If multiple rows exist or the volume UUID does not match the boot volume identified in Diagnostics, stop and contact Platform9 Support before proceeding.

3

Step 3 — Correct the Volume Format in the Database

This step directly modifies the Compute Service database. Ensure the backup from Step 1 is saved before proceeding.

Update the format field in connection_info from qcow2 to raw.

Nova DB

MySQL [nova]> UPDATE block_device_mapping
              SET connection_info = JSON_SET(connection_info, '$.data.format', 'raw')
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;

Sample Output

Query OK, 1 row affected (0.008 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Rows matched: 1 Changed: 1 confirms the update was applied to exactly one row.

4

Step 4 — Verify the Format Was Updated

Confirm the format field now reads "raw".

Nova DB

MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format')
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;

Sample Output

+-----------------------------------------------+
| JSON_EXTRACT(connection_info,'$.data.format') |
+-----------------------------------------------+
| "raw"                                         |
+-----------------------------------------------+
1 row in set (0.002 sec)

The result must be "raw" before proceeding.

5

Step 5 — Check VM Status and Reset State if Needed

Check the current server status using the OpenStack CLI.

OpenStack CLI

$ openstack server show <VM_UUID> -f value -c status

Sample Output

ACTIVE

If the status returns ACTIVE, proceed directly to Step 6. If the status returns ERROR, run the following command to reset the VM state before rebooting:

OpenStack CLI

$ openstack server set --state active <VM_UUID>

6

Step 6 — Hard Reboot the VM

Issue a hard reboot to restart the VM using the corrected volume format metadata.

OpenStack CLI

$ openstack server reboot --hard <VM_UUID>

Allow up to 60 seconds for the reboot to complete before verifying.

7

Step 7 — Verify the VM Is Running

Confirm the VM is ACTIVE with power state Running.

OpenStack CLI

$ openstack server show <VM_UUID> -f value -c status -c OS-EXT-STS:power_state

Sample Output

1
ACTIVE

Power state 1 indicates the VM is running. Optionally, confirm the libvirt domain is now present on the hypervisor:

Compute Host

$ virsh list --all | grep <VM_UUID>

Sample Output

 [DOMAIN_ID]   [VM_UUID]   running

The VM is successfully restored when the status is ACTIVE, power state is 1, and the libvirt domain appears as running on the hypervisor.

Rollback: If the VM does not start successfully after the hard reboot, restore the original connection_info value saved in Step 1 by running:

MySQL [nova]> UPDATE block_device_mapping
              SET connection_info = '<ORIGINAL_CONNECTION_INFO_JSON>'
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;

Then contact Platform9 Support for further assistance.

Additional Information

For issues where a volume creation operation fails due to an image format mismatch on NFS or Tintri storage, see Volume Creation Failure Due to Image Format Mismatch.