> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/compute/vm-fails-to-start-due-to-boot-volume-format-mismatch-in-block-device-mapping.md).

# VM Fails to Start Due to Boot Volume Format Mismatch in Block Device Mapping

## Problem

A virtual machine is active in the Private Cloud Director platform but fails to start or reboot. Every start and hard reboot attempt fails consistently. The VM console may display an "Instance not found" error.

The Compute Service logs on the hypervisor (`/var/log/pf9/ostackhost.log`) record the following error for every start attempt:

```
libvirt.libvirtError: internal error: process exited while connecting to monitor:
qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","driver":"qcow2",
"file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format
```

The VM reports as `ACTIVE` in the platform, but no corresponding libvirt domain exists on the hypervisor.

## Environment

* Private Cloud Director Virtualization - v2026.1 and Higher
* Self-Hosted Private Cloud Director Virtualization - v2026.1 and Higher
* Component: Compute Service
* NFS-backed boot volume (Tintri NFS or equivalent)

## Cause

The VM's boot volume is hosted on NFS storage. Under certain conditions — most commonly during a live migration — the on-disk format of the volume file can change from `qcow2` to `raw`. The `format` field inside the `connection_info` JSON column of the `block_device_mapping` table in the Compute Service database is not automatically updated when this format change occurs.

When the Compute Service subsequently attempts to start the VM, it reads the stale `"format": "qcow2"` metadata from the database and instructs QEMU to open the disk as `qcow2` via libvirt. QEMU rejects the file because the actual on-disk format is `raw`, causing every start and reboot attempt to fail with `Image is not in qcow2 format`.

## Diagnostics

The following checks confirm the VM is affected by this specific issue. Complete all steps before proceeding to [Workaround](#workaround).

{% stepper %}
{% step %}

#### Step 1 — Identify the Boot Volume and Hypervisor

Run the following command to retrieve the hypervisor hostname and boot volume UUID for the stuck VM.

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c OS-EXT-SRV-ATTR:hypervisor_hostname -c volumes_attached -c status
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+--------------------------------------+---------------------------------------------------+
| Field                                | Value                                             |
+--------------------------------------+---------------------------------------------------+
| OS-EXT-SRV-ATTR:hypervisor_hostname  | [HYPERVISOR_HOSTNAME]                             |
| status                               | ACTIVE                                            |
| volumes_attached                     | delete_on_termination='True', id='[VOLUME_UUID]'  |
+--------------------------------------+---------------------------------------------------+
```

{% endcode %}

Note the `[HYPERVISOR_HOSTNAME]` and the volume `id` (`[VOLUME_UUID]`) — both are required in subsequent steps.
{% endstep %}

{% step %}

#### Step 2 — Confirm the Libvirt Domain Is Absent

SSH to the hypervisor identified in Step 1 and verify no libvirt domain exists for the VM.

{% code title="Compute Host" %}

```bash
$ virsh list --all | grep <VM_UUID>
```

{% endcode %}

If no output is returned, the libvirt domain is absent. The VM has no running or stopped guest on the hypervisor, even though the platform reports it as `ACTIVE`.
{% endstep %}

{% step %}

#### Step 3 — Confirm the Actual On-Disk Volume Format

On the same hypervisor, locate the boot volume file and inspect its format using `qemu-img`.

First, locate the full path to the volume file:

{% code title="Compute Host" %}

```bash
$ ls /opt/pf9/data/state/mnt/*/volume-<VOLUME_UUID>
```

{% endcode %}

{% code title="Sample Output" %}

```bash
/opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID]
```

{% endcode %}

Then run `qemu-img info` on the full path returned above:

{% code title="Compute Host" %}

```bash
$ qemu-img info /opt/pf9/data/state/mnt/<MOUNT_UUID>/volume-<VOLUME_UUID>
```

{% endcode %}

{% code title="Sample Output" %}

```bash
image: /opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID]
file format: raw
virtual size: [DISK_SIZE] GiB ([DISK_BYTES] bytes)
disk size: [USED_SIZE] GiB
```

{% endcode %}

`file format: raw` confirms the on-disk format does not match the `qcow2` value stored in the database.
{% endstep %}

{% step %}

#### Step 4 — Confirm the Stale Format Metadata in the Database

{% hint style="info" %}
**Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf.
{% endhint %}

Query the `block_device_mapping` table in the Nova database to confirm the stale format entry.

{% code title="Nova DB" %}

```sql
MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format')
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+-----------------------------------------------+
| JSON_EXTRACT(connection_info,'$.data.format') |
+-----------------------------------------------+
| "qcow2"                                       |
+-----------------------------------------------+
1 row in set (0.002 sec)
```

{% endcode %}

A result of `"qcow2"` when the actual on-disk format is `raw` (confirmed in Step 3) confirms the database metadata mismatch causing the boot failure.
{% endstep %}
{% endstepper %}

If all checks above match — domain absent in Step 2, `file format: raw` in Step 3, and `"qcow2"` in Step 4 — proceed to [Workaround](#workaround).

## Resolution

No permanent code fix exists for this issue. Applying the [Workaround](#workaround) below permanently corrects the stale format metadata in the Compute Service database and restores normal VM operation.

## Workaround

The following procedure corrects the stale `format` value in the `block_device_mapping` table and restores the ability to start the VM.

{% hint style="info" %}
**Self-Hosted only (Steps 1–4):** Steps 1–4 require direct access to the Compute Service database on the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have the database correction applied on your behalf. Steps 5–7 use the OpenStack CLI and can be run from any machine with the OpenStack RC file sourced.
{% endhint %}

{% stepper %}
{% step %}

#### Step 1 — Back Up the Current connection\_info Row

Before making any changes, record the existing `connection_info` value so the original row can be restored if needed.

{% code title="Nova DB" %}

```sql
MySQL [nova]> SELECT connection_info
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;
```

{% endcode %}

Save the full output to a safe location before proceeding.
{% endstep %}

{% step %}

#### Step 2 — Verify a Single Boot Device Mapping Exists

Confirm there is only one block device mapping with `boot_index = 0` for the VM before modifying any records.

{% code title="Nova DB" %}

```sql
MySQL [nova]> SELECT id, device_name, source_type, boot_index, volume_id, deleted
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND deleted = 0;
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+------+-------------+-------------+------------+--------------------+---------+
| id   | device_name | source_type | boot_index | volume_id          | deleted |
+------+-------------+-------------+------------+--------------------+---------+
| [ID] | /dev/vda    | image       |          0 | [VOLUME_UUID]      |       0 |
+------+-------------+-------------+------------+--------------------+---------+
1 row in set (0.001 sec)
```

{% endcode %}

The output must show exactly one row with `boot_index = 0`. If multiple rows exist or the volume UUID does not match the boot volume identified in Diagnostics, stop and contact [Platform9 Support](https://support.platform9.com/) before proceeding.
{% endstep %}

{% step %}

#### Step 3 — Correct the Volume Format in the Database

{% hint style="warning" %}
This step directly modifies the Compute Service database. Ensure the backup from Step 1 is saved before proceeding.
{% endhint %}

Update the `format` field in `connection_info` from `qcow2` to `raw`.

{% code title="Nova DB" %}

```sql
MySQL [nova]> UPDATE block_device_mapping
              SET connection_info = JSON_SET(connection_info, '$.data.format', 'raw')
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;
```

{% endcode %}

{% code title="Sample Output" %}

```bash
Query OK, 1 row affected (0.008 sec)
Rows matched: 1  Changed: 1  Warnings: 0
```

{% endcode %}

`Rows matched: 1 Changed: 1` confirms the update was applied to exactly one row.
{% endstep %}

{% step %}

#### Step 4 — Verify the Format Was Updated

Confirm the `format` field now reads `"raw"`.

{% code title="Nova DB" %}

```sql
MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format')
              FROM block_device_mapping
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+-----------------------------------------------+
| JSON_EXTRACT(connection_info,'$.data.format') |
+-----------------------------------------------+
| "raw"                                         |
+-----------------------------------------------+
1 row in set (0.002 sec)
```

{% endcode %}

The result must be `"raw"` before proceeding.
{% endstep %}

{% step %}

#### Step 5 — Check VM Status and Reset State if Needed

Check the current server status using the OpenStack CLI.

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -f value -c status
```

{% endcode %}

{% code title="Sample Output" %}

```bash
ACTIVE
```

{% endcode %}

If the status returns `ACTIVE`, proceed directly to Step 6. If the status returns `ERROR`, run the following command to reset the VM state before rebooting:

{% code title="OpenStack CLI" %}

```bash
$ openstack server set --state active <VM_UUID>
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 6 — Hard Reboot the VM

Issue a hard reboot to restart the VM using the corrected volume format metadata.

{% code title="OpenStack CLI" %}

```bash
$ openstack server reboot --hard <VM_UUID>
```

{% endcode %}

Allow up to 60 seconds for the reboot to complete before verifying.
{% endstep %}

{% step %}

#### Step 7 — Verify the VM Is Running

Confirm the VM is `ACTIVE` with power state `Running`.

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -f value -c status -c OS-EXT-STS:power_state
```

{% endcode %}

{% code title="Sample Output" %}

```bash
1
ACTIVE
```

{% endcode %}

Power state `1` indicates the VM is running. Optionally, confirm the libvirt domain is now present on the hypervisor:

{% code title="Compute Host" %}

```bash
$ virsh list --all | grep <VM_UUID>
```

{% endcode %}

{% code title="Sample Output" %}

```bash
 [DOMAIN_ID]   [VM_UUID]   running
```

{% endcode %}

The VM is successfully restored when the status is `ACTIVE`, power state is `1`, and the libvirt domain appears as `running` on the hypervisor.

{% hint style="warning" %}
**Rollback:** If the VM does not start successfully after the hard reboot, restore the original `connection_info` value saved in Step 1 by running:

```sql
MySQL [nova]> UPDATE block_device_mapping
              SET connection_info = '<ORIGINAL_CONNECTION_INFO_JSON>'
              WHERE instance_uuid = '<VM_UUID>'
                AND volume_id = '<VOLUME_UUID>'
                AND deleted = 0;
```

Then contact [Platform9 Support](https://support.platform9.com/) for further assistance.
{% endhint %}
{% endstep %}
{% endstepper %}

## Additional Information

For issues where a volume creation operation fails due to an image format mismatch on NFS or Tintri storage, see [Volume Creation Failure Due to Image Format Mismatch](https://platform9.com/kb/pcd/compute/volume-creation-failure-due-to-image-format-mismatch).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/compute/vm-fails-to-start-due-to-boot-volume-format-mismatch-in-block-device-mapping.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.