> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/storage/vm-fails-to-start-after-live-migration-failure-due-to-fibre-channel-bdm-corruption.md). # VM Fails to Start After Live Migration Failure Due to Fibre Channel BDM Corruption ## Problem After a live migration fails on a host using Fibre Channel block storage, the affected VM shows `vm_state: error` while `power_state` remains `Running` on the original host. The VM continues to run until an operation such as a start, hard reboot, or a second migration attempt is issued. A start or hard reboot then fails during volume disconnect, the domain is destroyed, and the VM does not come back online. Additional observable symptoms: * `openstack volume show` for a non-multiattach volume lists two or more attachments, each on a different host * `openstack server event list` shows one or more `live-migration` events with a failure status * The `launched_at` timestamp of the instance predates the failed migrations, indicating the domain never moved {% hint style="danger" %} Do not start, hard-reboot, live-migrate, or reset the state of the affected VM before completing the [Workaround](#workaround). Each of these operations reads the corrupt Block Device Mapping and can remove SCSI paths from an unrelated volume that is in use by another VM on the same host. {% endhint %} ## Environment * Private Cloud Director Virtualization - All versions * Self-Hosted Private Cloud Director Virtualization - All versions * Component: Compute Service, Block Storage Service (Fibre Channel — HPE Alletra 9000/MP, Primera, 3PAR) ## Cause During a live migration, the Compute Service runs `pre_live_migration` on the destination host. This step creates a new volume attachment for the destination host in the Block Storage Service and rewrites the `connection_info` field of the Block Device Mapping (BDM) record in the `nova.block_device_mapping` table with the destination host's LUN ID and initiator WWPNs. The `attachment_id` column of the BDM is repointed to the destination host's attachment. If the migration fails after `pre_live_migration` runs and the rollback does not complete, the BDM record is never reverted. The domain still runs on the source host, but the BDM now describes the destination host's view of the volume. On 3PAR-lineage arrays (HPE Alletra 9000/MP, Primera, 3PAR) each host receives its own VLUN numbering. The same volume can be LUN 59 on one host, 36 on another, and 111 on a third. A LUN number is meaningless outside the host that the LUN was issued for. When the Compute Service later starts or hard-reboots the VM, os-brick builds `by-path` device names from the stale LUN number and resolves them on the source host. If an unrelated volume occupies that LUN number on the source host, os-brick attempts to flush and delete that volume's multipath map. The `map in use` protection in multipathd blocks the delete, but SCSI paths of the unrelated volume may already have been removed at that point. Resetting the VM state to `active` and retrying the migration clears the symptom without correcting the record. Each additional failed migration attempt leaks one more attachment per volume. Fibre Channel `connection_info` embeds the LUN in four fields and identifies the host in two, so a repair that only rewrites `target_lun` leaves the record inconsistent: | Field | Action | Notes | | ------------------------------- | ---------------- | ---------------------------------------------------------- | | `data.target_lun` | Rewrite | Scalar LUN for the host holding the domain | | `data.targets` | Rewrite | Four `[target_wwn, lun]` pairs | | `data.initiator_target_map` | Rekey | Keyed by initiator WWPN; targets listed per initiator | | `data.initiator_target_lun_map` | Rekey and re-LUN | Drives os-brick path derivation | | `attachment_id` (column) | Repoint | To the attachment whose connector host holds the domain | | `data.multipath_id` | Leave | Array WWID — host-independent | | `data.device_path` | Leave | Derived from the WWID | | `data.target_wwn` | Leave | Array target ports; verify the ports are zoned to the host | | `volume_id`, `serial` | Leave | Volume identity | Existing records list all four array target WWNs under each initiator, even where a given HBA only sees two of them. That is the 3PAR driver's own convention — replicate that layout rather than modelling per-HBA zoning. Paths that do not resolve are tolerated and only produce a warning. ## Diagnostics While the VM is still `Running`, every command in this section is read-only and safe. The domain is only alive because nothing has re-read the BDM yet. Capture the live multipath output before anything stops the VM — those paths are the cheapest source of the correct LUN. {% stepper %} {% step %} ### Step 1 — Confirm the VM state and capture the migration failure Record the state fields and the host the Compute Service believes is running the VM. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:power_state -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------------------+-----------------+ | Field | Value | +-----------------------------+-----------------+ | OS-EXT-SRV-ATTR:host | [SOURCE_HOST] | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | status | ERROR | +-----------------------------+-----------------+ ``` {% endcode %} Identify the failed live-migration event and record the reason for the failure. The failure reason does not block the repair, but a repaired VM must not be live-migrated again until the original error is understood. {% code title="OpenStack CLI" %} ```bash $ openstack server event list $ openstack server event show ``` {% endcode %} {% code title="Sample Output" %} ```bash +------------------------------------------+--------------------------------------+----------------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+----------------+----------------------------+ | [REQ_UUID] | [INSTANCE_UUID] | live-migration | [TIMESTAMP] | +------------------------------------------+--------------------------------------+----------------+----------------------------+ ``` {% endcode %} On the source compute host, search the Compute Service log for the instance and request ID: {% code title="Compute Host" %} ```bash $ sudo grep -E "|" /var/log/pf9/ostackhost.log ``` {% endcode %} {% hint style="info" %} A migration that fails with `No valid host was found. There are not enough hosts available` fails at the scheduler, before a destination is chosen. `pre_live_migration` never runs in that case and the BDM is intact — those VMs usually need only a state reset. Check such instances with the detection queries in [Region-Wide Detection Queries](#region-wide-detection-queries) anyway, because an earlier attempt on the same instance may have progressed further. Leaked Placement allocations from prior failed migrations are a common cause of that scheduler message. {% endhint %} {% endstep %} {% step %} ### Step 2 — Locate the running domain Do not trust the `OS-EXT-SRV-ATTR:host` value on its own. Run the following on every host that appears as a source or destination in the migration history. {% code title="Compute Host" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash [DOMAIN_ID] [INSTANCE_NAME] running ``` {% endcode %} Exactly one host must show the domain. If more than one host has the domain defined, stop and treat the situation as a potential dual-writer before going further. {% endstep %} {% step %} ### Step 3 — Read the Compute Service BDM record {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D nova" mysql > use nova; mysql > select volume_id, attachment_id, connection_info from nova.block_device_mapping where instance_uuid='' and deleted=0 \G; ``` {% endcode %} {% code title="Sample Output" %} ```sql *************************** 1. row *************************** volume_id: [VOLUME_UUID] attachment_id: [ATTACHMENT_UUID] connection_info: {"driver_volume_type": "fibre_channel", "data": {"target_lun": [LUN], "targets": [...], "initiator_target_map": {"[INITIATOR_WWPN]": [...]}, "initiator_target_lun_map": {...}, "multipath_id": "[WWID]", "target_wwn": [...], ...}} ``` {% endcode %} Record, per volume: `volume_id`, `attachment_id`, `data.target_lun`, the keys of `data.initiator_target_map`, and `data.multipath_id`. The `multipath_id` value is the array WWID and is host-independent, so the WWID is a safe lookup key even when every other value in the record is wrong. {% endstep %} {% step %} ### Step 4 — Read the Block Storage Service attachment records {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```sql mysql > select id, attach_status, deleted, json_extract(connector,'\$.host') as chost from cinder.volume_attachment where volume_id=''; ``` {% endcode %} {% code title="Sample Output" %} ```sql +--------------------------------------+---------------+---------+-----------------------+ | id | attach_status | deleted | chost | +--------------------------------------+---------------+---------+-----------------------+ | [SOURCE_ATTACHMENT_UUID] | attached | 0 | "[SOURCE_HOST]" | | [DEST_ATTACHMENT_UUID] | attached | 0 | "[DESTINATION_HOST]" | +--------------------------------------+---------------+---------+-----------------------+ ``` {% endcode %} Note every connector host, the `attach_status`, and the `deleted` flag. Each row with `attach_status: attached` on a non-multiattach volume is a live VLUN on the array, not a stale database row. {% endstep %} {% step %} ### Step 5 — Read the host's storage identity Run on the host that holds the domain (from Step 2). Use the WWID from `data.multipath_id` in Step 3. {% code title="Compute Host" %} ```bash $ sudo multipath -ll $ sudo grep . /sys/class/fc_host/host*/port_state /sys/class/fc_host/host*/port_name $ sudo systemctl is-active multipathd ``` {% endcode %} {% code title="Sample Output" %} ```bash [WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running /sys/class/fc_host/host[N]/port_state:Online /sys/class/fc_host/host[N]/port_name:0x[INITIATOR_WWPN] active ``` {% endcode %} The last field of the `H:C:T:L` column on the path lines is the real LUN for this host. The `port_name` values are the initiator WWPNs of this host. {% hint style="info" %} A `multipath -ll` entry printing `##,##` instead of `3PARdata,VV` has lost all paths — multipath cannot read vendor strings without a live path. That is local device state, not an array unmap. If the Block Storage Service still shows the attachment as `attached`, the array mapping is intact. {% endhint %} {% endstep %} {% step %} ### Step 6 — Compare the records Any one of the following three mismatches confirms the corruption: * The BDM `attachment_id` (Step 3) resolves to a `cinder.volume_attachment` row (Step 4) whose connector host is not the host holding the domain (Step 2) * `connection_info.data.target_lun` (Step 3) differs from the `H:C:T:L` LUN of `multipath -ll ` on the host holding the domain (Step 5) * The keys of `initiator_target_map` (Step 3) are WWPNs that do not appear in `/sys/class/fc_host/host*/port_name` on the host holding the domain (Step 5) Supporting evidence: a non-multiattach volume with several `attached` attachment rows (Step 4), and an instance in `vm_state: error` with `power_state: Running` whose `launched_at` predates the migrations. {% endstep %} {% endstepper %} If all checks above match, proceed to [Workaround](#workaround). ### Region-Wide Detection Queries The Compute Service and Block Storage Service share a MariaDB instance, so every affected instance in a region can be detected with cross-database queries instead of inspecting instances one by one. {% hint style="info" %} **Self-Hosted only:** These queries require direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} BDM points at another host's attachment — every row returned is an instance that will fail to boot and will act on an unrelated LUN when a start is attempted: {% code title="Nova DB" %} ```sql SELECT i.uuid AS instance, i.hostname, i.host AS nova_host, i.vm_state, i.power_state, bdm.volume_id, bdm.device_name, bdm.attachment_id, JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) AS attach_host, JSON_EXTRACT(bdm.connection_info,'$.data.target_lun') AS bdm_lun FROM nova.block_device_mapping bdm JOIN nova.instances i ON i.uuid = bdm.instance_uuid AND i.deleted = 0 JOIN cinder.volume_attachment va ON va.id = bdm.attachment_id WHERE bdm.deleted = 0 AND bdm.destination_type = 'volume' AND JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) <> i.host; ``` {% endcode %} Leaked multi-host mappings on non-multiattach volumes: {% code title="Nova DB" %} ```sql SELECT va.volume_id, COUNT(*) AS live_attachments, GROUP_CONCAT(JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) SEPARATOR ' ') AS hosts FROM cinder.volume_attachment va JOIN cinder.volumes v ON v.id = va.volume_id AND v.multiattach = 0 AND v.deleted = 0 WHERE va.deleted = 0 AND va.attach_status = 'attached' GROUP BY va.volume_id HAVING COUNT(*) > 1 ORDER BY live_attachments DESC; ``` {% endcode %} Live migrations that did not complete cleanly — the population at risk: {% code title="Nova DB" %} ```sql SELECT instance_uuid, source_compute, dest_compute, status, created_at, updated_at FROM nova.migrations WHERE migration_type = 'live-migration' AND status NOT IN ('completed', 'reverted', 'cancelled') ORDER BY updated_at DESC LIMIT 100; ``` {% endcode %} ## Resolution No product fix is tracked for this condition at this time. Correct the BDM record and clear the leaked attachments using the [Workaround](#workaround) below. The workaround does not address the reason the live migration failed. Two failures on one instance mean a third is coming — identify the original migration error from Step 1 of [Diagnostics](#diagnostics) before live-migrating the repaired VM again. ## Workaround The repair path depends on whether the source host's attachment is still alive. From Step 4 of [Diagnostics](#diagnostics), find the attachment whose connector host is the host holding the domain: * `attach_status = attached` and `deleted = 0` — the Block Storage Service never ran `terminate_connection` for that host, so the VLUN still exists on the array. Any missing paths are local state only, recoverable with an FC rescan. This is the common case, and the steps below apply. * No such row, or `deleted = 1` — the source attachment and the array VLUN were removed. The mapping must be recreated by the Block Storage Service with a fresh attachment. The steps below do not apply — contact [Platform9 Support](https://support.platform9.com/). See [Additional Information](#additional-information). {% hint style="warning" %} Repair the BDM first and reset the VM state last. `openstack server set --state active` followed by a retry clears the symptom, leaves the cause in place, and turns one leaked attachment per volume into two. {% endhint %} {% stepper %} {% step %} ### Step 1 — Clear pathless multipath map shells Run on the host holding the domain. Only remove maps with zero paths, and only once the domain is stopped or already destroyed. A map with live paths is the real mapping — leave that map alone. {% hint style="warning" %} `multipath -f` on a map that still has live paths removes the device backing a running VM. Confirm the map prints `##,##` with no path lines before flushing. {% endhint %} {% code title="Compute Host" %} ```bash $ sudo multipath -f ``` {% endcode %} {% endstep %} {% step %} ### Step 2 — Rescan the Fibre Channel HBAs The rescan is additive and safe: the rescan re-probes targets and adds what is found, never removes. The loop is scoped to FC hosts rather than every SCSI host. {% code title="Compute Host" %} ```bash $ for h in /sys/class/fc_host/host*; do echo "- - -" | sudo tee /sys/class/scsi_host/$(basename $h)/scan > /dev/null; done ``` {% endcode %} {% hint style="warning" %} Do not run a global `multipath -r`. The global reload suspends every map on the host, briefly interrupting every other VM's devices. multipathd picks up new devices from uevents on its own. If a specific map does not reappear after approximately 30 seconds, add only that one map: `sudo multipathd add map `. {% endhint %} {% endstep %} {% step %} ### Step 3 — Read the true LUN for each volume Run once per volume, one WWID per invocation. {% hint style="info" %} For a general procedure to map a volume to a LUN ID, see [How to Identify the LUN ID Associated with an Instance Volume](https://platform9.com/kb/pcd/storage/how-to-identify-the-lun-id-associated-with-an-instance-volume). {% endhint %} {% code title="Compute Host" %} ```bash $ sudo multipath -ll ``` {% endcode %} {% code title="Sample Output" %} ```bash [WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running ``` {% endcode %} Expect four `active ready running` paths. The last field of `H:C:T:L` on those path lines is the LUN to write into the BDM record. Never carry a LUN number from another host and never infer one host's LUN from another's. {% endstep %} {% step %} ### Step 4 — Back up the BDM rows {% hint style="info" %} **Self-Hosted only:** Steps 4 through 6 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have them completed on your behalf. {% endhint %} The backup is mandatory before the UPDATE. {% code title="Control Plane Node" %} ```sql mysql > create table nova.bdm_backup_ as select * from nova.block_device_mapping where instance_uuid='' and deleted=0;" ``` {% endcode %} {% endstep %} {% step %} ### Step 5 — Correct the BDM record Run one UPDATE per volume. The statement repoints `attachment_id` at the attachment of the host holding the domain and rewrites all four LUN-bearing fields plus the initiator WWPNs. Substitute every value for the host in front of you: * `` — from Step 3 * `` — the `id` of the attachment whose connector host holds the domain, from Step 4 of [Diagnostics](#diagnostics) * `` to `` — the four array target WWNs already present in the existing record's `data.target_wwn` * `` and `` — the two `port_name` values from `/sys/class/fc_host/` on the host holding the domain, from Step 5 of [Diagnostics](#diagnostics) {% hint style="warning" %} Repointing `attachment_id` matters as much as the LUN. If `attachment_id` is left on the destination's attachment, a later detach or delete removes the wrong attachment and orphans the live VLUN — the same failure in reverse. {% endhint %} {% code title="Nova DB" %} ```sql UPDATE nova.block_device_mapping SET attachment_id = '', connection_info = JSON_SET( JSON_SET( JSON_SET( JSON_SET(connection_info, '$.data.target_lun', ), '$.data.targets', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ))), '$.data.initiator_target_map', JSON_OBJECT( '', JSON_ARRAY('','','',''), '', JSON_ARRAY('','','',''))), '$.data.initiator_target_lun_map', JSON_OBJECT( '', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', )), '', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', )))) WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` {% endcode %} {% endstep %} {% step %} ### Step 6 — Verify the corrected record Expect the correct attachment, the LUN read from the rescan, and only the local host's initiators. {% code title="Control Plane Node" %} ```sql mysql > select volume_id, attachment_id, json_extract(connection_info,'\$.data.target_lun') as lun, json_keys(json_extract(connection_info,'\$.data.initiator_target_lun_map')) as inits from nova.block_device_mapping where instance_uuid='' and deleted=0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+--------------------------------------+------+--------------------------------------------+ | volume_id | attachment_id | lun | inits | +--------------------------------------+--------------------------------------+------+--------------------------------------------+ | [VOLUME_UUID] | [SOURCE_ATTACHMENT_UUID] | [LUN]| ["[INITIATOR_1]", "[INITIATOR_2]"] | +--------------------------------------+--------------------------------------+------+--------------------------------------------+ ``` {% endcode %} {% endstep %} {% step %} ### Step 7 — Start the VM while tailing the Compute Service log In a second session on the host holding the domain, tail the log before issuing any state change: {% code title="Compute Host" %} ```bash $ sudo tail -f /var/log/pf9/ostackhost.log | grep -i -E "|connect_volume|disconnect_volume|lun-|multipath" ``` {% endcode %} The VM must be in `stopped` state before the start. Reset the state only if the VM is in `error`, then stop, then start. {% hint style="info" %} For background on state resets, see [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states). {% endhint %} {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:task_state $ openstack server set --state active # only if vm_state is error $ openstack server stop $ openstack server start ``` {% endcode %} The `disconnect_volume` call inside `_hard_reboot` is the step that fails on a corrupt record. With the corrected record the call walks the correct LUNs and passes. {% endstep %} {% step %} ### Step 8 — Confirm the domain's backing devices Confirm the guest is on the right disks, then confirm the guest answers on its own addresses. {% code title="Compute Host" %} ```bash $ sudo virsh domblklist --details ``` {% endcode %} {% code title="Sample Output" %} ```bash Type Device Target Source ------------------------------------------------------------ block disk vda /dev/disk/by-id/dm-uuid-mpath-[WWID] block disk vdb /dev/disk/by-id/dm-uuid-mpath-[WWID] ``` {% endcode %} {% endstep %} {% step %} ### Step 9 — Delete the leaked attachments {% hint style="danger" %} The incident is not closed until this step runs. A booted VM with the leaked attachments still in place means a non-multiattach volume is exposed read-write to two or three hosts. That is the data-corruption risk the whole repair exists to remove. {% endhint %} Delete each attachment belonging to a host that is not holding the domain. Use the API — the API drives the driver's `terminate_connection`, which is what actually removes the VLUN from the array. {% hint style="warning" %} Never delete `volume_attachment` rows with SQL. A database delete skips `terminate_connection` and orphans the array export permanently, with no record left to clean up from. Editing `connection_info` is a different and much smaller risk; deleting attachments is not. {% endhint %} {% code title="OpenStack CLI" %} ```bash $ openstack --os-volume-api-version 3.27 volume attachment delete ``` {% endcode %} Confirm one attachment per volume, on the host holding the domain: {% code title="OpenStack CLI" %} ```bash $ openstack volume show -c status -c attachments ``` {% endcode %} {% code title="Sample Output" %} ```bash +-------------+------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------+------------------------------------------------------------------------------------------------------+ | attachments | [{'id': '[VOLUME_UUID]', 'attachment_id': '[SOURCE_ATTACHMENT_UUID]', 'host_name': '[SOURCE_HOST]', ...}] | | status | in-use | +-------------+------------------------------------------------------------------------------------------------------+ ``` {% endcode %} Confirm the maps are gone on each former destination host: {% code title="Compute Host" %} ```bash $ sudo multipath -ll | grep -E "|" ``` {% endcode %} {% code title="Sample Output" %} ```bash (no output) ``` {% endcode %} {% endstep %} {% step %} ### Step 10 — Clear leaked Placement allocations {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl -n "" deploy/nova-api-osapi -- nova-manage placement audit --verbose ``` {% endcode %} {% endstep %} {% step %} ### Step 11 — Check the collateral victims If the VM was started or rebooted while the record was corrupt, os-brick removed SCSI paths from whatever volumes occupied the stale LUN numbers on the host. The FC rescan in Step 2 restores those paths, but confirm the path counts — another VM may be running degraded. Identify what sat at each stale LUN on the host holding the domain, then check that volume's health: {% code title="Compute Host" %} ```bash $ ls -l /dev/disk/by-path/ | grep 'lun-' $ sudo multipath -ll ``` {% endcode %} {% code title="Sample Output" %} ```bash [VICTIM_WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running ``` {% endcode %} Find which instance owns the victim volume: {% hint style="info" %} **Self-Hosted only:** This query requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```sql mysql > select instance_uuid, volume_id, device_name from nova.block_device_mapping where connection_info like '%%' and deleted=0; ``` {% endcode %} {% hint style="success" %} The repair is complete when the VM is `ACTIVE`, `virsh domblklist` shows the expected multipath devices, every affected volume shows exactly one attachment on the host holding the domain, no maps for the volume remain on former destination hosts, and every collateral-victim volume shows four `active ready running` paths. {% endhint %} {% endstep %} {% endstepper %} ## Additional Information **Source attachment removed (`deleted = 1` or no row):** When the source host's attachment has been deleted, the array VLUN was removed by `terminate_connection` and the mapping must be recreated by the Block Storage Service with a fresh attachment rather than a corrected record. This is a longer procedure — contact [Platform9 Support](https://support.platform9.com/). The sequence in that case is: SQL-patch `connection_info` to a LUN that resolves to nothing on the host so the disconnect no-ops harmlessly, then run `nova-manage volume_attachment refresh`, then start the VM. **Why `nova-manage volume_attachment refresh` is the wrong tool for a corrupt record:** The command exists to regenerate `connection_info` from a host connector and would produce correct values, but the command cannot run against a corrupt record. `_do_refresh` sends `remove_volume_connection` to the compute host before creating the new attachment, and that disconnect uses the current, wrong `connection_info`. `remove_volume_connection` swallows only `exception.NotFound`, so os-brick's `ExceptionChainer` propagates and aborts the command. The `finally` block then leaves the BDM with a blank `attachment_id` and `connection_info = NULL`, which is strictly worse: the instance cannot start at all until a refresh succeeds. When the source attachment is intact, correct the record with SQL and skip `refresh` entirely. If `refresh` is run, four traps apply: | Trap | Consequence | Handling | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `CONF.host` read from the wrong config | Exit 7, `HostConflict` — Private Cloud Director sets `instance.host` to the host UUID, but `nova-manage` defaults to `/etc/nova/nova.conf` and yields the bare hostname | Always pass `--config-dir /opt/pf9/etc/nova/conf.d` | | `get_connector` has no `--path` option | Writes nothing; prints the connector | Redirect: `get_connector --json > file` | | `enforce_multipath=True` is hardcoded | Raises if multipathd is down | Check `systemctl is-active multipathd` first | | Requires `vm_state == stopped` exactly | Exit 5 on `error` | Reset to `active`, then stop | `get_connector` runs on the hypervisor; `refresh` needs database and Block Storage Service access, so `refresh` runs in a Compute Service pod on the control plane. The connector file has to be copied across. **Applicability:** The mechanism is the same on any array with per-host LUN numbering, including NetApp ONTAP igroups; only the array-side vocabulary changes. For NetApp iSCSI backends, follow the iSCSI article linked below. **Duplicate iSCSI initiator names:** In the originating case, both destination hosts reported the same iSCSI IQN — a cloned host image whose `/etc/iscsi/initiatorname.iscsi` was never regenerated. Harmless for Fibre Channel, dangerous the day either host touches an iSCSI backend. See the related article below. ### Related articles * [VM Fails to Start After Live Migration Failure Due to iSCSI BDM Corruption](https://platform9.com/kb/pcd/storage/vm-fails-to-start-after-live-migration-failure-due-to-iscsi-bdm-corruption) — the same failure on NetApp ONTAP iSCSI backends. The SQL in that article patches `target_lun` and `target_luns` only; `target_luns` does not exist in Fibre Channel records, so that SQL must not be used for Fibre Channel volumes. * [How to Identify the LUN ID Associated with an Instance Volume](https://platform9.com/kb/pcd/storage/how-to-identify-the-lun-id-associated-with-an-instance-volume) — general procedure for mapping a volume to a LUN ID on a compute host. * [Unable to Start VM due to Multipath issue](https://platform9.com/kb/pcd/storage/unable-to-start-vm-due-to-multipath-issue) — recovery of stale multipath devices blocking a VM start. * [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states) — resetting volumes stuck in intermediate states. * [How to Rename iSCSI Initiator Names to the Existing Hypervisors](https://platform9.com/kb/pcd/storage/how-to-rename-iscsi-initiator-names-to-the-existing-hypervisors) — regenerating duplicate IQNs on cloned hosts. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd/storage/vm-fails-to-start-after-live-migration-failure-due-to-fibre-channel-bdm-corruption.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.