VM Fails to Start After Live Migration Failure Due to Fibre Channel BDM Corruption

Problem

After a live migration fails on a host using Fibre Channel block storage, the affected VM shows vm_state: error while power_state remains Running on the original host. The VM continues to run until an operation such as a start, hard reboot, or a second migration attempt is issued. A start or hard reboot then fails during volume disconnect, the domain is destroyed, and the VM does not come back online.

Additional observable symptoms:

• openstack volume show for a non-multiattach volume lists two or more attachments, each on a different host
• openstack server event list shows one or more live-migration events with a failure status
• The launched_at timestamp of the instance predates the failed migrations, indicating the domain never moved

Do not start, hard-reboot, live-migrate, or reset the state of the affected VM before completing the Workaround. Each of these operations reads the corrupt Block Device Mapping and can remove SCSI paths from an unrelated volume that is in use by another VM on the same host.

Environment

• Private Cloud Director Virtualization - All versions
• Self-Hosted Private Cloud Director Virtualization - All versions
• Component: Compute Service, Block Storage Service (Fibre Channel — HPE Alletra 9000/MP, Primera, 3PAR)

Cause

During a live migration, the Compute Service runs pre_live_migration on the destination host. This step creates a new volume attachment for the destination host in the Block Storage Service and rewrites the connection_info field of the Block Device Mapping (BDM) record in the nova.block_device_mapping table with the destination host's LUN ID and initiator WWPNs. The attachment_id column of the BDM is repointed to the destination host's attachment.

If the migration fails after pre_live_migration runs and the rollback does not complete, the BDM record is never reverted. The domain still runs on the source host, but the BDM now describes the destination host's view of the volume.

On 3PAR-lineage arrays (HPE Alletra 9000/MP, Primera, 3PAR) each host receives its own VLUN numbering. The same volume can be LUN 59 on one host, 36 on another, and 111 on a third. A LUN number is meaningless outside the host that the LUN was issued for. When the Compute Service later starts or hard-reboots the VM, os-brick builds by-path device names from the stale LUN number and resolves them on the source host. If an unrelated volume occupies that LUN number on the source host, os-brick attempts to flush and delete that volume's multipath map. The map in use protection in multipathd blocks the delete, but SCSI paths of the unrelated volume may already have been removed at that point.

Resetting the VM state to active and retrying the migration clears the symptom without correcting the record. Each additional failed migration attempt leaks one more attachment per volume.

Fibre Channel connection_info embeds the LUN in four fields and identifies the host in two, so a repair that only rewrites target_lun leaves the record inconsistent:

Field	Action	Notes
data.target_lun	Rewrite	Scalar LUN for the host holding the domain
data.targets	Rewrite	Four [target_wwn, lun] pairs
data.initiator_target_map	Rekey	Keyed by initiator WWPN; targets listed per initiator
data.initiator_target_lun_map	Rekey and re-LUN	Drives os-brick path derivation
attachment_id (column)	Repoint	To the attachment whose connector host holds the domain
data.multipath_id	Leave	Array WWID — host-independent
data.device_path	Leave	Derived from the WWID
data.target_wwn	Leave	Array target ports; verify the ports are zoned to the host
volume_id, serial	Leave	Volume identity

Existing records list all four array target WWNs under each initiator, even where a given HBA only sees two of them. That is the 3PAR driver's own convention — replicate that layout rather than modelling per-HBA zoning. Paths that do not resolve are tolerated and only produce a warning.

Diagnostics

While the VM is still Running, every command in this section is read-only and safe. The domain is only alive because nothing has re-read the BDM yet. Capture the live multipath output before anything stops the VM — those paths are the cheapest source of the correct LUN.

1

Step 1 — Confirm the VM state and capture the migration failure

Record the state fields and the host the Compute Service believes is running the VM.

OpenStack CLI

$ openstack server show <INSTANCE_UUID> -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:power_state -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host

Sample Output

+-----------------------------+-----------------+
| Field                       | Value           |
+-----------------------------+-----------------+
| OS-EXT-SRV-ATTR:host        | [SOURCE_HOST]   |
| OS-EXT-STS:power_state      | Running         |
| OS-EXT-STS:task_state       | None            |
| OS-EXT-STS:vm_state         | error           |
| status                      | ERROR           |
+-----------------------------+-----------------+

Identify the failed live-migration event and record the reason for the failure. The failure reason does not block the repair, but a repaired VM must not be live-migrated again until the original error is understood.

OpenStack CLI

$ openstack server event list <INSTANCE_UUID>
$ openstack server event show <INSTANCE_UUID> <LIVE_MIGRATION_REQ_ID>

Sample Output

+------------------------------------------+--------------------------------------+----------------+----------------------------+
| Request ID                               | Server ID                            | Action         | Start Time                 |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| [REQ_UUID]                               | [INSTANCE_UUID]                      | live-migration | [TIMESTAMP]                |
+------------------------------------------+--------------------------------------+----------------+----------------------------+

On the source compute host, search the Compute Service log for the instance and request ID:

Compute Host

$ sudo grep -E "<INSTANCE_UUID>|<LIVE_MIGRATION_REQ_ID>" /var/log/pf9/ostackhost.log

A migration that fails with No valid host was found. There are not enough hosts available fails at the scheduler, before a destination is chosen. pre_live_migration never runs in that case and the BDM is intact — those VMs usually need only a state reset. Check such instances with the detection queries in Region-Wide Detection Queries anyway, because an earlier attempt on the same instance may have progressed further. Leaked Placement allocations from prior failed migrations are a common cause of that scheduler message.

2

Step 2 — Locate the running domain

Do not trust the OS-EXT-SRV-ATTR:host value on its own. Run the following on every host that appears as a source or destination in the migration history.

Compute Host

$ sudo virsh list --all | grep <INSTANCE_NAME>

Sample Output

 [DOMAIN_ID]   [INSTANCE_NAME]   running

Exactly one host must show the domain. If more than one host has the domain defined, stop and treat the situation as a potential dual-writer before going further.

3

Step 3 — Read the Compute Service BDM record

Self-Hosted only: This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Control Plane Node

$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")

$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')

$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D nova"
 
mysql > use nova;
mysql > select volume_id, attachment_id, connection_info from nova.block_device_mapping where instance_uuid='<INSTANCE_UUID>' and deleted=0 \G;      

Sample Output

*************************** 1. row ***************************
      volume_id: [VOLUME_UUID]
  attachment_id: [ATTACHMENT_UUID]
connection_info: {"driver_volume_type": "fibre_channel", "data": {"target_lun": [LUN], "targets": [...], "initiator_target_map": {"[INITIATOR_WWPN]": [...]}, "initiator_target_lun_map": {...}, "multipath_id": "[WWID]", "target_wwn": [...], ...}}

Record, per volume: volume_id, attachment_id, data.target_lun, the keys of data.initiator_target_map, and data.multipath_id. The multipath_id value is the array WWID and is host-independent, so the WWID is a safe lookup key even when every other value in the record is wrong.

4

Step 4 — Read the Block Storage Service attachment records

Self-Hosted only: This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Control Plane Node

mysql > select id, attach_status, deleted, json_extract(connector,'\$.host') as chost from cinder.volume_attachment where volume_id='<VOLUME_UUID>';

Sample Output

+--------------------------------------+---------------+---------+-----------------------+
| id                                   | attach_status | deleted | chost                 |
+--------------------------------------+---------------+---------+-----------------------+
| [SOURCE_ATTACHMENT_UUID]             | attached      |       0 | "[SOURCE_HOST]"       |
| [DEST_ATTACHMENT_UUID]               | attached      |       0 | "[DESTINATION_HOST]"  |
+--------------------------------------+---------------+---------+-----------------------+

Note every connector host, the attach_status, and the deleted flag. Each row with attach_status: attached on a non-multiattach volume is a live VLUN on the array, not a stale database row.

5

Step 5 — Read the host's storage identity

Run on the host that holds the domain (from Step 2). Use the WWID from data.multipath_id in Step 3.

Compute Host

$ sudo multipath -ll <WWID>
$ sudo grep . /sys/class/fc_host/host*/port_state /sys/class/fc_host/host*/port_name
$ sudo systemctl is-active multipathd

Sample Output

[WWID] dm-[N] 3PARdata,VV
size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
/sys/class/fc_host/host[N]/port_state:Online
/sys/class/fc_host/host[N]/port_name:0x[INITIATOR_WWPN]
active

The last field of the H:C:T:L column on the path lines is the real LUN for this host. The port_name values are the initiator WWPNs of this host.

A multipath -ll entry printing ##,## instead of 3PARdata,VV has lost all paths — multipath cannot read vendor strings without a live path. That is local device state, not an array unmap. If the Block Storage Service still shows the attachment as attached, the array mapping is intact.

6

Step 6 — Compare the records

Any one of the following three mismatches confirms the corruption:

• The BDM attachment_id (Step 3) resolves to a cinder.volume_attachment row (Step 4) whose connector host is not the host holding the domain (Step 2)
• connection_info.data.target_lun (Step 3) differs from the H:C:T:L LUN of multipath -ll <WWID> on the host holding the domain (Step 5)
• The keys of initiator_target_map (Step 3) are WWPNs that do not appear in /sys/class/fc_host/host*/port_name on the host holding the domain (Step 5)

Supporting evidence: a non-multiattach volume with several attached attachment rows (Step 4), and an instance in vm_state: error with power_state: Running whose launched_at predates the migrations.

If all checks above match, proceed to Workaround.

Region-Wide Detection Queries

The Compute Service and Block Storage Service share a MariaDB instance, so every affected instance in a region can be detected with cross-database queries instead of inspecting instances one by one.

Self-Hosted only: These queries require direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

BDM points at another host's attachment — every row returned is an instance that will fail to boot and will act on an unrelated LUN when a start is attempted:

Nova DB

SELECT i.uuid AS instance, i.hostname, i.host AS nova_host, i.vm_state, i.power_state,
       bdm.volume_id, bdm.device_name, bdm.attachment_id,
       JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) AS attach_host,
       JSON_EXTRACT(bdm.connection_info,'$.data.target_lun') AS bdm_lun
FROM nova.block_device_mapping bdm
JOIN nova.instances i ON i.uuid = bdm.instance_uuid AND i.deleted = 0
JOIN cinder.volume_attachment va ON va.id = bdm.attachment_id
WHERE bdm.deleted = 0
  AND bdm.destination_type = 'volume'
  AND JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) <> i.host;

Leaked multi-host mappings on non-multiattach volumes:

Nova DB

SELECT va.volume_id, COUNT(*) AS live_attachments,
       GROUP_CONCAT(JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) SEPARATOR ' ') AS hosts
FROM cinder.volume_attachment va
JOIN cinder.volumes v ON v.id = va.volume_id AND v.multiattach = 0 AND v.deleted = 0
WHERE va.deleted = 0 AND va.attach_status = 'attached'
GROUP BY va.volume_id
HAVING COUNT(*) > 1
ORDER BY live_attachments DESC;

Live migrations that did not complete cleanly — the population at risk:

Nova DB

SELECT instance_uuid, source_compute, dest_compute, status, created_at, updated_at
FROM nova.migrations
WHERE migration_type = 'live-migration'
  AND status NOT IN ('completed', 'reverted', 'cancelled')
ORDER BY updated_at DESC
LIMIT 100;

Resolution

No product fix is tracked for this condition at this time. Correct the BDM record and clear the leaked attachments using the Workaround below.

The workaround does not address the reason the live migration failed. Two failures on one instance mean a third is coming — identify the original migration error from Step 1 of Diagnostics before live-migrating the repaired VM again.

Workaround

The repair path depends on whether the source host's attachment is still alive. From Step 4 of Diagnostics, find the attachment whose connector host is the host holding the domain:

• attach_status = attached and deleted = 0 — the Block Storage Service never ran terminate_connection for that host, so the VLUN still exists on the array. Any missing paths are local state only, recoverable with an FC rescan. This is the common case, and the steps below apply.
• No such row, or deleted = 1 — the source attachment and the array VLUN were removed. The mapping must be recreated by the Block Storage Service with a fresh attachment. The steps below do not apply — contact Platform9 Support. See Additional Information.

Repair the BDM first and reset the VM state last. openstack server set --state active followed by a retry clears the symptom, leaves the cause in place, and turns one leaked attachment per volume into two.

1

Step 1 — Clear pathless multipath map shells

Run on the host holding the domain. Only remove maps with zero paths, and only once the domain is stopped or already destroyed. A map with live paths is the real mapping — leave that map alone.

multipath -f on a map that still has live paths removes the device backing a running VM. Confirm the map prints ##,## with no path lines before flushing.

Compute Host

$ sudo multipath -f <WWID>

2

Step 2 — Rescan the Fibre Channel HBAs

The rescan is additive and safe: the rescan re-probes targets and adds what is found, never removes. The loop is scoped to FC hosts rather than every SCSI host.

Compute Host

$ for h in /sys/class/fc_host/host*; do echo "- - -" | sudo tee /sys/class/scsi_host/$(basename $h)/scan > /dev/null; done

Do not run a global multipath -r. The global reload suspends every map on the host, briefly interrupting every other VM's devices. multipathd picks up new devices from uevents on its own. If a specific map does not reappear after approximately 30 seconds, add only that one map: sudo multipathd add map <WWID>.

3

Step 3 — Read the true LUN for each volume

Run once per volume, one WWID per invocation.

For a general procedure to map a volume to a LUN ID, see How to Identify the LUN ID Associated with an Instance Volume.

Compute Host

$ sudo multipath -ll <WWID>

Sample Output

[WWID] dm-[N] 3PARdata,VV
size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running
  `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running

Expect four active ready running paths. The last field of H:C:T:L on those path lines is the LUN to write into the BDM record. Never carry a LUN number from another host and never infer one host's LUN from another's.

4

Step 4 — Back up the BDM rows

Self-Hosted only: Steps 4 through 6 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact Platform9 Support to have them completed on your behalf.

The backup is mandatory before the UPDATE.

Control Plane Node

 mysql > create table nova.bdm_backup_<SHORT_UUID> as select * from nova.block_device_mapping where instance_uuid='<INSTANCE_UUID>' and deleted=0;"

5

Step 5 — Correct the BDM record

Run one UPDATE per volume. The statement repoints attachment_id at the attachment of the host holding the domain and rewrites all four LUN-bearing fields plus the initiator WWPNs.

Substitute every value for the host in front of you:

• <LUN> — from Step 3
• <CORRECT_HOST_ATTACHMENT_UUID> — the id of the attachment whose connector host holds the domain, from Step 4 of Diagnostics
• <TARGET_WWN_1> to <TARGET_WWN_4> — the four array target WWNs already present in the existing record's data.target_wwn
• <INITIATOR_1> and <INITIATOR_2> — the two port_name values from /sys/class/fc_host/ on the host holding the domain, from Step 5 of Diagnostics

Repointing attachment_id matters as much as the LUN. If attachment_id is left on the destination's attachment, a later detach or delete removes the wrong attachment and orphans the live VLUN — the same failure in reverse.

Nova DB

UPDATE nova.block_device_mapping
SET attachment_id = '<CORRECT_HOST_ATTACHMENT_UUID>',
    connection_info = JSON_SET(
      JSON_SET(
        JSON_SET(
          JSON_SET(connection_info, '$.data.target_lun', <LUN>),
          '$.data.targets', JSON_ARRAY(
            JSON_ARRAY('<TARGET_WWN_1>', <LUN>),
            JSON_ARRAY('<TARGET_WWN_2>', <LUN>),
            JSON_ARRAY('<TARGET_WWN_3>', <LUN>),
            JSON_ARRAY('<TARGET_WWN_4>', <LUN>))),
        '$.data.initiator_target_map', JSON_OBJECT(
          '<INITIATOR_1>',
            JSON_ARRAY('<TARGET_WWN_1>','<TARGET_WWN_2>','<TARGET_WWN_3>','<TARGET_WWN_4>'),
          '<INITIATOR_2>',
            JSON_ARRAY('<TARGET_WWN_1>','<TARGET_WWN_2>','<TARGET_WWN_3>','<TARGET_WWN_4>'))),
      '$.data.initiator_target_lun_map', JSON_OBJECT(
        '<INITIATOR_1>', JSON_ARRAY(
          JSON_ARRAY('<TARGET_WWN_1>', <LUN>), JSON_ARRAY('<TARGET_WWN_2>', <LUN>),
          JSON_ARRAY('<TARGET_WWN_3>', <LUN>), JSON_ARRAY('<TARGET_WWN_4>', <LUN>)),
        '<INITIATOR_2>', JSON_ARRAY(
          JSON_ARRAY('<TARGET_WWN_1>', <LUN>), JSON_ARRAY('<TARGET_WWN_2>', <LUN>),
          JSON_ARRAY('<TARGET_WWN_3>', <LUN>), JSON_ARRAY('<TARGET_WWN_4>', <LUN>))))
WHERE instance_uuid = '<INSTANCE_UUID>'
  AND volume_id = '<VOLUME_UUID>'
  AND deleted = 0;

6

Step 6 — Verify the corrected record

Expect the correct attachment, the LUN read from the rescan, and only the local host's initiators.

Control Plane Node

mysql > select volume_id, attachment_id, json_extract(connection_info,'\$.data.target_lun') as lun, json_keys(json_extract(connection_info,'\$.data.initiator_target_lun_map')) as inits from nova.block_device_mapping where instance_uuid='<INSTANCE_UUID>' and deleted=0;

Sample Output

+--------------------------------------+--------------------------------------+------+--------------------------------------------+
| volume_id                            | attachment_id                        | lun  | inits                                      |
+--------------------------------------+--------------------------------------+------+--------------------------------------------+
| [VOLUME_UUID]                        | [SOURCE_ATTACHMENT_UUID]             | [LUN]| ["[INITIATOR_1]", "[INITIATOR_2]"]         |
+--------------------------------------+--------------------------------------+------+--------------------------------------------+

7

Step 7 — Start the VM while tailing the Compute Service log

In a second session on the host holding the domain, tail the log before issuing any state change:

Compute Host

$ sudo tail -f /var/log/pf9/ostackhost.log | grep -i -E "<INSTANCE_UUID>|connect_volume|disconnect_volume|lun-|multipath"

The VM must be in stopped state before the start. Reset the state only if the VM is in error, then stop, then start.

For background on state resets, see How To Reset Volume States.

OpenStack CLI

$ openstack server show <INSTANCE_UUID> -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:task_state
$ openstack server set --state active <INSTANCE_UUID>    # only if vm_state is error
$ openstack server stop <INSTANCE_UUID>
$ openstack server start <INSTANCE_UUID>

The disconnect_volume call inside _hard_reboot is the step that fails on a corrupt record. With the corrected record the call walks the correct LUNs and passes.

8

Step 8 — Confirm the domain's backing devices

Confirm the guest is on the right disks, then confirm the guest answers on its own addresses.

Compute Host

$ sudo virsh domblklist <INSTANCE_NAME> --details

Sample Output

 Type   Device   Target   Source
------------------------------------------------------------
 block  disk     vda      /dev/disk/by-id/dm-uuid-mpath-[WWID]
 block  disk     vdb      /dev/disk/by-id/dm-uuid-mpath-[WWID]

9

Step 9 — Delete the leaked attachments

The incident is not closed until this step runs. A booted VM with the leaked attachments still in place means a non-multiattach volume is exposed read-write to two or three hosts. That is the data-corruption risk the whole repair exists to remove.

Delete each attachment belonging to a host that is not holding the domain. Use the API — the API drives the driver's terminate_connection, which is what actually removes the VLUN from the array.

Never delete volume_attachment rows with SQL. A database delete skips terminate_connection and orphans the array export permanently, with no record left to clean up from. Editing connection_info is a different and much smaller risk; deleting attachments is not.

OpenStack CLI

$ openstack --os-volume-api-version 3.27 volume attachment delete <LEAKED_ATTACHMENT_UUID>

Confirm one attachment per volume, on the host holding the domain:

OpenStack CLI

$ openstack volume show <VOLUME_UUID> -c status -c attachments

Sample Output

+-------------+------------------------------------------------------------------------------------------------------+
| Field       | Value                                                                                                |
+-------------+------------------------------------------------------------------------------------------------------+
| attachments | [{'id': '[VOLUME_UUID]', 'attachment_id': '[SOURCE_ATTACHMENT_UUID]', 'host_name': '[SOURCE_HOST]', ...}] |
| status      | in-use                                                                                               |
+-------------+------------------------------------------------------------------------------------------------------+

Confirm the maps are gone on each former destination host:

Compute Host

$ sudo multipath -ll | grep -E "<WWID_FRAGMENT_1>|<WWID_FRAGMENT_2>"

Sample Output

(no output)

10

Step 10 — Clear leaked Placement allocations

Self-Hosted only: This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Control Plane Node

$ kubectl -n "<REGION_NS>"  deploy/nova-api-osapi -- nova-manage placement audit --verbose

11

Step 11 — Check the collateral victims

If the VM was started or rebooted while the record was corrupt, os-brick removed SCSI paths from whatever volumes occupied the stale LUN numbers on the host. The FC rescan in Step 2 restores those paths, but confirm the path counts — another VM may be running degraded.

Identify what sat at each stale LUN on the host holding the domain, then check that volume's health:

Compute Host

$ ls -l /dev/disk/by-path/ | grep 'lun-<STALE_LUN>'
$ sudo multipath -ll <VICTIM_WWID>

Sample Output

[VICTIM_WWID] dm-[N] 3PARdata,VV
size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running
  |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running
  `- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running

Find which instance owns the victim volume:

Self-Hosted only: This query requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Control Plane Node

mysql > select instance_uuid, volume_id, device_name from nova.block_device_mapping where connection_info like '%<VICTIM_WWID_FRAGMENT>%' and deleted=0;

The repair is complete when the VM is ACTIVE, virsh domblklist shows the expected multipath devices, every affected volume shows exactly one attachment on the host holding the domain, no maps for the volume remain on former destination hosts, and every collateral-victim volume shows four active ready running paths.

Additional Information

Source attachment removed (deleted = 1 or no row): When the source host's attachment has been deleted, the array VLUN was removed by terminate_connection and the mapping must be recreated by the Block Storage Service with a fresh attachment rather than a corrected record. This is a longer procedure — contact Platform9 Support. The sequence in that case is: SQL-patch connection_info to a LUN that resolves to nothing on the host so the disconnect no-ops harmlessly, then run nova-manage volume_attachment refresh, then start the VM.

Why nova-manage volume_attachment refresh is the wrong tool for a corrupt record: The command exists to regenerate connection_info from a host connector and would produce correct values, but the command cannot run against a corrupt record. _do_refresh sends remove_volume_connection to the compute host before creating the new attachment, and that disconnect uses the current, wrong connection_info. remove_volume_connection swallows only exception.NotFound, so os-brick's ExceptionChainer propagates and aborts the command. The finally block then leaves the BDM with a blank attachment_id and connection_info = NULL, which is strictly worse: the instance cannot start at all until a refresh succeeds. When the source attachment is intact, correct the record with SQL and skip refresh entirely.

If refresh is run, four traps apply:

Trap	Consequence	Handling
CONF.host read from the wrong config	Exit 7, HostConflict — Private Cloud Director sets instance.host to the host UUID, but nova-manage defaults to /etc/nova/nova.conf and yields the bare hostname	Always pass --config-dir /opt/pf9/etc/nova/conf.d
get_connector has no --path option	Writes nothing; prints the connector	Redirect: get_connector --json > file
enforce_multipath=True is hardcoded	Raises if multipathd is down	Check systemctl is-active multipathd first
Requires vm_state == stopped exactly	Exit 5 on error	Reset to active, then stop

get_connector runs on the hypervisor; refresh needs database and Block Storage Service access, so refresh runs in a Compute Service pod on the control plane. The connector file has to be copied across.

Applicability: The mechanism is the same on any array with per-host LUN numbering, including NetApp ONTAP igroups; only the array-side vocabulary changes. For NetApp iSCSI backends, follow the iSCSI article linked below.

Duplicate iSCSI initiator names: In the originating case, both destination hosts reported the same iSCSI IQN — a cloned host image whose /etc/iscsi/initiatorname.iscsi was never regenerated. Harmless for Fibre Channel, dangerous the day either host touches an iSCSI backend. See the related article below.

Related articles

• VM Fails to Start After Live Migration Failure Due to iSCSI BDM Corruption — the same failure on NetApp ONTAP iSCSI backends. The SQL in that article patches target_lun and target_luns only; target_luns does not exist in Fibre Channel records, so that SQL must not be used for Fibre Channel volumes.
• How to Identify the LUN ID Associated with an Instance Volume — general procedure for mapping a volume to a LUN ID on a compute host.
• Unable to Start VM due to Multipath issue — recovery of stale multipath devices blocking a VM start.
• How To Reset Volume States — resetting volumes stuck in intermediate states.
• How to Rename iSCSI Initiator Names to the Existing Hypervisors — regenerating duplicate IQNs on cloned hosts.