VM Evacuation Fails Due to Volume Stuck in attaching State

Problem

Evacuation of a VM fails, leaving the instance in ERROR state. One of the attached volumes remains stuck in the attaching state, preventing Nova from completing the evacuation. The VM cannot boot because the required volume is not successfully attached.

Environment

• Storage Service
• Private Cloud Director Virtualization - v2025.4 and Higher
• Private Cloud Director Kubernetes – v2025.4 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
• Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher
• Component: Storage Service

Cause

The evacuation fails because one of the attached volumes is stuck in the attaching state. During evacuation, Nova requests volume service to attach all required volumes to the target compute host. If any volume does not reach available or downloading state and remains stuck, Nova cannot proceed with booting the instance, resulting in ERROR state.

Diagnostics

• Check the VM state:

  $ openstack server show <VM_ID>

The status field shows ERROR and volumes_attached lists all volumes; one may be stuck in attaching state.

• Check volume status:

  $ openstack volume show <VOLUME_ID>

Events show Invalid volume error for volumes stuck in attaching

• Verify volume attachment attempts:

  $ openstack server event show <VM_ID> <REQUEST_ID>

Shows traceback with Invalid volume due to attaching state

Resolution

1. Reset volumes to available

   $ openstack volume set --state available <BOOT_VOLUME_ID>
   $ openstack volume set --state available <VOLUME_ID_1>
   $ openstack volume set --state available <VOLUME_ID_2>
   $ openstack volume set --state available <VOLUME_ID_3>
   $ openstack volume set --state available <STUCK_VOLUME_ID>

2. Recreate the VM from the bootable volume

   $ openstack server create \
     --flavor <FLAVOR_ID> \
     --volume <BOOT_VOLUME_ID> \
     --nic net-id=<NETWORK_ID> \
     --availability-zone nova \
     <NEW_VM_NAME>

3. Reattach all data volumes

   $ openstack server add volume <NEW_VM_ID> <VOLUME_ID_1>
   $ openstack server add volume <NEW_VM_ID> <VOLUME_ID_2>
   $ openstack server add volume <NEW_VM_ID> <VOLUME_ID_3>
   $ openstack server add volume <NEW_VM_ID> <STUCK_VOLUME_ID>

Validation

1. Confirm VM status is ACTIVE:

   $ openstack server show <NEW_VM_ID> | grep status

2. Log into the guest OS and verify:

   1. VM boots successfully
   2. All data volumes are visible and mounted correctly

Additional Information

• This issue occurs when Cinder volumes remain in transitional states (attaching/reserved) due to incomplete detach/attach operations.
• Using openstack volume set --state available resets the volumes safely when no I/O is in progress.
• Preventive measures include monitoring volume states during evacuation and ensuring no stale Block Device Mappings exist in Nova for critical volumes.
• Similar issues may occur if live migration or evacuations are attempted while volumes are actively performing operations.