Instance Creation Failed due to erroneous volume

Problem

• Failed to create Instance due to volume being in error state.
• Instance stuck in the BUILD state during creation with the error mentioned below.

Nova Logs

2022-06-27 16:34:20.543 TRACE nova.compute.manager [instance: f788fd93-XXXX-XXXX-XXXX-XXXXXXXXfd56] nova.exception.VolumeNotCreated: Volume b52a7258-XXXX-XXXX-XXXX-XXXX9c74 did not finish being created even after we waited 65 seconds or 9 attempts. And its status is error.

Environment

• Platform9 Managed OpenStack - All versions
• Nova
• Cinder - Fiber Channel (FC)

Cause

• The error mentioned below found to be associated with the volume that got stuck in the ERROR state during creation.

Cinder Logs

2022-06-27 13:26:14.081 ERROR cinder.volume.utils [req-0d000c85-41af-4607-adec-67cfa4fa3f90 None transition1] Failed to copy image 13d05b01-XXXX-XXXX-XXXX-XXXXXXXXb40e to volume: 4de501bb-XXXX-XXXX-XXXX-XXXXXXXXd96a: os_brick.exception.NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume device.
2022-06-27 13:26:15.032 ERROR cinder.volume.flows.manager.create_volume [req-0d000c85-41af-4607-adec-67cfa4fa3f90 None transition1] Volume 4de501bb-XXXX-XXXX-XXXX-XXXXXXXXd96a: create failed

• The error indicates that the volume creation failed as cinder plugin was not able to find the associated Fiber Channel Volume at the backend SAN/NAS Storage .

Resolution

• Try to check the Fiber Channel Connectivity with backend SAN/NAS storage through SAN or Ethernet switch with the storage team assistance.
• If no connectivity or hardware issues are found then try to manually attach a test FC volume from same SAN/NAS storage to the Cinder Host.