VM Build Fails as Volume did not get Created

Problem

VM build fails with the error "Volume <volume-id> did not finish being created even after we waited 0 seconds or 1 attempts. And its status is error".

Server show

{u'message': u'Build of instance 056131da-cd68-469d-85d0-42f49835319e aborted: Volume <volume-id> did not finish being created even after we waited 0 seconds or 1 attempts. And its status is error.', u'code': 500, u'details': u'Traceback (most recent call last):<br>

Cinder Scheduler logs
2024-09-10 03:28:07.582 16841 ERROR cinder.scheduler.flows.create_volume [req-b9518d05-9d43-4ff8-a80b-d517e32e1d9a bb179168eb914673a13fe2dd9d26bc59 a861d7b610be4350a30e83fc70523627 - Federated default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available

Environment

• Platform9 Managed OpenStack - v4.0 and Higher

Cause

VM build tries to create the volume but the required _pf9-cindervolume-base_ service remains in active (exited) state on all cinder hosts, which doesn't allow creation of a new volume.

Service status

# systemctl status pf9-cindervolume-base
● pf9-cindervolume-base.service - LSB: OpenStack cinder volume
   Loaded: loaded (/etc/rc.d/init.d/pf9-cindervolume-base; bad; vendor preset: disabled)
   Active: active (exited) since Sat 2024-06-08 16:21:20 GMT; 3 months 2 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0
   Memory: 0B

Resolution

To fix the issue, restart the pf9-cindervolume-base service and verify if that service is active (running).

Service status

# systemctl restart pf9-cindervolume-base
# systemctl status pf9-cindervolume-base
● pf9-cindervolume-base.service - LSB: OpenStack cinder volume
   Loaded: loaded (/etc/rc.d/init.d/pf9-cindervolume-base; bad; vendor preset: disabled)
   Active: active (running) since Tue 2024-09-10 04:40:36 GMT; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 23919 ExecStop=/etc/rc.d/init.d/pf9-cindervolume-base stop (code=exited, status=0/SUCCESS)
  Process: 23998 ExecStart=/etc/rc.d/init.d/pf9-cindervolume-base start (code=exited, status=0/SUCCESS)
    Tasks: 1
   Memory: 80.9M

Additional Information

Platform9 has raised an internal bug IAAS-10791 to fix the issue with the service so that it should automatically be restarted if the service is in active (exited) state.