Forcefully Delete Cinder Volume of Non-existing Cinder Hosts.

Problem

Unable to delete a volume that resided on a Cinder host that no longer exists. Resulting the stack deletion failure.

Environment

• Platform9 Managed OpenStack - All Versions.

• Cinder.

• Cinder Cli tool should be installed.

Cause

Backend host for the cinder-volume is removed but the Cinder volume still points to the removed host.

Resolution

Error

CAUTION- DO NOT Perform below steps for volumes where the Cinder host backend is active. This will result in re-labelling all the volumes present the source cinder host to new cinder host. This will impact all the active volumes part of the source Cinder host attached to active VMs.

Add the volume to a new host as a temporary change to move the volume from a stale/orphaned state before proceeding with the deletion from the backend.

The steps to be followed are:

1. Confirm the details of the volume that needs to be deleted using below command:

Volume show output

# openstack volume show <VOLUME_ID> -c id -c os-vol-host-attr:host -c status -c attachments

Warning

The below steps can only be executed as admin by sourcing the RC file with admin credentials.

1. Change the volume pointing to old host to state available and detached using below command:

Cinder CLI

# openstack volume set --state available --detached <stale-volume-uuid>

OR

# cinder reset-state --state available --attach-status detached <stale-volume-uuid>

1. Update the cinder volume which is pointing to old host to the new host id. The new host can be any host with a cinder role applied within the same region.

Info

The cinder-manage binary is available only on the Platform9 Management Plane and Block Storage hosts directly. If running from a Block Storage host, the following will need to be prepended as shown.

Cinder CLI

# LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d volume update_host --currenthost <old-host-id@lvm-backend> --newhost <new-host-id@lvm-backend>

1. Force delete the volume once cinder host is updated.

Cinder CLI

# openstack volume delete --force <volume-uuid>

Additional Information

Verify if volume is deleted and do not have any old or stale data.

Volume show output

# openstack volume show <VOLUME_ID>