Forcefully Delete Cinder Volume of Non-existing Cinder Hosts.
Problem
Unable to delete a volume that resided on a Cinder host that no longer exists. Resulting the stack deletion failure.
Environment
- Platform9 Managed OpenStack - All Versions.
- Cinder.
- Cinder Cli tool should be installed.
Cause
Backend host for the cinder-volume is removed but the Cinder volume still points to the removed host.
Resolution
CAUTION- DO NOT Perform below steps for volumes where the Cinder host backend is active. This will result in re-labelling all the volumes present the source cinder host to new cinder host. This will impact all the active volumes part of the source Cinder host attached to active VMs.
Add the volume to a new host as a temporary change to move the volume from a stale/orphaned state before proceeding with the deletion from the backend.
The steps to be followed are:
- Confirm the details of the volume that needs to be deleted using below command:
# openstack volume show <VOLUME_ID> -c id -c os-vol-host-attr:host -c status -c attachmentsThe below steps can only be executed as admin by sourcing the RC file with admin credentials.
- Change the volume pointing to old host to state available and detached using below command:
# openstack volume set --state available --detached <stale-volume-uuid>OR# cinder reset-state --state available --attach-status detached <stale-volume-uuid>- Update the cinder volume which is pointing to old host to the new host id. The new host can be any host with a cinder role applied within the same region.
The cinder-manage binary is available only on the Platform9 Management Plane and Block Storage hosts directly. If running from a Block Storage host, the following will need to be prepended as shown.
# LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d volume update_host --currenthost <old-host-id@lvm-backend> --newhost <new-host-id@lvm-backend>- Force delete the volume once cinder host is updated.
# openstack volume delete --force <volume-uuid>Additional Information
Verify if volume is deleted and do not have any old or stale data.
# openstack volume show <VOLUME_ID>