Volume Shows Attached After Detach Operation Fails With Error " Unable to detach from guest transient domain

Problem

• Volume unavailable inside the instance after failure to detach the volume from the instance with error as follows.

DeviceDetachFailed: Device detach failed for vd[x]: Unable to detach from guest transient domain

• Volume status is "attached" and "in-use" in Cinder DB.

# openstack volume show volume_uuid --fit-width -c properties -c status+--------------------------------+--------------------------------------------------| Field | Value |+--------------------------------+--------------------------------------------------+| properties | attached_mode='rw' || status | in-use |+--------------------------------+---------------------------------------------------

Environment

• Platform9 Managed OpenStack - All Versions
• Cinder
• Nova
• Libvirt

Cause

• The volume detach at the Libvirt driver level happens in two steps:
  • Detach from the persistent domain (affect instance upon next reboot).
  • Detach from the live domain (affect running instance) - Transient Domain.
• A detach from a live domain is a request from the host to the guest, which the guest can choose to ignore. If the guest has a file open on the volume by some process, it might ignore the request to detach that volume because the file is in use and logs below error in ostackhost.log.

TRACE oslo.service.loopingcall DeviceDetachFailed: Device detach failed for vd[x]: Unable to detach from guest transient domain.

• If this scenario occurs, when a user retries to detach the volume, the volume is already detached from the persistent domain. Thus, it will fail the second attempt of detach operation with error:

ERROR nova.compute.manager [req-id  user-id] [instance: instance-id] Failed to detach volume volume_id from /dev/vd[x]

• This is the expected behavior if the volume is detached from the persistent domain and unavailable inside the instance.

Resolution

1. Restart of the instance is required to complete the detach operation.

Additional Information

A patch to log the error observed while detaching the volume from a persistent domain was added in the upstream OpenStack bugs launchpad #1633236.