Volume Attachment and Detachment Fails With Error "Flushing test_lv--test_vg failed: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'"
Problem
- Volume Detachment or Attachment fails with an error "Flushing testvg-test lv failed: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'".
- This issue can appear as VM boot failure or VM deletion failure as in both these situation volume associated with the VM either fails to attach or detach.
Environment
- Platform9 Managed OpenStack - All Versions
- Cinder (Block Storage)
- Nova Hosts on which LVM is in use for the base OS.
- iSCSI
Cause
- The underlying nova hosts which have LVM activated on them can sometime take control over the Logical Volumes (LVs) which are created inside the VM.
- This issue occurs when the cinder or iSCSI volume associated with any VM instance is first presented to the underlying nova host.
- Since the LVM is also activated on the underlying host it takes control over the LV created on the presented volume before the actual VM can take control over it.
Resolution
1) From the ostackhost logs, check and identify the name of the Logical Volume(LV) for which the flush call is failing.
2021-11-30 21:06:26.090 INFO nova.virt.libvirt.driver [-] [instance: a78f769a-dfe1-4dc3-ab21-c2da44d643db] Instance destroyed successfully....2021-11-30 21:06:26.808 WARNING os_brick.exception [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] Flushing test_lv--test_vg failed: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'2021-11-30 21:06:27.200 INFO nova.compute.manager [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] [instance: a78f769a-dfe1-4dc3-ab21-c2da44d643db] Successfully reverted task state from powering-on on failure for instance.2021-11-30 21:06:27.322 ERROR oslo_messaging.rpc.server [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'Command: multipath -f backup-lv--backupExit code: -Stdout: NoneStderr: None2) Login to the underlying Nova Host and check if the above mentioned LV is activated over there or not?
# lvs -ao +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices root centos -wi-ao---- 1000.00g /dev/sda3(0) swap centos -wi-ao---- 4.00g /dev/sda3(256000) test_lv test_vg -wi-a----- <200.00g /dev/sdh(0)3) a in the attributes for the test_lv -wi-a indicates that the Logical Volume is activated over underlying host.
4) Deactivate the Logical Volume using below command and verify if the LV is now deactivated or not.
# lvchange -an /dev/test_vg/test_lv# lvs -ao +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices root centos -wi-ao---- 1000.00g /dev/sda3(0) swap centos -wi-ao---- 4.00g /dev/sda3(256000) test_lv test_vg -wi------- <200.00g /dev/sdh(0)5) A permanent and more appropriate solution for this kind of issues is to implement LVM filters on the underlying Nova hosts where the Nova hosts is only allowed to take the control of the Logical Volumes from its own boot disk and should be restricted from taking the control over the Logical Volumes from another disks attached to it.
6) The Red Hat article linked here explains the similar implementation of the LVM filter for RHV hosts and it can be a good reference for LVM filter configuration.