# Volume Attachment and Detachment Fails With Error "Flushing test\_lv--test\_vg failed: oslo\_concurrency.processutils.ProcessExecutionError: \[Errno 2] No such file or directory: 'multipath'"

## Problem

* Volume Detachment or Attachment fails with an error "**Flushing testvg-test lv failed: oslo\_concurrency.processutils.ProcessExecutionError: \[Errno 2] No such file or directory: 'multipath'**".
* This issue can appear as VM boot failure or VM deletion failure as in both these situation volume associated with the VM either fails to attach or detach.

## Environment

* Platform9 Managed OpenStack - All Versions
* Cinder (Block Storage)
* Nova Hosts on which LVM is in use for the base OS.
* iSCSI

## Cause

* The underlying nova hosts which have LVM activated on them can sometime take control over the Logical Volumes (LVs) which are created inside the VM.
* This issue occurs when the cinder or iSCSI volume associated with any VM instance is first presented to the underlying nova host.
* Since the LVM is also activated on the underlying host it takes control over the LV created on the presented volume before the actual VM can take control over it.

## Resolution

1. From the ostackhost logs, check and identify the name of the Logical Volume(LV) for which the flush call is failing.

{% tabs %}
{% tab title="Ostackhost logs" %}

```bash
2021-11-30 21:06:26.090 INFO nova.virt.libvirt.driver [-] [instance: a78f769a-dfe1-4dc3-ab21-c2da44d643db] Instance destroyed successfully.
...
2021-11-30 21:06:26.808 WARNING os_brick.exception [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] Flushing test_lv--test_vg failed: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'

2021-11-30 21:06:27.200 INFO nova.compute.manager [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] [instance: a78f769a-dfe1-4dc3-ab21-c2da44d643db] Successfully reverted task state from powering-on on failure for instance.
2021-11-30 21:06:27.322 ERROR oslo_messaging.rpc.server [req-5ec6b4c7-669f-48f4-9481-5d1ace40ae0c gcooper@cloudera.com IOPS] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'multipath'
Command: multipath -f backup-lv--backup
Exit code: -
Stdout: None
Stderr: None
```

{% endtab %}
{% endtabs %}

2. Login to the underlying Nova Host and check if the above mentioned LV is activated over there or not?

{% tabs %}
{% tab title="Bash" %}

```bash
# lvs -ao +devices 
  LV                                             VG                                        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  root                                           centos                                    -wi-ao---- 1000.00g                                                     /dev/sda3(0)
  swap                                           centos                                    -wi-ao----    4.00g                                                     /dev/sda3(256000)
  test_lv                                        test_vg                                   -wi-a----- <200.00g                                                     /dev/sdh(0)
```

{% endtab %}
{% endtabs %}

3. `a` in the attributes for the test\_lv `-wi-a` indicates that the Logical Volume is activated over underlying host.
4. Deactivate the Logical Volume using below command and verify if the LV is now deactivated or not.

{% tabs %}
{% tab title="Javascript" %}

```javascript
# lvchange -an /dev/test_vg/test_lv

# lvs -ao +devices 
  LV                                             VG                                        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  root                                           centos                                    -wi-ao---- 1000.00g                                                     /dev/sda3(0)
  swap                                           centos                                    -wi-ao----    4.00g                                                     /dev/sda3(256000)
  test_lv                                        test_vg                                   -wi------- <200.00g                                                     /dev/sdh(0)
```

{% endtab %}
{% endtabs %}

5. A permanent and more appropriate solution for this kind of issues is to implement [LVM filters](https://access.redhat.com/solutions/65960) on the underlying Nova hosts where the Nova hosts is only allowed to take the control of the Logical Volumes from its own boot disk and should be restricted from taking the control over the Logical Volumes from another disks attached to it.
6. The Red Hat [article](https://access.redhat.com/solutions/2662261) linked here explains the similar implementation of the LVM filter for RHV hosts and it can be a good reference for LVM filter configuration.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pmo/solution/volume-attachment-and-detachment-fails-with-an-error-flushing-te.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.