Instance Failed To Start When Multipath Volume is Attached

Problem

VM is failing to start when multipath volumes are attached to the VM and ostackhost shows below error logs:

ostackhost logs

ERROR oslo_messaging.rpc.server [req-ID] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -f <multipath_ID>
Exit code: 1
Stdout: 'Apr 09 04:25:29 | <multipath_ID>-part7: map in use<br>Apr 09 04:25:29 | failed to remove multipath map <multipath_ID><br>'
Stderr: ''
TRACE oslo_messaging.rpc.server Traceback (most recent call last):
TRACE oslo_messaging.rpc.server   File "/opt/pf9/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
TRACE oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
TRACE oslo_messaging.rpc.server   File "/opt/pf9/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
TRACE oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)

Environment

• Platform9 Managed OpenStack - v5.1 and Higher

• Multi-path Volumes

Cause

The hosts is failing to flush the multipath map and creating stale entries.

Resolution

• Login to the host where the VM is running and list the paths to the LUN and volumes

YAML

# multipath -ll   <multipath_ID>
# lsblk /dev/dm-<DM_ID>

• Remove the volume entries

YAML

# dmsetup remove <volume-entries>

• Remove the named multipath device.

YAML

# multipath -f /dev/disk/by-id/dm-uuid-mpath-<ID>

• Now start the stop the server and start it again

YAML

# openstack server stop <UUID>
# openstack server start <UUID>