Instance Fails to Start/Power On: "blockdev: cannot open /dev/XXX: No such device or address"

Problem

Instance fails to start or be powered on and transitions into an ERROR state with the following fault/error message: "blockdev: cannot open /dev/XXX: No such device or address".

Environment

• Platform9 Managed OpenStack - All Versions

• Cinder (Block Storage)

• LVM

• iSCSI

Cause

A stale iSCSI target/connection – one that has not been logged out promptly following the shutdown of the VM – is present and the connection information is no longer valid.

Resolution

1. Identify which iSCSI target corresponds to the unknown device (i.e. /dev/XXX).

Bash

Target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6 (non-flash)
        Current Portal: 172.30.0.194:3260,1
        Persistent Portal: 172.30.0.194:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:6552111637
                Iface IPaddress: 172.30.0.152
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 477
                iSCSI Connection State: IN LOGIN
                iSCSI Session State: FREE
                Internal iscsid Session State: REOPEN
...
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 491 State: running
                scsi491 Channel 00 Id 0 Lun: 0
                Attached scsi disk sdcd State: transport-offline

1. Logout the iSCSI target.

Bash

$ sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6 -p 172.30.0.194:3260 --logout
Logging out of session [sid: 477, target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6, portal: 172.30.0.194,3260]
Logout of [sid: 477, target: iqn.2010-10.org.openstack:volume-0e173a77-46fc-4ca2-9955-0b7e534957e6, portal: 172.30.0.194,3260] successful.

1. Restart or start the Instance/VM.

Tip

The 'attached scsi disk' will be followed by the device name without the preceding /dev path. For example, in the output above, the corresponding device name is sdcd.