Instance in ERROR State, Write Lock on Disk File

Problem

The OpenStack instance fails to start and goes in ERROR state because of a write lock on the disk file of the instance. The following error can be seen in the ostackhost.log .

None

561 TRACE oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: XXXX-XX-XXTXX:XX:XX.084234Z qemu-kvm: -drive file=/mnt/nfs/instances/0e4d3940-d7c7-410f-b3f1-72ffda4d5605/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock

Environment

• Platform9 Managed OpenStack - v3.6.0 and Higher

Cause

Another process on the host is performing I/O operations on the disk file which causes the instance to fail during the start process as it cannot get a write-lock on the disk file.

Resolution

1. Check the hostname for the OpenStack instance.

None

# [[e-mail] user(admin)]# openstack server show 0e4d3940-d7c7-410f-b3f1-72ffda4d5605+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+| Field                              | Value                                                                                                                                                |+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+| OS-DCF:diskConfig                  | MANUAL                                                                                                                                                || OS-EXT-AZ:availability_zone        | HA-Win                                                                                                                                                || OS-EXT-PF9-SRV-RES-INFO:res_info    | {u'cfg_disk_gb': 60.0, u'vcpus': 6, u'cfg_memory_mb': 32768, u'cpu_mhz': 1000}                                                                        || OS-EXT-SRV-ATTR:host                | c4f909eb-56c4-41b2-a0a5-ccef71af8d71                                                                                                                  || OS-EXT-SRV-ATTR:hypervisor_hostname | demohost01                                                                                                                            || OS-EXT-SRV-ATTR:instance_name      | 0e4d3940-d7c7-410f-b3f1-72ffda4d5605                                                                                                                  || OS-EXT-STS:fault                    | {u'message': u'Instance 0e4d3940-d7c7-410f-b3f1-72ffda4d5605 could not be found on hypervisor.', u'code': 94043, u'details': u''}                    |                                                                                                    | OS-EXT-STS:vm_state                | error

1. Log in to the host on which the instance resides.
2. Navigate to the disk file location mentioned in the ostackhost.log error.

None

561 TRACE oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: XXXX-XX-XXTXX:XX:XX.084234Z qemu-kvm: -drive file=/mnt/nfs/instances/0e4d3940-d7c7-410f-b3f1-72ffda4d5605/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock

1. Create a backup copy of the original disk file in the same directory.

None

# cp disk disk.bak

1. Rename the original disk to disk.old.

None

# mv disk disk.old

1. Rename the backup disk to disk.

None

# mv disk.bak disk

1. Set the instance status to active.

None

# openstack server set --state active [instance-id]

1. Stop the server.

None

# openstack server stop [instance-id]

1. Start the server.

None

# openstack instance start [instance-id]