search
Ctrlk

Instance in ERROR State, Write Lock on Disk File

hashtag
Problem

The OpenStack instance fails to start and goes in ERROR state because of a write lock on the disk file of the instance. The following error can be seen in the ostackhost.log .

561 TRACE oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: XXXX-XX-XXTXX:XX:XX.084234Z qemu-kvm: -drive file=/mnt/nfs/instances/0e4d3940-d7c7-410f-b3f1-72ffda4d5605/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock

hashtag
Environment

  • Platform9 Managed OpenStack - v3.6.0 and Higher

hashtag
Cause

Another process on the host is performing I/O operations on the disk file which causes the instance to fail during the start process as it cannot get a write-lock on the disk file.

hashtag
Resolution

  1. Check the hostname for the OpenStack instance.

# [root@testregion.platform9.net user(admin)]# openstack server show 0e4d3940-d7c7-410f-b3f1-72ffda4d5605+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+| Field                              | Value                                                                                                                                                |+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+| OS-DCF:diskConfig                  | MANUAL                                                                                                                                                || OS-EXT-AZ:availability_zone        | HA-Win                                                                                                                                                || OS-EXT-PF9-SRV-RES-INFO:res_info    | {u'cfg_disk_gb': 60.0, u'vcpus': 6, u'cfg_memory_mb': 32768, u'cpu_mhz': 1000}                                                                        || OS-EXT-SRV-ATTR:host                | c4f909eb-56c4-41b2-a0a5-ccef71af8d71                                                                                                                  || OS-EXT-SRV-ATTR:hypervisor_hostname | demohost01                                                                                                                            || OS-EXT-SRV-ATTR:instance_name      | 0e4d3940-d7c7-410f-b3f1-72ffda4d5605                                                                                                                  || OS-EXT-STS:fault                    | {u'message': u'Instance 0e4d3940-d7c7-410f-b3f1-72ffda4d5605 could not be found on hypervisor.', u'code': 94043, u'details': u''}                    |                                                                                                    | OS-EXT-STS:vm_state                | error

  1. Log in to the host on which the instance resides.

  2. Navigate to the disk file location mentioned in the ostackhost.log error.

561 TRACE oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: XXXX-XX-XXTXX:XX:XX.084234Z qemu-kvm: -drive file=/mnt/nfs/instances/0e4d3940-d7c7-410f-b3f1-72ffda4d5605/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Failed to get "write" lock

  1. Create a backup copy of the original disk file in the same directory.

  1. Rename the original disk to disk.old.

  1. Rename the backup disk to disk.

  1. Set the instance status to active.

  1. Stop the server.

  1. Start the server.

PreviousHost(s) Unable to Connect to Management Plane With "Error: getaddrinfo ENOTFOUNDNextImage File Copied to Glance Host, Image Not Discovered

Last updated