Ostackhost Fails to Resize, Create and Migrate Instance with Error "ComputeHostNotFound

Problem

The cold migration of an instance fails with the following traceback observed in the ostackhost log located at /var/log/pf9 on the source host.

None

ERROR nova.compute.manager [req-ID] [instance: UUID] Setting instance vm_state to ERROR: ComputeHostNotFound: Compute host compute-prod-6 could not be found.TRACE nova.compute.manager [instance: UUID] Traceback (most recent call last):TRACE nova.compute.manager [instance: UUID] File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 8176, in _error_out_instance_on_exceptionTRACE nova.compute.manager [instance: UUID] yieldTRACE nova.compute.manager [instance: UUID] File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 3910, in do_confirm_resizeTRACE nova.compute.manager [instance: UUID] migration.source_node)TRACE nova.compute.manager [instance: UUID] File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4013, in _delete_allocation_after_moveTRACE nova.compute.manager [instance: UUID] cn_uuid = rt.get_node_uuid(nodename)TRACE nova.compute.manager [instance: UUID] File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 157, in get_node_uuidTRACE nova.compute.manager [instance: UUID] raise exception.ComputeHostNotFound(host=nodename)TRACE nova.compute.manager [instance: UUID] ComputeHostNotFound: Compute host compute-prod-6 could not be found.

Environment

• Platform9 Managed OpenStack - v4.0 and higher

• Nova

Cause

• Disk file for an instance is missing which prevents ostackhost from updating its resources.

None

2020-03-18 19:08:04.221 ERROR nova.compute.manager [req-ID] Error updating resources for node compute-prod-6: DiskNotFound: No disk at /opt/pf9/data/instances/90990d99-e350-430a-ae28-3b202bf6056e/disk2020-03-18 19:08:04.221 TRACE nova.compute.manager Traceback (most recent call last):2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 8000, in _update_available_resource_for_node2020-03-18 19:08:04.221 TRACE nova.compute.manager rt.update_available_resource(context, nodename)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 704, in update_available_resource2020-03-18 19:08:04.221 TRACE nova.compute.manager resources = self.driver.get_available_resource(nodename)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6661, in get_available_resource2020-03-18 19:08:04.221 TRACE nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8345, in _get_disk_over_committed_size_total2020-03-18 19:08:04.221 TRACE nova.compute.manager self._destroy(inst)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__2020-03-18 19:08:04.221 TRACE nova.compute.manager self.force_reraise()2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise2020-03-18 19:08:04.221 TRACE nova.compute.manager six.reraise(self.type_, self.value, self.tb)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8267, in _get_disk_over_committed_size_total2020-03-18 19:08:04.221 TRACE nova.compute.manager config, block_device_info)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8169, in _get_instance_disk_info_from_config2020-03-18 19:08:04.221 TRACE nova.compute.manager qemu_img_info = disk_api.get_disk_info(path)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/disk/api.py", line 98, in get_disk_info2020-03-18 19:08:04.221 TRACE nova.compute.manager return images.qemu_img_info(path)2020-03-18 19:08:04.221 TRACE nova.compute.manager File "/opt/pf9/venv/lib/python2.7/site-packages/nova/virt/images.py", line 58, in qemu_img_info2020-03-18 19:08:04.221 TRACE nova.compute.manager raise exception.DiskNotFound(location=path)2020-03-18 19:08:04.221 TRACE nova.compute.manager DiskNotFound: No disk at /opt/pf9/data/instances/90990d99-e350-430a-ae28-3b202bf6056e/disk

• This happens if an instance is resized or cold migrated and the confirm resize operation is not performed.

Resolution

1. Perform a confirm resize operation on the instance by running the following command.

None

# openstack server resize --confirm [server UUID]

1. If the instance goes into an error state, navigate to the instance storage path, identify the instance directory, and modify its name using the following command.

None

# cd /opt/pf9/data/instances/# mv 90990d99-e350-430a-ae28-3b202bf6056e_resize 90990d99-e350-430a-ae28-3b202bf6056e

1. After renaming the directory, set the instance into an ACTIVE state and perform a stop and start operation on the instance.

None

# openstack server set --state active [server_UUID]# openstack server stop [server_UUID]# openstack server start [server_UUID]

Additional Information

Related upstream bug: https://bugs.launchpad.net/nova/+bug/1774249