Instance Unexpectedly Not Found

Problem

• Instance operations (start, stop, reboot, resize, etc.) are failing for an instance and it is in 'ERROR' state.
• Instance 'fault' field is populated with an error similar to the following.

None

$ openstack server show [UUID] -f value -c fault{u'message': u'Instance [UUID] could not be found on hypervisor.', u'code': 94043, u'created': u'2019-07-18T23:08:58Z'}

• The hypervisor log shows that the instance is "unexpectedly not found".

-- /var/log/pf9/ostackhost.log --

None

WARNING nova.compute.manager [-] While synchronizing instance power states, found 14 instances in the database and 13 instances on the hypervisor.WARNING nova.compute.manager [-] [instance: [UUID]] Instance is unexpectedly not found. Ignore.INFO nova.compute.manager [-] Instance [UUID] marked as not found.

• The ephemeral disk file and XML metadata still exist on the instance storage path.

None

$ ll /pf9/instances/[UUID]/total 3468856-rw-r--r--. 1 pf9 pf9group       4614 Apr 27 01:50 cfg-[UUID].xml-rw-r--r--. 1 pf9 pf9group      29608 Apr 26 20:36 console.log-rw-r--r--. 1 pf9 pf9group 3552116736 Apr 26 20:36 disk-rw-r--r--. 1 pf9 pf9group         70 Mar 25 14:36 disk.info

Environment

• Platform9 Managed OpenStack - All Versions
• Nova

Cause

The instance failed to boot previously.

Resolution

1. Perform a hard-reboot of the instance to attempt to re-define the instance via Nova (CLI only).

None

$ openstack server reboot --hard [UUID]

1. If a hard-reboot doesn't work to address the issue, re-define the instance manually on the hypervisor using the 'virsh' utility.

None

# virsh define /path/to/instances/[UUID]/cfg-[UUID].xml