No VMs Displayed When 'View VMs in All Tenants' Is Enabled

When the “View VMs in all Tenants” option is enabled in the PCD UI, no VMs are displayed, and the associated API call fails with the following error:

{
    "computeFault": {
        "code": 500,
        "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.<br><class 'KeyError'>"
    }
}

Users could search and view specific VMs by name. However, when the “View VMs in all Tenants” option is disabled, VMs under the current tenant are displayed correctly.

CLI worked as expected for all-tenant queries (openstack server list --all-projects).

Environment

• Private Cloud Director Virtualization - till 2025.8-92

• Self-Hosted Private Cloud Director Virtualization - till 2025.8-92

• Compute API

Cause

A corrupted instance remained in the Compute database (nova-DB) due to a failure during VM provisioning (e.g., quota exceeded).

• The instance record was left in an incomplete or ERROR state.

• The corresponding request specification was missing from nova_api.request_specs.

• When Nova attempted to fetch all VMs across tenants, it raised a KeyError, resulting in a 500 error.

This behaviour is noted as PCD-4177 and improvements have been made starting with v2025.8-112 and above versions.

Diagnostics

The below 3 commands are applicable only to Self-Hosted PCD-V environments.

• Error logs from the nova-api-osapi pod exposed a KeyError involving a specific instance UUID, indicating corruption or missing data for that instance. Example log snippet:

ERROR nova.api.openstack.wsgi [None [REQ_ID] [CORRUPTED_VM_UUID]- - default default] Unexpected exception in API method: KeyError: '[CORRUPTED_VM_UUID]'
ERROR nova.api.openstack.wsgi KeyError: '[CORRUPTED_VM_UUID]'
<class 'KeyError'>

• Checking the nova database revealed the corrupted instance still present with vm_state=error and power_state=NOSTATE.

MySQL [nova]> select * from instances where uuid="<CORRUPTED_VM_UUID>"\G;
vm_state: error 
power_state: 0 
host: NULL 
availability_zone: [AZ]
display_name: [CORRUPTED_VM_NAME]

• Querying the request_specs table in the nova_api database for the same instance UUID returned no results, confirming a missing request spec entry:

MySQL [nova_api]> select * from request_specs where instance_uuid='<CORRUPTED_VM_UUID>'; 
Empty set (0.001 sec)

• Check VM status using below command :

$ openstack server show <CORRUPTED_VM_UUID> --fit 
| OS-EXT-STS:vm_state   | error                                         | 
| OS-EXT-STS:power_state| NOSTATE                                       | 
| fault                 | {'code': 413, 'message': 'Quota exceeded...'} |

• No event history for the VM, which confirms the VM entered the "Error" state during the creation process:

$ openstack server event list <CORRUPTED_VM_UUID> 
(no events)

Resolution

The corrupted VM deleted using below command:

openstack server delete <CORRUPTED_VM_UUID>

This restored normal functionality for both the UI and the API, allowing full visibility of VMs across tenant boundaries.

Validation

• Verified UI no longer shows blank list when “View VMs in All Tenants” is enabled.

• Cross-tenant VM visibility works as expected.

• No further 500 errors observed in related logs.

• API calls return valid responses without errors.