Stale OpenStack Regions in PCD

Problem

• Old OpenStack regions and endpoints still exist even after the PCD region is deleted.

Environment

• Private Cloud Director Virtualization

• Self-Hosted Private Cloud Director Kubernetes

Cause

• This is a known issue and a bug: PCD-2664 has been raised to track this matter.

CAUTION

• Do not create/delete an OpenStack region in PCD.

• To create/delete a PCD region:

  • PCD SaaS Users: Contact Platform9 Support

  • PCD Self-Hosted Users: Follow the airctl region creation/deletion process.

Diagnostics

1. Get the region list using the command given below, and see the stale region entry.

$ openstack region list

+----------------+---------------+-------------+
| Region         | Parent Region | Description |
+----------------+---------------+-------------+
| Infra          | None          |             |
| [REGION_1]     | Infra         |             |
| [STALE_REGION] | Infra         |             |
... more

1. Check if there are any endpoints available for the stale region.

$ openstack endpoint list --region <STALE_REGION_NAME> -c Region -c "Service Name" -c "Service Type"

+----------------+----------------+----------------+
| Region         | Service Name   | Service Type   |
+----------------+----------------+----------------+
| [STALE_REGION] | octavia        | load-balancer  |
| [STALE_REGION] | resmgr         | resmgr         |
| [STALE_REGION] | designate      | dns            |
| [STALE_REGION] | watcher        | infra-optim    |
| [STALE_REGION] | nova           | compute        |
| [STALE_REGION] | glance         | image          |
... more

Workaround

Proceed with extreme caution. The steps below are non-recoverable and could result in a complete system outage.

• Always double-check the region ID before deletion to avoid affecting active deployments.

• This operation is safe and non-disruptive if performed correctly.

NOTE: DO NOT PERFORM THE DELETE OPERATION ON ANY WORKING REGION.

1. Delete all endpoints associated with a given stale region, modify the script below using the stale region name, then execute it to remove the endpoints.

#!/bin/bash

echo "--- Capturing the endpoints in <stale-region-name>_endpoints_ids.txt ---"
openstack endpoint list --region <STALE_REGION_NAME> -c id --format value <STALE_REGION_NAME>_endpoints_ids.txt

while IFS= read -r epval; do
    echo "Deleting endpoint ID - $epval"
    openstack endpoint delete $epval
done < <STALE_REGION_NAME>_endpoints_ids.txt

1. Verify if there is any endpoint available for the stale region.

$ openstack endpoint list --region <STALE_REGION_NAME> -c Region -c "Service Name" -c "Service Type"

1. Once all stale endpoints associated with a given stale region are removed. Perform the region delete operation.

$ openstack region delete <STALE_REGION_NAME>

Delete all endpoints in the stale region first, or will encounter the issue below.

Failed to delete region with ID '[STALE_REGION_NAME]': Unable to delete region [STALE_REGION_NAME] because it or its child regions have associated endpoints. (HTTP 403) (Request-ID: [req_id])

Validation

• Validate if the stale region is deleted, the below command should not show any stale region information.

$ openstack region list