> This is part 1 of 11 of the full documentation (pages 1–100 of 1046). > The content is paginated: fetch every part to see all of it. > Next part: https://platform9.com/kb/llms-full.txt/1 > Page index: https://platform9.com/kb/llms.txt # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation This site has an agentic ask interface you may use to query the documentation dynamically by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/index.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. --- # KB Home Welcome to your team’s developer platform

Knowledge Base and Troubleshooting

Find solutions to common issues, troubleshooting guides, and best practices across Platform9 products. Search by topic or browse categories to quickly resolve issues. ### Private Cloud Director Find solutions to common issues, troubleshooting guides, and best practices for Private Cloud Director. Resolve VM, host, storage, and networking issues quickly.
Cover image
Knowledge Base

Find solutions to common issues, troubleshooting steps, and best practices for Private Cloud Director.

Browse by category or search for specific topics.

PCD-Knowledge Basepcd-kb.png
TroubleshootingDiagnose and resolve issues with service-specific debugging guides, log file locations, and step-by-step resolution paths.PCD-Troubleshootingpcd-tbs.png
#### Other Platform 9 Products
Cover image
Managed Kubernetes

Find solutions to common issues, troubleshooting steps, and best practices for Platform9 Managed Kubernetes (PMK).

Resolve cluster, node, networking, and storage issues.

Managed Kubernetesmanaged-k8-kb.png
Self Managed Cloud Platform

Find solutions to common issues, troubleshooting guides, and best practices for Platform9 Self Managed Cloud Platform.

Resolve installation, cluster, and air-gap deployment issues.

Self Managed Cloud Platformsmcp-kb.png
Managed OpenStack

Find solutions to common issues, troubleshooting steps, and best practices for Platform9 Managed OpenStack (PMO).

Resolve VM, networking, storage, and host issues.

Managed Open Stackpmo-kb.png
{% columns %} {% column %}
{% endcolumn %} {% column %} **Platform9 Generic** Find solutions to common issues, troubleshooting steps, and best practices for Platform9 products. Resolve authentication, deployment, configuration, and connectivity issues. Learn More {% endcolumn %} {% endcolumns %} # How to Check PCD Version? ## Problem Get the PCD Version information. ## Environment * Private Cloud Director - v2025.4 and Higher. * Self-Hosted Private Cloud Director Virtualisation – v2025.4 and Higher. ## Procedure **Using UI:** PCD version is displayed at the **bottom of the left side panel** of the management plane UI. **Using CLI:** {% hint style="info" %} `jq` binary should be present on the system used to check version. {% endhint %} The currently running PCD Version can also be checked through CLI with the below command: ```ruby $ curl -s https:///clarity/features.json | jq .releaseVersion ``` # DRR Not Triggering VM Migration Under Host Memory Pressure Leading to OOM-Induced VM Crashes ## Problem * Multiple VMs are unexpectedly transitioning to **SHUTOFF** state after crashing at the hypervisor level due to host memory pressure (OOM events). {% code title="Kernel Logs" %} ```log Out of memory: Killed process (qemu-system-x86) CPU x/KVM invoked oom-killer ``` {% endcode %} * Despite available memory capacity on other hosts in the cluster, **Dynamic Resource Rebalancing (DRR)** is not triggering live migrations. As a result, memory pressure continues to build on overloaded hosts, and the Linux kernel OOM killer terminates QEMU processes, leading to VM crashes. {% code title="Watcher Logs" %} ```log Node overloaded, attempting to reduce load No destination hosts suggested by nova scheduler ``` {% endcode %} ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher ## Cause DRR strategy detects overloaded hosts and attempts to rebalance workloads. However, during migration planning, the **Nova scheduler does not return any valid destination hosts**, even when sufficient capacity is available within the cluster. This results in: * No live migrations being triggered * Continued memory pressure on affected hosts * Kernel OOM killer terminating `qemu-system-x86` processes * VMs crashing and transitioning to **SHUTOFF** state This indicates a scheduler/placement decision issue rather than actual resource exhaustion. ## Workaround * Kill the **watcher decision engine pod**. Once restarted, it re-establishes the connection to RabbitMQ and DRR starts functioning again. ```bash kubectl get pods -n $NS | grep -i decision kubectl delete pod watcher-decision-engine-6fbd96b9b4-5l76d -n $NS ``` Once the pod restarts, DRR starts working again and migrations may resume. {% hint style="warning" %} This is a temporary workaround. The issue may reoccur, especially if RabbitMQ restarts. {% endhint %} ## Additional Information This issue is currently being tracked by the engineering team under PCD-1854, and a fix is planned to be included in the April 2026 release. # How to Mount ISO in a PCD VM? ## Problem Platform9 Private Cloud Director (PCD) does **not** support directly attaching an ISO image to a running instance (VM). Users who try to mount an ISO; for example, to install software, mount installation media, or load drivers inside a guest OS will find no direct option in the PCD UI. ## Environment * Private Cloud Director - v2025.4 and Higher. * Self-Hosted Private Cloud Director Virtualization – v2025.4 and Higher. * Component: ISO ## Procedure Currently, the only viable workaround to use ISO files in a VM is to convert the ISO into a Persistent Storage volume, then attach that volume to the existing VM. Once attached, the guest OS can mount it like any other block device. ### Prerequisites * PCD CLI access with the OpenStack client installed and configured (`openstack --version` works). * PCD project credentials sourced (e.g., `source pcdctl.rc`). * The VM must be in **STOPPED** state before mounting/unmounting ISO. * The ISO file is either already uploaded to Image Library, or available locally to upload. * The target VM exists, and the user must have permission to attach volumes to it. * Sufficient quota for an additional volume in the project. {% stepper %} {% step %} ### **Upload the ISO to Glance (skip if already present)** {% code overflow="wrap" %} ```bash openstack image create --os-interface admin --insecure --disk-format iso --container-format bare --file ``` {% endcode %} Verify the image is available: ```bash openstack image list | grep ``` {% endstep %} {% step %} ### Create a volume from the ISO image ```bash openstack volume create --image --size 5 ``` * `--size` must be **at least as large as the ISO**, rounded up to the next whole GB. * Replace `` with the image name or ID from Step 1. Wait for the volume to reach `available` status: ```bash openstack volume show -c status ``` {% endstep %} {% step %} ### Attach the volume to the existing VM ```bash openstack server add volume ``` To specify a device path inside the guest: ```bash openstack server add volume --device /dev/vdb ``` {% endstep %} {% step %} ### Verify the attachment ```bash openstack server show openstack volume show ``` The volume status should be `in-use` and the server should list the attached volume. {% endstep %} {% step %} ### Mount the ISO inside the guest OS * On a Linux guest: {% code title="Inside VM" %} ```bash sudo mkdir -p /mnt/iso sudo mount /dev/ /mnt/iso ls /mnt/iso ``` {% endcode %} * On a Windows guest, the new disk will appear in **Disk Management**. Assign a drive letter to access its contents. {% endstep %} {% step %} ### Detach the volume when finished * On a Linux guest: * Unmount inside the guest first: 
sudo umount /mnt/iso
* Then detach from the VM: ```bash openstack server remove volume ``` * Optionally delete the volume if no longer needed: ```bash openstack volume delete ``` * On a Windows guest: * Open **Disk Management** (`diskmgmt.msc`). * Right-click the ISO disk in the lower pane and select **Eject** (for the CD/ISO drive letter), or * Right-click the disk number on the left and select **Offline** {% endstep %} {% endstepper %} ## Additional Information: * Platform9 is devising a feature to enable users to mount ISO files to VMs via PCD UI via **PCD-924** # DRR Service Does Not Trigger Live Migration for VMs With NUMA Node Memory Exhaustion ## Problem Virtual machines pinned to a specific NUMA node via the `hw:numa_nodes=1` flavor extra spec are not automatically live-migrated by the DRR service when that NUMA node's memory is exhausted. The DRR service evaluates total available memory across the host when identifying candidates for live migration — it does not evaluate per-NUMA-node memory. When a single NUMA node is exhausted but the host has free memory available on other nodes, the DRR service considers the host within acceptable thresholds and does not trigger a migration. The affected VMs remain on the overloaded NUMA node and are vulnerable to kernel OOM kills that shut them down unexpectedly. The same gap exists at scheduling time: the Placement service tracks host-level memory inventory and does not account for per-NUMA-node memory availability. New VMs may be scheduled to a host where the target NUMA node is already under pressure. ## Environment * Private Cloud Director Virtualization — Version v2026.4-89 and below. * Self-Hosted Private Cloud Director Virtualization — Version v2026.4-89 and below. * Component: Watcher - DRR service ## Cause The DRR service and the Placement service do not model per-NUMA-node memory as a distinct resource dimension. **DRR service:** When evaluating whether to live-migrate a VM, the DRR service calculates host-level memory utilization — the sum of memory used across all NUMA nodes divided by total host memory. When a single NUMA node is fully exhausted but the host has substantial free memory on other nodes, the host-level utilization ratio remains within configured thresholds and no live migration is triggered. The VM stays on the exhausted NUMA node. **Placement service:** When scheduling new VMs, the Placement service checks whether the host has sufficient total memory to fulfill the flavor request. Per-NUMA-node memory availability is not tracked as a placement resource. The Placement service may return a host as a valid scheduling candidate even when the NUMA node the VM will be pinned to is already heavily loaded. The combined effect is that VMs with `hw:numa_nodes=1` that land on an overloaded NUMA node receive no automated relief — neither from the scheduler at creation time nor from the DRR service during operation. ## Diagnostics The following steps confirm the DRR service is running but not triggering live migration due to the NUMA memory evaluation gap — as opposed to the DRR service being down or misconfigured. {% stepper %} {% step %} #### Step 1 — Confirm NUMA Node Is Exhausted While Total Host Memory Is Healthy On the affected Compute Host, check memory availability per NUMA node and compare with total host memory. A scenario where one NUMA node shows near-zero free memory while overall host memory is healthy confirms the DRR evaluation gap. {% code title="Compute Host" overflow="wrap" %} ```bash $ for node in /sys/devices/system/node/node*/meminfo; do echo "=== $node ==="; grep -E "MemTotal|MemFree" "$node"; done ``` {% endcode %} {% code title="Sample Output" %} ``` === /sys/devices/system/node/node0/meminfo === Node 0 MemTotal: 1056888716 kB Node 0 MemFree: 504399044 kB === /sys/devices/system/node/node1/meminfo === Node 1 MemTotal: 1056888716 kB Node 1 MemFree: 529672 kB ``` {% endcode %} {% code title="Compute Host" %} ```bash $ free -h ``` {% endcode %} {% code title="Sample Output" %} ``` total used free shared buff/cache available Mem: [TOTAL] [USED] [FREE] [SHARE] [CACHE] [AVAIL] ``` {% endcode %} If one NUMA node shows near-zero `MemFree` while `free -h` reports substantial total free memory, the DRR service would not identify this host as a rebalancing candidate — confirming this issue. {% endstep %} {% step %} #### Step 2 — Verify the DRR Service Is Running Confirm all DRR service pods are in a healthy running state. If the DRR service is down, the absence of live migration is caused by a service failure rather than this NUMA evaluation gap. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl get pods -n | grep -i watcher ``` {% endcode %} {% code title="Sample Output" %} ``` watcher-api-[POD_ID] 2/2 Running 0 [TIMESTAMP] watcher-decision-engine-[POD_ID] 2/2 Running 0 [TIMESTAMP] watcher-applier-[POD_ID] 2/2 Running 0 [TIMESTAMP] ``` {% endcode %} All three pods — `watcher-api`, `watcher-decision-engine`, and `watcher-applier` — must be in `Running 2/2` state for the DRR service to be operational. If any pod is not running, contact Platform9 Support to investigate the DRR service failure separately. {% endstep %} {% step %} #### Step 3 — Confirm No Live Migration Was Triggered for the Affected VM Check the migration history for the affected VM to confirm the DRR service did not trigger a live migration before the OOM kill occurred. {% code title="OpenStack CLI" %} ```bash $ openstack server migration list --server ``` {% endcode %} {% code title="Sample Output" %} ``` +----+-------------+-----------+------+--------+ | Id | Source Node | Dest Node | Type | Status | +----+-------------+-----------+------+--------+ (no entries) +----+-------------+-----------+------+--------+ ``` {% endcode %} An empty migration history for a VM that was OOM-killed on a NUMA-exhausted host — while the host had overall free memory — confirms the DRR service did not intervene. {% endstep %} {% endstepper %} If all checks above match — one NUMA node near-zero while total host memory is healthy, DRR service pods are running, and no live migration was triggered — proceed to the [Workaround](#workaround) below. ## Resolution A fix for **PCD-8111** is targeted for release in version 2026.AUG. After upgrading, the DRR service will account for per-NUMA-node memory when identifying rebalancing candidates, and the Placement service will factor NUMA node memory availability into scheduling decisions. Until the fix is available, use the [Workaround](#workaround) below. ## Workaround The DRR service cannot be configured to evaluate per-NUMA-node memory in affected versions. Two manual mitigations are available. Apply both methods together for the most effective interim protection. ### Method 1 — Manually Monitor NUMA Node Memory and Live-Migrate VMs Proactively Identify Compute Hosts where a NUMA node is approaching exhaustion and manually live-migrate VMs off that host before the OOM killer triggers. This directly addresses the DRR gap and moves VMs to hosts with healthier NUMA node memory. {% stepper %} {% step %} #### Step 1 — Check NUMA Node Memory Across Affected Compute Hosts Run the following on each Compute Host to identify which NUMA node is under pressure. A `MemFree` value below 5% of `MemTotal` on any NUMA node signals risk of OOM kills. {% code title="Compute Host" %} ```bash $ for node in /sys/devices/system/node/node*/meminfo; do echo "=== $node ==="; grep -E "MemTotal|MemFree" "$node"; done ``` {% endcode %} {% code title="Sample Output" %} ``` === /sys/devices/system/node/node0/meminfo === Node 0 MemTotal: 1056888716 kB Node 0 MemFree: 504399044 kB === /sys/devices/system/node/node1/meminfo === Node 1 MemTotal: 1056888716 kB Node 1 MemFree: 529672 kB ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Identify VMs Pinned to the Exhausted NUMA Node List VMs running on the affected Compute Host, then confirm which ones are pinned to the exhausted node via their libvirt XML. {% code title="OpenStack CLI" %} ```bash $ openstack server list --host -c ID -c Name -c Status ``` {% endcode %} For each VM, check the NUMA node assignment. {% code title="Compute Host" %} ```bash $ virsh dumpxml | grep -A5 "numatune" ``` {% endcode %} {% code title="Sample Output" %} ```xml ``` {% endcode %} VMs with `nodeset=''` are the live-migration candidates. {% endstep %} {% step %} #### Step 3 — Live-Migrate Candidate VMs to a Less-Loaded Host Initiate a live migration for each candidate VM. The Compute Service scheduler selects a destination host. The VM remains running throughout the migration. {% code title="OpenStack CLI" %} ```bash $ openstack server migrate --live-migration ``` {% endcode %} Monitor migration status until the VM returns to `ACTIVE` on a new host. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c "OS-EXT-SRV-ATTR:host" ``` {% endcode %} {% code title="Sample Output" %} ``` +---------------------------+--------------------+ | Field | Value | +---------------------------+--------------------+ | status | ACTIVE | | OS-EXT-SRV-ATTR:host | [NEW_COMPUTE_HOST] | +---------------------------+--------------------+ ``` {% endcode %} Confirm the VM is `ACTIVE` on a new Compute Host. Verify the destination host has healthy NUMA node memory by repeating Step 1 on that host. {% endstep %} {% endstepper %} ### Method 2 — Apply the NUMA Memory Policy Fix to Prevent OOM Kills Changing the NUMA memory policy from `strict` to `preferred` ensures VMs can access memory on other NUMA nodes when their assigned node is exhausted — preventing the kernel OOM killer from terminating them. This mitigates the consequence of the DRR gap without resolving the underlying evaluation limitation. {% hint style="info" %} Method 2 protects VMs from OOM kills but does not cause the DRR service to rebalance overloaded hosts. Apply Method 1 alongside Method 2 for the most complete interim mitigation. {% endhint %} For full instructions, see [VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service](https://platform9.com/kb/pcd/compute/vms-shut-down-unexpectedly-due-to-strict-numa-memory-policy-in-compute-service). ## Additional Information This issue is tracked in **PCD-8111**. The engineering fix updates the DRR service to evaluate per-NUMA-node memory availability as a distinct resource dimension when selecting live-migration candidates, and updates the Placement service to factor NUMA node memory availability into scheduling decisions for VMs using NUMA pinning. **Related Articles:** * [VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service](https://platform9.com/kb/pcd/compute/vms-shut-down-unexpectedly-due-to-strict-numa-memory-policy-in-compute-service) — root cause of VMs being OOM-killed when a NUMA node is exhausted; workaround steps to change the NUMA memory policy from strict to preferred # Hotplug Memory Reporting Mismatch in Windows Guest OS ## Problem * When configuring Hotplug memory in a VM, the memory reported inside the Windows VM shows the maximum hotplug range instead of the configured hotplug memory value. * Example: With Hotplug RAM = 8 GB and Max Hotplug RAM = 64 GB, the Windows VM reports 64 GB instead of 8 GB. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher ## Cause This is a reporting mismatch inside Windows guests when the configured Hotplug RAM is less than the Max Hotplug RAM. * The VM is presented with the maximum hotplug range, which Windows reports as total memory. * Actual usable memory remains aligned to the configured hotplug value but may not be correctly displayed in Windows Resource Manager. ## Workaround * In some cases, performing a VM reboot may help the guest OS reflect the correct usable memory. * If the issue persists, a practical workaround is to configure the VM with *Hotplug RAM = Max Hotplug RAM* so that Windows reports the expected value. ## Additional Information This behavior is currently under review by the engineering team, and the enhancement is planned to be available in future releases of PCD under **PCD-2855**. # Hostagent Intermittently Fails to Converge with Virsh Domcapabilities Failures ## Problem After performing a host upgrade or re-onboarding (decommission + prep-node) of a compute host in Platform9 PCD, the hostagent intermittently fails to converge. The host may appear **Online** in the PCD UI, but the hostagent log continuously reports errors related to `virsh domcapabilities` and `fetch_gpu_info.py`. ## Environment * Private Cloud Director Virtualization — v2025.10-180 * Self-Hosted Private Cloud Director Virtualization — v2025.10-180 * Component**:** Hostagent, GPU. ## Diagnostics * Check for virsh domcapabilities failure and converging failures in `hostagent.log` 
sysinfo.py ERROR - Failed to get supported CPU models:
    Command '['virsh', 'domcapabilities']' returned non-zero exit status 1.
  Traceback (most recent call last):
    File ".../bbslave/sysinfo.py", in get_supported_cpu_models
      result = subprocess.run(['virsh', 'domcapabilities'],
  subprocess.CalledProcessError: Command '['virsh', 'domcapabilities']'
    returned non-zero exit status 1.
* Check for `fetch_gpu_info.py` extension failure 
session.py ERROR - /opt/pf9/hostagent/extensions/fetch_gpu_info.py command failed:
    Command '['/opt/pf9/hostagent/extensions/fetch_gpu_info.py']'
    returned non-zero exit status 1.
#### Affected Scenarios: This issue has been observed in the following scenarios: * Host upgrade (in-place package upgrade of `pf9-ostackhost`) * Host re-onboarding via `pcdctl decommission-node -f` followed by `pcdctl prep-node` ## Cause The `/opt/pf9/home` directory is incorrectly owned by `root:root` instead of `pf9:pf9group` after a host upgrade or re-onboarding. The hostagent runs `virsh domcapabilities` under the `pf9` user context. When `/opt/pf9/home` is owned by `root`, the `pf9` user cannot access its home directory, causing `virsh` to fail with a non-zero exit code. The root cause is in the Platform9 package installation script `/opt/pf9/hostagent/bin/pf9-apt`. The package manager runs as `root` and extracts package files with root ownership. When the `pf9-ostackhost` package is installed or upgraded, `/opt/pf9/home` is extracted/created with `root:root` ownership instead of `pf9:pf9group`. This is confirmed by the directory listing post-upgrade: ```bash $ ls -lart /opt/pf9/ drwxr-xr-x 3 root root 4096 [TIMESTAMP] home ← INCORRECT (should be pf9:pf9group) drwxr-xr-x 2 root root 4096 [TIMESTAMP] pf9-ostackhost drwxr-xr-x 8 pf9 pf9group 4096 [TIMESTAMP] venv drwxr-xr-x 2 pf9 pf9group 4096 [TIMESTAMP] cache ``` ## Resolution This is a known issue identified in PCD v2025.10-180, and the fix is to ensure the package installation/upgrade process correctly sets `pf9:pf9group` ownership to `/opt/pf9/home` during package extraction is now incorporated into **PCD v2026.1-260** and above versions. ## Workaround If upgrading immediately is not possible, apply the following immediate workaround on each affected compute host. {% stepper %} {% step %} #### Fix directory ownership ```bash chown pf9:pf9group /opt/pf9/home ``` {% endstep %} {% step %} #### Verify the ownership fix ```bash ls -la /opt/pf9/ | grep home ##Expected output: drwxr-xr-x 3 pf9 pf9group 4096 [TIMESTAMP] home ``` {% endstep %} {% step %} #### Verify virsh works under the pf9 user ```bash sudo -u pf9 virsh domcapabilities ``` Expected: XML output returned without errors. {% endstep %} {% step %} #### Check hostagent converge recovers automatically ```bash tail -f /var/log/pf9/hostagent.log | grep -i "converge\|error" ``` Expected: `--- Converging ---` followed by no `virsh` or `fetch_gpu_info` errors. In most cases, **no service restart is required**. The hostagent will automatically retry to converge and succeed once the ownership is corrected. {% endstep %} {% step %} #### (If needed) Restart hostagent manually If the hostagent does not recover automatically within 2–3 minutes: ```bash sudo systemctl restart pf9-hostagent sudo systemctl status pf9-hostagent ``` {% endstep %} {% endstepper %} ## Validation After applying the workaround or upgrade, validate in the following order: {% stepper %} {% step %} #### Confirm correct ownership ```bash ls -la /opt/pf9/ | grep home ##Expected file ownership: pf9 pf9group ``` {% endstep %} {% step %} #### Confirm virsh works under the pf9 user 
sudo -u pf9 virsh domcapabilities | head -5

## Expected XML output:
<domainCapabilities>
  <path>/usr/bin/qemu-system-x86_64</path>
  <domain>kvm</domain>
  <machine>pc-i440fx-focal</machine>
  <arch>x86_64</arch>
{% endstep %} {% step %} #### Confirm hostagent is converging cleanly ```bash tail -30 /var/log/pf9/hostagent.log ``` Expected: `--- Converging ---` with no `virsh` or `fetch_gpu_info` errors {% endstep %} {% step %} #### Confirm the host is Online in the PCD UI Navigate to: **Infrastructure → Hosts** Expected: Host status = **Online / Applied** {% endstep %} {% endstepper %} ## Additional Information * **GPU Hosts:** This issue has a higher impact on GPU compute hosts. In addition to the `virsh domcapabilities` failure, the `fetch_gpu_info.py` extension also fails, which can prevent GPU resource reporting to the hostagent and affect GPU VM scheduling. Apply the ownership fix on **all GPU hosts** after any upgrade or re-onboarding. * **Re-onboarding:** If a host was decommissioned (`pcdctl decommission-node -f`) and re-onboarded (`pcdctl prep-node`), the ownership issue may reoccur. Always verify `/opt/pf9/home` ownership after re-onboarding and apply the `chown` fix if needed. # PCD UI Port List Page Doesn’t Show Assigned Security Groups ## Problem In the PCD UI, the **Network > Ports** list page does not display security groups that are assigned to a port. The OpenStack CLI correctly shows the security group assignment for the same port, and the VM details page in the UI also shows the security group. The inconsistency is limited to the Ports list view. This can occur when ports are created with security groups assigned via Terraform, CLI, or any non-UI workflow and then attached to a VM. ## Environment * Private Cloud Director Virtualization - v2026.1 Release * Self Hosted Private Cloud Director Virtualization - v2026.1 Release * Components: PCD UI ## Cause The PCD UI Ports list page does not correctly render the security group data returned by Neutron for each port. The Neutron backend successfully returns port details including `security_group_ids`, and Neutron also successfully resolves the security group ID to a name. The issue is isolated to the UI data path for the Network > Ports list view — either the list request omits the required field, the UI fails to map security group IDs to names, or the column renderer does not display a valid value. The backend request path through ingress, Neutron, and CQRS is healthy with no 4xx/5xx errors observed ## Diagnostics 1. Navigate to **Network > Ports** in the PCD UI and locate the affected port. Confirm that the security group column is empty or missing for the port. 2. Navigate to the VM details page in the PCD UI for the VM that has the port attached. Confirm whether the security group is visible in the VM view. 3. Verify the port has a security group assigned using the CLI. ```bash $ openstack port show ``` In the output, check the `security_group_ids` field. If it contains a UUID, the security group is assigned at the backend level. 4. Verify the security group name resolves correctly. ```bash $ openstack security group show ``` If the security group is returned with a valid name, the backend data is intact. 5. Optionally, confirm the VM also shows the port attachment and security group. ```bash $ openstack server show ``` Check the `security_groups` field in the output. If the CLI confirms that the port has a security group assigned but the PCD UI Ports list does not display it, then this is a known UI defect. ## Resolution No workaround as this is a known UI issue tracked under in `PCD-6274`. A fix is planned in a future PCD release. To verify the current security group assignment for any port, use the OpenStack CLI as described in the [#diagnostics](#diagnostics "mention") section above. ## Additional Information * The security group is correctly enforced at the network level even when the UI does not display it. There is no security or functional impact from this display issue. * The issue affects the Ports list page specifically. The VM details page in the PCD UI correctly shows the security group. * The defect can be observed with ports created via Terraform, CLI, or any method — it is not limited to a specific port creation workflow. # Unable to Delete Network Host Config from UI ## Problem Unable to delete network host config from the UI, even if the host is not using it. ## Environment * Private Cloud Director Virtualization - till 2025.4-497 * Self-Hosted Private Cloud Director Kubernetes - till 2025.4-497 * Component: Blueprint ## Cause The host configuration was previously associated with a host that has since been decommissioned. However, a **stale mapping entry** remains between the host and the host configuration in the `host_to_hostconfig_map` table of the Resource Manager database. This issue is tracked under bug **PCD-407** and is now fixed in PCD 2025.6-151 and later releases, which prevent new stale mappings from being created. {% hint style="warning" %} Existing stale entries from earlier versions will persist after the upgrade; it's recommended to follow the steps mentioned in the Workaround to delete the stale entries. {% endhint %} ## Workaround * Retrieve the token: ```bash token=$(openstack token issue -f value -c id) ``` * Get the list of host configs: ```bash curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/hostconfigs | jq '.[] | {id, name}' # Sample Output: { "id": "[HOSTCONFIG_ID_1]", "name": "[HOSTCONFIG_NAME_1]" } { "id": "[HOSTCONFIG_ID_2]", "name": "[HOSTCONFIG_NAME_2]" } { "id": "[HOSTCONFIG_ID_3]", "name": "[HOSTCONFIG_NAME_3]" } ``` * Get the list of hosts that use the host config that needs to be deleted: ```bash curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/hosts | jq -c '.[] | select(.hostconfig_id == "") | {id, hostname: .info.hostname, hostconfig_id}' # Sample Output: { "id": "[HOST_ID_1]", "hostname": "[HOSTNAME_1]", "hostconfig_id": "[HOSTCONFIG_ID]" } { "id": "[HOST_ID_2]", "hostname": "[HOSTNAME_2]", "hostconfig_id": "[HOSTCONFIG_ID]" } ``` * Delete the stale host config mapping. Run below for each `[HOST_ID]`returned in the output of the command executed in the above output ```bash curl -s -X DELETE -H "X-Auth-Token: $token" "https:///resmgr/v2/hosts//hostconfig/" ``` * Delete the host Configuration after removing all the mappings ```bash curl -s -X DELETE -H "X-Auth-Token: $token" "https:///resmgr/v2/hostconfigs/" ``` ## Validation Check the UI to confirm that the previously stale host configuration no longer appears. # How to Delete Cluster BluePrint and HostConfig Using Resmgr API How to Delete Cluster Blueprint and HostConfig Using Resmgr API? ## Problem An existing cluster blueprint and host config cannot be deleted through the user interface (UI). ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component - Blueprint ## Procedure {% hint style="warning" %} Below steps should be used when there is a need to delete an existing blueprint or host configuration, or in cases where issues occur—such as failure to save or errors while creating a new blueprint. It is recommended to always list the existing blueprints to check for any stale entries before proceeding. {% endhint %} 1. Retrieve the token: ```bash $ token=$(openstack token issue -f value -c id) ``` 2. Get the list of Blueprints and Hostconfigs ```bash $ curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/blueprint | jq '.[] | {name}' ``` ```bash $ curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/hostconfigs | jq '.[] | {id, name}' ``` 3. Delete the required Blueprint and Hostconfig ```bash $ curl -s -X DELETE -H "X-Auth-Token: $token" "https:///resmgr/v2/blueprint/" ``` ```bash $ curl -s -X DELETE -H "X-Auth-Token: $token" "https:///resmgr/v2/hostconfigs/" ``` ## Validation ```bash $ curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/blueprint | jq '.[] | {name}' $ curl -s -X GET -H "X-Auth-Token: $token" https:///resmgr/v2/hostconfigs | jq '.[] | {id, name}' ``` # How to Setup CLI in Virtual Environment for Troubleshooting? ## Problem How to configure the OpenStack command-line client within a virtual environment to interact with an OpenStack cloud. ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Virtualization ## Procedure 1. Create a Virtual Environment (This command creates a new Python virtual environment named `openstack-venv` in your home directory): ```bash $ sudo python3 -m venv ~/openstack-venv ``` 2. Activate the Virtual Environment: ```bash $ source ~/openstack-venv/bin/activate ``` 3. Install OpenStack Client: ```bash $ sudo pip install python-openstackclient ``` 4. Verify OpenStack client by running the command: ```bash $ openstack --help ``` # How to Force Remove offline host from PCD Using API call? ## Problem API-driven forced removal of an offline host in PCD. ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Virtualization ## Procedure 1. Obtain an API token using the OpenStack CLI: ```bash $ openstack token issue -f value -c id ``` 2. Identify the Host ID of the offline host from the UI or using below api call: ```bash $ curl -X GET \\ -H "X-Auth-Token: " \\ "https:///resmgr/v2/hosts" \\ | jq '[.[] | {id, hostname: .info.hostname, responding: .info.responding}]' ``` {% code title="Sample Output" %} ```bash $ curl -s -X GET \\ -H "X-Auth-Token: " \\ "https://[example-region1.app.pcd.platform9.com]/resmgr/v2/hosts" \\ | jq '[.[] | {id, hostname: .info.hostname, responding: .info.responding}]' [ { "id": "[HOST_ID_1]", "hostname": "[HOST_NAME_1]", "responding": true }, { "id": "[HOST_ID_2]", "hostname": "[HOST_NAME_2]", "responding": false }, ] ``` {% endcode %} 3. Force Remove the Host: {% hint style="warning" %} The **DELETE** API operation can be perfomed by Admin users only. Also invoke the **DELETE** API for hosts that are **already offline or permanently unrecoverable and have no VMs or images tied to them**. Deleting a live host can create system inconsistencies—placing it in a limbo `deauthorized` state, causing it to re‑appear in inventory, corrupting metadata, or triggering erratic resource behavior. For active hosts, use [`deauthorize`](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#remove-all-roles-and-deauthorize-a-host) and [`decommission`](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#decommission-a-host) operations instead to ensure safe and consistent removal. {% endhint %} ```bash $ curl -X DELETE \\ -H "X-Auth-Token: " \\ "https:///resmgr/v2/hosts/" ``` 4. Once the Host is removed, the associated compute service and network agents must be deleted. 1. Note the ID of compute service and network agents. 2. Delete the compute service and network agents associated to Host. {% hint style="info" %} Delete all the network agents configured for the affected host in this case \ , listed in the output of command `openstack network agent list -c ID -c Agent_Type -c Host | grep ` {% endhint %} ```bash # Note the ID of service_agents $ openstack compute service list -c ID -c Binary -c Host | grep | [COMPUTE_SERVICE_ID] | nova-compute | [HOST_ID_2] $ openstack network agent list -c ID -c Agent_Type -c Host | grep | [NETWORK_METADATA_AGENT_ID] | OVN Metadata agent | [HOST_ID_2] | | [NETWORK_CONTROLLER_AGENT_ID] | OVN Controller Gateway agent | [HOST_ID_2] | ..... # Delete the service_agents associated to Host $ openstack compute service delete $ openstack network agent delete $ openstack network agent delete ``` # Hypervisor Stuck in N/A Scheduling State After Host Onboarding ## Problem * Host remains in *N/A* and *scheduling* state in the UI. * The command `openstack compute service list` shows the host as “*down*”. ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Virtualization ## Cause: * Reach out to the [Platform9 Support Team](https://support.platform9.com/) for assistance to figure out the cause as during the report of this issue, the cause was unknown. ## Workaround * Identify the host using the below command and confirm that it is marked as 'Down'. ```bash $ openstack compute service list ``` * Delete the problematic compute service ```bash $ openstack compute service delete ``` * Reset the *compute\_id* on the affected host through below steps (To be performed on the affected host): ```bash $ cat /opt/pf9/data/state/compute_id <--note this ID $ cp /opt/pf9/data/state/compute_id /tmp/compute_id_bkp $ rm /opt/pf9/data/state/compute_id ``` * Deauthorize the node using UI or CLI. ```bash $ pcdctl deauthorize-node ``` * Re-authorize the hypervisor role and wait for successful onboarding. ```bash $ pcdctl authorize-node ``` # Stale OpenStack Regions in PCD ## Problem * Old OpenStack regions and endpoints still exist even after the PCD region is deleted. ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Kubernetes ## Cause * This is a known issue and a bug: **PCD-2664** has been raised to track this matter. {% hint style="danger" %} CAUTION * Do not **create/delete** an OpenStack region in PCD. * To create/delete a PCD region: * PCD SaaS Users: [Contact Platform9 Support](https://support.platform9.com/hc/en-us) * PCD Self-Hosted Users: Follow the [airctl](https://platform9.com/docs/private-cloud-director/private-cloud-director/airctl) region creation/deletion process. {% endhint %} ## Diagnostics 1. Get the region list using the command given below, and see the stale region entry. ```bash $ openstack region list +----------------+---------------+-------------+ | Region | Parent Region | Description | +----------------+---------------+-------------+ | Infra | None | | | [REGION_1] | Infra | | | [STALE_REGION] | Infra | | ... more ``` 2. Check if there are any endpoints available for the stale region. ```bash $ openstack endpoint list --region -c Region -c "Service Name" -c "Service Type" +----------------+----------------+----------------+ | Region | Service Name | Service Type | +----------------+----------------+----------------+ | [STALE_REGION] | octavia | load-balancer | | [STALE_REGION] | resmgr | resmgr | | [STALE_REGION] | designate | dns | | [STALE_REGION] | watcher | infra-optim | | [STALE_REGION] | nova | compute | | [STALE_REGION] | glance | image | ... more ``` ## Workaround {% hint style="danger" %} **Proceed with extreme caution.** The steps below are non-recoverable and could result in a complete system outage. * Always double-check the **region ID** before deletion to avoid affecting active deployments. * This operation is safe and non-disruptive if performed correctly. **NOTE**: DO NOT PERFORM THE DELETE OPERATION ON ANY WORKING REGION. {% endhint %} 1. Delete all endpoints associated with a given stale region, **modify the script below using the stale region name, then execute it to remove the endpoints.** ```bash #!/bin/bash echo "--- Capturing the endpoints in _endpoints_ids.txt ---" openstack endpoint list --region -c id --format value _endpoints_ids.txt while IFS= read -r epval; do echo "Deleting endpoint ID - $epval" openstack endpoint delete $epval done < _endpoints_ids.txt ``` 2. Verify if there is any endpoint available for the stale region. ```bash $ openstack endpoint list --region -c Region -c "Service Name" -c "Service Type" ``` 3. Once all stale endpoints associated with a given stale region are removed. Perform the region delete operation. ```bash $ openstack region delete ``` {% hint style="warning" %} Delete all endpoints in the stale region first, or will encounter the issue below. `Failed to delete region with ID '[STALE_REGION_NAME]': Unable to delete region [STALE_REGION_NAME] because it or its child regions have associated endpoints. (HTTP 403) (Request-ID: [req_id])` {% endhint %} ## Validation * Validate if the stale region is deleted, the below command should not show any stale region information. ```bash $ openstack region list ``` # Multiple PCD Services Down and Hosts Unresponsive Due to Sealed Decco Vault ## Problem * Hosts become unresponsive and report as offline. * Critical Platform9 services on the host, such as `pf9-ostackhost` and `pf9-cindervolume-base` , `pf9-neutron-ovn-metadata-agententer` , `pf9-novncproxy` entered a failed state. ## Environment * Self-Hosted Private Cloud Director Virtualization ## Cause Following pods on the control plane were stuck in the *Initializing* phase: * **Decco Vault** in the `default` namespace * **Vouch Keystone** in the corresponding region namespace * **Vouch NoAuth** in the corresponding region namespace The **Decco Vault** was found to be in a **sealed** state. A sealed state is a security posture where the vault's data is encrypted and inaccessible. ## Diagnostics Most of the service logs report connectivity errors. {% code title="ostackhost logs:" %} ```bash ERROR oslo.messaging._drivers.impl_rabbit [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: . Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: ERROR oslo.messaging._drivers.impl_rabbit [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: ConnectionRefusedError: [Errno 111] ECONNREFUSED ``` {% endcode %} {% code title="cindervolume-base logs" overflow="wrap" %} ```bash ERROR oslo.messaging._drivers.impl_rabbit [-] [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: ConnectionRefusedError: [Errno 111] ECONNREFUSEDEADDRINUSE encountered during SNI client creation ... retrying in 20000 milliseconds/ERRORnn:qlscd ..lsvim novG/ERROR:qvim cinvG/ERRORmodel server went away: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')n ``` {% endcode %} ## Resolution 1. Cordoned the node where the **Decco Vault,Vouch Keystone,Vouch NoAuth** Pods are scheduled. ```bash $ kubectl cordon ``` {% hint style="info" %} If all three pods are located on the same node, only need to cordon that single node. However, if the pods are distributed across different nodes, cordon each node and restart the pods one by one. The primary goal here is to reschedule the pods from their current node. {% endhint %} 2. Performed a rollout restart of the following deployments in the same sequence * **Decco Vault** in the `default` namespace * **Vouch Keystone** in the corresponding region namespace * **Vouch NoAuth** in the corresponding region namespace ```bash $ kubectl rollout restart deployment -n ``` 3. Restarted the pf9-hostagent service on the affected host. ```none # systemctl restart pf9-hostagent ``` ## Validation 1. The hypervisor returned to a **healthy** state 2. All Platform9 services were observed to be running normally on the affected node. # VMHA Stuck in "Waiting" ## Problem **VMHA** remains stuck in the **"waiting"** state during enablement. ## Environment * Private Cloud Director Virtualization – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization – v2025.4 and Higher ## Cause A decommissioned host was still listed in Nova's service records. Because of this, VMHA tried to use that host during setup, which caused an error and left the VMHA stuck in the "waiting" state. ## Diagnostics {% hint style="info" %} For SAAS customers contact Platform9 Support Team to validate if you are hitting the issue mentioned in this article. {% endhint %} 1. Check VMHA logs: ```bash $ kubectl exec deploy/hamgr -n -- cat/var/log/pf9/hamgr/hamgr.log | grep -A1 'Enabling HA' ``` {% code title="Look for log entries like the following:" %} ```ruby Enabling HA on some of the hosts [...] including host '[HOST-ID]' WARNING Role status of host [HOST-ID] is not ok ``` {% endcode %} 2. List compute services and validate if any of the hypervisors are showing the "Status" as `disabled` and "State" `down` {% hint style="info" %} Identify services that are down, disabled, or associated with non-existent or decommissioned hosts. In the sample output the HOST2.EXAMPLE.COM is the decommissioned node. {% endhint %} ```bash $ openstack compute service list #sample output +--------------------+-------------+--------------------+-------+---------+-------+--------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +--------------------+-------------+--------------------+-------+---------+-------+--------------+ | [HOST1_SERVICE_ID] | nova-compute| [HOST1.EXAMPLE.COM]| [zone]| enabled | up | [TIMESTAMP] | | [HOST2_SERVICE_ID] | nova-compute| [HOST2.EXAMPLE.COM]| [zone]| disabled| down | [TIMESTAMP] | | [HOST3_SERVICE_ID] | nova-compute| [HOST3.EXAMPLE.COM]| [zone]| enabled | up | [TIMESTAMP] | +--------------------+-------------+--------------------+-------+---------+-------+--------------+ ``` 3. List hypervisors and validate host mapping. In the sample output, we see that the node `[HOST2.EXAMPLE.COM]` is in a `down`state. we can check its associated `service ID` to validate the host mapping 
$ openstack hypervisor list

#sample output:
+----------------+---------------------+-----------------+-------------+-------+
| ID             | Hypervisor Hostname | Hypervisor Type | Host IP     | State |
+----------------+---------------------+-----------------+-------------+-------+
| [HOST1_UUID]   | [HOST1.EXAMPLE.COM] | QEMU            | [IP-ADDR-1] | up    |
| [HOST2_UUID]   | [HOST2.EXAMPLE.COM] | QEMU            | [IP-ADDR-2] | down  |
+----------------+---------------------+-----------------+-------------+-------+

$ openstack hypervisor show <HYPERVISOR_ID>

#sample output 
$ openstack hypervisor show [HOST2_UUID]
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| aggregates          | []                                   |
| cpu_info            | None                                 |
| host_ip             | [IP-ADDR-2]                          |
| hypervisor_hostname | [HOST2.EXAMPLE.COM]                  |
| hypervisor_type     | QEMU                                 |
| hypervisor_version  | [HYPERVISOR_VERSION]                 |
| id                  | [HOST2_UUID]                         |
| service_host        | [SERVICE_HOST_UUID]                  |
| service_id          | [HOST2_SERVICE_ID]                   |
| state               | down                                 |
| status              | disabled                             |
+---------------------+--------------------------------------+
## Resolution 1. Identify the **stale** compute service entry from the output of the below command. In the sample output we see the node `HOST2.EXAMPLE.COM` is down. 
$ openstack compute service list

#sample output
+--------------------+-------------+--------------------+-------+---------+-------+--------------+
| ID                 | Binary      | Host               | Zone  | Status  | State | Updated At   |
+--------------------+-------------+--------------------+-------+---------+-------+--------------+
| [HOST1_SERVICE_ID] | nova-compute| [HOST1.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
| [HOST2_SERVICE_ID] | nova-compute| [HOST2.EXAMPLE.COM]| [zone]| disabled| down  | [TIMESTAMP]  |
| [HOST3_SERVICE_ID] | nova-compute| [HOST3.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
+--------------------+-------------+--------------------+-------+---------+-------+--------------+
2. Delete the **stale** service using below command, post deletion of the stale entry we will still have minimum two working hypervisors as per the requirement of enabling VMHA ```bash $ openstack compute service delete ``` 3. Wait for the VMHA to retry the operation automatically, or disable and re-enable VMHA to trigger a fresh attempt. ## Validation: 1. Ensure VMHA state transitions from `waiting` to `enabled`. 2. Confirm no additional stale hosts remain. ## Additional Information: At minimum two working hypervisors are needed for enabling [VMHA](https://platform9.com/docs/private-cloud-director/private-cloud-director/virtual-machine-high-availability--vm-ha-#pre-requisites) # Force Remove the Host Role From DB ## Problem Unable to remove a host role from a specific host via the Platform9 UI, `pcdctl` and `curl` command. The UI shows an `RoleUpdateConflict` error when attempting to deauthorize and decommission the host. ```bash $ curl -s -X DELETE -H "X-Auth-Token: $token" https:///resmgr/v2/hosts//roles/ -vvv ... "message":"RoleUpdateConflict: Role cannot be updated in the current state: Cannot remove role [ROLE_NAME] from host [HOST_UUID] in the current state". ``` ## Environment * Self-Hosted Private Cloud Director Virtualization * Component: ResMgr ## Cause The host role remained in a `not-applied` state in the backend database, which blocked further updates or role removal actions through the UI. This inconsistency prevented the host from being deauthorized or decommissioned. ## Diagnostics {% hint style="info" %} These steps are applicable only for Self-hosted Private Cloud Director product {% endhint %} To check the role state: 1. Login to the resmgr DB: ```bash $ NS= $ DB_PASS=$(kubectl exec -i -n "$NS" $(kubectl get po -n "$NS" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") -c resmgr -- /bin/bash -c "DB=\$(consul-dump-yaml --start-key customers/\$CUSTOMER_ID/regions/\$REGION_ID/db | awk '/dbserver:/ {print \$2}'); consul-dump-yaml --start-key customers/\$CUSTOMER_ID/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n $NS $(kubectl get po -n $NS -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") --template '{{.status.podIP}}') $ kubectl exec -it -n "$NS" $(kubectl get po -n "$NS" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS" ``` 2. Check the role state: 
MySQL [resmgr]> "SELECT * FROM host_role_map WHERE res_id = '<HOST_UUID>';"

+----------------+-------------+---------------+
| res_id         | rolename    | current_state |
+----------------+-------------+---------------+
| [HOST_UUID]    | [ROLE_NAME] |  not-applied  |
+----------------+-------------+---------------+
The `not-applied` state indicates that the role was not fully or correctly applied, resulting in a conflict when attempting UI-based role changes. ## Resolution {% hint style="danger" %} **Caution:** The following steps involve direct database manipulation and should only be performed at most care. {% endhint %} 1. Delete the role from the backend database: ```bash MySQL [resmgr]> "DELETE FROM host_role_map WHERE res_id='';" ``` 2. Verify that the role mapping has been removed: ```bash MySQL [resmgr]> "SELECT * FROM host_role_map WHERE res_id = '';" ## Expected output: Empty set ``` {% hint style="warning" %} **Caution:** The following steps only if need to delete the host as well. {% endhint %} 3. Optionally, delete the host from the database if needed: ```bash MySQL [resmgr]> "DELETE FROM hosts WHERE id='';" ``` 4. Purge PF9 packages (if host is being decommissioned): ```bash $ sudo dpkg --purge --force-all pf9-hostagent $ sudo dpkg --purge --force-all pf9-comms $ sudo dpkg --purge --force-all pf9-neutron-ovn-controller $ sudo dpkg --purge --force-all pf9-ostackhost ``` 5. Clean up PF9 services and directories: ```bash $ sudo systemctl stop 'pf9-*.service' $ sudo systemctl disable 'pf9-*.service' $ sudo systemctl daemon-reexec $ sudo systemctl daemon-reload $ rm -rf /etc/pf9 /opt/pf9 /var/log/pf9 /var/opt/pf9 ``` ## Validation * Confirm that the role mapping associated with the host has been removed * The host is no longer listed in the UI under Infrastructure > Hosts. * If re-onboarding, reauthorize the host and apply roles cleanly. * Review logs (`/var/log/pf9/hostagent.log`, `/var/log/pf9/comms/`) to ensure role application is successful. # Demystifying Deauthorization and Decommisioning of PCD Hosts ## Problem How does [Deauthorization](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#step-1---remove-all-roles-and-deauthorize-a-host) and [Decomissioning](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#step-2---decommission-a-host) work with the PCD hosts in the backend? ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Component - pcdctl ## Answer ### **Deauthorization of PCD Hosts** The deauthorization **removes roles** from an onboarded host and removes the host from the backend database. Ensure all the [prerequisites](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#remove-a-host) are taken care before triggering host deauthorization. This process is divided into four stages: #### 1. Initiating Deauthorization via CLI * The user triggers the deauthorization via: ```bash $ pcdctl deauthorize-node Do you wish to remove the roles present on the host? (y/n) Do you wish to delete the host? (y/n) ``` * It consists of two phases: Role Removal followed by Host Removal *\[optional]* #### 2. Role Removal (Deauthorization Phase) * PCD CLI internally calls `resmgr` role delete APIs, handling both: * **V1 roles** (Individual component roles like *pf9-glance-role, pf9-neutron-ovn-controller*) * **V2 roles** (Uber roles) — wrappers of V1 roles: *hypervisor, image-library, persistent-storage, dns.* * These uber roles further expand into multiple v1 roles: ```bash host_uber_roles = ("hypervisor", "image-library", "persistent-storage", "dns") hypervisor_roles = ["pf9-neutron-base", "pf9-ostackhost-neutron", "pf9-neutron-ovn-controller", "pf9-neutron-ovn-metadata-agent"] image_library_roles = ["pf9-glance-role"] persistent_storage_roles = ["pf9-cindervolume-base", "pf9-cindervolume-config"] dns_roles = ["pf9-designate"] ``` * Validation is performed before actual role removal. Each **v2 uber role** has its own validation logic that ensures whether the role's dependent components are in a healthy and clean state before proceeding with removal. * For Hypervisor role the checks include: * VMs are evacuated or deleted from the host. * No running workloads (nova instances). * Host is in disabled or maintenance state. * Host is not part of an aggregate/availability zone. * For Image Library role the checks include: * Host is not currently serving image storage. * No image data volume is mounted or in use. * All services related to Glance are stopped. * For Persistent Storage role the checks include: * No volumes are attached or in-use on the host. * All cinder backends (LVM, NFS, etc.) are unconfigured or inactive. * No active mount points from `/opt/pf9/pf9-cindervolume-*` * For DNS role the checks include: * The v1 role `pf9-designate` isn’t serving DNS on the host. * The CLI handles roles removal in reverse order of installation, as some roles have dependencies. For instance, v2 uber roles like `hypervisor` have their sub-roles \[v1 roles] removed collectively in reverse order; starting from `pf9-neutron-ovn-metadata-agent` to `pf9-neutron-base` . ```bash hypervisor_roles = [ "pf9-neutron-base", "pf9-ostackhost-neutron", "pf9-neutron-ovn-controller", "pf9-neutron-ovn-metadata-agent" ] ``` #### 3. Timeout Handling * The CLI waits for role deletion to complete and polls role status *(every 10 seconds)*. It polls for V2 role deletion completion. If deletion takes too long, it hits the `--timeout`*(default 5 minutes)* and exits. * Further, `resmgr` is queried again for any residual V1 roles, and are removed one by one. * A last polling mechanism takes place to ensure no roles remain unattended, resulting in either timeout or successful removal of roles. #### 4. Host Deletion * It deletes the host object from the control plane database (*resmgr),* condition being all the roles should have been removed for this phase to progress. * CLI calls `DELETE /resmgr/v2/hosts/{host_id}` , that deletes host metadata *(mapping, configs)* from `resmgr` database. > Host removal does not reattempt role cleanups, so skipping role removal earlier can leave stale role states. ### **Decommision of PCD Hosts** Decommissioning is the process of permanently **removing an onboarded host** from Private Cloud Director, it involves complete removal of the host and purges all PCD components, including binaries, libraries, packages, files on the host. The host be can safely **reused** in another PCD environment. Ensure all the [prerequisites](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#remove-a-host) are taken care before triggering host decommission. This process is divided into eight stages: #### 1. Initiating Decommission via CLI * The user triggers the decommission via: ```bash $ pcdctl decommission-node Do you wish to decommission the node? ``` * This command handles cleanup only after all roles have been removed (or if the role checks are skipped/forced via -r / *--skip-installed-role-check* option). * Forcefully decomissioning a node involves purging everything is enabled via `-f` / `--force` flag #### 2. Role Validation (Pre-check Phase) * Before cleanup, the CLI extracts host\_id from /`etc/pf9/host_id.conf` and queries resmgr API to retrieve all existing roles on the host. If roles exist, the CLI aborts unless --skip-installed-role-check is passed. #### 3. Cleanup Phase (Decommission Logic) * After validation, the CLI performs a multi-step cleanup \[4-9]. #### 4. Package removal * Remove following packages via `apt remove -y` ```bash pf9-hostagent pf9-ovn-controller ovn-host ovn-common ``` * Followed by, completely purging below two packages via apt-get purge -y , it ensures to completely remove the packages along with configuration and data files. ```bash pf9-comms pf9-hostagent ``` #### 5. Unmount Directories * Below PF9 directories are identified and unmounted: ```bash /opt/pf9/pf9-cindervolume-base/state/mnt/* /opt/data/instances /opt/pf9/etc/pf9-cindervolume-base/volumes/* /var/opt/imagelibrary/data/glance ``` * Also dynamically checks and unmounts any user-mounted PF9 directories via: `mount | awk '{print $3}' | grep -E "^/opt/pf9/|^/var/opt/pf9|^/var/log/pf9|^/root/pf9"` #### 6. Filesystem Cleanup * Deletes following directories when --force is not passed ```bash /etc/pf9 /var/log/pf9 /var/spool/mail/pf9 /root/pf9/ /var/opt/imagelibrary/data/glance/ /opt/data/instances/ /opt/pf9/data/state/compute_id /var/opt/pf9/neutron/metadata_proxy /opt/pf9/data/locks/ /opt/pf9/python/ /etc/pf9_environment ``` * When *--force* is passed, everything PF9-related, even parent directories that may include other nested content is deleted. It removes `/var/opt/pf9/ & /opt/pf9/` along with the aforementioned directories. #### 7. Network Cleanup * Lists all Open vSwitch bridges using the OVS API, deletes them one by one via ovs-vsctl del-br . This is followed execution of netplan apply to reconfigure networking defaults. - #### 8. Endpoint Removal * Removes OpenStack service entries from the two files. ```bash /etc/hosts /etc/cloud/templates/hosts.debian.tmpl ``` #### 9. Final System Reset * Ultimately, it executes `systemctl reset-failed` to clean up systemd states of failed services. # VMHA Fails to Work When Host Loses Connection to Shared Storage ## Problem * VMHA mechanism should be activated in scenarios when a host loses connectivity to shared storage. * In the event of a disconnection from shared storage, it is essential for the virtual machines (VMs) to be automatically evacuated from the affected host. This is necessary because, during such disconnections, the VMs can become unresponsive and enter a "*hang mode,*" rendering them read-only and inaccessible. ## Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Private Cloud Director Kubernetes – v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.6 and Higher * Component: VMHA ## Answer * This is a known product limitation, and an RFE (Request for Feature Enhancement) is reported internally with ID - **PCD-3448.** ## Additional Information * For further questions/concerns regarding the RFE, reach out to the [Platform9 Support Team](https://support.platform9.com/hc/en-us). # Tenant or Projects dropdown shows empty ## Problem It was observed that the tenants/projects list shows up empty when the user logs in to a Region with **a large number of tenants**. This was consistent for SSO and local users. The UI inspect was showing a lot of calls failing with `"net error":"ERR HTTP2 PROTOCOL ERROR"` ## Environment * Private Cloud Director Virtualization - till v2025.8 * Private Cloud Director Kubernetes – till v2025.8 * Self-Hosted Private Cloud Director Virtualization -till v2025.8 * Self-Hosted Private Cloud Director Kubernetes - till v2025.8 * Component: UI ## Cause The cause of the problem was the nginx response headers being too large and being dropped or truncated on response during the login process. This is now tracked on PCD-3830, and the fix is available from **v2025.10-115 or above versions.** ## Diagnostics 1. It was noticed that the error was seen only for a user with access to a large number of tenants, which could impact the response header or response body size 2. The issue was not reproduced for users with access to a lesser number of tenants. 3. On browser, run an inspect on the page and check for `"net error":"ERR HTTP2 PROTOCOL ERROR"` 4. Debug logs were enabled on the `ingress-nginx-controller` pod on the control plane side by adding `--v=5` to the startup params in the deployment. The nginx debug logs are shown below, which confirmed the issue ```javascript [info] 71#71: *484 client sent too large header field while processing HTTP_2 connection, client: [IP_ADDRESS], server: 0.0.0.0:443 [info] 71#71: *568 client sent too large header field while processing HTTP_2 connection, client: [IP_ADDRESS], server: 0.0.0.0:443 ``` {% hint style="info" %} Step 4 can be performed on Self-Hosted PCD {% endhint %} ## Workaround The following changes were made to the `ingress-nginx-controller` deployment on the control plane, which fixed the problem. a) The config map was edited to add buffer params `client-header-buffer-size` and `large-client-header-buffers` as below: ```bash $ kubectl edit cm ingress-nginx-controller -n apiVersion: v1 data: allow-snippet-annotations: "true" annotations-risk-level: Critical client-header-buffer-size: 64k large-client-header-buffers: 4 32k kind: ConfigMap ``` b) After making the changes the ingress-nginx-controller deployment was restarted ```bash $ kubectl rollout restart deploy_ingress-nginx-controller -n ``` ## Validation After applying the changes, the tenant dropdown list was verified as loading ok for various browsers and users. # Hypervisor List Fails With "500 Internal Server" Error When Using Compute API Microversion ## Problem The `openstack hypervisor list` command fails with a 500 Internal Server Error, even though `nova hypervisor-list` and `openstack hypervisor show ` work as expected. The failure occurs specifically when querying the `/os-hypervisors/detail` API endpoint, which is used by the OpenStack unified client. For Self-Hosted PCD users, the `nova-api-osapi`pod logs shows repeated warnings about invalid UUIDs, such as: ```bash Invalid uuid: nova. To ensure interoperability, identifiers should be a valid uuid. ``` ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: OpenStack-CLI version 8.1.0 and above. ## Cause This is an expected behavior, starting from **Compute API micro version 2.53**, Nova replaced integer-based service and hypervisor IDs with **UUIDs** to ensure global uniqueness across multi-cell deployments. In upgraded environments (from pre-Newton/Ocata releases), the `compute_nodes` or `services` tables may still contain old non-UUID identifiers (e.g., `nova`, `compute-01`, or blank fields). When `/os-hypervisors/detail` is queried using microversion ≥ 2.53, Nova attempts to serialize these IDs as UUIDs. If invalid values are encountered, an internal Python `OSError` or `ValueError` is raised, leading to the 500 error. Older microversions (≤ 2.52) continue to work as they still rely on integer-based IDs and do not enforce UUID validation. ## Diagnostics * Command failure observed: ```bash $ openstack hypervisor list HttpException: 500: Server Error for url: https://[PCD_FQDN]/nova/v2.1/[ID]/os-hypervisors/detail, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. ``` * Verification using older micro version: ```bash $ openstack --os-compute-api-version 2.1 hypervisor list +-----+--------------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +-----+--------------------------+-----------------+---------------+-------+ | [ID]| [HOSTNAME] | QEMU | [IP] | up | ``` ## Resolution Run with a supported micro version ≤ **2.52** (before UUID enforcement): ```bash $ openstack --os-compute-api-version 2.1 hypervisor list ``` ## Validation Command working after using older microversion: ```bash $ openstack --os-compute-api-version 2.1 hypervisor list ``` The command should return a valid hypervisor list with no 500 errors, and For Self Hosted PCD users, `nova-api-osapi` pod logs should no longer display “`Invalid uuid`” warnings. ## Additional Information **References:** * [Nova Compute API Microversion History — 2.53: Change service and hypervisor IDs to UUIDs](https://docs.openstack.org/nova/latest/reference/api-microversion-history.html) * [API Reference: GET /os-hypervisors/detail](https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail) * [Blueprint: Use UUIDs in services and os-hypervisors APIs](https://blueprints.launchpad.net/nova/+spec/service-hyper-uuid-in-api) * [Nova Source Commit — UUID Enforcement in Hypervisors API](https://opendev.org/openstack/nova/src/commit/614403370a877d6c8ab841cd593f822d4675319b/nova/api/openstack/compute/hypervisors.py) # VMHA is Stuck in ErrorRemoving state in the PCD GUI ## Problem * After adding host to the PCD cluster, the VMHA status in the cluster section turned to be `Error Removing` . * VMHA becomes non-functional after adding a node to the respective cluster. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: VMHA ## Cause * A stale compute-service entry was still listed in Nova's service records. The same stale host entry was being retrieved by the availability zone. Because of this, VMHA tried to use that host during setup, which caused an error and left the VMHA stuck in the `ErrorRemoving` state. ## Diagnostics {% hint style="info" %} **Info** For SAAS customers contact Platform9 Support Team to validate if you are hitting the issue mentioned in this article. {% endhint %} * Review VMHA logs for any errors being logged during performing the disable-enabled action with VMHA over PCD UI. Capture the VMHA server logs from `hamgr` pod running inside affected region. The logs are present inside `/var/log/pf9/hamgr/` ```bash $ kubectl exec -it deploy/hamgr -n -- bash ``` * Capture the errors in hamgr logs. {% code title="hamgr.log" overflow="wrap" %} ```bash vmha.hamgr.providers.nova ERROR Disable HA request failed for cluster cluster-name Traceback (most recent call last): requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://bbmaster.region-name.svc.cluster.local:8082/v1/hosts/[HOST_UUID]/apps vmha.hamgr.db.api WARNING Task state being updated from removing to error-removing vmha.hamgr.providers.nova INFO process enable request for Availability zone REGION_NAME vmha.hamgr.providers.nova WARNING Cluster REGION_NAME is running task error-removing, cannot enable vmha.__main__ INFO Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/eventlet/wsgi.py", line 614, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python3.9/site-packages/webob/dec.py", line 129, in __call__ ``` {% endcode %} * List compute services and availability zones and validate if there is any entry of the `HOST_UUID` captured in the error logs of hamgr. \ Review its `State` and `Service Status` . In the sample output the HOST2.EXAMPLE.COM is the decommissioned node that was not properly removed. 
$ openstack compute service list --service nova-compute
#sample output
+--------------------+-------------+--------------------+-------+---------+-------+--------------+
| ID                 | Binary      | Host               | Zone  | Status  | State | Updated At   |
+--------------------+-------------+--------------------+-------+---------+-------+--------------+
| [HOST1_SERVICE_ID] | nova-compute| [HOST1.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
| [HOST2_SERVICE_ID] | nova-compute| [HOST2.EXAMPLE.COM]| [zone]| disabled| down  | [TIMESTAMP]  |
| [HOST3_SERVICE_ID] | nova-compute| [HOST3.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
+--------------------+-------------+--------------------+-------+---------+-------+--------------+

$ openstack availability zone list --compute --long
#sample output
+-----------+-------------+---------------+---------------------+----------------+---------|--------+
| Zone Name | Zone Status | Zone Resource | Host Name           | Service Name   | Service | Status |
+-----------+-------------+---------------+---------------------+----------------+---------|--------+
| [zone]    | available   |               | [HOST1.EXAMPLE.COM] | nova-compute   | enabled |  :-) 	|	 
| [zone]    | available   |               | [HOST2.EXAMPLE.COM] | nova-compute   | enabled |  XXX 	|	 
| [zone]    | available   |               | [HOST3.EXAMPLE.COM] | nova-compute   | enabled |  :-)		|
## Resolution * Identify the **stale** compute service entry from the output of the below command. In the sample output we see the service `HOST2.EXAMPLE.COM` is down. 
$ openstack compute service list

  #sample output
  +--------------------+-------------+--------------------+-------+---------+-------+--------------+
  | ID                 | Binary      | Host               | Zone  | Status  | State | Updated At   |
  +--------------------+-------------+--------------------+-------+---------+-------+--------------+
  | [HOST1_SERVICE_ID] | nova-compute| [HOST1.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
  | [HOST2_SERVICE_ID] | nova-compute| [HOST2.EXAMPLE.COM]| [zone]| disabled| down  | [TIMESTAMP]  |
  | [HOST3_SERVICE_ID] | nova-compute| [HOST3.EXAMPLE.COM]| [zone]| enabled | up    | [TIMESTAMP]  |
  +--------------------+-------------+--------------------+-------+---------+-------+--------------+
2. Delete the **stale** service using below command: ``` openstack compute service delete ``` 3. Wait for the VMHA to retry the operation automatically, or disable and re-enable VMHA to trigger a fresh reconcile attempt. ## Validation * Ensure VMHA state transitions from `ErrorDeleting` to `Enabled`. * Confirm no additional stale hosts remain. # No VMs Displayed When 'View VMs in All Tenants' Is Enabled When the **“View VMs in all Tenants”** option is enabled in the PCD UI, no VMs are displayed, and the associated API call fails with the following error: ```bash { "computeFault": { "code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
" } } ``` Users could search and view specific VMs by name. However, when the **“View VMs in all Tenants”** option is disabled, VMs under the current tenant are displayed correctly. CLI worked as expected for all-tenant queries (`openstack server list --all-projects`). ## Environment * Private Cloud Director Virtualization - till 2025.8-92 * Self-Hosted Private Cloud Director Virtualization - till 2025.8-92 * Compute API ## Cause A corrupted instance remained in the Compute database (nova-DB) due to a failure during VM provisioning (e.g., quota exceeded). * The instance record was left in an incomplete or `ERROR` state. * The corresponding request specification was missing from `nova_api.request_specs`. * When Nova attempted to fetch all VMs across tenants, it raised a `KeyError`, resulting in a `500` error. This behaviour is noted as PCD-4177 and improvements have been made starting with **v2025.8-112 and above versions.** ## Diagnostics {% hint style="info" %} The below 3 commands are applicable only to Self-Hosted PCD-V environments. {% endhint %} * Error logs from the `nova-api-osapi` pod exposed a `KeyError` involving a specific instance UUID, indicating corruption or missing data for that instance. Example log snippet: ```bash ERROR nova.api.openstack.wsgi [None [REQ_ID] [CORRUPTED_VM_UUID]- - default default] Unexpected exception in API method: KeyError: '[CORRUPTED_VM_UUID]' ERROR nova.api.openstack.wsgi KeyError: '[CORRUPTED_VM_UUID]' ``` * Checking the `nova` database revealed the corrupted instance still present with `vm_state=error` and `power_state=NOSTATE`. ```bash MySQL [nova]> select * from instances where uuid=""\G; vm_state: error power_state: 0 host: NULL availability_zone: [AZ] display_name: [CORRUPTED_VM_NAME] ``` * Querying the `request_specs` table in the `nova_api` database for the same instance UUID returned **no results**, confirming a missing request spec entry: ```bash MySQL [nova_api]> select * from request_specs where instance_uuid=''; Empty set (0.001 sec) ``` * Check VM status using below command : ```bash $ openstack server show --fit | OS-EXT-STS:vm_state | error | | OS-EXT-STS:power_state| NOSTATE | | fault | {'code': 413, 'message': 'Quota exceeded...'} | ``` * No event history for the VM, which confirms the VM entered the "Error" state during the creation process: ```bash $ openstack server event list (no events) ``` ## Resolution The corrupted VM deleted using below command: ```bash openstack server delete ``` This restored normal functionality for both the UI and the API, allowing full visibility of VMs across tenant boundaries. ## Validation * Verified UI no longer shows blank list when “**View VMs in All Tenants**” is enabled. * Cross-tenant VM visibility works as expected. * No further 500 errors observed in related logs. * API calls return valid responses without errors. # Migrated VMs are Showing File System Errors ## Problem Multiple virtual machines (VMs) failed to boot due to file system corruption on their boot volumes. Both Windows and Linux VMs exhibited boot errors, showing file system inconsistencies. ```bash BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)_Pci(0x5,0x0) BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)_Pci(0x5,0x0) ``` ## Environment * Private Cloud Director Virtualization - v2025.7 & v2025.8 * Self-Hosted Private Cloud Director Virtualization - v2025.7 & v2025.8 * Component: VMHA ## Workaround 1. Identify Impacted VMs: 1. Review VM boot console output for file system errors or stuck boot sequences. 2. Cross-check recent migrations or evacuations triggered by VMHA from the ostackhost logs (`/var/log/pf9/ostackhost.logs`) from the source host. ```bash INFO nova.compute.manager [req-UUID masakari services] [instance: [INSTANCE_ID]] Evacuating instance INFO nova.compute.claims [req-UUID masakari services] [instance: [INSTANCE_ID]] Claim successful on node INFO nova.compute.resource_tracker [req-UUID masakari services] [instance: [INSTANCE_ID]] Updating resource usage from migration ERROR nova.scheduler.client.report [req-UUID masakari services] [req-UUID] Failed to update inventory to [{'MEMORY_MB': {'total': 2063836, 'min_unit': 1, 'max_unit': 2063836, 'step_size': 1, 'allocation_ratio': 1.5, 'reserved': 512}, 'VCPU': {'total': 128, 'min_unit': 1, 'max_unit': 128, 'step_size': 1, 'allocation_ratio': 16.0, 'reserved': 0}, 'DISK_GB': {'total': 86809, 'min_unit': 1, 'max_unit': 86809, 'step_size': 1, 'allocation_ratio': 9999.0, 'reserved': 0}}] for resource provider with UUID . Got 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\\n\\n resource provider generation conflict ", "code": "placement.concurrent_update", "request_id": "req-UUID"}]} ERROR nova.scheduler.client.report [req-UUID masakari services] [req-UUID] Failed to update inventory to [{'MEMORY_MB': {'total': 2063836, 'min_unit': 1, 'max_unit': 2063836, 'step_size': 1, 'allocation_ratio': 1.5, 'reserved': 512}, 'VCPU': {'total': 128, 'min_unit': 1, 'max_unit': 128, 'step_size': 1, 'allocation_ratio': 16.0, 'reserved': 0}, 'DISK_GB': {'total': 86808, 'min_unit': 1, 'max_unit': 86808, 'step_size': 1, 'allocation_ratio': 9999.0, 'reserved': 0}}] for resource provider with UUID . Got 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\\n\\n resource provider generation conflict ", "code": "placement.concurrent_update", "request_id": "req-UUID"}]} ``` 2. Temporarily Disable VMHA: 1. Disable VM High Availability (VMHA) across the affected clusters to prevent automatic evacuations. 2. Confirm no ongoing evacuation events in logs (`/var/log/pf9/ha/ha-agent.log`) from the source host. 3. Recover the VMs: 1. For Linux VMs: Boot into recovery mode and run `fsck -y /dev/` 2. For Windows VMs: Mount the volume to a helper instance, restore critical registry files, and reattach to the original VM or using a file system [checker](https://support.microsoft.com/en-us/topic/use-the-system-file-checker-tool-to-repair-missing-or-corrupted-system-files-79aa86cb-ca52-166a-92a3-966e85d4094e) tool. 4. Validate Cinder and NFS Mounts: 1. Check all Cinder hosts for correct NFS mount configurations as per the cluster blueprints. ## Cause VMHA (Virtual Machine High Availability) evacuation events triggered by network connectivity issues resulted in multiple VM instances simultaneously accessing the same storage volumes, causing data corruption. ## Resolution The long-term fix is available on **PCD-October** [release](https://platform9.com/docs/private-cloud-director/private-cloud-director/october-2025-release) (tracked under PCD-4211, PCD-4212). # Unable to Create Virtualized Cluster ## Problem You may run into an error when trying to create a new virtualized cluster that informs you that cluster creation failed because one or more of the Private Cloud Director services are not healthy. The error message might at times say "Hypervisor services are not ready" ## Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher ## Cause The most common case of this is when you attempt to create a new virtualized cluster **immediately after making updates to your cluster blueprint**. Changes to cluster blueprint may require a service restart of compute and / or networking service. If the service is being restarted, the service status health API will report the service to be unhealthy, which will result in this error when creating a cluster. Note however that you may also run into this error because the one or more of Private Cloud Director services are currently experiencing an issue or an outage. ## Diagnostics Refer to the service health dashboard for details on which services may currently be unhealthy. ## Resolution Refer to the debugging documentation for the individual service to identify the problem if any. Once all the services are healthy, retry the operation. # Virsh Commands run Infinitely Without Output / VM Console Inaccessible ## Problem * VM console inaccessible for VMs running on specific host. * `virsh` commands on hypervisor run infinitely without an output. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: `libvirtd` running on hypervisors, SCSI Storage Backend. ## Cause * It has been observed in such cases that one or more stale multipath devices remain on the host for the VM/s that have been already deleted. * These stale multipath/s cause the `libvirtd` service to enter in an endless I/O loop while communicating with these failed multipath/s devices. * This issue is already reported with ID: **PCD-7407** to better handle such situation in PCD. ## Diagnostics * Check if the `virsh` commands are infinitely running without any output. * Check if the console of the VMs running on the specific/affected host is inaccessible with execution stuck infinitely. * Identify the failing QEMU `libvirt` processes using the command below: {% code title="Affected Host - Sample output " %} ```bash $ sudo ps -aux | grep libvirt | grep /usr/bin/qemu-system-x86_64 libvirt+ 239349 65.8 12.7 140438780 134239804 ? SLl [TIMESTAMP] 15079:01 /usr/bin/qemu-system-x86_64 -name guest=[VM_UUID],debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-72-[VM_UUID]/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.ms.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/[VM_UUID]_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-6.2,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram -accel kvm -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,erms=off,fsrm=off -global driver=cfi.pflash01,property=secure,value=on -m 131072 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":137438953472} -overcommit mem-lock=off -smp 16,sockets=16,dies=1,cores=1,threads=1 -uuid [VM_UUID] -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=5.6.0,serial=[VM_UUID],uuid=[VM_UUID],family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=39,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_ticket_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0 -device pcie-root-port,port=19,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x6 -device piix3-usb-uhci,id=usb,bus=pci.4,addr=0x1 -device virtio-scsi-pci,id=scsi0,bus=pci.3,addr=0x0 -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=[VM_UUID],drive=libvirt-2-format,id=ua-[VM_UUID],bootindex=1,write-cache=off,serial=[VM_UUID] -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=[VM_UUID],drive=libvirt-1-format,id=ua-[VM_UUID],write-cache=off,serial=[VM_UUID] -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=44 -device virtio-net-pci,host_mtu=1500,netdev=hostnet0,id=net0,mac=[VM_UUID],bus=pci.2,addr=0x0 -add-fd set=3,fd=40 -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc 0.0.0.0:0,audiodev=audio1 -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/[VM_UUID],display=off,bus=pci.5,addr=0x0 -device virtio-balloon-pci,id=balloon0,free-page-reporting=on,bus=pci.6,addr=0x0 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on libvirt+ 239345 61.8 15.8 140438783 134239808 ? SLl [TIMESTAMP] 15079:03 /usr/bin/qemu-system-x86_64 -name guest=[VM_UUID],debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-72-[VM_UUID]/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.ms.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/[VM_UUID]_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-6.2,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram -accel kvm -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,erms=off,fsrm=off -global driver=cfi.pflash01,property=secure,value=on -m 131072 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":137438953472} -overcommit mem-lock=off -smp 16,sockets=16,dies=1,cores=1,threads=1 -uuid [VM_UUID] -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=5.6.0,serial=[VM_UUID],uuid=[VM_UUID],family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=39,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_ticket_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0 -device pcie-root-port,port=19,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x6 -device piix3-usb-uhci,id=usb,bus=pci.4,addr=0x1 -device virtio-scsi-pci,id=scsi0,bus=pci.3,addr=0x0 -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=[VM_UUID],drive=libvirt-2-format,id=ua-[VM_UUID],bootindex=1,write-cache=off,serial=[VM_UUID] -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=[VM_UUID],drive=libvirt-1-format,id=ua-[VM_UUID],write-cache=off,serial=[VM_UUID] -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=44 -device virtio-net-pci,host_mtu=1500,netdev=hostnet0,id=net0,mac=[VM_UUID],bus=pci.2,addr=0x0 -add-fd set=3,fd=40 -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc 0.0.0.0:0,audiodev=audio1 -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/[VM_UUID],display=off,bus=pci.5,addr=0x0 -device virtio-balloon-pci,id=balloon0,free-page-reporting=on,bus=pci.6,addr=0x0 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on ``` {% endcode %} * In this list, identify `guest=[VM_UUID]`. This UUID corresponds to the VM's UUID. * These processes are the actual VMs running on the host. * In this list, there is a high possibility, as observed in the such issues, that a process would exist for a VM that is already deleted. * That process will be a Zombie process `Z` and in `defunct` state. {% code title="Sample Output of Defunct Process" %} ```bash $ sudo ps -aux | grep libvirt | grep 1912566 libvirt+ 1912566 330 0.0 0 0 ? Zl [TIMESTAMP] 756698:25 [qemu-system-x86 LONG_CONTENT] ``` {% endcode %} * Identify `guest=[VM_UUID]` from this process. ## Resolution * Move the `.xml` file of this `VM_UUID` from `/etc/libvirt/qemu/.xml` to any other location like `/tmp` or user's home dir. * After moving the file, this defunct process should get killed automatically and that can be verified using the below command: {% code title="Returns Empty Output" %} ```bash $ sudo ps -aux | grep ``` {% endcode %} * Read the `.xml` of the VM UUID that was copied earlier and search for `` section. * In there, identify the line `` {% hint style="info" %} **Example:** The **disk mapper device name** will appear as `dm-11` \ `` {% endhint %} * Identify the WWN of that multipath using the command below: {% code title="Sample Output" %} ```bash $ multipath -ll | grep [WWN] [DISK_MAPPER_DEVICE_NAME] NETAPP, LUN C-Mode ``` {% endcode %} * Get full details of this multipath using: 
multipath -ll <WWN_UUID>
size=100G features='2 pg_init_retries 50' hwhandler='1 alua' wp=rw
-+- policy='service-time 0' prio=0 status=enabled
I '- 6:0:0:3 sda 8:112 failed faulty running
'-+- policy='service-time 0' prio=0 status=enabled
'- 7:0:0:3 sdb 8:128 failed faulty running
* Identify that there are `failed faulty` block devices for this multipath. `sda` and `sdb` are examples and the actual output will vary. * Delete the faulty block devices using the below command for each one: {% code title="" %} ```bash echo 1 | sudo tee /sys/block//device/delete ``` {% endcode %} * `libvirtd` service auto-recovers from here. ## Validation * Check if `virsh` commands are executing on the host. * Check if the VM Console is accessible. * On the affected host, check if the faulty multipath is flushed automatically and the below command should give empty response {% code title="Affected Host" %} ```bash $ multipath -ll ``` {% endcode %} * `systemctl status libvirtd` should display status as `Active` `Running`. # How To Manually Apply Roles On a Host Using API Calls ## Problem In normal scenarios, the roles are applied through the PCD UI. However, the following steps can be used to apply the roles manually using API calls. ## Environment * Private Cloud Director Virtualization - v2026.1 and Higher * Self-Hosted Private Cloud Director Virtualization - v2026.1 and Higher * Component - Resource Manager ## Procedure {% stepper %} {% step %} **Retrieve Token** {% code title="Export RC file and Perform" overflow="wrap" expandable="true" %} ```bash $ openstack token issue -f value -c id ``` {% endcode %} **Set necessary variables.** ```bash $ export OS_TOKEN= $ export DU= $ export HOST_ID= ``` {% endstep %} {% step %} **List the applied roles on the host. \[If applicable]** {% code title="Sample Output" overflow="wrap" %} ```bash $ curl -sS -H "x-auth-token: $OS_TOKEN" '$DU/resmgr/v1/hosts/$HOST_ID' | jq '{hostconfig_id, roles, role_status}' { "hostconfig_id": "[HOSTCONFIG_UUID]", "roles": [ "pf9-cindervolume-base", "pf9-cindervolume-config", "pf9-glance-role", "pf9-ha-slave", "pf9-neutron-base", "pf9-ostackhost-neutron", "pf9-support" ], "role_status": "ok" } ``` {% endcode %} {% endstep %} {% step %} **APl call to apply the required role** * **Hypervisor Role** 
$ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-ostackhost-neutron

  $ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-neutron-base

  $ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-neutron-ovn-controller

  $ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-neutron-ovn-metadata-agent

  $ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-ip-discovery
* **Persistent Storage** 
$ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-cindervolume-base

  $ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-cindervolume-config
* **Image Library** 
$ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-glance-role
* **Advanced Remote Support** 
$ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-support
* **High Availabilty** 
$ curl -X PUT -H "x-auth-token: $OS_TOKEN" -H 'content-type: application/json' '$DU/resmgr/v1/hosts/$HOST_ID/roles/pf9-ha-slave
{% endstep %} {% step %} **Observe installation progress** * Observe the hostagent logs 
$ tail -f /var/log/pf9/hostagent.log
* After applying a package wait for converge to complete before applying the next. {% endstep %} {% endstepper %} ## Validation * Check the packages that are installed on the host. On Ubuntu: 
$ dpkg -l | grep pf9
On Platform9 OS: 
$ rpm -qa | grep pf9
* Check the roles applied using: 
$ curl -sS -H "x-auth-token: $OS_TOKEN" '$DU/resmgr/v1/hosts/$HOST_ID' | jq '{hostconfig_id, roles, role_status}'
# Unable to Reconfigure SSO in PCD UI Using the Same Entity ID ## Problem After disabling SSO in the Platform9 PCD UI, attempting to reconfigure SSO with the same entity ID is not allowed. The UI displays an error "*Identity provider already exists*" or does not permit using the previous entity ID for SSO setup. ## Environment * Private Cloud Director Virtualization - till 2025.7-47 * Private Cloud Director Kubernetes – till 2025.7-47 * Self-Hosted Private Cloud Director Virtualization - till 2025.7-47 * Self-Hosted Private Cloud Director Kubernetes - till 2025.7-47 * Component - SSO ## Cause Disabling SSO from the PCD UI removes the SSO settings from the backend service (Consul) but does not delete the Identity Provider (IDP) object from OpenStack. This means the entity ID stays in OpenStack, stopping reuse during reconfiguration. This was a known issue which is tracked under **PCD-3227** and is now fixed starting with the August release **(v2025.8-92) and higher.** ## Diagnostics To allow SSO to be reconfigured with the same entity ID **Step 1:** List existing identity provider: 
$ openstack identity provider list

+---------------------+---------------------------------------------------------------+
| Field               | Value                                                         |
+---------------------+---------------------------------------------------------------+
| authorization_ttl   | None                                                          |
| description         | None                                                          |
| domain_id           | default                                                       |
| enabled             | True                                                          |
| id                  | [IDP1_VALUE]                                                  |
| remote_ids          | https://[URL]                                                 |
+---------------------+---------------------------------------------------------------+
**Step 2:** Verify Existing IDP Configuration: Use the following commands to check the current IDP state (replace **\** with the actual identity provider name): ```bash $ openstack mapping list $ openstack identity provider show --fit-width $ openstack federation protocol list --identity-provider $ openstack federation protocol show saml2 --identity-provider $ openstack group list $ openstack group show ``` Run these commands to review and confirm the existing identity provider (IDP) configuration including its details, associated federation protocols, mappings, and any related OpenStack groups. The above step is to ensure that you are deleting the correct identity provider and do not accidentally remove configurations that may be used by other SSO integrations, mappings, or user groups. Proceed with deletion only if you are certain that the information shown matches the identity provider you intend to remove. ## Workaround Delete the stale identity provider using the below Openstack command: ```bash $ openstack identity provider delete ``` ## Validation After deleting the identity provider, refresh the PCD UI and reconfigure SSO with the desired entity ID. # SSO Disabled Post Management Plane Upgrade ## Problem * SSO appears disabled in the UI after upgrading the PCD management plane, but the backend SSO configuration is functioning correctly. ## Environment * Private Cloud Director - till v2026.6-159 * Self-Hosted Private Cloud Director - till v2026.6-159 * Component: SSO ## Cause * After upgrading from the FEB release to the APR release, the **"Enable SSO" flag appears disabled** in the Enterprise SSO UI section, even though SSO is actually configured and functional (e.g., SAML group details are visible). * The issue occurs because the **signing attribute** is not included when creating the SSO config, and fetching it later from Consul fails. As a result, the UI treats the SSO setup as incomplete and displays the **"Enable SSO"** flag as **disabled**. The bug was tracked in PCD-2029 and is now fixed with **v2025.7-47 and above** **releases**. ## Diagnostics * After upgrading the PCD management plane, navigate to Settings > Enterprise SSO in the user interface. The SSO feature will be disabled following the upgrade. ![](https://uploads.developerhub.io/prod/K9yg/ihk3au0jbp3s50hxswh5zme9ire8zhbjm7nijamwqk0zureg8gnzsl3g9d7649n6.png) ## Workaround {% hint style="info" %} For PCD (SaaS) users, the platform9 support team will apply the steps below. Please open a [Support Ticket](https://support.platform9.com/hc/en-us). For Self-Hosted PCD users, perform the steps below from the management plane cluster. {% endhint %} 1. Install consul CLI, refer this [documentation](https://developer.hashicorp.com/consul/install). 2. Get `hagrid` deployment configuration. ```ruby $ kubectl get deploy/hagrid -o yaml -n > hagrid-deployment.yaml ``` 3. Search inside the `hagrid-deployment.yaml` file for below variables. 1. `CONSUL_HTTP_ADDR` 2. `CONSUL_HTTP_TOKEN` 3. `CUSTMOR_ID` 4. Export values into current shell session. ```ruby export CONSUL_HTTP_ADDR="" export CONSUL_HTTP_TOKEN="" export CUSTMOR_ID="" ``` 5. Once the environment is configured, run the following command to store the signing flag. ```ruby $ consul kv put customers/$CUSTOMER_ID/keystone/sso/saml2/IDP1/signing false ``` ## Validation * Navigate to Settings > Enterprise SSO in the user interface. The SSO feature will be enabled. ![](https://uploads.developerhub.io/prod/K9yg/9v3xymhcimgyccctczh0p1daylcsglfl3i1o010renfikid0hidbusdu3fmtuzcz.png) # Tenant Rename Creates Duplicate Tenants and Deleted Tenants Reappear ## Problem When renaming a tenant in Private Cloud Director, a new tenant is created with the new name while the original tenant remains intact, resulting in two duplicate tenant entries. Additionally, when the duplicate or stale tenants are manually deleted, they reappear approximately one hour later. ## Environment * Private Cloud Director Virtualization — versions prior to v2026.1.-427 * Self-Hosted Private Cloud Director Virtualization — versions prior to v2026.1.-427 * Component: SSO, Identity Service * Prerequisite: Enterprise SSO (Microsoft Entra ID or other SAML provider) configured with group role mappings that include project (tenant) names ## Cause SAML group role mappings in Private Cloud Director previously supported specifying project (tenant) names directly within the mapping configuration. When an SSO user belonging to a mapped group logged in, the Identity Service processed the mapping and recreated any listed project that was not currently present — including projects that had been renamed or deleted. This caused two symptoms: * **Tenant rename appears as a new tenant creation.** The Identity Service detected the original project name as missing after a rename and recreated it alongside the renamed tenant, leaving both entries present. * **Deleted tenants reappear after approximately one hour.** The Identity Service recreated deleted tenants on the next SSO user login. The project-level access entries in the mapping also caused role assignments to be applied at both the SSO group level and the individual user level simultaneously. This made the user-level assignments persist as static entries, causing access to survive SSO group membership changes. This behavior is tracked under **PCD-5778** and has been resolved in the [January 2026 Patch 2 Release](https://docs.platform9.com/release-notes/january-2026-release/january-2026-patch-2-release). After upgrading and re-saving the SSO configuration, project-level entries are removed from group role mappings automatically. ## Diagnostics Use the following steps to confirm the environment is affected by this specific issue before applying the workaround. {% stepper %} {% step %} #### Confirm the SSO group mapping contains project names Navigate to **Settings > Authentication** and open the SSO configuration. Inspect the group role mapping entries. If any mapping entry contains a `projects` array listing specific tenant names, the environment matches this issue. {% code title="Example SSO Group Mapping with Project Names (affected configuration)" %} ```json { "local": [ { "group": { "id": "" }, "projects": [ { "name": "", "roles": [{ "name": "admin" }] } ] } ], "remote": [...] } ``` {% endcode %} {% endstep %} {% step %} #### Confirm duplicate tenants are present in the UI Navigate to **Settings > Tenants** in the PCD UI. Check whether both the original tenant name and the renamed tenant name appear as separate entries. Confirmation of the issue shows two distinct tenant entries with the same intended purpose but different UUIDs. {% endstep %} {% endstepper %} If both checks above match, proceed to [Resolution](#resolution) or [Workaround](#workaround). ## Resolution Upgrade to v2026.1.-427 ([January 2026 Patch 2 Release](https://docs.platform9.com/release-notes/january-2026-release/january-2026-patch-2-release)) or later. After upgrading, navigate to **Settings > Authentication**, open the SSO configuration, and re-save without making changes. Re-saving triggers the removal of project-level entries from the group role mappings automatically. Follow the [Workaround](#workaround) steps to clean up any duplicate tenants that were created before the upgrade. ## Workaround Apply the following steps for environments not yet upgraded to v2026.1.2, or to clean up duplicate tenants that already exist. {% hint style="warning" %} Only SSO-derived role assignments are recreated automatically on the next user login. Local or manually-created role assignments are **not** automatically recreated after deletion. Document any manually-managed role assignments before proceeding. {% endhint %} {% stepper %} {% step %} #### Remove affected tenant names from the SSO group mapping Navigate to **Settings > Authentication** in the PCD UI and open the SSO group role mapping configuration. Remove all `projects` entries from each group mapping. Do not remove the group-to-role mapping entries themselves — only remove the `projects` array and its contents from the mapping structure. Save the SSO configuration after making the changes. {% endstep %} {% step %} #### Delete the duplicate or stale tenants Navigate to **Settings > Tenants** and delete the duplicate or stale tenants. Deleting a tenant automatically removes all role assignments associated with that tenant. Pre-removing role assignments individually before deletion is not required. {% hint style="info" %} If both the original tenant and the duplicate both need to be removed, delete both. Role assignments for both tenants are removed automatically as part of deletion. If only the stale duplicate needs to be removed, delete the unwanted tenant and retain the one with the correct name and resources. {% endhint %} {% endstep %} {% step %} #### Verify deleted tenants do not reappear Wait approximately 60–90 minutes — enough time for at least one SSO login cycle to complete — then navigate to **Settings > Tenants** and confirm the deleted tenants are absent. {% hint style="success" %} The deleted tenants should no longer appear in the tenant list. If tenants reappear, confirm the SSO configuration was saved successfully in Step 1 and that no `projects` entries remain in any group mapping. {% endhint %} {% endstep %} {% step %} #### Confirm SSO user access on retained tenants Have SSO users log out and log back in to the PCD environment. SSO-derived role assignments on retained tenants are recreated automatically at login based on the group-level mapping. {% hint style="info" %} Only role assignments derived from SSO group mappings are recreated automatically on login. Local or manually-created role assignments that were removed must be reassigned manually. {% endhint %} {% endstep %} {% endstepper %} ## Additional Information The root cause and fix are tracked under **PCD-5778**, included in the [January 2026 Patch 2 Release](https://docs.platform9.com/release-notes/january-2026-release/january-2026-patch-2-release) (released April 3, 2026). For guidance on configuring Microsoft Entra ID SSO with Private Cloud Director, see [Setup Microsoft Entra ID for PCD](https://docs.platform9.com/private-cloud-director/identity-and-multi-tenancy/enterprise-sso/setup-microsoft-entra-id-for-pcd) in the Platform9 documentation. # MetalLB VIP Connectivity Issues When Accessing VIP Externally ## Problem * In environments where port security features are enforced, users may observe that external VIPs assigned by MetalLB (in Layer 2 mode) are not accessible from other virtual machines or hosts (hypervisors). ```ruby curl: (7) Failed to connect: No route to host ``` * This typically occurs when attempting to connect to a service exposed via a MetalLB-assigned external IP. ## Environment * Private Cloud Director - till 2025.7-47 * Self-Hosted Private Cloud Director Virtualization – till 2025.7-47 * Component - Networking ## Cause * When using MetalLB in Layer 2 mode, it responds to ARP requests using the MAC address of one of the Kubernetes nodes. However, strict port security settings often block traffic from IP/MAC combinations that are not explicitly permitted, resulting in dropped ARP replies or traffic not being routed correctly. * This was a known limitation tracked KAAP-677 is now fixed in **2025.8-92 and above versions.** ## Resolution To allow the specific IP and MAC address combinations used by MetalLB for VIPs. The fix involves adding Allowed Address Pairs for each MetalLB VIP and the corresponding VM node MAC address. Steps to Add IP/MAC Allowed Address Pairs: 1. Identify the MetalLB VIP that is not reachable. 2. Determine the MAC address of the VM Node 3. Now, to add the IP/MAC Pair navigate to Network and Security page and select Physical Networks 4. Select Ports section and click on edit the Port (Kubernetes worker node VM) 5. Now go to the Allowed Address Pairs section. Add a new entry with: IP Address: The MetalLB VIP (e.g., 192.168.1.240) MAC Address: MAC address of the VM Node 6. Click Update Port to save changes. 7. Repeat this for all worker nodes in the cluster. ## Additional Information * The fix for the bug is currently planned for future releases. * For further questions/concerns regarding the bug, reach out to the [Platform9 Support Team](https://support.platform9.com/hc/en-us). # VM Reverse DNS Lookup Returns Wrong Domain Due to OVN DNS Interception ### Problem Virtual machines in Private Cloud Director Virtualization environments receive the wrong DNS search domain via DHCP. Short-name queries resolve against the global blueprint domain instead of the expected per-network or per-site domain, causing incorrect FQDNs in lookups and failures in site-specific name resolution. The issue affects both Linux and Windows guest operating systems. Virtual machines running on non-PCD hypervisors (for example, VMware) on the same network resolve correctly using the expected domain. Running `nslookup` or `dig` from inside an affected virtual machine returns FQDNs constructed from the global blueprint domain: {% code title="Affected VM" %} ``` $ nslookup Server: [DNS_SERVER_IP] Address: [DNS_SERVER_IP]#53 Name: [SHORT_NAME].[GLOBAL_DOMAIN] Address: [VM_IP_ADDRESS] ``` {% endcode %} The search domain in the guest `/etc/resolv.conf` or Windows DNS settings reflects the global blueprint domain rather than the expected per-site or per-network domain. ### Environment * Private Cloud Director Virtualization — All versions * Self-Hosted Private Cloud Director Virtualization — All versions * Component: Networking Service, OVN ML2 ### Cause The Networking Service OVN ML2 driver populates the OVN internal DNS table using the global `dns_domain` value from `neutron.conf`, regardless of any per-port DNS configuration. The `dns_domain` attribute on a Networking Service network object is used by the Designate (DNS as a Service) integration only — it does not configure DHCP Option 15 and has no effect on what virtual machines receive in `/etc/resolv.conf`. OVN intercepts all DNS queries originating from virtual machines at the logical switch level and answers from this internal table — the external authoritative DNS server is never consulted for these queries. Virtual machines therefore receive DHCP Option 15 (DNS domain suffix) derived from the global domain only. Per-port DHCP domain configuration via `--extra-dhcp-option name=domain-name` is not set by default. The only way to deliver the correct per-site domain to guests via DHCP is to configure this option explicitly on each port — or, after PCD-6289 is available, via the new per-network `pf9_dhcp_domain` attribute. ### Diagnostics {% stepper %} {% step %} **Step 1 — Confirm the port DNS assignment shows the wrong domain** Retrieve the port attached to the affected virtual machine and inspect the `dns_assignment` field. The `fqdn` value reflects the global domain rather than the expected per-network domain. {% code title="OpenStack CLI" %} ```bash $ openstack port list --server -c id -c dns_assignment ``` {% endcode %} {% code title="Sample Output — Port dns\_assignment with Incorrect Domain" %} ``` +--------------------------------------+------------------------------------------------------------------------------------+ | id | dns_assignment | +--------------------------------------+------------------------------------------------------------------------------------+ | [PORT_UUID] | [{"hostname": "[VM_HOSTNAME]", "ip_address": "[VM_IP]", | | | "fqdn": "[VM_HOSTNAME].[GLOBAL_DOMAIN]."}] | +--------------------------------------+------------------------------------------------------------------------------------+ ``` {% endcode %} If the `fqdn` shows the global blueprint domain instead of the expected per-network domain, proceed to Step 2. {% endstep %} {% step %} **Step 2 — Confirm OVN is intercepting DNS queries locally using dig +trace** Run `dig +trace` from inside the affected virtual machine. An OVN-intercepted response returns in `0 ms` from what appears to be the authoritative or root DNS server — an impossibly fast round-trip that confirms OVN is answering from its internal table rather than forwarding the query to the external DNS server. The returned FQDN also reflects the global domain, not the expected per-network domain. {% code title="Inside VM" %} ```bash $ dig +trace ``` {% endcode %} {% code title="Sample Output — OVN Intercepting DNS Locally" %} ``` ; <<>> DiG [VERSION] <<>> +trace [SHORT_NAME] . 518400 IN NS [ROOT_SERVER]. ;; Received [BYTES] bytes from [DNS_SERVER_IP]#53([DNS_SERVER_IP]) in 0 ms [EXPECTED_DOMAIN]. 3600 IN NS [AUTHORITATIVE_NS]. ;; Received [BYTES] bytes from [DNS_SERVER_IP]#53([DNS_SERVER_IP]) in 1 ms [SHORT_NAME].[GLOBAL_DOMAIN]. 300 IN A [VM_IP] ;; Received [BYTES] bytes from [DNS_SERVER_IP]#53([DNS_SERVER_IP]) in 0 ms ``` {% endcode %} A response time of `0 ms` at each delegation hop, combined with a returned FQDN using `[GLOBAL_DOMAIN]` rather than `[EXPECTED_DOMAIN]`, confirms OVN is intercepting and answering the query locally. A correctly resolving virtual machine would show non-zero round-trip times with a full delegation chain to the authoritative DNS server. {% endstep %} {% step %} **Step 3 — Confirm no per-port DHCP domain-name option is configured** Verify that the affected virtual machine port does not have a `domain-name` DHCP option set via `extra_dhcp_opts`. An empty result confirms no per-port override has been applied and the guest is receiving the global domain via DHCP by default. {% code title="OpenStack CLI" %} ```bash $ openstack port show -c extra_dhcp_opts ``` {% endcode %} {% code title="Sample Output — No per-port DHCP domain configured (issue confirmed)" %} ``` +------------------+-------+ | Field | Value | +------------------+-------+ | extra_dhcp_opts | [] | +------------------+-------+ ``` {% endcode %} An empty `extra_dhcp_opts` field confirms that no `domain-name` DHCP option has been set on this port. Proceed to Step 4. {% hint style="info" %} The `dns_domain` attribute visible on `openstack network show` is used by the Designate (DNS as a Service) integration only. Setting it does not configure DHCP Option 15 and does not affect the domain name delivered to virtual machines via DHCP. {% endhint %} {% endstep %} {% step %} **Step 4 — Verify OVN DNS records use the global domain** Inspect the OVN NB database DNS table to confirm that records are populated with the global domain rather than the per-network domain. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl exec -n -- ovn-nbctl list DNS ``` {% endcode %} {% code title="Sample Output — OVN DNS Records Using Global Domain" %} ``` _uuid : [DNS_RECORD_UUID] external_ids : {neutron:network_id="[NETWORK_UUID]"} records : {"[VM_HOSTNAME].[GLOBAL_DOMAIN]"="[VM_IP]"} ``` {% endcode %} DNS entries in the OVN NB table using the global domain instead of the expected per-network domain confirm the root cause. {% endstep %} {% step %} **Step 5 — Confirm OVN DNS interception at the OVS flow level** On the compute host running the affected virtual machine, inspect the Open vSwitch flow table to confirm that OVN has installed a flow rule intercepting all UDP DNS traffic (port 53) before it reaches the physical network. {% code title="Compute Host" %} ```bash $ ovs-ofctl dump-flows br-int table=32 | grep "tp_dst=53" ``` {% endcode %} {% code title="Sample Output — OVN DNS Intercept Flow Rule Present" %} ``` cookie=0x0, duration=[DURATION]s, table=32, n_packets=[PACKET_COUNT], n_bytes=[BYTE_COUNT], priority=100,udp,tp_dst=53 actions=controller(userdata=[USERDATA],pause) ``` {% endcode %} The presence of a flow rule at `table=32` with `tp_dst=53` and `actions=controller(...pause)` confirms that OVN is intercepting all UDP DNS queries at the logical switch. A non-zero `n_packets` count indicates that DNS traffic has already been intercepted. Virtual machines running on non-PCD hypervisors (for example, VMware) on the same network do not have this flow rule and resolve DNS correctly through the external authoritative server. If all checks above match, proceed to Workaround. {% endstep %} {% endstepper %} ### Resolution PCD-6289 introduces a new Networking Service network attribute `pf9_dhcp_domain` that allows DHCP Option 15 (DNS search domain) to be configured per-network. This replaces the need for per-port `--extra-dhcp-option name=domain-name` configuration. However, PCD-6289 does not stop OVN from intercepting DNS queries — OVN DNS interception continues to operate after this change. A complete resolution requires both setting `pf9_dhcp_domain` per-network (once PCD-6289 is available in Private Cloud Director Virtualization 2026.AUG) and removing the `dns_domain_ports` extension driver to disable OVN DNS interception. In the meantime, use the Workaround below. ### Workaround The workaround requires two actions applied together. First, set the correct DHCP domain name on each affected virtual machine port (Step 2). Second, remove the `dns_domain_ports` OVN ML2 extension driver from the Networking Service configuration (Steps 3–6). Setting the per-port domain name alone does not stop OVN from intercepting DNS queries — reverse lookups continue to return the global domain until the extension driver is removed. Both steps are required for full resolution. {% hint style="warning" %} Removing the `dns_domain_ports` extension driver disables OVN-internal DNS resolution entirely. After applying this workaround, virtual machines cannot resolve each other by hostname using OVN's internal table — all name resolution relies on the external DNS servers specified in the subnet `dns_nameservers` field. Confirm that external DNS entries exist for all virtual machines that require hostname-based access before proceeding. {% endhint %} {% stepper %} {% step %} **Step 1 — Verify all affected subnets have external DNS nameservers configured** Before proceeding, confirm that every subnet in the affected region has `dns_nameservers` pointing to an authoritative external DNS server. Subnets with no `dns_nameservers` configured will have no DNS resolution for hosted virtual machines after the workaround is applied. {% code title="OpenStack CLI" %} ```bash $ openstack subnet list --network -c id -c name -c dns_nameservers ``` {% endcode %} {% code title="Sample Output — Subnet DNS Nameservers" %} ``` +--------------------------------------+---------------+------------------+ | id | name | dns_nameservers | +--------------------------------------+---------------+------------------+ | [SUBNET_UUID] | [SUBNET_NAME] | [DNS_SERVER_IP] | +--------------------------------------+---------------+------------------+ ``` {% endcode %} The `dns_nameservers` field must be non-empty for all subnets where virtual machines must resolve DNS. If any subnet has an empty `dns_nameservers` field, add the authoritative DNS server IP before proceeding: {% code title="OpenStack CLI" %} ```bash $ openstack subnet set --dns-nameserver ``` {% endcode %} {% endstep %} {% step %} **Step 2 — Set the correct DHCP domain-name on affected virtual machine ports** For each affected virtual machine port, set the `domain-name` DHCP option to the expected per-site domain using the `--extra-dhcp-option` flag. This configures DHCP Option 15 on the port and updates the guest OS `/etc/resolv.conf` search domain on the next DHCP lease renewal or virtual machine reboot. {% code title="OpenStack CLI" %} ```bash $ openstack port set --extra-dhcp-option name=domain-name,value= ``` {% endcode %} Confirm the option is set: {% code title="OpenStack CLI" %} ```bash $ openstack port show -c extra_dhcp_opts ``` {% endcode %} {% code title="Sample Output — DHCP domain-name option confirmed" %} ``` +------------------+-----------------------------------------------------------------------------------+ | Field | Value | +------------------+-----------------------------------------------------------------------------------+ | extra_dhcp_opts | [{"opt_name": "domain-name", "opt_value": "", "ip_version": 4}] | +------------------+-----------------------------------------------------------------------------------+ ``` {% endcode %} After applying this option, renew the DHCP lease inside the virtual machine or reboot it to pick up the updated domain. Forward short-name lookups from inside the guest will now resolve correctly. Reverse lookups will still return the global domain until the `dns_domain_ports` extension driver is removed in the following steps. {% hint style="info" %} After PCD-6289 is available, the `pf9_dhcp_domain` attribute can be set per-network instead of requiring per-port configuration: ```bash $ openstack network set --pf9-dhcp-domain ``` {% endhint %} {% endstep %} {% step %} **Step 3 — Remove dns\_domain\_ports from the Networking Service ML2 extension drivers** Edit the Networking Service ML2 configuration to remove `dns_domain_ports` from the `extension_drivers` list in the `[ml2]` section of `ml2_conf.ini`. On Self-Hosted deployments, this configuration is stored in a Kubernetes ConfigMap or Secret in the region namespace. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com) to have this completed on your behalf. {% endhint %} Locate the ConfigMap or Secret containing `ml2_conf.ini`: {% code title="Control Plane Node" %} ```bash $ kubectl get configmap -n | grep -i ml2 $ kubectl get secret -n | grep -i ml2 ``` {% endcode %} Edit the configuration and remove `dns_domain_ports` from the `extension_drivers` line: {% code title="ml2\_conf.ini — Before" %} ```ini [ml2] extension_drivers = port_security,qos,dns_domain_ports ``` {% endcode %} {% code title="ml2\_conf.ini — After" %} ```ini [ml2] extension_drivers = port_security,qos ``` {% endcode %} Apply the updated ConfigMap or Secret to the cluster after saving the change. {% endstep %} {% step %} **Step 4 — Restart the Networking Service** Restart the Networking Service deployment to load the updated ML2 configuration. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl rollout restart deployment/neutron-server -n $ kubectl rollout status deployment/neutron-server -n ``` {% endcode %} {% code title="Sample Output — Networking Service Restart Successful" %} ``` deployment.apps/neutron-server successfully rolled out ``` {% endcode %} Wait for the rollout to complete fully before proceeding to Step 5. {% endstep %} {% step %} **Step 5 — Clear existing OVN DNS records from all logical switches** After restarting the Networking Service, clear existing OVN DNS records from every logical switch in the region. Without this step, previously created DNS records remain in the OVN NB database and virtual machines continue to receive the wrong domain until the next port operation triggers a rebuild. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com) to have this completed on your behalf. {% endhint %} List all OVN NB DNS records to identify the network UUIDs requiring cleanup: {% code title="Control Plane Node" %} ```bash $ kubectl exec -n -- ovn-nbctl list DNS ``` {% endcode %} {% code title="Sample Output — OVN DNS Records Before Cleanup" %} ``` _uuid : [DNS_RECORD_UUID_1] external_ids : {neutron:network_id="[NETWORK_UUID_1]"} records : {"[VM_HOSTNAME_1].[GLOBAL_DOMAIN]"="[VM_IP_1]"} _uuid : [DNS_RECORD_UUID_2] external_ids : {neutron:network_id="[NETWORK_UUID_2]"} records : {"[VM_HOSTNAME_2].[GLOBAL_DOMAIN]"="[VM_IP_2]"} ``` {% endcode %} Note each `[NETWORK_UUID]` value from the `external_ids` field. Clear the DNS records for each affected logical switch. Run the following command once per network UUID: {% code title="Control Plane Node" %} ```bash $ kubectl exec -n -- \ ovn-nbctl clear logical_switch neutron- dns_records ``` {% endcode %} Confirm all DNS records have been cleared: {% code title="Control Plane Node" %} ```bash $ kubectl exec -n -- ovn-nbctl list DNS ``` {% endcode %} {% code title="Sample Output — OVN DNS Records Cleared" %} ``` (no output) ``` {% endcode %} {% endstep %} {% step %} **Step 6 — Verify DNS resolution from inside an affected virtual machine** Log in to a previously affected virtual machine and confirm DNS resolution now returns the correct domain. The query must reach the external authoritative DNS server and return the expected per-site FQDN. {% code title="Inside VM" %} ```bash $ nslookup ``` {% endcode %} {% code title="Sample Output — DNS Resolution Using Correct Domain" %} ``` Server: [DNS_SERVER_IP] Address: [DNS_SERVER_IP]#53 Name: [SHORT_NAME].[EXPECTED_DOMAIN] Address: [VM_IP_ADDRESS] ``` {% endcode %} The returned FQDN must use the expected per-site or per-network domain, not the global blueprint domain. DNS resolution is confirmed as working correctly. {% endstep %} {% endstepper %} ### Additional Information This issue is tracked in PCD-6289. PCD-6289 introduces the `pf9_dhcp_domain` network attribute for per-network DHCP domain configuration. It does not stop OVN from intercepting DNS queries — removing the `dns_domain_ports` extension driver is still required to disable OVN DNS interception. * The `dns_domain_ports` OVN ML2 extension driver enables per-port DNS hostname assignment and populates OVN's internal DNS table. Removing this driver stops OVN from intercepting DNS queries but also disables OVN-internal hostname resolution between virtual machines. After applying this workaround, all hostname-based communication between virtual machines must rely on external DNS entries. * OVN DNS records are automatically regenerated on any port operation (create, update, or delete). Running `ovn-nbctl clear logical_switch neutron- dns_records` alone provides only temporary relief — the permanent solution is to remove `dns_domain_ports` from the Networking Service `extension_drivers` and restart the Networking Service before clearing OVN records. * Setting `--extra-dhcp-option name=domain-name,value=` on individual **ports** (not subnets) configures DHCP Option 15 and updates the guest OS `/etc/resolv.conf` search domain on the next DHCP lease renewal. This corrects forward short-name lookups within the guest but does not stop OVN from intercepting DNS queries — reverse lookups continue to return the global FQDN. Both the per-port `domain-name` option and removing the `dns_domain_ports` extension driver are required for complete resolution. * The `dns_domain` attribute on a Networking Service network object (visible via `openstack network show -c dns_domain`) is used exclusively by the Designate (DNS as a Service) integration. It does not configure DHCP Option 15 and has no effect on the search domain delivered to virtual machines via DHCP or on OVN internal DNS table entries. Setting `openstack network set --dns-domain` will not resolve this issue. * If DNS resolution remains incorrect after applying the workaround, contact [Platform9 Support](https://support.platform9.com) for assistance. # How to Change the Default DNS IPs Added to the VMs? ## Problem * Default DNS IPs `8.8.8.8` and `1.1.1.1` are getting added to the VMs when there is no DNS configured on a subnet. * Default IPs are configured within the `neutron-server` `/etc/neutron/plugins/ml2/ml2_conf.ini` file. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher. * Self-Hosted Private Cloud Director Virtualization – v2025.4 and Higher. * Component - Networking ## Procedure {% hint style="info" %} For SaaS environments, the[ ](https://support.platform9.com/hc/en-us/requests/new?ticket_form_id=360000924873)[Platform9 support](https://support.platform9.com/hc/en-us)[ ](https://support.platform9.com/hc/en-us/requests/new?ticket_form_id=360000924873)team should be contacted to implement the changes. For Self-Hosted environments, the changes should be carried out directly from the Management Plane cluster. {% endhint %} * A DNS entry is populating from the secret `neutron-etc` as shown below, which is referred in the `neutron-server` deployment. The config file is in the form of base64 encoded within the secret. * The secret can be found in the corresponding region namespace ```bash $ kubectl get secret -n ``` 
$ kubectl get secret -n <REGION_NAMESPACE> neutron-etc -oyaml |grep ml2_conf.ini | awk '{print $2}' | base64 -d
[agent]
extensions =
[ml2]
extension_drivers = dns,port_security
mechanism_drivers = openvswitch,ovn,l2population
tenant_network_types = vxlan
type_drivers = flat,vlan,local,geneve,vxlan
[ml2_type_flat]
flat_networks = *
[ml2_type_geneve]
max_header_size = 38
vni_ranges = 4:4000
[ml2_type_vlan]
network_vlan_ranges = ext_net,physnet
[ml2_type_vxlan]
vni_ranges = 4:4000
vxlan_group = 239.1.1.1
[ovn]
dns_servers = 8.8.8.8,1.1.1.1   <<------------   DNS IP
enable_distributed_floating_ip = true
neutron_sync_mode = repair
ovn_l3_scheduler = leastloaded
To modify the default IPs, follow the below steps * Redirect the config to a file. ```bash $ kubectl get secret -n neutron-etc -oyaml |grep ml2_conf.ini | awk '{print $2}' | base64 -d > secret_neutron-etc_ml2_conf.ini ``` * Modify the file by replacing the IPs against `dns_servers` with your preferred IPs and Encode the file using base64. ```bash $ vi secret_neutron-etc_ml2_conf.ini ## Update prefered DNS IPs in dns_servers field $ base64 secret_neutron-etc_ml2_conf.ini -w0 ## Encoding the file with new values ``` * Edit the secret and replace the existing base64 encoded data with the new one on the variable `ml2_conf.ini` within the secret. ```bash $ kubectl edit secret -n neutron-etc secret_neutron-etc edited ``` * Run the following command to verify whether a new IP is being retrieved from the secret. ```bash $ kubectl get secret -n neutron-etc -oyaml |grep ml2_conf.ini | awk '{print $2}' | base64 -d ``` * Scale the `neutron-server` deployment to `0`. Once the pods are successfully terminated, scale it back to `2`. ```bash $ kubectl scale deploy_neutron-server -n --replicas=0 deployment.apps_neutron-server scaled $ kubectl scale deploy_neutron-server -n --replicas=2 deployment.apps_neutron-server scaled ``` * Verify that new DNS IPs are being received within the pod. ```bash $ kubectl exec -it -n -c neutron-server -- cat _etc_neutron_plugins_ml2_ml2_conf.ini ``` * Now, a VM should be created by attaching it to a network or subnet that does **not** have DNS configured manually. Once the VM is up and running, log in to it and execute the commands below to verify that the new IPs are reflected. **Note**: These changes will not be reflected in the existing virtual machines. ```bash $ resolvctl dns $ ip route | grep ``` # VMs Inaccessible due to Underlying Hypervisors Frequently Going Offline ## Problem The VMs are inaccessible via SSH and console as the Hypervisor connectivity is flapping frequently. ```bash $ ssh ubuntu@ Connection closed by [VM_IP] port 22 ``` ## Environment * Private Cloud Director Virtualization - till v2025.4-497 * Self-Hosted Private Cloud Director Virtualization - till v2025.4-497 * Component - Networking ## Cause * When the DHCP lease expires, the host interface gets reconfigured. During this process, interfaces that are attached to the bridge also receive IP addresses unexpectedly. Because of this, `ovn-controller` enters a failed state while processing its configuration updates. * This is a known issue and is reported internally with ID: **PCD-2492.** ## Diagnostics {% code title="neutron logs:" %} ```ruby pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via [IP] dev br-phy1 metric 0 failed. 2:b'RTNETLINK answers: File exists\\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 625, in handle_msg current_config = get_current_config() File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 262, in get_current_config config = app_db.get_current_config() File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_app_db.py", line 80, in get_current_config 'config': app.get_config(), File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_pf9_app.py", line 311, in get_config raise ConfigOperationError() pf9app.exceptions.ConfigOperationError ``` {% endcode %} Network related files: ```bash $ ls -lt /run/systemd/network/ -rw-r----- 1 root systemd-network 213 Jun 11 06:25 10-netplan-bond0.600.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.601.netdev -rw-r----- 1 root systemd-network 130 Jun 11 06:25 10-netplan-bond0.601.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.602.netdev -rw-r----- 1 root systemd-network 131 Jun 11 06:25 10-netplan-bond0.602.network -rw-r----- 1 root systemd-network 65 Jun 11 06:25 10-netplan-bond0.3001.netdev -rw-r----- 1 root systemd-network 155 Jun 11 06:25 10-netplan-bond0.3001.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.600.netdev -rw-r----- 1 root systemd-network 196 Jun 11 06:25 10-netplan-bond0.netdev -rw-r----- 1 root systemd-network 195 Jun 11 06:25 10-netplan-bond0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f0.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f1.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f1.network ``` {% code title="/var/log/pf9/hostagent.log" %} ```ruby pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via dev br-phy12 metric 0 failed. 2:b'RTNETLINK answers: File exists\\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 625, in handle_msg current_config = get_current_config() File "_opt_pf9_hostagent_lib_python3.9_site-packages_bbslave_session.py", line 262, in get_current_config config = app_db.get_current_config() File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_app_db.py", line 80, in get_current_config 'config': app.get_config(), File "_opt_pf9_hostagent_lib_python3.9_site-packages_pf9app_pf9_app.py", line 311, in get_config raise ConfigOperationError() pf9app.exceptions.ConfigOperationError ``` {% endcode %} ## Resolution The fix for the bug is available in the v2025.6 and above releases. ## Workaround Check if the interface IP is assigned to the bridge and the default route is associated with the bridge. * If yes, delete the default route via *bondx.x*. * If not, reboot the host and check again. ```ruby $ ip r default via [VM_IP] dev br-phy12 default via [VM_IP] dev bond0.10 proto static 1.1.1.1_24 dev br-phy12 proto kernel scope link src 1.1.1.251 1.1.1.2_24 dev bond0.10 proto kernel scope link src 1.1.1.251 1.1.1.3_24 dev br-phy15 proto kernel scope link src 1.1.1.251 1.1.1.4_24 dev bond0.50 proto kernel scope link src 1.1.1.251 $ sudo ip route del default via dev bond0.10 ``` ## Additional Information * For further questions/concerns regarding the bug, reach out to the [Platform9 Support Team](https://support.platform9.com/). # Tenant Router Unreachable Due to Missing OVN Localnet Port on Provider Network ## Problem Tenant router gateway becomes unreachable from the external network even though the router and provider network configuration appear correct. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component - Networking ## Cause The issue occurs due to a race condition between Neutron worker processes during network and port creation. When the network and associated ports are created concurrently, different Neutron workers may process these requests before the logical switch state is fully synchronized in OVN. As a result, the required localnet port is not created in the OVN logical switch. Without this port, the logical network cannot connect to the physical provider network bridge, preventing ARP responses from the router gateway and causing external connectivity failure. This is a known bug in neutron. An upstream fix has been released in latest version. Please see [Neutron Commit 71ff8ee](https://github.com/openstack/neutron/commit/71ff8ee881a2c387ff318df92c61e7b8333e7014) for more information. ## Diagnostics 1. Verify that the router status is ACTIVE and get the \[NETWORK\_ID]: ``` $ openstack router show ``` 2. Verify provider network configuration: ``` $ openstack network show ``` {% code title="Example Output" %} ``` | provider:network_type | vlan | | provider:physical_network | [phprod] | | provider:segmentation_id | [266] | | router:external | External | | status | ACTIVE | ``` {% endcode %} * Physical network mapping should match the expected provider network * VLAN ID should match the configured external VLAN
3. Verify that OVS bridge mappings on the gateway host: Run the following command on the compute host where the router gateway port is scheduled. ``` $ sudo ovs-vsctl list open ``` {% code title="Example Output" %} ``` external_ids : {ovn-bridge-mappings="[phprod:br-phy1]"} ``` {% endcode %} The provider network, in this example (`phprod`), must be mapped to the correct physical bridge (`br-phy1`)
4. Verify that ARP requests reach the compute host: Run the following commands on the compute host where the router gateway port is scheduled. This helps confirm whether ARP requests from the external network are reaching the host. {% code title="Sample command:" %} ``` $ tcpdump -i -evvvn arp and host ``` {% endcode %} or on the provider bridge: ``` $ sudo tcpdump -eni br-phy1 vlan and arp ``` {% code title="Example:" %} ``` $ sudo tcpdump -eni br-phy1 vlan 266 and arp ``` {% endcode %} {% code title="Sample Output" %} ``` 14:40:10.100908 00:09:0f:09:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42 14:40:11.149943 00:09:0f:19:0a:bb > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 266, p 0, ethertype ARP (0x0806), Request who-has 100.77.1.50 tell 100.77.1.40, length 42 ``` {% endcode %} In this example: * `100.77.1.40` → External firewall * `100.77.1.50` → Router gateway IP * VLAN `266` → Provider network VLAN The ARP request will typically appear as: ``` Request who-has tell ``` If ARP requests are visible but no ARP reply is observed, it indicates that even though traffic is successfully reaching the compute host via the physical network, OVN is not responding to the ARP request.
5. Verify OVN logical switch configuration: ``` $ ovn-nbctl show ``` Locate the logical switch corresponding to the network. {% code title="Example:" %} ``` switch ([neutron-dgsdtn-c819-4b6a-b9ff-3459fndf9d8cd]) ``` {% endcode %} The expected configuration should include a `localnet` port: ``` port type: localnet options: network_name= ``` If the `localnet` port is missing, the provider network cannot connect to the physical network, which prevents traffic from reaching the router gateway. {% hint style="info" %} *Note: Please check the **Additional Information** section for more information on how to run OVN commands depending on your environment* {% endhint %} 6. Compare with a working provider network Run the following command to inspect a working external network. ``` $ openstack network show ``` {% code title="Example:" %} ``` | provider:segmentation_id | [270] | ``` {% endcode %} If routers connected to this network work correctly while the failing network does not, the issue is isolated to the affected VLAN network. ## Resolution This issue is caused by a known Neutron race condition that has been addressed in the upstream project. An internal engineering tracker has been created for this issue: **PCD-5897** ## Workaround As an immediate workaround, recreate the affected provider network so that the OVN logical switch and localnet port are created correctly. 1. Delete the affected Network. Run the following command from the controller node: ``` $ openstack network delete ``` 2. Recreate the provider network: {% code title="Example command" %} ``` $ openstack network create \ --provider-network-type vlan \ --provider-physical-network [phprod] \ --provider-segment [266] \ --external \ ``` {% endcode %} 3. Recreate the router gateway: Attach the router to the recreated network: ``` $ openstack router set --external-gateway ``` 4. Verify OVN localnet port creation: Run: ``` $ ovn-nbctl show ``` Confirm the logical switch now contains: ``` port type: localnet ``` Once the localnet port is present, traffic will be correctly patched between `br-int` and `br-phy`, restoring router connectivity. ## Additional Information * For further questions/concerns regarding the bug, reach out to the [Platform9 Support Team](https://support.platform9.com/). * To run `ovn-*` commands on the hosts onboarded to PCD, execute below steps. 1. Create an environment file `ovs-alias.rc` as below: ``` EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}') export NBDB=tcp:${EXTERNAL_ID}:6641 export SBDB=tcp:${EXTERNAL_ID}:6642 alias ovn-sbctl="ovn-sbctl --db=$SBDB" alias ovn-nbctl="ovn-nbctl --db=$NBDB" alias ovn-trace="ovn-trace --db=$SBDB" ``` 2. Export the rc file and start using the ovn commands: ``` $ source ovs-alias.rc $ ovn-nbctl ls-list ``` {% hint style="info" %} **NOTE:** `ovn` commands can also be executed from inside the OVN North Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the steps below. {% endhint %} 1. Access the OVN North Bond Pod in the Management Cluster using the command below. ``` $ kubectl -n exec -it -- bash --kubeconfig ``` 2. Run the below command to validate: ``` $ ovn-trace neutron- 'inport=="" && eth.src== && ip4.src== && ip4.dst== && icmp' ``` # Unable to Ping Network Gateway From VM With Security Group Enabled ## Problem * Unable to Ping Network Gateway From VM With Security Group Enabled ```bash $ ping ``` * This behaviour is observed when the security group attached does not have Ingress rule defined. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization – v2025.4 and Higher * Component - Networking ## Cause * When using a security group, inbound rule in the Security Group is not defined to allow the incoming traffic. ## Resolution * To allow the incoming traffic for the Tenant Network Gateway IP, add the rule in the Security Group. * Steps to add ingress rule in the security group 1. Identify the security group UUID. 2. Run the below command to add the ingress rule to allow all incoming traffic. ```bash $ openstack security group rule create \\ --protocol any \\ --ethertype IPv4 \\ --direction ingress \\ --remote-ip 0.0.0.0_0 ``` ## Additional Information * No additional action needs to be performed on the virtual machine or security group. * Ingress Rule is required for the ping to be successful. Refer this [Upstream Documentation](https://docs.openstack.org/nova/latest/user/security-groups.html). * To understand Logical Tracing of the ONV Packet Flow, Refer this[ Upstream Documentation](https://docs.ovn.org/en/latest/tutorials/ovn-openstack.html#logical-tracing). * To run `ovn-*` commands on the hosts onboarded to PCD, execute below steps. 1. Create an environment file `ovs-alias.rc` as below ```bash EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}') export NBDB=tcp:${EXTERNAL_ID}:6641 export SBDB=tcp:${EXTERNAL_ID}:6642 alias ovn-sbctl="ovn-sbctl --db=$SBDB" alias ovn-nbctl="ovn-nbctl --db=$NBDB" alias ovn-trace="ovn-trace --db=$SBDB" ``` 2. Export the rc file and start using the ovn commands; ```bash $ source ovs-alias.rc $ ovn-nbctl ls-list ``` {% hint style="info" %} `ovn` commands can also be executed from inside the OVN North Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the below steps. {% endhint %} 1. Access the OVN North Bond Pod in the Management Cluster using the below command. ```bash $ kubectl -n exec -it -- bash --kubeconfig ``` 2. Run the below command to validate. ```bash $ ovn-trace neutron- 'inport=="" && eth.src== && ip4.src== && ip4.dst== && icmp' ``` # Network MTU Description States Possible Values of 1280-9000, But MTU can only be set up to 1500 ## Problem * In the PCD UI, the MTU description indicates that acceptable values range from **1280 to 9000.** However, when attempting to set the MTU value above **1500**, an error message is displayed in the UI, indicating that the MTU is capped at 1500. This suggests that, despite the stated range, the maximum MTU value that can be configured in the current environment is limited to 1500. ![](https://uploads.developerhub.io/prod/K9yg/a69wu4b3idsm2bqbmh67wdf3sazgkfgsjwvngr1ptchg4eq0i4bymw7m7b2phu4l.png) ## Environment * Private Cloud Director Virtualization - till v2025.7-47 * Private Cloud Director Kubernetes – till v2025.7-47 * Self-Hosted Private Cloud Director Virtualization - till v2025.7-47 * Self-Hosted Private Cloud Director Kubernetes - till v2025.7-47 * Component: Neutron ## Cause * Even though the MTU description states that acceptable values range from 1280 to 9000, the MTU is capped at 1500 as part of the default neutron configuration. * This issue is tracked in PCD-3215 and a fix is available from **2025.8-92 versions and above**. ## Workaround {% hint style="info" %} **PCD SaaS Users:** Reach out to the [Platform9 support team](https://support.platform9.com/hc/en-us) to update the MTU value on the management plane. **PCD Self-Hosted Users**: Please apply the steps below to update the MTU value in the Neutron configuration. **NOTE:** The steps provided below are for a temporary workaround. It will won't persist management plane upgrade. {% endhint %} **1.** Backup neutron-etc secret on management plane cluster. ```ruby $ kubectl get secret neutron-etc -n --kubeconfig -o yaml > neutron-etc-secret.yaml ``` {% hint style="info" %} All the configuration file data in the `neutron-etc` secret is in base64 encoding. To see the actual data, decode it from base64. {% endhint %} **2.** Modify neutron-etc secret. * Update `neutron.conf` and add `global_physnet_mtu = 9000` under **DEFAULT** section. ```ruby $ kubectl get secret neutron-etc -n --kubeconfig \\ -o jsonpath="{.data.neutron\\.conf}" | base64 -d | \\ sed '_^\\[DEFAULT\\]_a global_physnet_mtu = 9000' | \\ base64 | tr -d '\\n' | \\ xargs -I{} kubectl patch secret neutron-etc -n --kubeconfig \\ --type='json' -p='[{"op":"replace","path":"_data_neutron.conf","value":"{}"}]' ``` * Modify `ml2_conf.ini` and add `path_mtu = ` under **ml2** section. ```ruby $ kubectl get secret neutron-etc -n --kubeconfig \\ -o jsonpath="{.data.ml2_conf\\.ini}" | \\ base64 -d | \\ sed '_^\\[ml2\\]_a path_mtu = ' | \\ base64 | tr -d '\\n' | \\ xargs -I{} kubectl patch secret neutron-etc -n --kubeconfig \\ --type='json' -p='[{"op":"replace","path":"_data_ml2_conf.ini","value":"{}"}]' ``` **3.** Restart Neutron server. ```ruby $ kubectl rollout restart deployment neutron-server -n --kubeconfig ``` ## Validation {% hint style="info" %} The virtual networks have an associated offset, meaning that the maximum MTU value that a virtual network can support is determined by subtracting **50** from the **path\_mtu** value. {% endhint %} * Update the MTU value of a network from the UI to a value greater than 1500. # VMs Running in a Specific Compute Host Were Not Reachable ## Problem Multiple VMs running in a specific compute host were not reachable from the compute host itself and also from the external network when checked using network connectivity tests. Identified that the affected VMs were part of a common VLAN with port security disabled. ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Networking Service ### Cause The issue is that stale entries in OVN-SB-DB are causing a complete VM network outage. When VMs are connected to a provider network with **port security disabled**, any attempt to reach the VM IP addresses, PCD does not recognize this and prevents the FDB table from learning the associated MAC addresses. As a result, traffic is continuously broadcast, ultimately this affected VM is causing a total network outage for the VM. An[ upstream](https://bugs.launchpad.net/neutron/+bug/2012069) issue was identified and is tracked as PCD-4889 in PCD ## Diagnostics * Identify target VMs on the host from the affected network. Usually, all the VMs running on the same host from a specific network are affected ```bash openstack server list --all --host | grep -i ``` * Find recently migrated/created VMs from above ```bash openstack server event list ``` * Check if Port Security is disabled. * Check if the VMs are reachable inside the corresponding subnet. * Power off suspect VM**s** one by one (with confirmation) and check network reachability for other VMs. * Once the problematic VM is identified and powered off, it allows the rest of the VMs in a running state and using the affected network to be reachable; change the IP and MAC address of the network port attached to the problematic VM. * In the OVN flows for the affected VM, the evident traces of a stale route can be tracked as shown in the below example 
65. reg15=0x58,metadata=0x14, priority 100, cookie 0xf8315af4
      output:274
  ##In the OVN flow 0x58 to decimal is 88.
* The corresponding rule to 88 was the following, which caused the stale route. 
_uuid               : [UUID]
  dp_key              : 20
  mac                 : "[MAC_ADDRESS]"
  port_key            : 88
### Workaround 1 * Delete the existing port and recreate the port with the same IP so that the MAC address of the VM is changed.
### Workaround 2 * To run `ovn-*` commands on the hosts onboarded to PCD, execute below steps. * Create an environment file `ovs-alias.rc` as below: ``` EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}') export NBDB=tcp:${EXTERNAL_ID}:6641 export SBDB=tcp:${EXTERNAL_ID}:6642 alias ovn-sbctl="ovn-sbctl --db=$SBDB" alias ovn-nbctl="ovn-nbctl --db=$NBDB" alias ovn-trace="ovn-trace --db=$SBDB" ``` * Export the rc file and start using the ovn commands: ``` $ source ovs-alias.rc $ ovn-sbctl show ``` * Delete the stale FDB from the `ovn-sb` pod using the command:
$ ovn-sbctl destroy fdb [FDB_UUID]
* Post this change, the network connectivity of the impacted VMs in the host will be resolved. ### Resolution Instead of disabling port security entirely, we recommend keeping the basic MAC-address validation enabled and applying a security group that allows all traffic. This approach ensures that: * The VM ports continue to enforce correct MAC-address learning. * The gateway MAC address is not mistakenly learned on VM interfaces. * All inbound and outbound traffic continues to flow without restriction. This provides the required functionality while maintaining the minimal level of protection needed to prevent incorrect MAC entries from causing connectivity issues. ### Validation The VMs will respond to ping tests and be accessible via SSH and virsh console. # VM Network Port Attachment Failure and High CPU Usage on OVNDB ### Problem PCD environment is experiencing a critical failure in VM network port attachment operations, preventing virtual machines from connecting to the network. The service is consuming 88-95% CPU, causing it to become unavailable for legitimate API requests. {% code title="OVN Logs" %} ``` |poll_loop|INFO|Dropped 79 log messages in last 94 seconds (most recently, 91 seconds ago) due to excessive rate |poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage) ``` {% endcode %} {% code title="ostackhost Logs" %} ``` nova.exception.PortInUse: Port is still in use. ``` {% endcode %} ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher ### Cause The OVN Database (OVN-DB) service has entered a non-responsive state characterised by a high-CPU spin loop. ### Diagnostics {% hint style="info" %} Step 2 -3 only accessible and applicable for the Self-Hosted Private Cloud Director. For SaaS environments, please reach out to Platform9 support. {% endhint %} 1. VM deployment fails with the below exception shown on the `pf9-ostackhost` logs located on the hypervisor at location `/var/log/pf9/`. 
nova.exception.PortInUse: Port <PORT_UUID> is still in use.
2. Review the `ovn-ovsdb-sb-0` pod logs from the workload region namespace. ``` $ kubectl logs -n ``` 3. Validate the above pod logs and check CPU usage messages like those shown in the below snippet. ``` |poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage) ``` ## Resolution {% hint style="info" %} Only accessible and applicable for the Self-Hosted Private Cloud Director. For SaaS environments, please reach out to Platform9 support. {% endhint %} 1. Take the backup of OVNDB database. ``` $ kubectl cp /:/var/lib/ovn/ovnnb_db.db ./ovnnb_db_backup.db $ kubectl cp /:/var/lib/ovn/ovnsb_db.db ./ovnsb_db_backup.db ``` 2. Once backup is taken, execute compaction command ``` $ kubectl exec -n -- ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/compact $ kubectl exec -n -- ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/compact ``` 3. Enable memory compaction on the below pods ``` $ kubectl exec -n -- \ ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/memory-trim-on-compaction on $ kubectl exec -n -- \ ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/memory-trim-on-compaction on ``` 4. Restart the below pods ``` $ kubectl delete -n $ kubectl delete -n ``` ## Validation {% hint style="info" %} Step 1-2 is applicable for the Self-Hosted Private Cloud Director. For SaaS environments, please reach out to Platform9 support. {% endhint %} 1. Check the pods' status; it should be running ``` $ kubectl get pods -n | grep -i ovn ``` 2. Review the `ovn-ovsdb` pod logs from the workload region namespace and confirm if no high CPU usage messages are logged in these pods. ``` $ kubectl logs -n ``` 3. Deploy a new VM and review the `pf9-ostackhost` logs from the hypervisor (host); it no longer logs `Port is still in use` messages. # VMs Unable to Retrieve Metadata From Cloud-Init ## Problem Virtual machines in a specific subnet fail to reach metadata service on `169.254.169.254`. As a result, cloud-init did not apply user-data configurations such as password injection. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component: * Networking ( Neutron/OVN) ## Cause The root cause of the issue is a missing **network namespace** used for metadata routing. This network namespace is critical for OVN to route metadata requests from VMs to the metadata proxy service. ## Diagnostics 1. VMs failed to reach `169.254.169.254` via `ping` or `curl`. 2. Gateway IP was also unreachable from within the instance. 3. `ip netns ls` command on impacted hosts showed no namespaces. 4. Found no distributed Neutron port on the host. 5. `pf9-neutron-ovn-metadata-agent` service was confirmed to be listening on port `8775`. ## Resolution 1. Check if a distributed neutron port exists for the network: ```bash $ openstack port list --network --device-owner network:distributed #sample output: +--------------+-------------+---------------+-----------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------+-------------+---------------+-----------------------------------------------------+--------+ | [PORT_UUID] | [PORT_NAME] | [MAC_ADDRESS] | ip_address='[IP_ADDRESS]', subnet_id='[SUBNET_UUID]'| DOWN | +--------------+-------------+---------------+-----------------------------------------------------+--------+ ``` 2. If the distributed Neutron port is present, proceed to **Step 3**. If it is missing, manually create the distributed Neutron port using the command: ```bash $ openstack port create --network --fixed-ip subnet=,ip-address= --device-owner network:distributed metadata-proxy-port- +-------------------------+-----------------------------------------------------+ | Field | Value | +-------------------------+-----------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | data_plane_status | None | | device_id | | | device_owner | network:distributed | | device_profile | None | | extra_dhcp_opts | | | . | | | fixed_ips | ip_address='[IP-Address]', subnet_id='[subnet-id]' | |. | | | hardware_offload_type | None | | security_group_ids | | | status | DOWN | +-------------------------+-----------------------------------------------------+ ``` > The \[IP-Address] is any free IP Address available in the DHCP pool of the subnet. 3. Restart pf9-neutron-ovn-metadata-agent on all impacted compute nodes. {% code title="Affected Host:" %} ```bash $ sudo systemctl restart pf9-neutron-ovn-metadata-agent ``` {% endcode %} 4. Verify that ip netns ls showed namespaces post-restart. {% code title="Affected Host:" %} ```bash $ ip netns list ovnmeta-[ovnnetns-id-1] (id: 0) ``` {% endcode %} 5. Re-test metadata access from inside VMs using curl. 6. Spawn a new VM with Ubuntu image and confirm that cloud-init correctly applied the configured password and user-data. > Cirros OS image do not have cloud init service for handling the user data injection. Use image with cloud init service ex: Ubuntu or Rocky Linux for validation ## Validation * From within the VM, below *curl* command returns the expected user-data content : {% code title="Getting meta\_data within the VM" %} ```bash $ curl http://169.254.169.254/openstack/latest/meta_data.json {"uuid":"[UUID]", availability_zone":"[AZ]", "hostname":"[hostname]", "name": "[name]","launch_index": 0, "random_seed": "[random-id]", "devices":[], "dedicated_cpus":[]} ``` {% endcode %} {% code title="Getting user\_data within the VM" %} ```bash $ curl http://169.254.169.254/openstack/latest/user_data #cloud-config password: [Password] chpasswd: { expire: False } ssh_pwauth: True manage_etc_hosts: true runcmd: - ['sh' , '-c', 'echo "Hello World" › /tmp/helloworld.txt' ] ``` {% endcode %} * Cloud-init logs inside the Ubuntu VM (*/var/log/cloud-init.log*) confirmed successful metadata retrieval and password configuration. * VM SSH access was successful using credentials configured via user-data. ## Additional Information * In OVN-based OpenStack environments, the distributed metadata port must be explicitly present for each subnet requiring metadata access. # VM Creation Fails with No AvaiIable IP ### Problem Unable to create a VM on a subnet because the UI reports **no available IPs**, even though the allocation pool still has free addresses. ### Environment * Private Cloud Director Virtualization - till v2026.1-360 * Private Cloud Director Kubernetes – till v2026.1-360 * Self-Hosted Private Cloud Director Virtualization - till v2026.1-360 * Self-Hosted Private Cloud Director Kubernetes - till v2026.1-360 * Component: Networking Service ### Cause The issue is caused by incorrect IP availability calculation. The system subtracts all used addresses in the subnet (including those outside the allocation pool) instead of only considering addresses within the defined allocation pool. This is a known bug tracked PCD-3647 and fix is available from **v2026.1-360 and above** ### Diagnostics 1. Check subnet details and allocation pool configuration. Look for **allocation\_pools** field and confirm the defined IP range. ```bash openstack subnet list --long openstack subnet show ``` 2. List all ports and IPs allocated in the subnet.Compare allocated IPs with allocation pool; verify how many pool IPs are actually in use. ```bash openstack port list --fixed-ip subnet= -c ID -c Fixed_IPs -c Device_ID ``` 3. Verify mismatch in UI: 1. In the **UI → Network & Security → Physical/Virtual Networks→ Subnets**, check the **Available IPs** field. 2. Compare UI’s reported value (e.g., 0 or negative) against the manual calculation.
### Workaround Since the UI incorrectly reports available IPs, create VMs directly from the CLI using a free IP from the allocation pool. ``` openstack server create \ --flavor \ --image \ --nic net-id=,v4-fixed-ip= \ --security-group \ --key-name \ --availability-zone \ ``` # VM Creation Failed Due to Port Binding Failure ## Problem VM creation failed due to the following error: {% code title="ostackhost.logs" %} ```bash nova.exception.PortBindingFailed: Binding failed for port [PORT-UUID] please check neutron logs for more information. ``` {% endcode %} ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component: Networking Service ## Cause This issue occurs when Neutron agents on compute nodes are unable to bind the requested port to the VM. Common causes include: * OVN controller on the compute node is unresponsive or out of sync with the OVN Southbound database. * Neutron server-to-OVN database communication issues. * Stale port binding records in OVN or Neutron. ## Diagnostics * Review ostackhost logs on the affected compute host: ```bash $ cat /var/log/pf9/ostackhost.log | grep -i ``` * Check Neutron agent status: ```bash $ openstack network agent list ``` * Review OVN controller logs on the compute node: ```bash $ sudo journalctl -u ovn-controller -f $ cat /var/log/ovn/ovn-controller.log ``` ## Resolution Restart OVN services on the affected compute node(s): ```bash $ sudo systemctl restart ovn-controller ``` {% hint style="warning" %} Restarting `ovn-controller` on a compute node generally does not disrupt existing VM traffic since OVS flows remain programmed, but it temporarily prevents new port bindings, VM spawns, or security group updates {% endhint %} ## Validation * Confirm the VM builds successfully * Verify the Neutron port binding: ```bash $ openstack port list --server ``` # Network Agents Missing for Re-onboarded Host Due to Stale OVN Chassis Entry ### Problem * Network agents for a host are missing after re-onboarding * Host does not appear correctly in: ```shellscript $ openstack network agent list ``` * OVN agents (ovn-controller / metadata agent) are not visible or not alive * OVN controller logs show following error in `/var/log/pf9/ovn/ovn-controller.log` : ```ruby transaction error: constraint violation multiple rows in "Encap" table with same (geneve, [IP]) ``` * Host networking functionality is degraded due to missing OVN registration ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component - Networking ### Cause This issue occurs specifically on re-onboarded hosts where a stale OVN chassis entry remains in the OVN Southbound database. This typically happens when the host is not properly decommissioned prior to re-onboarding, resulting in residual OVN state persisting in the control plane. To prevent such issues, ensure the host is fully [deauthorized and decommissioned](https://docs.platform9.com/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/hypervisor-role-deauthorization-and-reauthorization) by following the official Platform9 documentation. More details on how deauthorization and decommission work can be found in [Demystifying Deauthorization and Decommissioning of PCD Hosts](https://platform9.com/kb/pcd/generic/demystifying-deauthorization-and-decommisioning-of-pcd-hosts). * During re-onboarding, the host attempts to register a new chassis * The old chassis entry (from before decommissioning) still exists with the same Geneve IP * This leads to a **constraint violation** in the OVN database * As a result: * New chassis registration fails * OVN agents do not come up * Network agents appear missing in PCD. ### Diagnostics #### 1. Check Network Agents ```shellscript $ openstack network agent list ``` * Confirm that the re-onboarded host is missing or agents are not alive. #### 2. Check OVN Controller Logs (on the re-onboarded host) ```shellscript $ cat /var/log/pf9/ovn/ovn-controller.log ``` Look for: ```ruby |ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"Encap\" table to have identical values (geneve and \"[HOST_IP_ADDRESS]\") for index on columns \"type\" and \"ip\". First row, with UUID [UUID_1], was inserted by this transaction. Second row, with UUID [UUID_2], existed in the database before this transaction and was not modified by the transaction.","error":"constraint violation"} ``` ### Resolution 1. Identify the existing (stale) chassis entry: {% hint style="info" %} Refer the Additional Information section to understand how to execute ovn-\* commands. {% endhint %} ```bash $ ovn-sbctl show ``` {% code title="Example Output (Before Cleanup):" %} ```ruby #example output Chassis "[OLD_CHASSIS_UUID]" hostname: "[OLD_HOST_ID]" Encap geneve ip: "[HOST_GENEVE_IP]" options: {csum="true"} ``` {% endcode %} Observation: * A chassis entry already exists for the Geneve IP * This entry is stale and preventing new registration * The expected (current) host is not present in the output 2. Delete the stale chassis entry: ```shellscript $ ovn-sbctl chassis-del ``` 3. Restart OVN services on the host: ```bash $ systemctl restart ovn-controller $ systemctl restart pf9-neutron-ovn-metadata-agent ``` 4. Verify new chassis registration: ```bash $ ovn-sbctl show #example output Chassis "[NEW_CHASSIS_UUID]" hostname: "[NEW_HOST-ID]" Encap geneve ip: "[HOST_GENEVE_IP]" options: {csum="true"} ``` 5. Verify network agents are now visible: ```bash $ openstack network agent list ``` ## **Validation** * Network agents for the host are visible in `openstack network agent list` * Agents show `Alive :-)` and `State UP` * No constraint violation errors in OVN logs ### Additional Information * To run `ovn-*` commands on the hosts onboarded to PCD, execute below steps. 1. Create an environment file `ovs-alias.rc` as below: ``` EXTERNAL_ID=$(sudo ovs-vsctl get open . external_ids:ovn-remote | awk -F: '{print $2}') export NBDB=tcp:${EXTERNAL_ID}:6641 export SBDB=tcp:${EXTERNAL_ID}:6642 alias ovn-sbctl="ovn-sbctl --db=$SBDB" alias ovn-nbctl="ovn-nbctl --db=$NBDB" alias ovn-trace="ovn-trace --db=$SBDB" ``` 2. Export the rc file and start using the ovn commands: ```shellscript $ source ovs-alias.rc $ ovn-sbctl show ``` {% hint style="info" %} `ovn` commands can also be executed from inside the OVN South Bond Pod on the Management Cluster. Only Self-Hosted Private Cloud Director Virtualization users can run the steps below. {% endhint %} 3. Access the OVN South Bond Pod in the Management Cluster using the command below. ```shellscript $ kubectl -n exec -it -- bash --kubeconfig ``` * Always ensure that hosts are properly decommissioned before re-onboarding to avoid stale OVN entries. Refer to the [official documentation.](https://docs.platform9.com/private-cloud-director/virtualized-clusters/troubleshooting-and-log-files/hypervisor-role-deauthorization-and-reauthorization) # MTU for Virtual Networks Cannot be Modified ## Problem * MTU for Virtual Networks cannot be modified. * PCD UI does not provide any option to increase the MTU for Virtual Networks. ## Environment * Private Cloud Director Virtualization - v2026.1 and Higher * Self Hosted Private Cloud Director Virtualization - v2026.1 and Higher ## Cause * This is a known issue and is already reported with ID: **PCD-6454**. {% hint style="info" %} Refer the [**Workaround**](#workaround) section on how to set the MTU for Virtual Networks. {% endhint %} ## Workaround * MTU for Virtual Networks can be modified through CLI. * In the RC file that is needed for the communication between `pcdctl` CLI and PCD, change the variable `OS_INTERFACE` value from `public` to `admin`. * Set the relevant tenant name in the `OS_PROJECT_NAME` variable if applicable. * Re-source the RC file. * Change the MTU using the below command: {% code title="" %} ```bash pcdctl network set --mtu ``` {% endcode %} ## Additional Information * Reach out to Platform9 Support with the issue ID mentioned in the [**Cause**](#cause) section if there are any questions/concerns. # MTU of Virtual Networks Cannot be Modified ### Problem * MTU of the Virtual Networks created in PCD cannot be modified. * PCD UI does not provide any option to edit the MTU of Virtual Networks ### Environment * Private Cloud Director Virtualization - v2026.1 and Higher * Private Cloud Director Kubernetes – v2026.1 and Higher ### Cause * This is a known product limitation and is already reported internally as feature request with ID: **PCD-6454** ### Workaround * MTU of Virtual Networks can be changed through CLI. ### Additional Information * This article will be updated promptly on the Fi * For any information on the # Configuring Windows Network Load Balancer (NLB) over PCD ## Problem * Microsoft Windows Network Load Balancer (NLB) cluster VIPs are unreachable when the NLB virtual machines are attached to a network that has port security enabled. * Symptoms typically include: * The NLB cluster VIP cannot be pinged or reached from other VMs or hosts. * ARP resolution for the VIP fails or resolves inconsistently. * Client applications relying on the NLB VIP (for example, ADFS proxy services) intermittently or completely fail to connect. ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Virtualization * Component - Networking ## Cause Windows NLB is designed to share a single Virtual IP (VIP) across all cluster nodes. Depending on the cluster's operating mode, NLB associates that VIP with a special shared MAC address: | NLB mode | VIP MAC behaviour | | ------------------------------- | ------------------------------------------------------------------------------------------------- | | Unicast | A single shared unicast MAC (`02:BF:...`) is assigned to every node for the VIP. | | Multicast (default/recommended) | A multicast MAC (`03:BF:...`) is assigned to the VIP, while each node keeps its own physical MAC. | | Multicast with IGMP | A multicast MAC (`01:00:5E:...`) is assigned to the VIP. | In all three modes, cluster nodes deliberately advertise a MAC address for the VIP that does **not** match the node's own physical port MAC, and multiple nodes answer for the same VIP. This is fundamental to how NLB distributes traffic across the cluster. PCD networking uses OVN, which enforces **port security** (anti-spoofing) by default. Port security only permits traffic whose source IP/MAC matches the addresses configured on the port. Because NLB's VIP traffic intentionally uses a shared or virtual MAC that differs from the port's own MAC, port security treats it as spoofed traffic and drops it. This is the same reason VMware requires the equivalent security options to be relaxed for NLB (see [Additional Information](#additional-information)). Selectively permitting NLB traffic while keeping port security enabled is not a reliable option: it would have to account for every protocol NLB and its workloads use (ARP, ICMP, TCP, UDP, and many others), which is not sustainable. The supported approach is therefore to isolate NLB workloads on a dedicated network with port security disabled. ## Resolution Deploy Windows NLB VMs on a **dedicated, isolated, VLAN-backed provider network with port security disabled**, and terminate the NLB traffic on your **external/physical router** rather than a Neutron (tenant) router. This confines the relaxed security settings to a single, purpose-built broadcast domain used only for NLB, so the rest of the cloud remains protected by port security. ### Prerequisites * Administrator access to PCD with permissions to create provider networks and manage ports. * A dedicated VLAN provisioned on the physical/underlay network for the NLB traffic, trunked to the PCD hosts. * The physical network label configured in the OVN bridge mappings on the hosts where the NLB VMs will run. * Gateway/routing for the NLB VIP subnet provided by your external router (see step 4). ### Step 1 — Create a dedicated VLAN-backed provider network Create a provider network exclusively for the NLB VMs. Do not attach any other production workloads to it. ```bash $ openstack network create \ --provider-network-type vlan \ --provider-physical-network \ --provider-segment \ --external \ nlb-provider-net ``` ### Step 2 — Disable port security on the network Disabling port security at the network level ensures every port created on it inherits the setting. ```bash $ openstack network set --disable-port-security nlb-provider-net ``` Create the NLB subnet on this network. Set the gateway to your external router's address on the NLB VLAN: ```bash $ openstack subnet create \ --network nlb-provider-net \ --subnet-range \ --gateway \ nlb-subnet ``` ### Step 3 — Attach the NLB VMs and confirm port security is disabled Attach each Windows NLB VM to `nlb-provider-net`. For any port that already exists, explicitly disable port security. A port cannot have port security disabled while security groups are attached, so remove them first: ```bash $ openstack port set --no-security-group --disable-port-security ``` Verify the setting on every NLB VM port: ```bash $ openstack port show -c port_security_enabled -c security_group_ids ``` Expected output: ``` | port_security_enabled | False | | security_group_ids | | ``` Repeat for every NLB node's port. ### Step 4 — Terminate NLB traffic on the external router Route and terminate the NLB VIP traffic on your **external/physical router**, not on a Neutron router. The dedicated provider network is created as `--external` and mapped to the VLAN so that Layer 2 (including the NLB multicast/shared-MAC frames) stays within the isolated broadcast domain and is handed off directly to the physical network. Configure the default gateway of the NLB subnet to point at the external router, and configure any north-south routing for the VIP on that external router. ### Step 5 — Configure Windows NLB On the Windows guests, configure the NLB cluster as usual, using **Multicast mode** (the generally recommended mode). Once the cluster nodes are attached to the isolated, port-security-disabled network, the VIP becomes reachable and the cluster converges normally. ## Risks Disabling port security removes the anti-spoofing protection on the affected ports. This introduces the following risks: * A compromised VM can impersonate other devices (**MAC spoofing**). * Exposure to **ARP poisoning** and **Man-in-the-Middle (MITM)** attacks. * **Network instability** caused by conflicting or duplicate MAC addresses. Because of these risks, port security must **never** be disabled on a shared or production network. It should only be disabled on the dedicated, isolated NLB network described above. ## Preventive Measures To keep the exposure strictly contained: * **Isolate the network.** Use a dedicated VLAN-backed provider network exclusively for the Windows NLB VMs. Do not attach any other production VMs to this network. * **Restrict the broadcast domain.** Placing the NLB VMs on a dedicated VLAN limits the impact of MAC spoofing or ARP anomalies to that VLAN only. * **Terminate on the external router.** Keep NLB traffic termination on the external/physical router so the relaxed-security segment never mixes with tenant routed traffic. * **Monitor continuously.** Continuously monitor the NLB network segment for unusual traffic patterns, unexpected MAC addresses, and ARP table anomalies. ## Additional Information * This approach mirrors how other platforms handle Windows NLB. VMware supports NLB only when `Promiscuous mode = Accept`, `Forged Transmits = Accept`, and `MAC Address Changes = Accept` are set on a dedicated port group, the combination of which is equivalent to disabling port security in PCD. See the [VMware knowledge base article on configuring NLB](https://knowledge.broadcom.com/external/article?articleNumber=414968) for more detail. * The same dedicated-VLAN, isolated-broadcast-domain pattern is the standard guidance for running multicast-dependent workloads on top of OpenStack-based clouds generally. See [Cisco's guidance on multicast with OpenStack](https://community.cisco.com/t5/cloud-blogs/multicast-with-openstack-using-cisco-s-nexus9000-and-ucs/ba-p/3661925) for background. * For further questions or assistance configuring Windows NLB on PCD, reach out to the [Platform9 Support Team](https://support.platform9.com/). # Cinder Volume Operations Failing With Vmstore “NotFound” Error on Tintri Backend ## Problem Volume operations in PCD Storage Service fail when using the Tintri (vmstore) backend. Users may observe: * Volume stuck in **`creating`** state and eventually failing. * Failures during operations such as: * Volume create * Volume clone * Other CRUD operations * PCD UI throws error like: ``` Could not find any available weighted backend ``` * On the persistent storage host, `cindervolume-base` logs reports: ``` VmstoreException: Volume driver reported an error: No details (typeId: VmstoreError, code: NotFound) ``` ## Environment * Private Cloud Director Virtualization * Self-Hosted Private Cloud Director Virtualization * Component - Storage Service, Tintri * Tintri - v3.0.5 and below ## Resolution * The Tintri Driver needs to be upgraded to the version `3.0.6` and above, followed by a rollout restart of the `cinder-scheduler` workload for the respective PCD region. * Customers need to apply the Tintri driver upgrade patch to the Persistent Storage Nodes. Reach out to the Tintri Support Team to implement this patch. * Once the Tintri driver is upgraded, reach out to the Platform9 Team about a rollout restart of `cinder-scheduler` workload in the affected region. {% hint style="info" %} **Only Self-hosted PCD customers can run the command to restart the cinder-scheduler:** `$ kubectl rollout restart deployment/cinder-scheduler -n $REGION_NAMESPACE` {% endhint %} ## Additional Information * Post PCD upgrade, customers need to upgrade the Tintri driver to 3.0.6 or above versions. # How To Reset Volume States ## Problem Private Cloud Director block storage volumes can be in the following states: "reserved", “error”, “creating”, “deleting”, “in-use”, “attaching”, “detaching”, “error\_deleting” or “maintenance”. Sometimes the volumes might be in a state that does not reflect their present condition. For example, a failed live migration might set the volume state to "error". However, it is possible to access and edit the data inside the volume. In such a case, it becomes necessary to reset the volume state to reflect its actual condition. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher ## Procedure 1. These steps can only be run as admin. Source the RC file with the admin credentials. ```bash $ source ``` 2. Get the UUID of the volume whose status that needs to be changed. ```bash $ openstack volume list ``` 3. Change the status using the following command. ```bash $ openstack volume set --state ``` ## Validation Verify if the volume state has changed. ```bash $ openstack volume show ``` ## Additional Information * [Set other volume properties](https://docs.openstack.org/python-openstackclient/ocata/command-objects/volume.html?#volume-set) # Cinder Driver Fails to Decrypt Storage Passwords set via Blueprint ## Problem Following the PCD v2025.4 release, storage passwords within the `cinder.conf` file, specifically in the `secret_mapping.conf` section, are encrypted. However, the Cinder service fails to decrypt these passwords before passing them to the storage driver. This results in authentication failures, manifesting as 401 errors on the storage array. {% code title="Cinder Logs" %} ```ruby REQ: curl -i https://:8080/api/v1/credentials -X POST -H "User-Agent: python-3parclient" -H "Accept: application/json" -H "Content-Type: application/json" from (pid=3173923) _http_log_req /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/hpe3parclient/http.py:170 REQ BODY: {"user": "3paruser", "password": "xxxxxx"} DEBUG hpe3parclient.http [[req-ID] None None] RESP BODY:{"code":5,"desc":"invalid username or password"} ERROR cinder.volume.manager [[req-ID] None None] Failed to initialize driver.: hpe3parclient.exceptions.HTTPForbidden: Forbidden (HTTP 403) 6 - invalid session key ``` {% endcode %} ## Environment * Private Cloud Director Virtualization - till v2025.4-497 * Self-Hosted Private Cloud Director Virtualisation – till v2025.4-497. * Component - Storage ## Cause * The v2025.4 release introduces encryption for storage passwords in Cinder's configuration. A defect exists where the Cinder service does not decrypt these passwords when providing them to the storage driver. * This is a known issue that was reported as PCD-2162 and a fix is available from **v2025.6-151 and above versions.** ## Resolution * The fix for this issue is now available in the v2025.6 Release. * After the Management Plane and the hosts are fully upgraded to v2025.6 Release, remove the file `secret_mapping_override.conf` and provide the password in the Blueprint Volume Config. ## Workaround To mitigate this issue, do not set storage passwords in the blueprint. Instead, set password-related options directly on the Cinder host as part of an override configuration file. 1. Navigate to the Cinder configuration directory on the Cinder host: ```bash $ cd /opt/pf9/etc/pf9-cindervolume-base/conf.d/ ``` 2. Comment the below mentioned lines in cinder.conf as it forces to use the *"config\_source = secrets" section* 
[DEFAULT]
...
# config_source = secrets
...
# [secrets]
# driver = castellan
# config_file = /opt/pf9/etc/pf9-cindervolume-base/conf.d/castellan.conf
# mapping_file = /opt/pf9/etc/pf9-cindervolume-base/conf.d/secret_mapping.conf
3. Edit the `cinder.conf` file. Add the necessary password configurations for your storage backend. For example, for an HPE 3PAR backend named `storage-3par-fc`, the content might look like this: > *Replace* *`your_actual_password`* *with the correct password for your environment.* ```ruby [storage-3par-fc] hpe3par_debug = True san_password = your_actual_password hpe3par_password = your_actual_password ``` 4. Restart the cinder service on the host: ```bash systemctl restart pf9-cindervolume-base.service ``` ## Additional Information For further questions/concerns regarding the bug, reach out to the [Platform9 Support Team](https://support.platform9.com/hc/en-us) # How to Delete Volume Attached to Stale VM? ## Problem A Persistent Storage volume cannot be deleted because it is still marked as **attached to a VM**, but: * The VM is **deleted / missing / stale** * The attachment is **orphaned** * Standard delete fails with errors like: {% code overflow="wrap" %} ``` $ openstack volume delete Failed to delete volume with name or ID '[VOLUME_UUID]': Invalid volume: Volume is in status 'in-use', expected one of ('available', 'error', 'error_restoring', 'error_extending', 'error_managing'). (HTTP 400) (Request-ID: [REQ_ID]) ``` {% endcode %} ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Storage Service ## Cause This usually happens due to: * Failed VM deletion * Incomplete detach workflow * Messaging issues (RabbitMQ / Persistent Storage / Compute Service sync gap) * DB state inconsistency between **Persistent Storage** and **Compute Service** ## Diagnostics * **Step 1: Identify the Volume** 
$ openstack volume show <VOLUME_ID>
* *Check:* **status** -> likely `in-use`\ **attachments** -> contains stale server UUID * **Step 2: Verify VM Existence** 
$ openstack server show <VM_ID>
* Check: Not found -> stale attachment confirmed ## Resolution 1. Force the volume to an available state for Persistent Storage to ignore the stale VM 
$ openstack volume set --state available <VOLUME_UUID>
2. Delete the stale attachment: ``` $ openstack volume attachment delete ``` 3. Delete the volume: ``` $ openstack volume delete --force ``` ## Validation * The volume is deleted successfully {% code overflow="wrap" %} ``` $ openstack volume show No volume with a name or ID of '[VOLUME_ID]' exists. ``` {% endcode %} {% hint style="info" %} Below step is applicable for Self-Hosted Private Cloud Director Users who can access the persistent storage and compute service databases. {% endhint %} * No stale entries present in the following tables: * `volume_attachment` * `nova.block_device_mapping` # Persistent Storage Volume Creation Fails With VolumeSizeExceedsAvailableQuota (HTTP 413) ## Problem * A volume operation: creating a new volume, expanding an existing one, or migrating a VM via a tool such as vjailbreak, fails with an `HTTP 413` response from the Cinder API. {% code title="cinder-api Logs" overflow="wrap" %} ```bash WARNING cinder.quota_utils [None [REQ_ID] [USER_ID] [TENANT_ID] - - default default] Quota exceeded for [TENANT_ID], tried to create [SIZE_IN_Gi] volume ([USAGE_SIZE] of [TOTAL_SIZE] already consumed).: cinder.exception.OverQuota: Quota exceeded for resources: ['gigabytes'] ``` {% endcode %} {% hint style="info" %} In SaaS environments, above verification is handled by Platform9 on the backend. Please contact Platform9 Support for verification details if needed. {% endhint %} * Via vJailbreak migration: {% code title="vJailbreak agent Logs" overflow="wrap" %} ```bash Failed to migrate VM: failed to add volumes to host: failed to create volume: failed to create volume: Expected HTTP response code [202] when accessing [POST https://[FQDN]/cinder/v3/[TENANT_UUID]/volumes], but got 413 instead: {"overLimit": {"code": 413, "message": "VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested [SIZE], quota is [TOTAL_SIZE] and [USAGE_SIZE] has been consumed.", "retryAfter": "0"}}. ``` {% endcode %} * The disk utilization via `df` on the persistent storage host shows plenty of free space yet the volume creation fails with the aforementioned errors. ## Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.6 and Higher * Component - Quota ## Cause The error is a logical (quota) rejection from Persistent Storage, not a physical capacity problem. Persistent Storage Service tracks two independent quota tiers per tenant, and the request is rejected if **either** is exceeded: 1. **Tenant-wide `gigabytes` quota** wherein the total volume capacity allowed across all backends in the project is configured. 2. **Per-backend `gigabytes_` quota** wherein a separate cap for each Persistent Storage backend is configured The physical capacity reported by `df` on the storage host reflects the **backend** size. Persistent Storage Service does not consult `df` when admitting a new volume rather it only compares the requested size against the tenant's logical quota counters. Expanding the underlying storage (LUN, volume group, any storage provider volume etc.) increases what the backend can hold but does **not** raise the Persistent Storage tenant quota. The two must be adjusted independently. ## **Diagnostics** 1. **Review the quota and current usage side-by-side for the reported tenant:** ```bash openstack quota show --usage | grep -i gigabyte ``` Output columns are `Limit`, `In Use`, and `Reserved`. Compare: * If the project-wide `gigabytes` row is close to or at its limit, the global cap is the constraint. * If a specific `gigabytes_` row is at its limit, that backend's per-backend cap is the constraint. * A non-zero `Reserved` column indicates in-flight allocations that may be stuck. {% code title="Example Output" overflow="wrap" %} ```bash openstack quota show --usage | grep -i gigabyte | Resource | Limit | In Use | Reserved | | gigabytes | 10000 | 9995 | 0 | | gigabytes_primary_test | -1 | 5000 | 0 | | gigabytes_secondary_test | -1 | 3000 | 0 | | gigabytes_archive_test | -1 | 1500 | 0 | | gigabytes_dev_test | -1 | 495 | 0 | | gigabytes_dr_test | -1 | 0 | 0 | | gigabytes___DEFAULT__ | -1 | 0 | 0 | ``` {% endcode %} **How to read above example output to understand the example configured limits:** * **The first row, `gigabytes`, is the tenant-wide aggregate cap** across all Persistent Storage backends. Limit is 10,000 GB and 9,995 GB is in use; only 5 GB of headroom. Any request larger than 5 GB will be rejected with `VolumeSizeExceedsAvailableQuota`. * **Every `gigabytes_` row is a per-backend cap.** All show `-1` (unlimited), so none of them are blocking the request. If one of them had a finite `Limit` and was at it, *that backend's cap would be the bottleneck instead;* even if the global row had plenty of headroom. Either tier can trip the 413, so check both. * **Counter sanity check: the per-backend `In Use` values should sum to the global `In Use`.** Here, `5000 + 3000 + 1500 + 495 = 9995`, which matches the global row exactly. * **If the sum did not match the global `In Use`**, the counters have drifted from reality. A `cinder-manage quota check/sync` (admin operation) is needed before raising the cap; otherwise the new limit will be measured against incorrect usage data. * **`Reserved` is `0` on every row.** No in-flight allocations are stuck. A non-zero value here indicates volumes mid-creation that may be hung that needs further investigation before raising the quota, since they may eventually succeed or fail and shift `In Use` in either direction. * **Backends showing `0` in use** (`dr_test`, `__DEFAULT__` in the example) are configured but unused. **2. Confirm that the request would have landed on the expected backend.** Check the volume type used by the failing request, and which backend it maps to: ```bash openstack volume type show ``` Look at the `properties` for `volume_backend_name`. The corresponding per-backend quota is `gigabytes_`. **3. Check for orphaned volumes that may be inflating `in_use`.** ```bash openstack volume list --tenant --long --status error openstack volume list --tenant --long --status error_deleting openstack volume list --tenant --long --status available ``` Volumes in `error` or `error_deleting` state, and unattached `available` volumes left over from failed operations, all count against the quota. ## **Resolution** **Pick the path that matches the diagnosis:** * **If the counters are consistent and the cap is genuinely too low.** * Raise the quota that is constraining the request. To raise the project-wide cap: 
openstack quota set --gigabytes <NEW_SIZE_GB> <TENANT_UUID>
* Set the new value above the committed physical capacity for the backend(s), with headroom for growth. {% hint style="info" %} Use `-1` for configuring no limits on a specific backend. {% endhint %} * **If Orphaned volumes are inflating usage.** * Delete leftover volumes from prior failed operations: 
openstack volume delete <VOLUME_UUID>
* For volumes stuck in `error` or `error_deleting`, [reset state](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states) first: 
openstack volume set --state available <VOLUME_UUID>
    openstack volume delete <VOLUME_UUID>
* After cleanup, re-run the `quota show --usage` command to confirm the `in_use` figure dropped. * **If Counters have drifted from reality.** * If the per-backend `in_use` totals don't match the global `gigabytes` `in_use`, the cached quota counters have diverged from the actual volume records and must be reconciled before raising the cap. 
LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d quota check --project-id <TENANT_UUID>
* For Self-Hosted users, the above command can *also* be executed from Control Plane, `cinder-api` pod 
cinder-manage quota check --project-id <TENANT_UUID>
    cinder-manage quota sync --project-id <TENANT_UUID>
* This recomputes usage from the actual volume records and rewrites the quota counters. The first command `check` scans the tenant's volume records, compares them against the cached quota counters, and reports any discrepancies. **It does not modify anything.** * However, `sync` **rewrites** the cached `In Use` and `Reserved` values to match what is actually present in the volume tables. * After `sync` completes, re-run the diagnostic from step 1 of the [#diagnostics](#diagnostics "mention") section; the per-backend `In Use` values should now sum to the global `In Use`. Only then is it safe to raise the cap with the `quota set` command 
openstack quota show --usage <TENANT_UUID> | grep -i gigabyte
    openstack quota set --gigabytes <NEW_SIZE_GB> <TENANT_ID>
# How to Enable LUKS Encryption for Cinder Volumes? ### Problem How to implement LUKS encryption for Cinder volume backends to enhance data security by encrypting storage volumes at rest.. This ensures that sensitive data remains protected, even if the underlying storage devices are compromised. ### Environment * Private Cloud Director Virtualisation - v2025.4 and Higher. * Self-Hosted Private Cloud Director Virtualisation – v2025.4 and Higher. * Component - Storage (Cinder, Barbican, LVM) {% hint style="info" %} With the 2025.6-159 and above releases, Cinder API has integrated native support for volume encryption (covered in steps 6 and 7 in the Procedure section), removing the need for manual configuration. This release currently supports only basic (LUKS/plain) encryption. {% endhint %} ### Procedure 1. Ensure Barbican is running and integrated with Keystone: ```bash $ openstack service list | grep barbican | [service-uuid] | barbican | key-manager | ``` 2. Create an Encryption Key in Barbican. ```bash $ openstack secret store \ --name luks-key \ --payload-content-type 'application/octet-stream' \ --payload-content-encoding base64 \ --payload "$(openssl rand -base64 32)" ## Sample Output +---------------+------------------------------------------------------+ | Field | Value | +---------------+------------------------------------------------------+ | Secret href | https://[PCD_FQDN]/barbican/v1/secrets/[secret-id] | | Name | luks-key | | Created | None | | Status | None | | Content types | {'default': 'application/octet-stream'} | | Algorithm | aes | | Bit length | 256 | | Secret type | opaque | | Mode | cbc | | Expiration | None | +---------------+------------------------------------------------------+ ``` 3. Create an Encrypted Volume Type ``` $ openstack volume type create luks-encrypted +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | description | None | | id | [Volume-Type-Id] | | is_public | True | | name | luks-encrypted | +-------------+--------------------------------------+ ``` {% hint style="info" %} To run `cinder` commands, install `python-cinderclient==9.4.0`. In some OpenStack versions, the `openstack` CLI does not support creating encrypted volume types. In such cases, use the `cinder` CLI instead. {% endhint %} 4. Associate Encryption with the Volume Type ``` $ ~/local/bin/cinder --os-volume-api-version 3.70 encryption-type-create luks-encrypted luks \ --cipher aes-xts-plain64 \ --key-size 256 \ --control-location front-end +--------------------------------------+----------+-----------------+----------+-----------------+ | Volume Type ID | Provider | Cipher | Key Size | Control Location| +--------------------------------------+----------+-----------------+----------+-----------------+ | [Volume-Type-Id] | luks | aes-xts-plain64 | 256 | front-end | +--------------------------------------+----------+-----------------+----------+-----------------+ ``` 5. Confirm volume type and secret: ``` $ openstack volume type list +--------------------------------------+----------------+-----------+ | ID | Name | Is Public | +--------------------------------------+----------------+-----------+ | [Luks-Volume-Type-Id] | luks-encrypted | True | | [Other-Volume-Type-Id] | lvm | True | | [Other-Volume-Type-Id] | __DEFAULT__ | True | +--------------------------------------+----------------+-----------+ $ openstack secret list -c "Secret href" -c Name -c Status -c Algorithm -c "Secret type" -c Mode +--------------------------------------------------------+----------+--------+-----------+------------+------+ | Secret href | Name | Status | Algorithm | Secret type| Mode | +--------------------------------------------------------+----------+--------+-----------+------------+------+ | https://[PCD_FQDN]/barbican/v1/secrets/[secret-uuid] | luks-key | ACTIVE | aes | opaque | cbc | +--------------------------------------------------------+----------+--------+-----------+------------+------+ ``` 6. On the PCD GUI :angle-double-right:Select Cluster blueprint :angle-double-right:Edit desired volume :angle-double-right:add below properties :angle-double-right:save the blueprint. 
encryption_cipher = aes-xts-plain64
   target_ip_address = <IP-address>
   encryption_key_size = 256
   encryption_provider = luks
7. Configure the key manager in the `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` on all cinder hosts. ``` [key_manager] backend = barbican ``` 8. After the above changes, restart the `pf9-cindervolume-base` service on all cinder hosts. ``` $ sudo systemctl restart pf9-cindervolume-base ``` 9. Confirm whether the following highlighted configurations are present in the `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` file: 
[key_manager]
   backend = barbican
   
   [lvm2]
   volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
   volume_backend_name = lvm2
   pf9_identifier_timestamp = <timestamp>
   lvm_type = default
   volumes_dir = /opt/pf9/etc/pf9-cindervolume-base/volumes/
   volume_group = cinder-volumes
   target_protocol = iscsi
   encryption_cipher = aes-xts-plain64
   target_ip_address = <IP-address>
   encryption_key_size = 256
   encryption_provider = luks
10. Create Encrypted volume. ``` $ openstack volume create --size --type luks-encrypted ``` 11. Confirm the Encryption status of the volume. ``` $ openstack volume show --fit +--------------------------------+------------------------------------------------+ | Field | Value | +--------------------------------+------------------------------------------------+ | ...... | | | encryption_key_id | [ENCRYPTION_KEY_ID] | | id | [VOLUME_ID] | | name | [VOLUME_NAME] | | ...... | | | status | available | | type | luks-encrypted | | .. | | | volume_type_id | [VOLUME_TYPE_ID] | +--------------------------------+------------------------------------------------+ ``` # Unable to Start VM due to Multipath issue ## Problem * VM fails to reboot with error: ```ruby multipath -f [DEVICE_ID]: map in use ``` * VM stuck in "spawning" or "rebooting" task state in Nova database * Error in `/var/log/pf9/ostackhost.log`: ```ruby Cannot reboot instance: Command: multipath -f [DEVICE_ID]Exit code: 1 Stderr: ': map in use' ``` ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component - iSCSI storage backend ## Cause The multipath device lock occurs when the device-mapper has active references to a storage volume that cannot be released during VM reboot/shutdown operations. This is typically caused by: **Accumulated State Corruption:** Weeks of failed operations leave: * Stale device-mapper references * Unclosed iSCSI sessions * Incomplete volume detach operations * Orphaned multipath devices When a reboot is initiated, Nova attempts to disconnect volumes by flushing the multipath device. However, accumulated stale references from previous failed operations prevent the flush, causing the `map in use` error even though no active processes are using the device. ## Diagnostics #### Step 1: Verify the Multipath Device Lock ```bash # On the compute node, check multipath status $ sudo multipath -ll # Attempt to flush the specific device (will fail with "map in use") $ sudo multipath -f # Check device-mapper status $ sudo dmsetup ls $ sudo dmsetup info # Check for processes using the device (may show none) $ sudo lsof /dev/mapper/ $ sudo fuser -vm /dev/mapper/ ``` **Expected output:** * `multipath -f` returns exit code 1 with error: `[DEVICE_ID]: map in use` * `lsof` and `fuser` may show no processes, indicating stale kernel references #### Step 2: Check VM State in Nova ```bash # From control plane or with OpenStack credentials $ openstack server show # Check task state (look for stuck states) $ openstack server show -f value -c OS-EXT-STS:task_state # Check VM state $ openstack server show -f value -c OS-EXT-STS:vm_state ``` **Expected output:** * `task_state` may show: `rebooting`, `powering-on`, or similar * `vm_state` may show: `error`, `stopped`, or mismatch with actual hypervisor state #### Step 3: Check iSCSI Sessions ```bash # On compute node $ sudo iscsiadm -m session # Check for stale sessions related to the volume $ sudo iscsiadm -m session -P 3 | grep -A 20 ``` #### Step 4: Review Compute Logs ```bash # Check for the specific error $ sudo grep -A 5 "Cannot reboot instance" /var/log/pf9/ostackhost.log # Check for device cleanup warnings $ sudo grep "leftovers may remain" /var/log/pf9/ostackhost.log ``` ## Resolution #### Method 1: Quick Recovery - VM Rebuild **Use when:** Service restoration is priority, time is limited, production environment **Time:** 15-20 minutes **Steps:** 1. **Document current VM configuration:** ```bash $ openstack server show -f json > vm-config-backup.json $ openstack server volume list > vm-volumes-backup.txt $ openstack port list --server > vm-ports-backup.txt ``` 2. **Verify volume preservation flag:** ```bash # Check if delete_on_termination is false for volumes $ openstack server show # If needed, update the flag $ openstack server volume set --preserve-on-termination ``` 3. **Create volume snapshot through the UI or using following commands:** ```bash # For each attached volume $ openstack volume snapshot create --volume \ --name "snapshot--$(date +%Y%m%d-%H%M%S)" # Wait for snapshot to complete $ openstack volume snapshot list --volume ``` 4. **Delete the VM (preserving volumes):** ```bash $ openstack server delete # Wait for deletion to complete (30-60 seconds) $ openstack server list | grep # Verify volumes still exist $ openstack volume list | grep ``` 5. **Clean up multipath device on compute node:** ```bash # SSH to the compute node $ sudo multipath -r # Verify device is removed $ sudo multipath -ll | grep ``` 6. **Recreate VM with same configuration:** ```bash # Use saved configuration from step 1 $ openstack server create \ --flavor \ --volume \ --nic port-id= \ --availability-zone \ # Monitor creation $ openstack server show ``` 7. **Verify VM is running:** ```bash $ openstack server show # Test connectivity $ ping $ ssh ``` #### Method 2: Manual Recovery - Multipath Cleanup (For Investigation/Non-Production) **Use when:** time available for investigation, want to preserve VM ID **Time:** 45-60 minutes **Steps:** 1. **Stop the VM:** ```bash $ openstack server stop ``` 2. **On compute node, identify the stuck device:** ```bash $ sudo multipath -ll $ sudo ls -la /dev/disk/by-id/ | grep $ sudo dmsetup ls ``` 3. **Check what's using the device:** ```bash $ sudo lsof /dev/mapper/ $ sudo fuser -vm /dev/mapper/ $ mount | grep ``` 4. **Force stop VM in libvirt (if running):** ```bash $ sudo virsh list --all | grep $ sudo virsh destroy ``` 5. **Remove device-mapper device:** ```bash # Try normal remove first $ sudo dmsetup remove # If fails, force remove $ sudo dmsetup remove --force ## If the above --force is failing with "Device or resource busy" and no consumers or backing path $ sudo systemctl restart multipathd ``` 6. **Flush multipath device:** ```bash # Try reload first $ sudo multipath -r # If reload doesn't work, try flush $ sudo multipath -f # Verify removal $ sudo multipath -ll | grep ``` 7. **Rescan SCSI/iSCSI:** ```bash # Rescan SCSI hosts for host in /sys/class/scsi_host/host*; do echo "- - -" > $host/scan done # Rescan iSCSI sessions sudo iscsiadm -m session --rescan ``` 8. **Restart compute service:** ```bash $ sudo systemctl restart pf9-ostackhost # Verify service is running $ sudo systemctl status pf9-ostackhost ``` 9. **Start the VM:** ```bash $ openstack server start # Monitor startup $ openstack server show ``` ## Validation * Before starting the VM verify Multipath Cleanup: ```bash # On compute node $ sudo multipath -ll | grep # Should return no results $ sudo dmsetup ls | grep # Should return no results $ sudo lsof | grep # Should return no results ``` * Once VM is started verify VM Status: ```bash $ openstack server show # Should show: status=ACTIVE, power_state=Running, task_state=None $ openstack server list --all | grep # Should show: Status=ACTIVE # Test connectivity $ ping $ ssh ``` * Verify Compute Service: ```bash $ openstack compute service list # Should show: State=up, Status=enabled ``` * Check for Errors: ```bash # On compute node $ sudo tail -100 /var/log/pf9/ostackhost.log | grep -i error # Should show no recent multipath or device-mapper errors ``` ## Additional Information **Regular Health Checks:** ```bash # Check for VMs in ERROR state $ openstack server list --status ERROR # Check multipath health on compute nodes $ sudo multipath -ll | grep -i fail ``` # Volume Services Flapping Due to EADDRINUSE in pf9-comms Service ## Problem The volume provisioning failed intermittently as the Storage services on one or more hosts were observed to be repeatedly transitioning between `up` and `down` states, leading to a degraded Persistent Storage Service status. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Persistent Storage ## Cause The issue was caused by communication socket exhaustion on the affected node, leading to port binding failures. Specifically, the `pf9-comms` service was repeatedly logging the following error in `/var/log/pf9/comms/comms.log`: ```bash EADDRINUSE encountered during SNI client creation ... retrying in 20000 milliseconds ``` This indicates that the required port was already in use or in TIME\_WAIT state, preventing successful socket binding and disrupting communication between services (including volume driver interactions with the backend). ## Diagnostics 1. The following symptoms were observed: * The`openstack volume service list` showed cinder-volume in down state intermittently. * The `/var/log/pf9/comms/comms.log` contained repeated `EADDRINUSE` errors. 2. The `/var/log/pf9/cindervolume-base` logs on the cinder host showed API communication failures to the Tintri backend, including: ```bash reached maximum retry attempts: 2, reason: Volume driver reported an error: (source: CinderDriver, typeId: com.tintri.api.rest.v310.dto.domain.beans.TintriError, code: ERR-API-8001) ``` Network and Tintri API endpoints were reachable, and credentials were valid, ruling out direct storage backend issues. ## Resolution 1. Restarted the pf9-comms service on the affected Cinder volume host: ```bash $ sudo systemctl restart pf9-comms ``` 2. Post-restart, the EADDRINUSE messages no longer appeared in comms.log. 3. The `cinder-volume` services returned to `up` state: ```bash $ openstack volume service list -c Binary -c Host -c Status -c State +-----------------+----------------------------------+---------+------+ | Binary | Host | Status | State| +-----------------+----------------------------------+---------+------+ | cinder-scheduler| cinder-scheduler-[ID] | enabled | up | | cinder-volume | [CINDER_HOST_ID]@[CINDER_BACKEND]| enabled | up | +-----------------+----------------------------------+---------+------+ ``` 4. Verified volume provisioning functionality and ensured that backend API communication was stable. ## Validation * Monitored the `cinder-volume` service status using the `openstack volume service list`for **30 minutes** and confirmed that the services remained in the `up` state. * No further `EADDRINUSE` errors were observed in `comms.log`. * Volume creation and attachment operations succeeded. ## Additional Information * `EADDRINUSE` indicates that a socket binding operation failed due to the port already being in use. * This is often caused by: * A large number of short-lived TCP connections * Delays in socket cleanup * Insufficient ephemeral port ranges or low timeout settings * Restarting the communication layer (`pf9-comms`) forces the cleanup of stale sockets and refreshes the SNI client sessions. # How to Rename iSCSI Initiator Names to the Existing Hypervisors? ## Problem Changing an iSCSI Initiator Name (IQN) on a hypervisor is an inherently disruptive process. It requires modifying a configuration file and restarting critical services like `iscsid` and `multipathd`. This guide is a clear and reliable procedure to rename these iSCSI initiator names (IQN). ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component - Storage - iSCSI ## Procedure {% hint style="warning" %} To minimise impact, it is **strongly recommended to shut down all VMs** on the hypervisor where you are applying the change before proceeding, starting with the least critical one. {% endhint %} 1. "Enabling maintenance mode" on the hypervisor. On the left-hand side of PCD GUI, click on **Infrastructure** -> select **Cluster Hosts** -> select hypervisor -> Click on the "Other" dropdown section -> [Enable maintenance mode](https://platform9.com/docs/private-cloud-director/private-cloud-director/maintenance-mode). ![Enable Maintenance Mode](https://uploads.developerhub.io/prod/K9yg/4isrhvexg0n87wstdgnksfbk38qi0k8rmdnbft3fo2rf8ismg62gsk7gvcys31tr.png) 2. Migrate all VMs from the hypervisor to another hypervisor. It is recommended to perform cold-migration (Stop VM and perform migrate) to avoid data losses. 3. SSH to the hypervisor, and identify the current iSCSI Initiator Name (IQN): {% hint style="warning" %} This will immediately **break all existing iSCSI sessions**, making any attached virtual machine (VM) disks temporarily unavailable. {% endhint %} ```bash $ sudo cat /etc/iscsi/initiatorname.iscsi ``` 4. Update the `InitiatorName` with the new desired IQN value and save the file. 5. Reboot the hypervisor. ## Validation 1. Service `pf9-ostackhost` and other PF9 services should be running. 2. Run `iscsiadm -m session` on the host to verify new IQN 3. Hypervisor compute services should show up. Use the command given below: ```bash $ openstack compute service list ``` 3. Create a test VM on the hypervisor and validate if the VM build is working as expected. # How to Delete Stale Storage Host Entries in PCD ## Problem Unable to launch new VMs using Self Hosted PCD Management plane UI, Cinder-scheduler pods logs show volume service down errors. {% code title="cinder-scheduler.log" %} ```bash WARNING cinder.scheduler.host_manager [None req-ID] volume service is down. (host: [HOST-UUID@STORAGE-NAME]) ``` {% endcode %} The volume service list has multiple volumes in down state: ```shell $ openstack volume service list +------------------+-----------------------+------+---------+------+-------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-----------------------+------+---------+------+-------------+ | cinder-scheduler |cinder-scheduler-[UUID]| nova | enabled | down | [Timestamp] | +------------------+-----------------------+------+---------+------+------------+ ``` ## Environment * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher. * Component: Storage ## Procedure Steps to delete the stale volumes from the Self Hosted PCD: 1. List the storage volume services, to identify the stale storage host. Execute below command from the Host having the `PersistentStorage` role applied: ```bash $ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service list ``` 2. Validate the above command output with below command output, capture the information about stale storage host UUID. ```bash $ openstack volume service list ``` 3. After confirmation storage volume service can be deleted. Execute the below command from the `PersistentStorage` role host. ```sh $ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service remove cinder-scheduler ``` ## Validation The stale volumes should be removed in the volumes list. It can be verified using: 1. Export the `.rc` file and execute the below command: ``` $ openstack volume service list ``` 2. From the `PersistentStorage` role host., execute the below command to ensure the stale entries are removed. ``` $ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service list ``` # How To Delete a Volume Stuck in "Reserved" State ## Problem Users may encounter situations where a block storage volume cannot be deleted because it is stuck in the `reserved` state. This condition usually occurs when a volume operation (such as attach, detach, or delete) was initiated but did not complete successfully. As a result, the volume remains marked as reserved in the database even though it is not actively attached to any instance. When a volume is in this state, attempts to delete it will fail until the state is manually reset. Example output showing a volume stuck in `reserved` state: The volume service list has multiple volumes in down state: 
$ openstack volume show <VOLUME_UUID>
+--------------------------------+-------------------------------+
| Field                          | Value                         |
+--------------------------------+-------------------------------+
| id                             | [VOLUME_UUID]                 |
| status                         | reserved                      |
| size                           | [VOLUME_SIZE]                 |
| attachments                    | []                            |
| multiattach                    | False                         |
| name                           | [VOLUME_NAME]                 |
| bootable                       | false                         |
| created_at                     | [CREATION_TIMESTAMP           |
+--------------------------------+-------------------------------+
## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher ## Procedure These steps can only be run as admin. Source the RC file with the admin credentials. ```bash $ source ``` 1. Identify the Volume - Retrieve the volume UUID. ```bash $ openstack volume list ``` 2. Reset the Volume state to Available: ```bash $ openstack volume set --state available ``` 3. Delete the Volume: ```bash $ openstack volume delete ``` ## Validation Verify that the volume has been successfully deleted. ```bash $ openstack volume show ``` If the volume is deleted successfully, the command should return a message similar to: ```bash No volume with a name or ID of exists. ``` # How to Identify the LUN ID Associated with an Instance Volume ## Problem To identify the LUN ID on the backend storage that corresponds to a particular instance (VM) volume. This is useful for troubleshooting, storage mapping, and validating multipath connectivity. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component - Storage / Block Storage (Cinder, Multipath, LUN Mapping) ## Procedure 1. List the VM’s attached volumes Run the following command on the compute node hosting the VM: ``` $ virsh domblklist ``` {% code title="Example" %} ```bash $ virsh domblklist vm-101 Target Source ------------------------------------------------ vda /var/lib/libvirt/images/vm1.qcow2 vdb /dev/disk/by-id/dm-name-mpathcq ``` {% endcode %} The above command lists all block devices (disks) attached to the specified VM and shows their mapping to host devices. In the above example output, we can see vdb is a block device mapped via multipath. 2. Obtain the WWID of the multipath device Identify the multipath name from Step 1 (mpathcq) and run: ``` $ multipath -ll | grep -A5 ``` {% code title="Example" %} ```bash $ multipath -ll | grep -A5 mpathcq mpathcq (123123123123123abc25abc) dm-9 3PARdata,VV size=50G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='round-robin 0' prio=50 status=active |- 0:0:1:2 sdv 65:80 active ready running |- 2:0:3:2 sdab 65:176 active ready running |- 0:0:0:2 sdz 65:144 active ready running ``` {% endcode %} * The WWID is `123123123123123abc25abc` (this uniquely identifies the volume). * The devices (sdv, sdab, sdz) show their `Host:Channel:Target:LUN (H:C:T:L)` tuples. 3. Identifying the LUN ID from above output * In the tuples 0:0:1:**2** , 2:0:3:**2** , 0:0:0:**2** → the last number = LUN ID. * So, this volume maps to LUN ID 2 on the storage backend. 4. We can also cross check using by-path ``` $ ls -l _dev_disk_by-path_ | grep ``` {% code title="Example" %} ```bash $ ls -l _dev_disk_by-path_ | grep sdv lrwxrwxrwx 1 root root 9 Aug 20 10:15 pci-0000:3b:00.0-fc-0x1235000abc-lun-2 -> .._.._sdv ``` {% endcode %} Output will show a symlink including lun-2, confirming the backend LUN mapping. ## Additional Information * In the SCSI address format `Host:Channel:Target:LUN (H:C:T:L)` , the last digit (L) represents the LUN ID. * `virsh domblklist` helps you trace which device (e.g., /dev/dm-\*) is attached to a VM. * `multipath -ll` maps that device to its WWID and shows the backend paths, where you can identify the LUN ID. * `/dev/disk/by-path/` provides human-readable symlinks, including names like lun-\, making it easier to confirm the LUN number. # Change Default Volume Type for Block Storage ## Problem How to set a default volume type in PCD per project. * A request to delete the currently configured `default_volume_type` type will fail. (Manual volume-type deletion is possible, but won't work if the value is set as a configuration option.) * If the `default_volume_type_` is misconfigured with the value referring to a non-existent volume-type, requests that rely on the default volume-type (a volume-create request without specifying the volume-type) will result in a HTTP 500 response. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component - Block Storage ## Cause * The `default_volume_type` configuration option is required to have a value. The default value is `__DEFAULT__`. * There must always be at least one volume-type defined in a Cinder installation. The type-delete call enforces this. ## Procedure {% hint style="info" %} A new volume creation request without explicitly specifying the volume type (explicit or in the source), Cinder will look for the default volume type specified in the Database for the specific project and use it; if there isn’t one, it will continue using the default type from `cinder.conf`. {% endhint %} 1. Use the below cinder commands to set, unset or list the default volume type per project. {% hint style="warning" %} Cinder binary needs to be downloaded and installed on the machine from which you are running the command, as the OpenStack client won't work to set/unset the volume type. {% endhint %} ```bash # To set default volume type per project $ cinder default-type-set # To unset default volume type per project $ cinder default-type-unset # To list default volume type per project $ cinder default-type-list [--project ] ``` 2. To set a default volume type in `cinder.conf` . * Edit the file`/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` to add the below in the`[DEFAULT]` section: ```bash $ vim /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf [DEFAULT] #Add below to existing values default_volume_type = ``` 3. Restart Block Storage (cinder) service on the host ```bash $ systemctl restart pf9-cindervolume-base ``` ## Validation Create a new volume in the project without specifying the volume type, and the new volume should be created in the default volume configured at project level. Verify the same using below: ```bash $ openstack volume show -c type ``` # New Storage Configuration in Cluster Blueprint is Not Reflecting Persistent Role Host ### Problem The newly added hosts which have the persistent storage role applied are stuck in the converging state. ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Blueprint and Persistent Storage ### Cause The cinder configurations applied from the clusterblueprint is not getting reflected at the cinder backend host due to syntax errors in `cinder.conf` ### Diagnostics Check the hostagent logs for the errors: {% code title="hostagent.logs" %} ```bash pf9_app.py ERROR - pf9-cindervolume-config:get_config failed: Source contains parsing errors: '/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf' [line 147]: 'volume_dri\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): ``` {% endcode %} ### Resolution In the new host, having the persistent storage role applied, perform the following manual steps to apply the cinder configurations: 1. Edit the Cinder configuration file in the cinder host `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` 2. Add the new backend sections in the end: 
[HOSTNAME]
   volume_driver = cinder.volume.drivers.vmstore.nfs.VmstoreNfsDriver
   volume_backend_name = [VOLUME_NAME]
   nas_host = [HOST_IP]
   vmstore_user = cinder
   nas_share_path = [PATH]
   vmstore_password = [VMSTORE_PASSWORD]
   nfs_mount_options = vers=3,proto=tcp,lookupcache=pos,nolock,noacl
   nfs_snapshot_support = true
   vmstore_rest_address = [VMSTORE_ADDRESS]
   vmstore_qcow2_volumes = true
   vmstore_rest_retry_count = 1
3. Update the enabled backends In the \[DEFAULT] section, append the new backends: 
enabled_backends = [BACKEN_1],[BACKEN_2],[BACKEND_3]
4. Restart the pf9-cindervolume-base service: ```bash sudo systemctl restart pf9-cindervolume-base.service ``` ## Validation 1. Check that the new backends are mounted on the host: 
$ mount | grep -E '10.18.8.6|10.18.8.7'
2. Check backend pool 
$ openstack volume backend pool list --long | grep -E '[CINDER_HOSTNAME]'
3. If both commands show the mounts and the new backends, the configuration is applied successfully. # Unable to Create Clone Volumes From Source Volume ### Problem The VM shows an attached bootable volume in the PCD UI. However, the actual disk file being used by the hypervisor does **not** match the volume registered in the storage service. This resulted in Failure when creating a clone of the volume present in the Cinder but without having a valid backend file. ### Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Private Cloud Director Kubernetes – v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.6 and Higher * Component - Block Storage ### Cause This issue is caused by a metadata inconsistency between the Compute service and Block storage service which likely occurred during a failed volume operation. The file mapped for `/dev/vda` bootable volume as per `virsh dumpxml` does not exist in Block storage, and therefore is not manageable by Block Storage APIs. Meanwhile, the Cinder-registered volume has no backend file present on the Shared mount used for tintri, confirming metadata drift between Compute Service and Block storage service . #### Diagnostics The following command outputs confirm the mismatch between the control plane metadata and the data plane reality. 1. **Verify the Volume Status:** This shows that the reported boot volume is `in-use`, but a search for the actual disk file will fail in the Block storage backend: ``` openstack volume show --fit +--------------------------------+----------------+ | Field | Value | +--------------------------------+----------------+ | name | [VOLUME_NAME] | | status | in-use | +--------------------------------+----------------+ ``` 2. **Inspect the VM XML on the Hypervisor:** Log in to the compute host where the VM is running and inspect the VM domain XML. This reveals the actual disk file in use. \ **Note:** The output shows the *serial* matching the Cinder-reported ID, but the *source file* uses the unmanaged ID 
$ sudo virsh dumpxml <VM_UUID> | grep -B5 -i volume
   <...>
     <disk type='file' device='disk'>
       <driver name='qemu' type='raw' cache='none' io='native'/>
       <source file='/path/to/volume-<VOLUME_UUID_2>' index='2'/>
       <backingStore/>
       <target dev='vda' bus='virtio'/>
   <...>
3. **Confirm the Actual Volume is Unmanaged:** Searching for the volume ID found in the `virsh dumpxml` confirms that **Block storage service** does not know about this active disk. ```bash $ openstack volume show --fit No volume with a name or ID of '' exists. ``` 4. **Check the Disk Format:** Inspecting the file on the hypervisor confirms it is a `raw` disk 
$ qemu-img info /path/to/volume-<VOLUME_UUID_2>
   volume-<VOLUME_UUID_2>
   image: /path/to/volume-<VOLUME_UUID_2>
   file format: raw
   virtual size: 160 GiB (171798691840 bytes)
   disk size: 162 GiB
The Nova domain XML on the compute host points to the correct, functional disk file (ID **\**), but that volume is missing from the **Cinder** database, leaving it an **unmanaged** resource. The original Cinder volume record **\** is a ghost record with no physical file, which is why attempts to clone or snapshot fail. #### Resolution The resolution involves directly recovering the active disk file, converting it to a standard image format, and creating a new Cinder-managed volume and instance from that recovered image. 1. On the compute hypervisor, copy the active, **raw** disk file to a temporary location. ```bash $ cp /path/to/volume- /tmp/volume-.raw ``` 2. Convert the **raw** disk image to the recommended **qcow2** format using **qemu-img** to save space and enable sparse image features. ```bash $ qemu-img convert -f raw -O qcow2 /tmp/volume-.raw /tmp/.qcow2 ``` 3. Upload the recovered *.qcow2* file to **Image Library**. ```bash $ openstack image create --disk-format qcow2 --container-format bare \ --property os_type='windows' \ --file /tmp/.qcow2 \ "" ``` 4. Launch a new instance using the newly created **Image Library Service** image to create a boot volume. 5. Verify that the new instance, boots successfully and operates normally. #### Additional Information * This recovery method bypasses the corrupted Block Storage metadata by extracting the live, functional disk data from the hypervisor. * The cleanup of the original instance and its associated volume records is required to fully resolve the metadata corruption. * If the original instance has a second attached volume, the secondary non-bootable volume should be carefully detached and re-attached to the recovered instance if it contains user data. # Shared Storage Requirements for VMHA ## Problem VMHA evacuation behavior depends on the VM's storage type and the shared storage mounts. ## Background VMHA can trigger an evacuation when a compute host is: * Powered off * Network isolated (management plane can’t reach the host) Storage configuration affects whether the evacuated VM preserves data. It can also introduce NFS lock conflicts. ## VM storage types ### Ephemeral (local instance disk) * VM disk data lives under `/opt/data/instances`. * If `/opt/data/instances` is on shared storage and mounted on every hypervisor host, VM data is preserved during evacuation. * If `/opt/data/instances` is not shared, evacuation may still succeed, but the instance disk data is lost. ### Volume-backed (boot from volume) * VM volumes are mounted under a path like `/opt/pf9/data/state/mnt/`. * VM disk data is not stored under `/opt/data/instances`. * Whether `/opt/data/instances` is shared does not affect data retention for volume-backed VMs. ## NFS locking (`nolock`) considerations If your shared storage backend is NFS, mount options can change failover behavior: * With `nolock`, the evacuated VM will start without NFS lock checks. * Without `nolock`, the evacuated VM will fail to start if the old VM still holds locks. {% hint style="warning" %} `nolock` disables NFS file locking. This can allow two VM processes to write the same files. Use it only if you have fencing in place and the source VM cannot keep writing to the same storage. {% endhint %} ## Expected VM Evacuation Outcomes The table below shows the expected VM evacuation status as determined by the VM type (ephemeral or volume-backed), whether `/opt/data/instances` is shared (with or without nolock) for ephemeral VMs, and the host failure type (power loss or network isolation). | **VM Type** | **`/opt/data/instances` Mount Status** | **Host Failure Type** | **VM Evacuation Status** | | ------------- | -------------------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ephemeral | Not shared/mounted | Power loss | Success, with data loss | | Ephemeral | Not shared/mounted | Network isolation | Success, with data loss | | Ephemeral | Shared and mounted with `nolock` | Power loss | Success, no data loss | | Ephemeral | Shared and mounted with `nolock` | Network isolation | Success, no data loss. VMs are paused on the source host | | Ephemeral | Shared and mounted without `nolock` | Power loss | Success, no data loss | | Ephemeral | Shared and mounted without `nolock` | Network isolation |

Failure with Failed to write lock

VM still running on isolated host

| | Volume-backed | Doesn't matter | Power loss | Success, no data loss | | Volume-backed | Doesn't matter | Network isolation (NFS without `nolock`) |

Failure with Failed to write lock

VM still running on isolated host and continues to hold /opt/pf9/data/state/mnt/\/volume-\

| | Volume-backed | Doesn't matter | Network isolation (NFS with `nolock`) | Success, no data loss. VMs are paused on the source host | # VM Cloning Fails For Self-Service Users ### Problem * VM clone operation gets stuck at the `VM snapshot initiated` phase and never progresses to subsequent phases. * The instance eventually goes into ERROR with scheduler retries exceeded.\ Relevant error line seen in logs: ``` ERROR nova.scheduler.utils ... Exceeded max scheduling attempts 3 ... Failed to call glance method data ``` ### Environment * Private Cloud Director Virtualization - v2025.10 **till v2026.1-260** * Self-Hosted Private Cloud Director Virtualization - v2025.10 **till v2026.1-260** * Component: Compute Service * Self-Service Users ### Cause * The cloning attempts are carried out by a **self-service user**, which is where the cloning workflow encounters issues. Sometimes, volume cloning deletes the snapshot before the virtual machine finishes downloading the image. * Based on the behaviour, the self-service user does not appear to have the required privileges to complete the full snapshot/image lifecycle used in the cloning flow (snapshot creation + image access during boot + snapshot cleanup). ### Resolution * Bug **PCD-5572** has been reported, and the fix is available from **v2026.1-309**. #### Workaround 1. Assign admin-level privileges for cloning. To use the cloning feature reliably, the **self-service user** will need **admin-level privileges.** 2. If admin privileges cannot be provided to the self-service user, a workable alternative is: 1. Take a **volume snapshot** of the source VM’s boot volume 2. Create a new VM from that **volume snapshot** # Volume Creation or Clone Requests Delayed Due to Storage Service Flapping ### Problem The Block Storage service on various hosts exhibits "flapping" behavior, where the service frequently transitions between up and down states in the scheduler. This causes volume tasks to fail or get stuck because the service is taking too long to report its status, sometimes taking over 15 minutes to finish a task that should only take seconds ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component - Block Storage ### Cause The default Linux kernel setting for `net.ipv4.tcp_retries2` is **`15`**. In the event of temporary network latency or a hiccup between the Block Storage service and the RabbitMQ server, a value of **`15`** causes the TCP stack to wait between 13 to 30 minutes before timing out a broken connection.\ Because of this, the Block Storage service completely freezes. It stops sending heartbeats, which makes the Scheduler think the host is dead and causes the flapping (switching between UP and DOWN). ### Diagnostics Review the following logs to confirm this issue: {% hint style="info" %} In SaaS environments, Step 1 verification is handled by Platform9 on the backend. Please contact Platform9 Support for verification details if needed. {% endhint %} Step 1: Cinder Scheduler Logs (`cinder-scheduler` log) Look for warnings indicating the volume service is down for specific host IDs: {% code overflow="wrap" %} ```bash WARNING cinder.scheduler.host_manager [None [req-UUID] [USER_ID] [PROJECT_ID] - - - -] volume service is down. (host: [STORAGE_UUID]@[BACKEND_STORAGE_HOST]) ``` {% endcode %} Step 2: Block Storage Logs (`/var/log/pf9/cindervolume-base.log`) Look for report\_state outlasting its interval: {% code overflow="wrap" %} ```bash WARNING oslo.service.backend.eventlet.loopingcall ... Function 'cinder.service.Service.report_state' run outlasted interval by 914.44 sec ``` {% endcode %} Database/Messaging Errors (`/var/log/pf9/cindervolume-base.log`) Identify intermittent DB connection losses or RabbitMQ heartbeat misses: {% code overflow="wrap" %} ```bash oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') AMQP server on 127.0.0.1:5673 is unreachable: Server unexpectedly closed connection ``` {% endcode %} ### Resolution In order to tune this , we update the kernel parameter `net.ipv4.tcp_retries2` from `15` to **`7`** so the service stops hanging on dead connections and reconnects faster. Step 1: Create the persistence configuration file ```bash $ echo "net.ipv4.tcp_retries2 = 7" | sudo tee /etc/sysctl.d/99-cinder-tuning.conf ``` Step 2: Apply the parameter to the live kernel ```bash $ sudo sysctl -p /etc/sysctl.d/99-cinder-tuning.conf ``` Step 3: Restart the affected service ```bash $ sudo systemctl restart pf9-cindervolume-base.service ``` Step 4: Verify the change ```bash $ sudo sysctl net.ipv4.tcp_retries2 ``` ### Validation 1. Verify Kernel Value: Run `sysctl net.ipv4.tcp_retries2` and confirm the output is **`7`** 2. Check Persistence: Confirm `/etc/sysctl.d/99-cinder-tuning.conf` contains the correct value. 3. Monitor Logs: Ensure `/var/log/pf9/cindervolume-base.log` no longer shows **report\_state warnings exceeding 60 seconds**. ### Additional Information This issue is currently being worked on and is tracked under enhancement PCD-5500. Until an official recommendation is available, this manual tuning procedure must be followed for all affected Block Storage hosts. # VM Evacuation Fails Due to Volume Stuck in attaching State ## Problem Evacuation of a VM fails, leaving the instance in `ERROR` state. One of the attached volumes remains stuck in the `attaching` state, preventing Nova from completing the evacuation. The VM cannot boot because the required volume is not successfully attached. ## Environment * Storage Service * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component: Storage Service ## Cause The evacuation fails because one of the attached volumes is stuck in the `attaching` state. During evacuation, Nova requests volume service to attach all required volumes to the target compute host. If any volume does not reach `available` or `downloading` state and remains stuck, Nova cannot proceed with booting the instance, resulting in `ERROR` state. ### Diagnostics * **Check the VM state:** ```bash $ openstack server show ``` The status field shows ERROR and `volumes_attached` lists all volumes; one may be stuck in `attaching` state.
* **Check volume status**: ```bash $ openstack volume show ``` Events show `Invalid volume` error for volumes stuck in `attaching`
* **Verify volume attachment attempts**: ```bash $ openstack server event show ``` Shows traceback with `Invalid volume` due to `attaching` state ## Resolution 1. Reset volumes to `available` ``` $ openstack volume set --state available $ openstack volume set --state available $ openstack volume set --state available $ openstack volume set --state available $ openstack volume set --state available ``` 2. Recreate the VM from the bootable volume ``` $ openstack server create \ --flavor \ --volume \ --nic net-id= \ --availability-zone nova \ ``` 3. Reattach all data volumes ``` $ openstack server add volume $ openstack server add volume $ openstack server add volume $ openstack server add volume ``` ## Validation 1. Confirm VM status is ACTIVE: ```bash $ openstack server show | grep status ``` 2. Log into the guest OS and verify: 1. VM boots successfully 2. All data volumes are visible and mounted correctly ### Additional Information * This issue occurs when Cinder volumes remain in transitional states (`attaching`/`reserved`) due to incomplete detach/attach operations. * Using `openstack volume set --state available` resets the volumes safely when no I/O is in progress. * Preventive measures include monitoring volume states during evacuation and ensuring no stale Block Device Mappings exist in Nova for critical volumes. * Similar issues may occur if live migration or evacuations are attempted while volumes are actively performing operations. # How to Direct New Volumes to a Specific Block Storage Backend in PCD ## Problem When two or more Block Storage backends are configured under the same volume type in PCD, new volumes and VM boot disks may consistently land on one backend even though another backend is available — or even newer and emptier. This occurs because the Block Storage scheduler selects backends using a capacity-based algorithm, not a round-robin or "newest first" policy. This article explains how the scheduler makes its selection, how to confirm which backend is being chosen and why, and three options for directing new volumes to a specific backend. ## Prerequisites * OpenStack CLI or pcdctl is configured and the environment `.rc` file sourced * The volume type name used by the affected VM or volume ## Understanding How the Scheduler Selects a Backend The Block Storage scheduler uses the `CapacityWeigher` algorithm by default. When a new volume is requested against a volume type that maps to multiple backends, the scheduler ranks all eligible backend pools by **absolute free space in GB** and places the volume on the pool with the most free space. This means: * For example, a backend with 73 TB free is always selected over a backend with 59 TB free — even if the 59 TB backend is brand new and the 73 TB backend is older and more heavily used. These are illustrative values; the same logic applies to any two backends with different free capacity. * Adding a new backend does not cause the scheduler to prefer it. The new backend only becomes preferred once it reports more free space than all other backends sharing the same volume type. * The behavior is by design and is not a bug. Two backends are treated as members of the same logical pool when both stanzas in the storage configuration share the same `volume_backend_name` value. The volume type resolves to that shared name, and the scheduler picks between the two pool members on every volume creation request. ## Procedure {% stepper %} {% step %} #### Step 1 — Confirm which backends are active and check free capacity Run the following commands to see all backends registered under the Block Storage service and each backend's reported capacity. {% code title="OpenStack CLI" %} ```bash $ openstack volume service list $ openstack volume backend pool list --long ``` {% endcode %} {% code title="Sample Output — volume service list" %} ```bash +------------------+--------------------------------------------+---------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+--------------------------------------------+---------+---------+-------+----------------------------+ | cinder-volume | [CINDER_HOST_UUID]@[BACKEND_A] | nova | enabled | up | [TIMESTAMP] | | cinder-volume | [CINDER_HOST_UUID]@[BACKEND_B] | nova | enabled | up | [TIMESTAMP] | +------------------+--------------------------------------------+---------+---------+-------+----------------------------+ ``` {% endcode %} {% code title="Sample Output — volume backend pool list --long (key fields)" %} ```bash | [CINDER_HOST_UUID]@[BACKEND_A]#[NFS_SHARE_PATH_A] | free_capacity_gb='59012', total_capacity_gb='59150', allocated_capacity_gb='0', volume_backend_name='[VOLUME_BACKEND_NAME]' | | [CINDER_HOST_UUID]@[BACKEND_B]#[NFS_SHARE_PATH_B] | free_capacity_gb='73218', total_capacity_gb='107812', allocated_capacity_gb='12252', volume_backend_name='[VOLUME_BACKEND_NAME]' | ``` {% endcode %} The backend with the higher `free_capacity_gb` value is the one the scheduler will select for all new volumes until the free capacity values change. {% endstep %} {% step %} #### Step 2 — Confirm both backends share the same volume type Verify that the volume type in use maps to a shared `volume_backend_name` that both backends report. {% code title="OpenStack CLI" %} ```bash $ openstack volume type show ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------+----------------------------------------------+ | Field | Value | +--------------------+----------------------------------------------+ | name | [VOLUME_TYPE_NAME] | | properties | volume_backend_name='[VOLUME_BACKEND_NAME]' | +--------------------+----------------------------------------------+ ``` {% endcode %} If both backends in the `volume backend pool list` output report the same `volume_backend_name`, they are competing for placement under the same volume type and the `CapacityWeigher` result in Step 1 explains the selection. {% endstep %} {% step %} #### Step 3 — Choose an option to control placement Three options are available depending on the goal. Select the option that best matches the use case: | Option | Best for | Effort | Reversible? | | ---------------------------------------- | ---------------------------------------------------------------------- | ------ | ------------------------------ | | **A — Disable the unwanted backend** | Temporarily forcing all new volumes to one backend | Low | Yes — re-enable when done | | **B — Tune the scheduler weigher** | Biasing placement toward a specific backend without service disruption | Medium | Yes — revert config when done | | **C — Split into separate volume types** | Deterministic, permanent per-backend placement control | High | Partial — volume types persist | | {% endstep %} | | | | | {% endstepper %} | | | | *** ### Option A — Temporarily Disable the Unwanted Backend Disabling a Block Storage backend service prevents the scheduler from selecting it for new volume placements. Existing volumes on the disabled backend continue to function normally — reads, writes, snapshots, clones, resizes, and deletes on existing volumes are not affected. All new volume creation requests will land on the remaining enabled backend. {% hint style="warning" %} Disabling a backend service affects all volume creation across the entire cluster for that volume type — not just for a specific project or VM workflow. All teams using the same volume type will be affected until the service is re-enabled. Admin-level API credentials are required to run these commands. {% endhint %} {% stepper %} {% step %} #### Step A1 — Disable the backend to deprioritize Replace `` with the full host string from the `openstack volume service list` output — for example, `[CINDER_HOST_UUID]@[BACKEND_B]`. {% code title="OpenStack CLI" %} ```bash $ openstack volume service set \ --disable \ --disable-reason "" \ ``` {% endcode %} {% code title="Sample Output" %} ```bash (no output on success) ``` {% endcode %} {% endstep %} {% step %} #### Step A2 — Confirm the service is disabled {% code title="OpenStack CLI" %} ```bash $ openstack volume service list | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash | cinder-volume | [CINDER_HOST_UUID]@[BACKEND_B] | nova | disabled | up | [TIMESTAMP] | ``` {% endcode %} The `Status` column shows `disabled`. New volumes will now be placed on the remaining enabled backend. {% endstep %} {% step %} #### Step A3 — Re-enable the backend when the rebalance is complete {% code title="OpenStack CLI" %} ```bash $ openstack volume service set --enable ``` {% endcode %} {% code title="Sample Output" %} ```bash (no output on success) ``` {% endcode %} After re-enabling, the `CapacityWeigher` resumes selecting between both backends based on free capacity. {% endstep %} {% endstepper %} *** ### Option B — Tune the Scheduler Weigher The `CapacityWeigher` uses a backend's **virtual free capacity** as its weight. Virtual free capacity is calculated as: ``` virtual_free_capacity = (total_capacity_gb × max_over_subscription_ratio) − allocated_capacity_gb ``` Increasing `max_over_subscription_ratio` on the preferred backend inflates its virtual free capacity, making the scheduler rank it above the other backend. This change does not affect actual storage or data — it only changes how the scheduler calculates the weight. The `max_over_subscription_ratio` setting is part of the backend configuration managed by Blueprint. To make this change via Cluster Blueprint, navigate through Infrastructure -> Cluster Blueprint -> Persistent Storage Connectivity -> Select the desired Volume Backend -> Edit the `max_over_subscription_ratio` field as per requirement. {% hint style="warning" %} Over-subscription ratio changes inflate reported virtual capacity. Set a realistic value that does not exceed the actual expected workload on the backend. Revert `max_over_subscription_ratio` to its original value once placement has been rebalanced to the desired state. {% endhint %} **Example calculation:** If `[BACKEND_A]` has 59 TB total capacity and `max_over_subscription_ratio = 2`: * Current virtual free ≈ `59,150 × 2 − 0 = 118,300 GB` If `[BACKEND_B]` has 108 TB total with 12,252 GB allocated and `max_over_subscription_ratio = 2`: * Current virtual free ≈ `107,812 × 2 − 12,252 = 203,372 GB` To make `[BACKEND_A]` rank higher, increase its ratio to 4: * New virtual free for `[BACKEND_A]` ≈ `59,150 × 4 − 0 = 236,600 GB` — now higher than `[BACKEND_B]` After the changes are applied, verify the scheduler now selects the preferred backend by checking that new volumes land on `[BACKEND_A]`: {% code title="OpenStack CLI" %} ```bash $ openstack volume show | grep "os-vol-host-attr:host" ``` {% endcode %} {% code title="Sample Output" %} ```bash | os-vol-host-attr:host | [CINDER_HOST_UUID]@[BACKEND_A]#[NFS_SHARE_PATH_A] | ``` {% endcode %} *** ### Option C — Split Backends into Separate Volume Types This is the permanent, deterministic solution. Assigning each backend a unique `volume_backend_name` and a matching volume type allows explicit control over which backend receives each volume creation request. VM migration tools and provisioning automation can then specify the exact target backend by selecting the corresponding volume type. {% stepper %} {% step %} #### Step C1 — Request backend configuration split * Set `[BACKEND_A]` stanza: `volume_backend_name = ` * Set `[BACKEND_B]` stanza: `volume_backend_name = ` Example naming convention: * `svr-06` → `volume_backend_name = vmstore_06` * `svr-07` → `volume_backend_name = vmstore_07` Post this change, restart the affected Block Storage volume services. Wait for confirmation before proceeding to Step C2. {% endstep %} {% step %} #### Step C2 — Create a volume type for each backend After the configuration change is live, create a volume type for each backend that maps to its new unique `volume_backend_name`. {% code title="OpenStack CLI" %} ```bash $ openstack volume type create \ --property volume_backend_name= \ $ openstack volume type create \ --property volume_backend_name= \ ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------+----------------------------------------------+ | Field | Value | +--------------------+----------------------------------------------+ | name | [NEW_VOLUME_TYPE_NAME_A] | | properties | volume_backend_name='[NEW_BACKEND_NAME_A]' | +--------------------+----------------------------------------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step C3 — Verify the new volume types are visible Confirm both new volume types appear in the volume type list. {% code title="OpenStack CLI" %} ```bash $ openstack volume type list ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+---------------------------+ | ID | Name | +--------------------------------------+---------------------------+ | [UUID] | [NEW_VOLUME_TYPE_NAME_A] | | [UUID] | [NEW_VOLUME_TYPE_NAME_B] | | [UUID] | [LEGACY_VOLUME_TYPE_NAME] | +--------------------------------------+---------------------------+ ``` {% endcode %} The legacy volume type (e.g., `noi_vmstore`) can remain active for existing volumes. Existing volumes are not affected by this change and continue to use the original backend each volume was created on. {% endstep %} {% step %} #### Step C4 — Update provisioning tools to use the new volume types Update any VM migration tools or provisioning automation to specify `` or `` when creating volumes or migrating VMs. The tool or workflow that previously used `` must now specify the exact target volume type to deterministically control backend placement. After updating, verify a new volume lands on the correct backend: {% code title="OpenStack CLI" %} ```bash $ openstack volume show | grep "os-vol-host-attr:host" ``` {% endcode %} {% code title="Sample Output" %} ```bash | os-vol-host-attr:host | [CINDER_HOST_UUID]@[NEW_BACKEND_NAME_A]#[NFS_SHARE_PATH_A] | ``` {% endcode %} {% endstep %} {% endstepper %} # VM Fails to Start After Live Migration Failure Due to iSCSI BDM Corruption ## Problem After a live migration fails on a host using iSCSI storage (NetApp ONTAP), the affected VM remains in `error` state and cannot be started or hard-rebooted. The VM was running normally before the migration was initiated. Post-migration-failure symptoms include: * The VM shows `status: ERROR` or `vm_state: error` in PCD * Hard reboot attempts fail without the VM coming back online * The VM's volume remains attached in the platform but the instance cannot mount it on start * `virsh list --all` on the source compute host shows the VM as absent or in a failed domain state This article covers the recovery process for the **Block Device Mapping (BDM) corruption** that occurs as a consequence of the failed migration. If `libvirtd` on the compute host is also in a failed state and `virsh` commands are unresponsive, resolve that condition first by following [Virsh Commands Run Infinitely Without Output / VM Console Inaccessible](https://platform9.com/kb/pcd/generic/virsh-commands-run-infinitely-without-output-vm-console-inaccessible) before proceeding with the steps below. ## Environment * Private Cloud Director Virtualization - **All versions** * Self-Hosted Private Cloud Director Virtualization - **All versions** * Component: Compute Service, iSCSI Storage (NetApp ONTAP) ## Cause When a live migration is initiated, the Compute Service runs `pre_live_migration` on the destination host. This step maps the volume to the destination host's igroup on the NetApp array and updates the Block Device Mapping (BDM) record in the Nova database with the destination LUN ID. If the migration fails after `pre_live_migration` runs but before the rollback completes, two problems persist: 1. **BDM has the wrong LUN ID** — The BDM `connection_info` field was updated with the destination LUN ID. The source host's igroup has a different LUN ID assigned for the same volume. The Compute Service uses the BDM to attach the disk when starting the VM, so the mismatch prevents the VM from mounting its boot volume. 2. **Volume is dual-mapped on the NetApp array** — The volume remains mapped to the destination host's igroup even though the migration never completed. ## Diagnostics {% stepper %} {% step %} #### Step 1 — Confirm the VM state and identify the attached volume Retrieve the VM state and the UUID of the affected volume. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c name -c "OS-EXT-SRV-ATTR:host" $ openstack server volume list ``` {% endcode %} {% code title="Sample Output" %} ```bash +---------------------------+-----------------+ | Field | Value | +---------------------------+-----------------+ | OS-EXT-SRV-ATTR:host | [SOURCE_HOST] | | name | [VM_NAME] | | status | ERROR | +---------------------------+-----------------+ +--------------------------------------+----------+--------------------------------------+------+ | ID | Server ID| Volume ID | Dev | +--------------------------------------+----------+--------------------------------------+------+ | [VOLUME_UUID] | [VM_UUID]| [VOLUME_UUID] | vda | +--------------------------------------+----------+--------------------------------------+------+ ``` {% endcode %} Note the `[VOLUME_UUID]` for use in the following steps. {% endstep %} {% step %} #### Step 2 — Check the BDM for the wrong LUN ID Connect to the Nova database and retrieve the `connection_info` for the volume. The `target_lun` field is the LUN ID the Compute Service will use when attaching the volume to the VM. {% hint style="info" %} **Self-Hosted only:** Database access requires access to the MySQL pods running in the control plane. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to check the BDM. {% endhint %} Open a MySQL session on the Nova database using the following commands on the control plane node. Replace ``, ``, and `` with the values for the affected region: {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D nova" ``` {% endcode %} {% code title="Sample Output" %} ```bash Welcome to the MariaDB monitor. ... MySQL [nova]> ``` {% endcode %} At the `MySQL [nova]>` prompt, run the following query: {% code title="Nova DB" %} ```sql SELECT id, instance_uuid, volume_id, JSON_EXTRACT(connection_info, '$.data.target_lun') AS current_lun, JSON_EXTRACT(connection_info, '$.data.target_luns') AS current_luns FROM block_device_mapping WHERE volume_id = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ``` +------+--------------------------------------+--------------------------------------+-------------+--------------+ | id | instance_uuid | volume_id | current_lun | current_luns | +------+--------------------------------------+--------------------------------------+-------------+--------------+ | [ID] | [VM_UUID] | [VOLUME_UUID] | 5 | [5, 5, 5, 5] | +------+--------------------------------------+--------------------------------------+-------------+--------------+ ``` {% endcode %} Note the `current_lun` value. This will be compared against the source host's igroup LUN ID in Step 3 to confirm whether the BDM is corrupted. {% endstep %} {% step %} #### Step 3 — Find the correct LUN ID from the NetApp array Navigate to the NetApp ONTAP System Manager UI and locate the LUN for the affected volume. 1. Go to **Storage → LUNs** and find the LUN by name or path (the volume UUID is typically part of the LUN path) 2. Click into the LUN and open the **LUN Mappings** tab 3. Review the list of igroups the LUN is mapped to The expected state after a failed migration is that the LUN appears in **two igroup entries** — one for the source host and one for the destination host. Each igroup will have a different LUN ID assigned. Note: * The **source host's igroup LUN ID** — this is the correct value and must match the BDM * The **destination host's igroup LUN ID** — this is the stale entry to be removed in Step 2 of the Recovery If the source host's igroup is **not listed** at all, the mapping was removed by a failed hard reboot attempt. Proceed to Step 4 before fixing the BDM. {% endstep %} {% step %} #### Step 4 — Re-add the source igroup mapping if missing Skip this step if the source host's igroup is still listed on the LUN. If the source igroup mapping is absent: 1. In the NetApp ONTAP System Manager UI, go to **Storage → LUNs**, open the LUN, and click the **LUN Mappings** tab 2. Click **Map** and select the source host's igroup 3. After saving, note the LUN ID assigned to the source host's igroup — NetApp assigns this automatically 4. Return to the LUN Mappings tab and confirm the source igroup now appears with its assigned LUN ID After re-mapping, rescan iSCSI on the source host to pick up the new path: {% code title="Compute Host" %} ```bash $ sudo iscsiadm -m session --rescan $ sudo multipath -ll ``` {% endcode %} {% code title="Sample Output" %} ```bash [WWID] dm-[N] NETAPP,LUN C-Mode size=[SIZE]G features='2 pg_init_retries 50' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- [H:C:T:L] sd[X] [MAJOR:MINOR] active ready running | |- [H:C:T:L] sd[X] [MAJOR:MINOR] active ready running `-+- policy='service-time 0' prio=10 status=enabled |- [H:C:T:L] sd[X] [MAJOR:MINOR] active ready running `- [H:C:T:L] sd[X] [MAJOR:MINOR] active ready running ``` {% endcode %} Confirm the LUN appears under the correct dm device and that paths show `active ready` rather than `failed faulty`. {% endstep %} {% endstepper %} ## Resolution ### Fix the BDM and Restore VM Operation {% stepper %} {% step %} #### Step 1 — Update the BDM with the correct LUN ID Using the source host's igroup LUN ID from Diagnostics Step 3, update the `connection_info` in the Nova database. The `target_luns` array has one entry per iSCSI path — set all entries to the same LUN ID (NetApp typically presents 4 paths per volume). {% hint style="info" %} **Self-Hosted only:** Database write access requires access to the MySQL pods running in the control plane. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to apply the BDM correction. {% endhint %} Open a MySQL session using the same commands from Diagnostics Step 2 before running the UPDATE below. {% hint style="warning" %} Always verify the `id`, `instance_uuid`, and `volume_id` values from the SELECT query in Diagnostics Step 2 before running the UPDATE. An incorrect UPDATE to the BDM can prevent the VM from attaching any volume. {% endhint %} {% code title="Nova DB" %} ```sql UPDATE nova.block_device_mapping SET connection_info = JSON_SET( JSON_SET(connection_info, '$.data.target_lun', ), '$.data.target_luns', JSON_ARRAY(, , , ) ) WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash Query OK, 1 row affected ``` {% endcode %} Verify both fields updated correctly: {% code title="Nova DB" %} ```sql SELECT instance_uuid, JSON_EXTRACT(connection_info, '$.data.target_lun') AS bdm_lun, JSON_EXTRACT(connection_info, '$.data.target_luns') AS bdm_luns FROM nova.block_device_mapping WHERE instance_uuid = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+---------+----------------------------+ | instance_uuid | bdm_lun | bdm_luns | +--------------------------------------+---------+----------------------------+ | [VM_UUID] | 3 | [3, 3, 3, 3] | +--------------------------------------+---------+----------------------------+ ``` {% endcode %} Both `bdm_lun` and all values in `bdm_luns` must match the source host's LUN ID. {% endstep %} {% step %} #### Step 2 — Remove the stale destination igroup mapping Remove the volume from the **destination host's igroup** on the NetApp array. This is the stale mapping left behind by the failed migration. In the NetApp ONTAP System Manager UI: 1. Go to **Storage → LUNs**, open the LUN, and click the **LUN Mappings** tab 2. Find the destination host's igroup entry 3. Click **Unmap** to remove it {% hint style="warning" %} Until this mapping is removed, the volume is accessible from two compute hosts simultaneously. This is unsafe for non-cluster-aware filesystems and must be cleaned up before the VM is restarted. {% endhint %} After unmapping, confirm only the source host's igroup remains in the LUN Mappings tab. {% endstep %} {% step %} #### Step 3 — Reset the VM state and hard reboot Reset the VM from `error` state to `active` and perform a hard reboot to trigger volume attachment with the corrected BDM as mentioned in [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states) {% code title="OpenStack CLI" %} ```bash $ openstack server reset-state --active $ openstack server reboot --hard ``` {% endcode %} {% endstep %} {% step %} #### Step 4 — Confirm the VM is running Monitor the VM state until it reaches `ACTIVE`. {% code title="OpenStack CLI" %} ```bash $ openstack server show | grep status ``` {% endcode %} {% code title="Sample Output" %} ```bash | status | ACTIVE | ``` {% endcode %} Confirm the volume is attached and the domain is visible on the source compute host: {% code title="Compute Host" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash [DOMAIN_ID] [VM_UUID] running ``` {% endcode %} {% endstep %} {% endstepper %} ## Additional Information **Related articles:** * [Virsh Commands Run Infinitely Without Output / VM Console Inaccessible](https://platform9.com/kb/pcd/generic/virsh-commands-run-infinitely-without-output-vm-console-inaccessible) — covers the `libvirtd` failure and stale multipath recovery that commonly precedes the BDM corruption scenario described in this article. Resolve that issue first if `virsh` commands are unresponsive on the affected compute host. * [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states) — covers resetting volumes stuck in intermediate states if the volume shows an error state in addition to the VM. # How to Migrate the NFS Filer Backing an NFS Backend ## Problem When the storage team has replicated NFS exports to a new filer and the goal is to redirect the existing PCD control plane and hypervisors to the new network path — without re-creating volumes, re-attaching disks, or rebuilding VMs — the Cinder NFS backend, hypervisor mounts, and underlying databases must all be updated in a coordinated sequence. This article describes that process for migrating from one NFS filer (including non-NetApp servers) to another. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component: Storage Service ## Procedure Complete all steps in order during a scheduled maintenance window. VM downtime is required. ### Prerequisites * A maintenance window with planned VM downtime. * A freeze on new VM creation and VM power-on is in place for the hypervisors being modified during the window. * The storage team has already replicated the NFS exports to the new filer and the new exports are accessible from all hypervisors. * The following placeholder values are known and ready to substitute before running any command: | Placeholder | Meaning | Example | | -------------------------- | ---------------------------------------------------------- | -------------------------- | | `` | FQDN of the current NFS filer | `oldfiler.example.com` | | `` | FQDN of the target NFS filer | `newfiler.example.com` | | ``, `` | NFS export paths used by Cinder | `/nfs_pcd01`, `/nfs_pcd02` | | `` | Name of the Cinder volume backend in the cluster blueprint | `site_netapp` | | `` | Hostnames of the hypervisors mounting the backend | `hv0001`, `hv0002` | {% stepper %} {% step %} #### Shut Down All VMs on the Affected Hypervisors VMs must be cleanly powered off before the storage path is changed. From the PCD UI, select the VMs and stop them in batches (preferred for large clusters). Wait until every VM reaches `SHUTOFF`: {% code title="Controller / OpenStack CLI" %} ```bash $ openstack server list --all-projects --status ACTIVE ``` {% endcode %} Confirm at the libvirt layer on each hypervisor — the output must be empty: {% code title="Hypervisor Node" %} ```bash $ sudo virsh list ``` {% endcode %} {% endstep %} {% step %} #### Update the PCD Cluster Blueprint In the PCD UI, navigate to: **Infrastructure → Cluster Blueprint → Volume Backend Configuration → ``** Update the following fields: * **`nfs_mount_points`** — change from `:` and `:` to `:` and `:`. * **`nfs_mount_options`** — leave unchanged unless the new filer requires different options. A typical value is `vers=3,proto=tcp,lookupcache=pos,nolock,noacl`. Click **Apply**. Verify on each hypervisor that the new mounts are active: {% code title="Hypervisor Node" %} ```bash $ mount | grep -i pcd ``` {% endcode %} {% code title="Sample Output" %} ```bash [NEW_FILER_FQDN]:/nfs_pcd01 on /opt/pf9/... type nfs (...) [NEW_FILER_FQDN]:/nfs_pcd02 on /opt/pf9/... type nfs (...) ``` {% endcode %} The new `` mounts should appear. Old `` entries may still appear transiently and will be cleaned up in Step 5. {% endstep %} {% step %} #### Update the Cinder and Nova Databases These database updates rewrite the embedded NFS network path on every existing volume and block device mapping so that volume operations and VM boot continue to find their backing files on the new filer. {% hint style="info" %} **Self-Hosted only:** Self-hosted customers can run these updates against their own control-plane database. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf. {% endhint %} Use the following command from Control Plane nodes to login to Cinder DB {% code title="Control Plane Node" overflow="wrap" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D cinder" ``` {% endcode %} Run each block independently. Capture the `Rows matched` output from each `UPDATE` statement for the record. **3a — Cinder `provider_location`** {% code title="Control Plane Node — MySQL - Cinder" %} ```bash USE cinder; SELECT COUNT(*) AS to_update FROM volumes WHERE provider_location LIKE ':%' AND deleted = 0; UPDATE volumes SET provider_location = REPLACE(provider_location, ':', ':') WHERE provider_location LIKE ':%' AND deleted = 0; SELECT COUNT(*) AS still_old FROM volumes WHERE provider_location LIKE ':%' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +----------+ | to_update| +----------+ | 42 | +----------+ Query OK, 42 rows affected +----------+ | still_old| +----------+ | 0 | <-- must return 0 +----------+ ``` {% endcode %} **3b — Cinder `host`** {% code title="Control Plane Node — MySQL" %} ```bash USE cinder; UPDATE volumes SET host = REPLACE(host, ':', ':') WHERE host LIKE '%:%' AND deleted = 0; ``` {% endcode %} **3c — Nova `block_device_mapping.connection_info`** {% code title="Control Plane Node — MySQL" %} ```bash USE nova; UPDATE block_device_mapping SET connection_info = REPLACE(connection_info, ':', ':') WHERE connection_info LIKE '%:%' AND deleted = 0; ``` {% endcode %} {% hint style="info" %} **Why these updates matter:** The Cinder updates ensure all existing volumes report the new network path so subsequent volume operations (snapshot, clone, extend, detach/attach) target the new filer. The Nova block device mapping update is required for libvirt to mount each VM's volumes from the new path when VMs are powered back on in Step 6. {% endhint %} {% endstep %} {% step %} #### Update `/etc/fstab` on Each Hypervisor Run the following sequence on every hypervisor that mounts the backend: {% code title="Hypervisor Node" %} ```bash $ sudo systemctl stop pf9-nova-compute libvirtd $ mount | egrep "images|instances" $ sudo umount /var/lib/glance/images $ sudo umount /opt/data/instances $ sudo sed -i 's|||g' /etc/fstab $ sudo mount -a $ mount | egrep "images|instances" ``` {% endcode %} {% code title="Sample Output" %} ```bash [NEW_FILER_FQDN]:/nfs_pcd01 on /var/lib/glance/images type nfs (...) [NEW_FILER_FQDN]:/nfs_pcd01 on /opt/data/instances type nfs (...) ``` {% endcode %} The output must show only `` entries — no `` entries. Bring services back up: {% code title="Hypervisor Node" %} ```bash $ sudo systemctl start libvirtd pf9-ostackhost ``` {% endcode %} Confirm each hypervisor returns to a healthy state in the PCD UI before proceeding to the next hypervisor. {% endstep %} {% step %} #### Clean Up Orphan Old Mounts If any `` mounts are still present after Step 4, clean them up on each affected hypervisor. First, confirm no processes have open file handles on the old filer: {% code title="Hypervisor Node" %} ```bash $ sudo lsof | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash (empty — must return no results) ``` {% endcode %} Unmount the orphan paths: {% code title="Hypervisor Node" %} ```bash $ sudo umount /opt/pf9/etc/pf9-cindervolume-base/volumes/ $ sudo umount /opt/pf9/etc/pf9-cindervolume-base/volumes/ $ sudo umount /opt/pf9/data/state/mnt/ $ sudo umount /opt/pf9/data/state/mnt/ ``` {% endcode %} Verify no old mounts remain: {% code title="Hypervisor Node" %} ```bash $ mount | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash (empty — must return no results) ``` {% endcode %} **Checkpoint:** Every mount on every hypervisor should now reference ``. None should reference ``. {% endstep %} {% step %} #### Power VMs Back On Start the VMs in batches from the PCD UI. Spot-check a representative sample of VMs after each batch: {% code title="Controller / OpenStack CLI" %} ```bash $ openstack server show -c status -f value ``` {% endcode %} {% code title="Sample Output" %} ```bash ACTIVE ``` {% endcode %} Verify disk paths reference the new filer mount on the hypervisor: {% code title="Hypervisor Node" %} ```bash $ sudo virsh domblklist instance- ``` {% endcode %} {% code title="Sample Output" %} ```bash Target Source --------------------------------------------------------- vda /opt/pf9/data/state/mnt/[NEW_HASH_EXPORT]/[volume-file] ``` {% endcode %} The source path must reference the new filer mount hash — not the old one. Monitor the cluster for 10–15 minutes after the last VM is powered on to confirm no late-failing volume operations or NFS timeouts. {% endstep %} {% endstepper %} ### Validation After all VMs are back online, run the following end-to-end checks against the new filer: 1. **Create** a test volume and confirm it is created on ``. 2. **Attach** the test volume to a running VM 3. **Snapshot** the test volume. 4. **Extend** the test volume. 5. **Boot** a new test VM and confirm it comes up using the new filer path. # NetApp NFS Block Storage Service get\_volume\_stats High Latency ## Problem The NetApp ONTAP Cluster-mode NFS driver's `get_volume_stats` operation consistently takes 30–40 seconds per stats cycle on Block Storage Service hosts. Warning log entries similar to the following appear repeatedly at approximately 60-second intervals: {% code title="Sample Output" %} ```bash WARNING cinder.volume.manager [req-[REQ_UUID] None None] The NetAppCmodeNfsDriver volume driver's get_volume_stats operation ran for [DURATION] seconds. This may indicate a performance problem with the backend which can lead to instability. ``` {% endcode %} Because the stats cycle runs synchronously in the Block Storage Service processing thread, volume create, snapshot, and other storage operations queue behind each active cycle. Operations that complete in 2–6 seconds on the storage backend appear to take 60–70 seconds from the user's perspective. ## Environment * Private Cloud Director Virtualization - All current versions * Self-Hosted Private Cloud Director Virtualization - All current versions * Component: Block Storage Service, NetApp ONTAP Cluster-mode NFS Driver ## Cause The NetApp ONTAP Cluster-mode NFS driver uses Python's `HTTPBasicAuthHandler` for all ONTAP management API calls. This handler is stateless — no credential caching and no session reuse between calls. Each management API call triggers a `401 Unauthorized` challenge from ONTAP, after which ONTAP validates the credentials against Active Directory (AD) before returning a response. Additionally, `HTTPBasicAuthHandler` does not pre-send the `Authorization` header — the handler waits for the 401 challenge before attaching credentials — resulting in two HTTP round-trips per API call. The `_update_volume_stats` path fans out approximately `2N + 3` API calls per cycle, where N is the number of FlexVols or NFS shares managed by the backend. When the configured NetApp service account is AD-backed, each call incurs the AD domain controller lookup latency (\~350–400 ms) on top of normal ONTAP processing time (\~10–50 ms). A backend with 47 FlexVols makes approximately 97 API calls per cycle — at \~390 ms per call, this produces a 37–40-second stats cycle. The stats cycle runs synchronously in the Block Storage Service processing thread. A 37–40-second blocking operation on the default 60-second `periodic_interval` starves the thread and prevents RPC messages — volume create, snapshot, clone, and attach requests — from being processed during that window. ## Diagnostics {% stepper %} {% step %} #### Step 1 — Check for Warning Log Entries Check the Block Storage Service logs on each Persistent Storage Host for the `get_volume_stats` warning. {% code title="Persistent Storage Host - /var/log/pf9/cindervolume-base.log" %} ```bash $ grep -i "get_volume_stats" /var/log/pf9/cindervolume-base.log ``` {% endcode %} {% code title="Sample Output" %} ```bash [TIMESTAMP] WARNING cinder.volume.manager [req-[REQ_UUID] None None] The NetAppCmodeNfsDriver volume driver's get_volume_stats operation ran for [DURATION] seconds. This may indicate a performance problem with the backend which can lead to instability. ``` {% endcode %} Repeated warning entries at approximately 60-second intervals confirm the issue is active. {% endstep %} {% step %} #### Step 2 — Identify the Service Account Authentication Method Read the `netapp_login` value from the active Block Storage Service configuration on each Persistent Storage Host. {% code title="Persistent Storage Host" %} ```bash $ sudo grep 'netapp_login' /opt/pf9/etc/pf9-cindervolume-base/cinder.conf ``` {% endcode %} {% code title="Sample Output" %} ```bash netapp_login = [DOMAIN]\[SERVICE_ACCOUNT] ``` {% endcode %} A login value in the format `DOMAIN\username` or `username@domain.com` indicates an AD-backed account. To confirm the account type, ask the NetApp administrator to run the following command on the ONTAP cluster: `security login show -vserver -user-or-group-name `. An `Authentication Method` value of `domain` confirms the account is AD-backed and this issue applies. {% endstep %} {% endstepper %} If both checks above match, proceed to [Workaround](#workaround). ## Resolution The upstream fix (Cinder bug #2128652, merged in Cinder 28.0/Gazpacho) will be available in PCD once the upstream Cinder version is adopted. We are tracking this under `PCD-7975`. In the meantime, use the [Workaround](#workaround) below. ## Workaround Switching the NetApp service account from an AD-backed account to a local ONTAP account eliminates the AD lookup overhead per API call. This reduces `get_volume_stats` cycle time from 37–40 seconds to approximately 2 seconds on typical deployments. {% hint style="info" %} Before proceeding, ask the NetApp administrator to create a local ONTAP account on the relevant SVM with `ontapi` application access and `password` authentication: `security login create -vserver -user-or-group-name -application ontapi -authmethod password`. Note the account name and password before continuing. {% endhint %} {% hint style="info" %} **SaaS customers:** The configuration override file (Method 1) is not available for SaaS deployments. Use [#method-2-apply-via-cluster-blueprint](#method-2-apply-via-cluster-blueprint "mention") to apply the credential change, or contact [Platform9 Support](https://support.platform9.com/) for assistance. {% endhint %} Apply the new credentials using one of the following methods. Method 1 is a quick validation approach and does not persist across blueprint reapplication. Method 2 is the permanent configuration change. ### Method 1 — Apply via Configuration Override File {% stepper %} {% step %} #### Step 1 — Edit the Configuration Override File On each Persistent Storage Host, edit or create the configuration override file at `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf`. Add the local ONTAP account credentials under each backend section. Backend section names match those defined in `/opt/pf9/etc/pf9-cindervolume-base/cinder.conf`. {% code title="Persistent Storage Host" %} ```bash $ sudo vi /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf ``` {% endcode %} {% code title="Persistent Storage Host" %} ```ini [] netapp_login = netapp_password = [] netapp_login = netapp_password = ``` {% endcode %} Add one stanza per backend section. Omit sections that do not use an AD-backed account. {% endstep %} {% step %} #### Step 2 — Restart the Block Storage Service {% hint style="warning" %} Restarting the Block Storage Service briefly interrupts volume operation processing. In-flight operations resume after the service restarts. {% endhint %} {% code title="Persistent Storage Host" %} ```bash $ sudo systemctl restart pf9-cindervolume-base ``` {% endcode %} Repeat Step 1 and Step 2 on all Persistent Storage Hosts in the affected region. {% endstep %} {% step %} #### Step 3 — Verify the Warning Entries Stop Monitor the Block Storage Service logs for 5–10 minutes after restart. No `get_volume_stats` warning entries should appear. {% code title="Persistent Storage Host" %} ```bash $ sudo journalctl -u pf9-cindervolume-base -f | grep -i "get_volume_stats" ``` {% endcode %} Confirm all Block Storage Service backends are in `up` state: {% code title="OpenStack CLI" %} ```bash $ openstack volume service list ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------+---------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +-----------------+---------------------------+------+---------+-------+----------------------------+ | cinder-volume | [HOSTNAME]@[BACKEND_NAME] | nova | enabled | up | [TIMESTAMP] | +-----------------+---------------------------+------+---------+-------+----------------------------+ ``` {% endcode %} All `State` values must show `up` and all `Status` values must show `enabled`. {% endstep %} {% endstepper %} {% hint style="warning" %} The configuration override file does not persist through blueprint reapplication. After validating with Method 1, promote the change to the cluster blueprint using Method 2 and remove the override entries from `cinder_override.conf`. {% endhint %} ### Method 2 — Apply via Cluster Blueprint {% stepper %} {% step %} #### Step 1 — Update the NetApp Backend Credentials in the Blueprint In the PCD management interface, open the cluster and navigate to the cluster blueprint. Locate the NetApp NFS backend configuration and update the `netapp_login` and `netapp_password` fields with the local ONTAP account credentials. Save and apply the blueprint. PCD deploys the updated configuration to all Persistent Storage Hosts and restarts the Block Storage Service automatically. {% endstep %} {% step %} #### Step 2 — Verify the Configuration Is Applied On each Persistent Storage Host, confirm the local ONTAP account name is now in the active configuration. {% code title="Persistent Storage Host" %} ```bash $ sudo grep 'netapp_login' /opt/pf9/etc/pf9-cindervolume-base/cinder.conf ``` {% endcode %} {% code title="Sample Output" %} ```bash netapp_login = [LOCAL_ONTAP_USER] ``` {% endcode %} The output must show the local ONTAP account name with no domain prefix. {% endstep %} {% step %} #### Step 3 — Verify the Warning Entries Stop Monitor the Block Storage Service logs for 5–10 minutes after blueprint application. No `get_volume_stats` warning entries should appear. {% code title="Persistent Storage Host" %} ```bash $ sudo journalctl -u pf9-cindervolume-base -f | grep -i "get_volume_stats" ``` {% endcode %} Confirm all Block Storage Service backends are in `up` state: {% code title="OpenStack CLI" %} ```bash $ openstack volume service list ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------+---------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +-----------------+---------------------------+------+---------+-------+----------------------------+ | cinder-volume | [HOSTNAME]@[BACKEND_NAME] | nova | enabled | up | [TIMESTAMP] | +-----------------+---------------------------+------+---------+-------+----------------------------+ ``` {% endcode %} All `State` values must show `up` and all `Status` values must show `enabled`. {% endstep %} {% endstepper %} # Volume Snapshots Stuck in Error Deleting on Tintri VMstore ## Problem One or more Block Storage volume snapshots are stuck in `error_deleting` status on a Tintri VMstore-backed deployment and cannot be deleted via the PCD UI or CLI. Deletion attempts fail silently — no explicit error is surfaced — and the snapshot reverts to `error_deleting` after each attempt. Running `openstack volume snapshot list` shows one or more snapshots in `error_deleting` status: {% code title="OpenStack CLI" %} ```bash $ openstack volume snapshot list --all-projects | grep error_deleting ``` {% endcode %} {% code title="Sample Output — Snapshots Stuck in Error Deleting" %} ```bash | [SNAPSHOT_UUID_1] | [SNAPSHOT_NAME_1] | [VOLUME_NAME_1] | error_deleting | [SIZE] GiB | | [SNAPSHOT_UUID_2] | [SNAPSHOT_NAME_2] | [VOLUME_NAME_2] | error_deleting | [SIZE] GiB | ``` {% endcode %} Attempting to delete the snapshot via CLI produces no error output, but the snapshot returns to `error_deleting`: {% code title="OpenStack CLI" %} ```bash $ openstack volume snapshot delete ``` {% endcode %} ## Environment * Private Cloud Director Virtualization - All versions * Self-Hosted Private Cloud Director Virtualization - All versions * Component: Block Storage Service, Tintri VMstore ## Cause Tintri VMstore enforces referential integrity on snapshot objects. When a volume is created by cloning a snapshot — either explicitly or as a result of volume creation from an image that was snapshot-backed — Tintri increments the `cloneReferenceCount` on the source snapshot. Tintri does not permit deletion of a snapshot whose `cloneReferenceCount` is greater than zero, even when the snapshot is flagged as `isOrphaned: true` on the Tintri side. When the Block Storage Service issues a delete call to the Tintri API, the API silently blocks the operation rather than returning an explicit error. The Block Storage Service receives no failure signal, releases the coordination lock, and the snapshot transitions back to `error_deleting`. This cycle repeats on every subsequent delete attempt. Block Storage Service force-delete options do not bypass this constraint — the block is enforced at the Tintri storage layer, not within the Block Storage Service itself. ## Diagnostics {% stepper %} {% step %} #### Step 1 — Confirm the snapshot is stuck in error\_deleting {% code title="OpenStack CLI" %} ```bash $ openstack volume snapshot show -c status -c volume_id -c name ``` {% endcode %} {% code title="Sample Output — Snapshot Stuck in Error Deleting" %} ```bash +-----------+--------------------------------------+ | Field | Value | +-----------+--------------------------------------+ | name | [SNAPSHOT_NAME] | | status | error_deleting | | volume_id | [SOURCE_VOLUME_UUID] | +-----------+--------------------------------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Identify the Block Storage host and backend for the source volume {% code title="OpenStack CLI" %} ```bash $ openstack volume show -c status -c "os-vol-host-attr:host" -c type ``` {% endcode %} {% code title="Sample Output — Volume on Tintri Backend" %} ```bash +----------------------+----------------------------------------------------+ | Field | Value | +----------------------+----------------------------------------------------+ | os-vol-host-attr:host| [BLOCK_STORAGE_HOST_UUID]@[BACKEND_NAME]#[POOL_PATH] | | status | available | | type | [VOLUME_TYPE] | +----------------------+----------------------------------------------------+ ``` {% endcode %} Confirm the backend name contains a Tintri-specific path (e.g., `/tintri/`). If the host shows a different backend, this workaround may not apply. {% endstep %} {% step %} #### Step 3 — Inspect the Block Storage pod log for repeated delete cycles Identify the Block Storage volume pod name, then grep the logs for the snapshot UUID to confirm repeated delete lock cycles with no state change. {% hint style="info" %} **Self-Hosted only:** This step requires access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl get pods -n | grep cinder-api $ kubectl logs -n | grep ``` {% endcode %} {% code title="Sample Output — Block Storage Log Showing Repeated Delete Attempts" %} ```bash INFO cinder.volume.drivers.vmstore.nfs [...] Deleting snapshot snapshot-[SNAPSHOT_UUID] DEBUG cinder.volume.drivers.vmstore.api [...] Start request hook on session request GET https://[TINTRI_APPLIANCE_FQDN]:443/api/v310/snapshot?contain=snapshot-[SNAPSHOT_UUID] None and session response 401 {"code":"ERR-API-0104","message":"Invalid session."} DEBUG cinder.volume.drivers.vmstore.api [...] Getting list of VolumeSnapshot with path: snapshot?contain=snapshot-[SNAPSHOT_UUID] DEBUG cinder.coordination [...] Lock "cinder-[SNAPSHOT_UUID]-delete_snapshot" released by "delete_snapshot" :: held [DURATION]s ``` {% endcode %} {% hint style="info" %} The `ERR-API-0104 Invalid session` response is a normal Tintri session expiry that auto-recovers — it does not indicate an authentication failure. The delete cycle silently fails after re-authentication succeeds, with no `ERROR` line in the log. {% endhint %} {% endstep %} {% step %} #### Step 4 — Verify the clone reference count on the Tintri backend Log in to the Tintri VMstore management UI or query the Tintri API directly from the Block Storage host to confirm `cloneReferenceCount` is blocking deletion. {% code title="Persistent Storage Host" %} ```bash $ curl -sk -H "Content-Type: application/json" \ https:///api/v310/snapshot/ \ | python3 -m json.tool | grep -E "cloneReferenceCount|isOrphaned|description" ``` {% endcode %} {% code title="Sample Output — Snapshot Blocked by Clone Reference" %} ```bash "cloneReferenceCount": 1, "description": "snapshot-[SNAPSHOT_UUID]", "isOrphaned": true, ``` {% endcode %} A `cloneReferenceCount` value greater than `0` confirms that the Tintri backend is blocking deletion. {% endstep %} {% endstepper %} If all checks above match, proceed to [Workaround](#workaround). ## Resolution No permanent fix exists within the Block Storage Service for this behavior. The constraint is enforced at the Tintri VMstore storage layer and cannot be bypassed through Block Storage Service configuration. Use the [Workaround](#workaround) below to release the clone reference before deleting the snapshot. ## Workaround The following steps release the clone reference on the Tintri backend, allowing the snapshot to be deleted. The order of these steps is critical — deleting the snapshot from PCD before the reference is released at the storage layer will return the snapshot to `error_deleting`. {% hint style="warning" %} Complete all steps in the order listed. Skipping or reordering steps will cause the snapshot to return to `error_deleting`. {% endhint %} {% stepper %} {% step %} #### Step 1 — Migrate the source volume to a different backend Identify the source volume UUID from the snapshot details confirmed in Step 1 of the Diagnostics section. Migrate the volume off the Tintri backend to any other available backend. This causes Tintri to release the clone reference on the snapshot. {% code title="OpenStack CLI" %} ```bash $ openstack volume migrate --host ``` {% endcode %} {% code title="Sample Output — Migration Initiated" %} ```bash +----------------------------+-----------+ | Field | Value | +----------------------------+-----------+ | migration_status | migrating | | status | available | +----------------------------+-----------+ ``` {% endcode %} `` must be a valid backend host string in the format `@#`. Run `openstack volume backend pool list` to find available backends. Monitor the volume status until `migration_status` returns to `None` and `status` is `available` before proceeding to the next step. {% hint style="info" %} If the source volume has already been deleted in PCD (the snapshot list shows `None` in the volume column), the migration step is not applicable. Skip to [Additional Information](#additional-information) for the alternative approach. {% endhint %} {% endstep %} {% step %} #### Step 2 — Wait for the clone reference count to drop to zero Log in to the Tintri VMstore management UI for the relevant appliance and navigate to the snapshot list for the affected VM. Wait until the snapshot entry shows `cloneReferenceCount: 0`. Do not proceed to Step 3 until this is confirmed. Alternatively, poll the Tintri API from the Block Storage host: {% code title="Persistent Storage Host" %} ```bash $ curl -sk -H "Content-Type: application/json" \ https:///api/v310/snapshot/ \ | python3 -m json.tool | grep cloneReferenceCount ``` {% endcode %} {% code title="Sample Output — Reference Released" %} ```bash "cloneReferenceCount": 0, ``` {% endcode %} {% endstep %} {% step %} #### Step 3 — Delete the snapshot object from the Tintri VMstore UI After confirming `cloneReferenceCount` is `0`, delete the snapshot directly from the Tintri VMstore management UI. This removes the underlying storage object before the Block Storage Service attempts deletion. Navigate to the snapshot list in the Tintri UI, locate the snapshot by its description (`snapshot-`), and delete it. {% hint style="warning" %} Delete the snapshot from the Tintri UI before proceeding to Step 4. Deleting the PCD snapshot first will return the snapshot to `error_deleting`. {% endhint %} {% endstep %} {% step %} #### Step 4 — Delete the snapshot from PCD and verify removal With the underlying Tintri object removed, the Block Storage Service delete call will succeed because no storage object remains to block it. {% code title="OpenStack CLI" %} ```bash $ openstack volume snapshot delete ``` {% endcode %} Confirm the snapshot no longer appears in the list: {% code title="OpenStack CLI" %} ```bash $ openstack volume snapshot show ``` {% endcode %} {% code title="Sample Output — Snapshot Successfully Deleted" %} ```bash No snapshot with a name or ID of '[SNAPSHOT_UUID]' exists. ``` {% endcode %} The snapshot has been successfully removed from both the Tintri backend and PCD. {% endstep %} {% endstepper %} ## Additional Information * The `cloneReferenceCount` field on a Tintri snapshot reflects the number of volumes created by cloning that snapshot. This count must reach `0` before Tintri permits snapshot deletion — either via the Tintri UI or via a Block Storage Service driver call. * The `isOrphaned: true` flag on the Tintri snapshot indicates that Tintri has lost the association between the snapshot and its parent VM. This flag does not remove the clone reference block. * Deleting the PCD snapshot before clearing the Tintri storage object will return the snapshot to `error_deleting`. The order in the Workaround section must be followed exactly. * If the source volume has already been deleted in PCD but the snapshot remains stuck, the volume migration step is not applicable. Work directly with the Tintri support team to manually remove the clone reference from the VMstore before attempting PCD deletion. * In some cases, the Block Storage log may contain a Tintri `ERR-API-3107` error with the message `Cannot delete the shared snapshot because it has younger or older siblings`. This indicates the snapshot has sibling dependencies on the Tintri side in addition to the clone reference. Engage the Tintri support team to identify and remove all sibling and clone references before reattempting deletion. * Platform9 is engaged with Tintri Engineering to address this behavior and provide a more direct resolution path in a future release. * If the standard workaround does not resolve the issue, contact [Platform9 Support](https://support.platform9.com/) for assistance. A Block Storage database soft-delete is available as a last resort and must only be performed under Platform9 guidance. # VM Creation Fails on Dell PowerStore Backend With Duplicate Host Registration Error ## Problem New virtual machine creation fails immediately at the volume-attachment stage on a Dell PowerStore iSCSI backend. Boot-from-volume instances enter ERROR state within seconds of launch, while standalone (empty) volume creation continues to work normally. The instance fault reports a storage backend failure of the form: `Build of instance [INSTANCE_UUID] aborted: Unable to update attachment.(Bad or unexpected response from the storage volume backend API: ... Failed to create PowerStore host [HOST_UUID]-iSCSI.). (HTTP 500)`. The failure typically surfaces immediately after a management-plane upgrade, but the upgrade is not the underlying cause. The trigger is the number of hosts registered on the PowerStore array crossing a threshold, which commonly coincides with an upgrade because the upgrade is the next significant cluster event. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component: Block Storage Service (Dell PowerStore iSCSI backend) ## Cause The Dell PowerStore Block Storage driver looks up existing array hosts through `get_all_hosts()` in `client.py`, which issues a single unpaginated `GET /host` request to the PowerStore REST API. The PowerStore API enforces a default response limit of 100 objects per query. When the array has more than 100 registered hosts, the response is truncated to the first 100. If the compute node's iSCSI initiator name (IQN) is registered under a host that falls outside those first 100 results, the driver does not see it and concludes the host does not exist. The driver then attempts to create a new host with that IQN, and the PowerStore array rejects the request because it enforces global IQN uniqueness, returning HTTP 400 with code `0xE0A01001002D` — "One of the provided port names is already registered with another host." This is not a naming or IQN-mismatch defect. The iSCSI initiator name in `/etc/iscsi/initiatorname.iscsi` is not modified during a management-plane upgrade, and the IQN on affected compute nodes remains unchanged. The defect is latent: while the array holds 100 or fewer hosts, every host is returned and the lookup always succeeds. Once the array grows past 100 hosts, any host beyond the first 100 becomes invisible to the driver. An upgrade often appears to "introduce" the failure only because it is the first VM-creation activity after the host count crosses the threshold. Standalone volume creation does not register an iSCSI host on the array and is therefore unaffected. Only boot-from-volume VM creation — which triggers host registration at attachment time — fails. This is tracked as a bug under **PCD-6294**. ## Diagnostics {% stepper %} {% step %} #### Confirm the failure pattern Confirm that boot-from-volume VM creation fails while standalone volume creation succeeds. Create an empty volume to verify the Block Storage backend itself is functional, then inspect the failed instance. {% code title="OpenStack CLI" %} ```bash $ openstack server show --fit ``` {% endcode %} The instance fault confirms the failure occurs during volume attachment. {% code title="Sample Output" %} ```bash | OS-EXT-STS:vm_state | error | | fault | {'code': 500, 'message': 'Build of instance [INSTANCE_UUID] aborted: | | | Unable to update attachment.(Bad or unexpected response from the | | | storage volume backend API: Driver initialize connection failed | | | (error: ... Failed to create PowerStore host [HOST_UUID]-iSCSI.).). | | | (HTTP 500) (Request-ID: [REQ_UUID])'} | | image | N/A (booted from volume) | | status | ERROR | ``` {% endcode %} {% endstep %} {% step %} #### Confirm the PowerStore rejection in the Block Storage logs On the Persistent Storage role host serving the backend, inspect the Block Storage volume log for the PowerStore REST rejection. The definitive signature is the `0xE0A01001002D` error code returned by the array. {% code title="Persistent Storage Host" %} ```bash $ grep -E "0xE0A01001002D|Failed to create PowerStore host" /var/log/pf9/cindervolume-base.log ``` {% endcode %} {% code title="Sample Output" %} ```bash ERROR cinder.volume.drivers.dell_emc.powerstore.client REST Request: POST https://[POWERSTORE_IP]/api/rest/host with body {"name": "[HOST_UUID]-iSCSI", "os_type": "Linux", "initiators": [{"port_name": "[HOST_IQN]", "port_type": "iSCSI"}]} ERROR cinder.volume.drivers.dell_emc.powerstore.client REST Response: 400 with data {"messages":[{"code":"0xE0A01001002D","severity":"Error","message_l10n":"One of the provided port names is already registered with another host. Please use port names unique to all hosts."}]} ERROR cinder.volume.drivers.dell_emc.powerstore.client Failed to create PowerStore host [HOST_UUID]-iSCSI. ``` {% endcode %} {% endstep %} {% step %} #### Confirm the IQN is already registered on the array Verify the compute node's IQN is already present on the PowerStore array under an existing host object. A match confirms there is no IQN mismatch — the host exists, but the driver cannot see it. {% hint style="info" %} The compute node IQN is read from `/etc/iscsi/initiatorname.iscsi` on the affected hypervisor. This value is not changed by a management-plane upgrade. {% endhint %} {% code title="Persistent Storage Host" %} ```bash $ curl -k -u ':' \ 'https:///api/rest/host?select=id,name,host_initiators&host_initiators->0->>port_type=eq.iSCSI' \ | python3 -c "import json,sys; h=json.load(sys.stdin); print([x['name'] for x in h if any('' in str(i) for i in x.get('host_initiators',[]))])" ``` {% endcode %} {% endstep %} {% step %} #### Confirm the pagination limit is the cause Compare an unpaginated query against one with an explicit high limit. If the default query returns exactly 100 hosts while the explicit-limit query returns more, the pagination limit is confirmed as the cause. Confirm the affected compute node's IQN is absent from the 100-host result but present in the full result. {% code title="Persistent Storage Host" %} ```bash $ curl -k -u ':' \ 'https:///api/rest/host?select=id,name,host_initiators&host_initiators->0->>port_type=eq.iSCSI' \ | python3 -c "import json,sys; print(len(json.load(sys.stdin)), 'hosts returned')" ``` {% endcode %} {% code title="Sample Output" %} ```bash 100 hosts returned # actual total registered on the array: [ACTUAL_HOST_COUNT] ``` {% endcode %} {% endstep %} {% endstepper %} If all checks above match, proceed to the [Workaround](#workaround). ## Resolution A permanent fix is tracked under **PCD-6294**. The corrected host-lookup logic is expected from the upstream Dell PowerStore driver and will be backported to the Platform9 Block Storage driver once it is merged. No fixed release is available at this time. In the meantime, apply the [Workaround](#workaround) below. ## Workaround Add an explicit high page limit to the PowerStore host-lookup query so that all registered hosts are returned. Apply this change on every host that has the Persistent Storage role assigned. {% hint style="warning" %} This workaround edits a file inside a Platform9-managed package. The change is reverted whenever the host is upgraded or the `pf9-cindervolume-base` package is updated, and it must be reapplied. The change must also be applied to any new host that is later assigned the Persistent Storage role. {% endhint %} {% stepper %} {% step %} #### Edit the PowerStore driver host-lookup query SSH into the Persistent Storage role host and update the `get_all_hosts` function to include an explicit `limit`. {% code title="Persistent Storage Host" %} ```bash $ sudo vi /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powerstore/client.py ``` {% endcode %} Replace the `get_all_hosts` function with the following: {% code title="Persistent Storage Host" %} ```python def get_all_hosts(self, protocol): r, response = self._send_get_request( "/host", params={ "select": "id,name,host_initiators", "host_initiators->0->>port_type": "eq.%s" % protocol, "limit": 1000, } ) if r.status_code not in self.ok_codes: msg = _("Failed to query PowerStore hosts.") LOG.error(msg) raise exception.VolumeBackendAPIException(data=msg) return response ``` {% endcode %} {% endstep %} {% step %} #### Restart the Block Storage volume service Restart the service so the driver change takes effect. {% code title="Persistent Storage Host" %} ```bash $ sudo systemctl restart pf9-cindervolume-base.service ``` {% endcode %} {% endstep %} {% step %} #### Verify all hosts are now returned Re-run the host query with the explicit limit and confirm the count exceeds 100 and includes the previously missing IQN. {% code title="Persistent Storage Host" %} ```bash $ curl -k -u ':' \ 'https:///api/rest/host?select=id,name,host_initiators&host_initiators->0->>port_type=eq.iSCSI&limit=1000' \ | python3 -c "import json,sys; print(len(json.load(sys.stdin)), 'hosts returned')" ``` {% endcode %} {% code title="Sample Output" %} ```bash [ACTUAL_HOST_COUNT] hosts returned ``` {% endcode %} {% endstep %} {% step %} #### Confirm VM creation succeeds Create a boot-from-volume test VM and confirm it reaches ACTIVE state, then delete the test VM. {% code title="OpenStack CLI" %} ```bash $ openstack server create --flavor --volume --network $ openstack server show -c status ``` {% endcode %} {% code title="Sample Output" %} ```bash | status | ACTIVE | ``` {% endcode %} A successful ACTIVE result confirms the workaround is effective. {% endstep %} {% endstepper %} {% hint style="info" %} If a Persistent Storage host must be brought back to a clean state instead of patched, the stale PowerStore host entry for the affected node can be removed so the driver re-creates it on the next attachment. See [How to Delete Stale Storage Host Entries in PCD](https://platform9.com/kb/pcd/storage/how-to-delete-stale-volume-entries-from-using-openstack-in-pcd). This is a per-node relief that recurs for any host outside the first 100 results until the driver fix is in place. {% endhint %} ## Additional Information This issue is tracked in **PCD-6294**. The permanent fix is expected from the [upstream Dell PowerStore driver](https://review.opendev.org/c/openstack/cinder/+/986606) and will be backported to the Platform9 Block Storage driver once merged. The workaround above is temporary and does not survive a host upgrade or package update. # VM Fails to Start After Live Migration Failure Due to Fibre Channel BDM Corruption ## Problem After a live migration fails on a host using Fibre Channel block storage, the affected VM shows `vm_state: error` while `power_state` remains `Running` on the original host. The VM continues to run until an operation such as a start, hard reboot, or a second migration attempt is issued. A start or hard reboot then fails during volume disconnect, the domain is destroyed, and the VM does not come back online. Additional observable symptoms: * `openstack volume show` for a non-multiattach volume lists two or more attachments, each on a different host * `openstack server event list` shows one or more `live-migration` events with a failure status * The `launched_at` timestamp of the instance predates the failed migrations, indicating the domain never moved {% hint style="danger" %} Do not start, hard-reboot, live-migrate, or reset the state of the affected VM before completing the [Workaround](#workaround). Each of these operations reads the corrupt Block Device Mapping and can remove SCSI paths from an unrelated volume that is in use by another VM on the same host. {% endhint %} ## Environment * Private Cloud Director Virtualization - All versions * Self-Hosted Private Cloud Director Virtualization - All versions * Component: Compute Service, Block Storage Service (Fibre Channel — HPE Alletra 9000/MP, Primera, 3PAR) ## Cause During a live migration, the Compute Service runs `pre_live_migration` on the destination host. This step creates a new volume attachment for the destination host in the Block Storage Service and rewrites the `connection_info` field of the Block Device Mapping (BDM) record in the `nova.block_device_mapping` table with the destination host's LUN ID and initiator WWPNs. The `attachment_id` column of the BDM is repointed to the destination host's attachment. If the migration fails after `pre_live_migration` runs and the rollback does not complete, the BDM record is never reverted. The domain still runs on the source host, but the BDM now describes the destination host's view of the volume. On 3PAR-lineage arrays (HPE Alletra 9000/MP, Primera, 3PAR) each host receives its own VLUN numbering. The same volume can be LUN 59 on one host, 36 on another, and 111 on a third. A LUN number is meaningless outside the host that the LUN was issued for. When the Compute Service later starts or hard-reboots the VM, os-brick builds `by-path` device names from the stale LUN number and resolves them on the source host. If an unrelated volume occupies that LUN number on the source host, os-brick attempts to flush and delete that volume's multipath map. The `map in use` protection in multipathd blocks the delete, but SCSI paths of the unrelated volume may already have been removed at that point. Resetting the VM state to `active` and retrying the migration clears the symptom without correcting the record. Each additional failed migration attempt leaks one more attachment per volume. Fibre Channel `connection_info` embeds the LUN in four fields and identifies the host in two, so a repair that only rewrites `target_lun` leaves the record inconsistent: | Field | Action | Notes | | ------------------------------- | ---------------- | ---------------------------------------------------------- | | `data.target_lun` | Rewrite | Scalar LUN for the host holding the domain | | `data.targets` | Rewrite | Four `[target_wwn, lun]` pairs | | `data.initiator_target_map` | Rekey | Keyed by initiator WWPN; targets listed per initiator | | `data.initiator_target_lun_map` | Rekey and re-LUN | Drives os-brick path derivation | | `attachment_id` (column) | Repoint | To the attachment whose connector host holds the domain | | `data.multipath_id` | Leave | Array WWID — host-independent | | `data.device_path` | Leave | Derived from the WWID | | `data.target_wwn` | Leave | Array target ports; verify the ports are zoned to the host | | `volume_id`, `serial` | Leave | Volume identity | Existing records list all four array target WWNs under each initiator, even where a given HBA only sees two of them. That is the 3PAR driver's own convention — replicate that layout rather than modelling per-HBA zoning. Paths that do not resolve are tolerated and only produce a warning. ## Diagnostics While the VM is still `Running`, every command in this section is read-only and safe. The domain is only alive because nothing has re-read the BDM yet. Capture the live multipath output before anything stops the VM — those paths are the cheapest source of the correct LUN. {% stepper %} {% step %} ### Step 1 — Confirm the VM state and capture the migration failure Record the state fields and the host the Compute Service believes is running the VM. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:power_state -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------------------+-----------------+ | Field | Value | +-----------------------------+-----------------+ | OS-EXT-SRV-ATTR:host | [SOURCE_HOST] | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | status | ERROR | +-----------------------------+-----------------+ ``` {% endcode %} Identify the failed live-migration event and record the reason for the failure. The failure reason does not block the repair, but a repaired VM must not be live-migrated again until the original error is understood. {% code title="OpenStack CLI" %} ```bash $ openstack server event list $ openstack server event show ``` {% endcode %} {% code title="Sample Output" %} ```bash +------------------------------------------+--------------------------------------+----------------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+----------------+----------------------------+ | [REQ_UUID] | [INSTANCE_UUID] | live-migration | [TIMESTAMP] | +------------------------------------------+--------------------------------------+----------------+----------------------------+ ``` {% endcode %} On the source compute host, search the Compute Service log for the instance and request ID: {% code title="Compute Host" %} ```bash $ sudo grep -E "|" /var/log/pf9/ostackhost.log ``` {% endcode %} {% hint style="info" %} A migration that fails with `No valid host was found. There are not enough hosts available` fails at the scheduler, before a destination is chosen. `pre_live_migration` never runs in that case and the BDM is intact — those VMs usually need only a state reset. Check such instances with the detection queries in [Region-Wide Detection Queries](#region-wide-detection-queries) anyway, because an earlier attempt on the same instance may have progressed further. Leaked Placement allocations from prior failed migrations are a common cause of that scheduler message. {% endhint %} {% endstep %} {% step %} ### Step 2 — Locate the running domain Do not trust the `OS-EXT-SRV-ATTR:host` value on its own. Run the following on every host that appears as a source or destination in the migration history. {% code title="Compute Host" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash [DOMAIN_ID] [INSTANCE_NAME] running ``` {% endcode %} Exactly one host must show the domain. If more than one host has the domain defined, stop and treat the situation as a potential dual-writer before going further. {% endstep %} {% step %} ### Step 3 — Read the Compute Service BDM record {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D nova" mysql > use nova; mysql > select volume_id, attachment_id, connection_info from nova.block_device_mapping where instance_uuid='' and deleted=0 \G; ``` {% endcode %} {% code title="Sample Output" %} ```sql *************************** 1. row *************************** volume_id: [VOLUME_UUID] attachment_id: [ATTACHMENT_UUID] connection_info: {"driver_volume_type": "fibre_channel", "data": {"target_lun": [LUN], "targets": [...], "initiator_target_map": {"[INITIATOR_WWPN]": [...]}, "initiator_target_lun_map": {...}, "multipath_id": "[WWID]", "target_wwn": [...], ...}} ``` {% endcode %} Record, per volume: `volume_id`, `attachment_id`, `data.target_lun`, the keys of `data.initiator_target_map`, and `data.multipath_id`. The `multipath_id` value is the array WWID and is host-independent, so the WWID is a safe lookup key even when every other value in the record is wrong. {% endstep %} {% step %} ### Step 4 — Read the Block Storage Service attachment records {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```sql mysql > select id, attach_status, deleted, json_extract(connector,'\$.host') as chost from cinder.volume_attachment where volume_id=''; ``` {% endcode %} {% code title="Sample Output" %} ```sql +--------------------------------------+---------------+---------+-----------------------+ | id | attach_status | deleted | chost | +--------------------------------------+---------------+---------+-----------------------+ | [SOURCE_ATTACHMENT_UUID] | attached | 0 | "[SOURCE_HOST]" | | [DEST_ATTACHMENT_UUID] | attached | 0 | "[DESTINATION_HOST]" | +--------------------------------------+---------------+---------+-----------------------+ ``` {% endcode %} Note every connector host, the `attach_status`, and the `deleted` flag. Each row with `attach_status: attached` on a non-multiattach volume is a live VLUN on the array, not a stale database row. {% endstep %} {% step %} ### Step 5 — Read the host's storage identity Run on the host that holds the domain (from Step 2). Use the WWID from `data.multipath_id` in Step 3. {% code title="Compute Host" %} ```bash $ sudo multipath -ll $ sudo grep . /sys/class/fc_host/host*/port_state /sys/class/fc_host/host*/port_name $ sudo systemctl is-active multipathd ``` {% endcode %} {% code title="Sample Output" %} ```bash [WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running /sys/class/fc_host/host[N]/port_state:Online /sys/class/fc_host/host[N]/port_name:0x[INITIATOR_WWPN] active ``` {% endcode %} The last field of the `H:C:T:L` column on the path lines is the real LUN for this host. The `port_name` values are the initiator WWPNs of this host. {% hint style="info" %} A `multipath -ll` entry printing `##,##` instead of `3PARdata,VV` has lost all paths — multipath cannot read vendor strings without a live path. That is local device state, not an array unmap. If the Block Storage Service still shows the attachment as `attached`, the array mapping is intact. {% endhint %} {% endstep %} {% step %} ### Step 6 — Compare the records Any one of the following three mismatches confirms the corruption: * The BDM `attachment_id` (Step 3) resolves to a `cinder.volume_attachment` row (Step 4) whose connector host is not the host holding the domain (Step 2) * `connection_info.data.target_lun` (Step 3) differs from the `H:C:T:L` LUN of `multipath -ll ` on the host holding the domain (Step 5) * The keys of `initiator_target_map` (Step 3) are WWPNs that do not appear in `/sys/class/fc_host/host*/port_name` on the host holding the domain (Step 5) Supporting evidence: a non-multiattach volume with several `attached` attachment rows (Step 4), and an instance in `vm_state: error` with `power_state: Running` whose `launched_at` predates the migrations. {% endstep %} {% endstepper %} If all checks above match, proceed to [Workaround](#workaround). ### Region-Wide Detection Queries The Compute Service and Block Storage Service share a MariaDB instance, so every affected instance in a region can be detected with cross-database queries instead of inspecting instances one by one. {% hint style="info" %} **Self-Hosted only:** These queries require direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} BDM points at another host's attachment — every row returned is an instance that will fail to boot and will act on an unrelated LUN when a start is attempted: {% code title="Nova DB" %} ```sql SELECT i.uuid AS instance, i.hostname, i.host AS nova_host, i.vm_state, i.power_state, bdm.volume_id, bdm.device_name, bdm.attachment_id, JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) AS attach_host, JSON_EXTRACT(bdm.connection_info,'$.data.target_lun') AS bdm_lun FROM nova.block_device_mapping bdm JOIN nova.instances i ON i.uuid = bdm.instance_uuid AND i.deleted = 0 JOIN cinder.volume_attachment va ON va.id = bdm.attachment_id WHERE bdm.deleted = 0 AND bdm.destination_type = 'volume' AND JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) <> i.host; ``` {% endcode %} Leaked multi-host mappings on non-multiattach volumes: {% code title="Nova DB" %} ```sql SELECT va.volume_id, COUNT(*) AS live_attachments, GROUP_CONCAT(JSON_UNQUOTE(JSON_EXTRACT(va.connector,'$.host')) SEPARATOR ' ') AS hosts FROM cinder.volume_attachment va JOIN cinder.volumes v ON v.id = va.volume_id AND v.multiattach = 0 AND v.deleted = 0 WHERE va.deleted = 0 AND va.attach_status = 'attached' GROUP BY va.volume_id HAVING COUNT(*) > 1 ORDER BY live_attachments DESC; ``` {% endcode %} Live migrations that did not complete cleanly — the population at risk: {% code title="Nova DB" %} ```sql SELECT instance_uuid, source_compute, dest_compute, status, created_at, updated_at FROM nova.migrations WHERE migration_type = 'live-migration' AND status NOT IN ('completed', 'reverted', 'cancelled') ORDER BY updated_at DESC LIMIT 100; ``` {% endcode %} ## Resolution No product fix is tracked for this condition at this time. Correct the BDM record and clear the leaked attachments using the [Workaround](#workaround) below. The workaround does not address the reason the live migration failed. Two failures on one instance mean a third is coming — identify the original migration error from Step 1 of [Diagnostics](#diagnostics) before live-migrating the repaired VM again. ## Workaround The repair path depends on whether the source host's attachment is still alive. From Step 4 of [Diagnostics](#diagnostics), find the attachment whose connector host is the host holding the domain: * `attach_status = attached` and `deleted = 0` — the Block Storage Service never ran `terminate_connection` for that host, so the VLUN still exists on the array. Any missing paths are local state only, recoverable with an FC rescan. This is the common case, and the steps below apply. * No such row, or `deleted = 1` — the source attachment and the array VLUN were removed. The mapping must be recreated by the Block Storage Service with a fresh attachment. The steps below do not apply — contact [Platform9 Support](https://support.platform9.com/). See [Additional Information](#additional-information). {% hint style="warning" %} Repair the BDM first and reset the VM state last. `openstack server set --state active` followed by a retry clears the symptom, leaves the cause in place, and turns one leaked attachment per volume into two. {% endhint %} {% stepper %} {% step %} ### Step 1 — Clear pathless multipath map shells Run on the host holding the domain. Only remove maps with zero paths, and only once the domain is stopped or already destroyed. A map with live paths is the real mapping — leave that map alone. {% hint style="warning" %} `multipath -f` on a map that still has live paths removes the device backing a running VM. Confirm the map prints `##,##` with no path lines before flushing. {% endhint %} {% code title="Compute Host" %} ```bash $ sudo multipath -f ``` {% endcode %} {% endstep %} {% step %} ### Step 2 — Rescan the Fibre Channel HBAs The rescan is additive and safe: the rescan re-probes targets and adds what is found, never removes. The loop is scoped to FC hosts rather than every SCSI host. {% code title="Compute Host" %} ```bash $ for h in /sys/class/fc_host/host*; do echo "- - -" | sudo tee /sys/class/scsi_host/$(basename $h)/scan > /dev/null; done ``` {% endcode %} {% hint style="warning" %} Do not run a global `multipath -r`. The global reload suspends every map on the host, briefly interrupting every other VM's devices. multipathd picks up new devices from uevents on its own. If a specific map does not reappear after approximately 30 seconds, add only that one map: `sudo multipathd add map `. {% endhint %} {% endstep %} {% step %} ### Step 3 — Read the true LUN for each volume Run once per volume, one WWID per invocation. {% hint style="info" %} For a general procedure to map a volume to a LUN ID, see [How to Identify the LUN ID Associated with an Instance Volume](https://platform9.com/kb/pcd/storage/how-to-identify-the-lun-id-associated-with-an-instance-volume). {% endhint %} {% code title="Compute Host" %} ```bash $ sudo multipath -ll ``` {% endcode %} {% code title="Sample Output" %} ```bash [WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[LUN] sd[X] [MAJOR:MINOR] active ready running ``` {% endcode %} Expect four `active ready running` paths. The last field of `H:C:T:L` on those path lines is the LUN to write into the BDM record. Never carry a LUN number from another host and never infer one host's LUN from another's. {% endstep %} {% step %} ### Step 4 — Back up the BDM rows {% hint style="info" %} **Self-Hosted only:** Steps 4 through 6 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have them completed on your behalf. {% endhint %} The backup is mandatory before the UPDATE. {% code title="Control Plane Node" %} ```sql mysql > create table nova.bdm_backup_ as select * from nova.block_device_mapping where instance_uuid='' and deleted=0;" ``` {% endcode %} {% endstep %} {% step %} ### Step 5 — Correct the BDM record Run one UPDATE per volume. The statement repoints `attachment_id` at the attachment of the host holding the domain and rewrites all four LUN-bearing fields plus the initiator WWPNs. Substitute every value for the host in front of you: * `` — from Step 3 * `` — the `id` of the attachment whose connector host holds the domain, from Step 4 of [Diagnostics](#diagnostics) * `` to `` — the four array target WWNs already present in the existing record's `data.target_wwn` * `` and `` — the two `port_name` values from `/sys/class/fc_host/` on the host holding the domain, from Step 5 of [Diagnostics](#diagnostics) {% hint style="warning" %} Repointing `attachment_id` matters as much as the LUN. If `attachment_id` is left on the destination's attachment, a later detach or delete removes the wrong attachment and orphans the live VLUN — the same failure in reverse. {% endhint %} {% code title="Nova DB" %} ```sql UPDATE nova.block_device_mapping SET attachment_id = '', connection_info = JSON_SET( JSON_SET( JSON_SET( JSON_SET(connection_info, '$.data.target_lun', ), '$.data.targets', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ))), '$.data.initiator_target_map', JSON_OBJECT( '', JSON_ARRAY('','','',''), '', JSON_ARRAY('','','',''))), '$.data.initiator_target_lun_map', JSON_OBJECT( '', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', )), '', JSON_ARRAY( JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', ), JSON_ARRAY('', )))) WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` {% endcode %} {% endstep %} {% step %} ### Step 6 — Verify the corrected record Expect the correct attachment, the LUN read from the rescan, and only the local host's initiators. {% code title="Control Plane Node" %} ```sql mysql > select volume_id, attachment_id, json_extract(connection_info,'\$.data.target_lun') as lun, json_keys(json_extract(connection_info,'\$.data.initiator_target_lun_map')) as inits from nova.block_device_mapping where instance_uuid='' and deleted=0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+--------------------------------------+------+--------------------------------------------+ | volume_id | attachment_id | lun | inits | +--------------------------------------+--------------------------------------+------+--------------------------------------------+ | [VOLUME_UUID] | [SOURCE_ATTACHMENT_UUID] | [LUN]| ["[INITIATOR_1]", "[INITIATOR_2]"] | +--------------------------------------+--------------------------------------+------+--------------------------------------------+ ``` {% endcode %} {% endstep %} {% step %} ### Step 7 — Start the VM while tailing the Compute Service log In a second session on the host holding the domain, tail the log before issuing any state change: {% code title="Compute Host" %} ```bash $ sudo tail -f /var/log/pf9/ostackhost.log | grep -i -E "|connect_volume|disconnect_volume|lun-|multipath" ``` {% endcode %} The VM must be in `stopped` state before the start. Reset the state only if the VM is in `error`, then stop, then start. {% hint style="info" %} For background on state resets, see [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states). {% endhint %} {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:vm_state -c OS-EXT-STS:task_state $ openstack server set --state active # only if vm_state is error $ openstack server stop $ openstack server start ``` {% endcode %} The `disconnect_volume` call inside `_hard_reboot` is the step that fails on a corrupt record. With the corrected record the call walks the correct LUNs and passes. {% endstep %} {% step %} ### Step 8 — Confirm the domain's backing devices Confirm the guest is on the right disks, then confirm the guest answers on its own addresses. {% code title="Compute Host" %} ```bash $ sudo virsh domblklist --details ``` {% endcode %} {% code title="Sample Output" %} ```bash Type Device Target Source ------------------------------------------------------------ block disk vda /dev/disk/by-id/dm-uuid-mpath-[WWID] block disk vdb /dev/disk/by-id/dm-uuid-mpath-[WWID] ``` {% endcode %} {% endstep %} {% step %} ### Step 9 — Delete the leaked attachments {% hint style="danger" %} The incident is not closed until this step runs. A booted VM with the leaked attachments still in place means a non-multiattach volume is exposed read-write to two or three hosts. That is the data-corruption risk the whole repair exists to remove. {% endhint %} Delete each attachment belonging to a host that is not holding the domain. Use the API — the API drives the driver's `terminate_connection`, which is what actually removes the VLUN from the array. {% hint style="warning" %} Never delete `volume_attachment` rows with SQL. A database delete skips `terminate_connection` and orphans the array export permanently, with no record left to clean up from. Editing `connection_info` is a different and much smaller risk; deleting attachments is not. {% endhint %} {% code title="OpenStack CLI" %} ```bash $ openstack --os-volume-api-version 3.27 volume attachment delete ``` {% endcode %} Confirm one attachment per volume, on the host holding the domain: {% code title="OpenStack CLI" %} ```bash $ openstack volume show -c status -c attachments ``` {% endcode %} {% code title="Sample Output" %} ```bash +-------------+------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------+------------------------------------------------------------------------------------------------------+ | attachments | [{'id': '[VOLUME_UUID]', 'attachment_id': '[SOURCE_ATTACHMENT_UUID]', 'host_name': '[SOURCE_HOST]', ...}] | | status | in-use | +-------------+------------------------------------------------------------------------------------------------------+ ``` {% endcode %} Confirm the maps are gone on each former destination host: {% code title="Compute Host" %} ```bash $ sudo multipath -ll | grep -E "|" ``` {% endcode %} {% code title="Sample Output" %} ```bash (no output) ``` {% endcode %} {% endstep %} {% step %} ### Step 10 — Clear leaked Placement allocations {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl -n "" deploy/nova-api-osapi -- nova-manage placement audit --verbose ``` {% endcode %} {% endstep %} {% step %} ### Step 11 — Check the collateral victims If the VM was started or rebooted while the record was corrupt, os-brick removed SCSI paths from whatever volumes occupied the stale LUN numbers on the host. The FC rescan in Step 2 restores those paths, but confirm the path counts — another VM may be running degraded. Identify what sat at each stale LUN on the host holding the domain, then check that volume's health: {% code title="Compute Host" %} ```bash $ ls -l /dev/disk/by-path/ | grep 'lun-' $ sudo multipath -ll ``` {% endcode %} {% code title="Sample Output" %} ```bash [VICTIM_WWID] dm-[N] 3PARdata,VV size=[SIZE]G features='[FEATURES]' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running |- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running `- [H]:[C]:[T]:[STALE_LUN] sd[X] [MAJOR:MINOR] active ready running ``` {% endcode %} Find which instance owns the victim volume: {% hint style="info" %} **Self-Hosted only:** This query requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```sql mysql > select instance_uuid, volume_id, device_name from nova.block_device_mapping where connection_info like '%%' and deleted=0; ``` {% endcode %} {% hint style="success" %} The repair is complete when the VM is `ACTIVE`, `virsh domblklist` shows the expected multipath devices, every affected volume shows exactly one attachment on the host holding the domain, no maps for the volume remain on former destination hosts, and every collateral-victim volume shows four `active ready running` paths. {% endhint %} {% endstep %} {% endstepper %} ## Additional Information **Source attachment removed (`deleted = 1` or no row):** When the source host's attachment has been deleted, the array VLUN was removed by `terminate_connection` and the mapping must be recreated by the Block Storage Service with a fresh attachment rather than a corrected record. This is a longer procedure — contact [Platform9 Support](https://support.platform9.com/). The sequence in that case is: SQL-patch `connection_info` to a LUN that resolves to nothing on the host so the disconnect no-ops harmlessly, then run `nova-manage volume_attachment refresh`, then start the VM. **Why `nova-manage volume_attachment refresh` is the wrong tool for a corrupt record:** The command exists to regenerate `connection_info` from a host connector and would produce correct values, but the command cannot run against a corrupt record. `_do_refresh` sends `remove_volume_connection` to the compute host before creating the new attachment, and that disconnect uses the current, wrong `connection_info`. `remove_volume_connection` swallows only `exception.NotFound`, so os-brick's `ExceptionChainer` propagates and aborts the command. The `finally` block then leaves the BDM with a blank `attachment_id` and `connection_info = NULL`, which is strictly worse: the instance cannot start at all until a refresh succeeds. When the source attachment is intact, correct the record with SQL and skip `refresh` entirely. If `refresh` is run, four traps apply: | Trap | Consequence | Handling | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `CONF.host` read from the wrong config | Exit 7, `HostConflict` — Private Cloud Director sets `instance.host` to the host UUID, but `nova-manage` defaults to `/etc/nova/nova.conf` and yields the bare hostname | Always pass `--config-dir /opt/pf9/etc/nova/conf.d` | | `get_connector` has no `--path` option | Writes nothing; prints the connector | Redirect: `get_connector --json > file` | | `enforce_multipath=True` is hardcoded | Raises if multipathd is down | Check `systemctl is-active multipathd` first | | Requires `vm_state == stopped` exactly | Exit 5 on `error` | Reset to `active`, then stop | `get_connector` runs on the hypervisor; `refresh` needs database and Block Storage Service access, so `refresh` runs in a Compute Service pod on the control plane. The connector file has to be copied across. **Applicability:** The mechanism is the same on any array with per-host LUN numbering, including NetApp ONTAP igroups; only the array-side vocabulary changes. For NetApp iSCSI backends, follow the iSCSI article linked below. **Duplicate iSCSI initiator names:** In the originating case, both destination hosts reported the same iSCSI IQN — a cloned host image whose `/etc/iscsi/initiatorname.iscsi` was never regenerated. Harmless for Fibre Channel, dangerous the day either host touches an iSCSI backend. See the related article below. ### Related articles * [VM Fails to Start After Live Migration Failure Due to iSCSI BDM Corruption](https://platform9.com/kb/pcd/storage/vm-fails-to-start-after-live-migration-failure-due-to-iscsi-bdm-corruption) — the same failure on NetApp ONTAP iSCSI backends. The SQL in that article patches `target_lun` and `target_luns` only; `target_luns` does not exist in Fibre Channel records, so that SQL must not be used for Fibre Channel volumes. * [How to Identify the LUN ID Associated with an Instance Volume](https://platform9.com/kb/pcd/storage/how-to-identify-the-lun-id-associated-with-an-instance-volume) — general procedure for mapping a volume to a LUN ID on a compute host. * [Unable to Start VM due to Multipath issue](https://platform9.com/kb/pcd/storage/unable-to-start-vm-due-to-multipath-issue) — recovery of stale multipath devices blocking a VM start. * [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states) — resetting volumes stuck in intermediate states. * [How to Rename iSCSI Initiator Names to the Existing Hypervisors](https://platform9.com/kb/pcd/storage/how-to-rename-iscsi-initiator-names-to-the-existing-hypervisors) — regenerating duplicate IQNs on cloned hosts. # High CPU Usage Due to Orphaned VM Processes on Hypervisor Nodes ### Problem In some environments, compute nodes may experience elevated CPU utilization caused by orphaned virtual machine (QEMU) processes. This occurs when certain instances are deleted or migrated at the control-plane level (Compute DB), but their corresponding QEMU processes continue to run on the hypervisor. Symptoms include: * QEMU processes consuming CPU although the VM no longer exists in the database. * Mismatch between the number of instances reported in Nova DB versus running on the hypervisor. * Periodic warnings in `nova.compute.manager` logs during instance power-state synchronization. ### Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Private Cloud Director Kubernetes – v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.6 and Higher * Component: Compute Service ### Cause Compute Service periodically performs a synchronization cycle where it validates running instances on the hypervisor against entries in the Compute database. In this case: * Several VMs were deleted or migrated, but their QEMU processes persisted on the source hypervisor. * These orphaned processes continued running because the default Nova behavior did not automatically clean them up. * As a result, CPU utilization on the affected hypervisor increased unnecessarily. Additionally, frequent instance deletions/migrations caused temporary discrepancies in instance counts during sync cycles, contributing to repeated warnings in the logs. #### Diagnostics **Power State Synchronization Warnings** Compute service logs indicate mismatches between DB-reported VM count and hypervisor-reported VM count: ```bash $ less /var/log/pf9/ostackhost.log WARNING nova.compute.manager [...] While synchronizing instance power states, found 83 instances in the database and 86 instances on the hypervisor. ``` These messages demonstrate: * The presence of additional QEMU processes not tracked in DB. * Consistent mismatch over multiple cycles. #### Resolution {% hint style="info" %} Below configuration is included as part of standard deployment from **PCD - v2025.10-180 and higher** {% endhint %} 1. To ensure Compute Service automatically cleans up orphaned QEMU processes, the following configuration was added to ***/opt/pf9/etc/nova/conf.d/nova\_override.conf*** on all affected compute hosts: 
$ vi /opt/pf9/etc/nova/conf.d/nova_override.conf 
   [DEFAULT]
   running_deleted_instance_action = reap
2. Restart the pf9-ostackhost service post adding the above change on the hypervisors. The restart of this service will also cleanup any VMs which are stuck in the deleting phase as per the Compute service database. 
$ sudo systemctl restart pf9-ostackhost
3. Effect of this configuration: When Nova detects an instance running on the hypervisor that does not exist in the Compute Service database: 1. Frees compute node CPU/memory resources. 2. Ensures hypervisor state aligns with Nova DB state. 3. Orphaned VMs are now automatically removed within the standard 30-minute sync cycle # How to Setup resource allocation ratios in PCD for the hosts in cluster? ## Problem How to setup and configure host resource like cpu, memory and disk allocation ratio in PCD? How to ensure the instances are spawned on hosts with specific resource allocation ratio mapped? ## Environment * Private Cloud Director Virtualization - v2025.6 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.6 and Higher ## Procedure > It is necessary to ensure that the target set of hosts to be used for the isolation should belong to a PCD host cluster and should be added to the newly created aggregate from the UI * From the PCD UI create a [**host cluster**](https://platform9.com/docs/private-cloud-director/private-cloud-director/virtualized-cluster#creating-a-cluster) and add the required hosts * From the PCD UI create an [**aggregate**](https://platform9.com/docs/private-cloud-director/private-cloud-director/host-aggregate) with required hosts mapped into it. * Set the required metadata tag to the aggregate: ![](https://uploads.developerhub.io/prod/K9yg/zc8hs146z4mbx2a4zuvewuwti9arkaw6t27qrdq7hm1o1rfizekb1w8sd0p7bqri.png) * Create a flavor with matching metadata to the aggregate thus created: ![](https://uploads.developerhub.io/prod/K9yg/4fteecpwoj4iqqvfdz98ukx85denqg858hqz6xta5fhu03wx6p3sro0tvmy4m60q.png) * From the clusters host page, select the specific host that is part of the host aggregate as well the host cluster and set the allocation ratio values: ![](https://uploads.developerhub.io/prod/K9yg/2frknp3vbd6of8e9o1w9r3ya83c53ztmikdv0gw58wv6u3u2dh27y9uhd38dc7zw.png) ## Validation * Deploy an instance by selecting the cluster to which the specific host belongs: ![](https://uploads.developerhub.io/prod/K9yg/ab6qkj6cct3skz2bvtlvthf22attn6h7j1z9fxdjl7vxgyaat9tja4344nib6dk2.png) * In the next section select the flavor that has the specific tag matching and applied: ![](https://uploads.developerhub.io/prod/K9yg/ilxasn6iyj90aaxs12c40f32h1ag9kvw6p8skae2p2ath6ck9e9sn1qk17buj23j.png) * Complete the subsequent instance creation form and continue to deploy the VM. ## Additional Information * The actual allocation ratios will be placed within the placement API DB instead of host-level changes. * When successfully applied, the values can be reviewed by looking at the resource provider inventory list for the specific hosts: ```bash $ openstack resource provider list +-------------------+---------------------+------------+--------------------+----------------------+ | uuid | name | generation | root_provider_uuid | parent_provider_uuid | +-------------------+---------------------+------------+--------------------+----------------------+ | [RESOURCE1_UUID] | [HOST1.EXAMPLE.COM] | 4 | [PROVIDER1_UUID] | None | | [RESOURCE2_UUID] | [HOST2.EXAMPLE.COM] | 8 | [PROVIDER2_UUID] | None | | [RESOURCE3_UUID] | [HOST3.EXAMPLE.COM] | 4 | [PROVIDER3_UUID] | None | | [RESOURCE4_UUID] | [HOST4.EXAMPLE.COM] | 3 | [PROVIDER4_UUID] | None | +-------------------+---------------------+------------+--------------------+----------------------+ $ openstack resource provider inventory list +----------------+------------------+----------+----------+----------+-----------+-------+------+ | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | VCPU | 1.0 | 1 | 8 | 0 | 1 | 8 | 0 | | MEMORY_MB | 1.0 | 1 | 15988 | 512 | 1 | 15988 | 0 | | DISK_GB | 9999.0 | 1 | 125 | 0 | 1 | 125 | 0 | +----------------+------------------+----------+----------+----------+-----------+-------+------+ ``` # VM Build Failure Due to Nova Timeout ## Problem The VM build is failing, and the following errors are seen in `ostackhost log`: 
INFO nova.compute.resource_tracker [[REQ_UUID] None None] Instance [VM_UUID] has allocations against this compute host but is not found in the database.
ERROR nova.compute.manager [[REQ_UUID] None None] Error updating resources for node [HOSTNAME]: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID [MSG_UUID]
## Environment * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component: Compute Service ## Cause The root cause is the RPC response timeout threshold being exceeded for communication between **Nova Compute** and **Nova Conductor** via **RabbitMQ**. > Optimizing the timeout value based on environmental factors can improve performance. In scenarios with slower networks or other delays, adjusting the Nova configuration helps ensure efficient request processing. ## Workaround Increase the following parameters in the Nova configuration to allow more time for RPC communication and service reporting: | Parameter | Default Value | Updated Value | | ---------------------- | ------------- | ------------- | | `rpc_response_timeout` | 120s | **180s** | | `report_interval` | 60s | **120s** | **1. On the Management Cluster (Control Plane)** * Update the `nova-etc` Kubernetes secret with the new values. * Restart the `nova-api-osapi` pods to pick up the changes. * Verify the updated values are reflected in: ```bash /etc/pf9/nova/conf.d/nova.conf ``` **2. On Each Compute Host** * Update the following file with the new parameter values: ```bash /opt/pf9/etc/nova/conf.d/nova.conf ``` Add or modify under the `[DEFAULT]` section: ```ini [DEFAULT] rpc_response_timeout = 180 report_interval = 120 ``` * Restart the `pf9-ostackhost` service: ```bash sudo systemctl restart pf9-ostackhost ``` > **Note:** This change must be applied on **all compute hosts**. ## Validation After applying the changes to both the control plane and all compute hosts: 1. Attempt to provision multiple VMs concurrently (burst provisioning). 2. Monitor the `ostackhost.log` on the compute hosts for any recurrence of `MessagingTimeout` errors. 3. Check RabbitMQ pod logs for a reduction in `missed heartbeats from client` errors. 4. Confirm VMs are successfully built without entering ERROR or stuck BUILD state. # VM Fails to Start After Flavor Resize ## Problem After successfully resizing the flavor of the VM, it is unable to start. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Compute Service ## Cause There is a transient desynchronization between the Compute service and the hypervisor, where the instance state in the database did not match the actual state on the compute node. ## Diagnostics 1. Verify if the reported VM is present in the libvirt database on the compute host. {% code title="Host" overflow="wrap" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} 2. Review the `/var/log/pf9/ostackhost.logs` over the compute host for the impacted VM. It complains around the soft reboot of the VM after successful flavor resize, has not completely deleted the older allocations \[from the old flavor]. {% code title="/var/log/pf9/ostackhost.logs" overflow="wrap" %} ```bash INFO nova.compute.manager [[REQUEST_ID] [USER_NAME] service] [instance: [INSTANCE_ID]] Deleting allocations for old flavor on source node [COMPUTE_ID] after finish_resize failure. You may be able to recover the instance by hard rebooting it. ``` {% endcode %} 3. The `ostackhost.log` also points to a mismatch in the instance count and intermittent "*Instance not found*” messages. {% code title="/var/log/pf9/ostackhost.logs" overflow="wrap" %} ```bash WARNING nova.compute.manager: While synchronizing instance power states, found 3 instances in the database and 2 instances on the hypervisor. WARNING nova.compute.manager [instance: [INSTANCE_ID]]: Instance is unexpectedly not found. Ignore. ``` {% endcode %} ## Resolution * Perform a [**HARD REBOOT**](https://docs.platform9.com/private-cloud-director/virtualized-clusters/virtualmachine/virtual-machine-actions#hard-reboot) of the affected VM to force a resynchronization between Nova and the hypervisor, which will bring the VM back to a healthy state. ## Validation * Confirm the VM is now in **ACTIVE** state: ```bash $ openstack server show -c status ``` * Ensure it is hosted correctly: ```bash $ openstack server show -c OS-EXT-SRV-ATTR:hypervisor_hostname ``` * Ensure the VM is present in the active libvirt domains on the hypervisor. {% code title="Host" overflow="wrap" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} # Unable to Create VM On Compute Host- Aggregate Sync Issue ## Problem When attempting to create a VM on a newly added compute node, the following error is observed in the UI or through the CLI: ```ruby No valid host was found. There are not enough hosts available. ``` Additionally, the nova-scheduler pod and nova-api-osapi logs show the following messages: ```ruby $ kubectl logs deploy/nova-scheduler -n Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up. ``` ```ruby $ kubectl logs deploy/nova-api-osapi -m WARNING nova.compute.api [None [REQ_ID] [RESROUCE_UUID] - - default default] Failed to associate [PROVIDER_NAME] with a placement aggregate: No such resource provider [PROVIDER_NAME].. This may be corrected after running nova-manage placement sync_aggregates.: nova.exception.ResourceProviderNotFound: No such resource provider [PROVIDER_NAME]. ``` The above message in logs indicate that the issue is the resource provider mapping for the new/existing compute node is incorrect ## Environment * Platform9 Private Cloud Director - v2025.4 and Higher * Self Hosted Private Cloud Director Virtualization – v2025.4 and Higher * Component - Compute ## Cause This issue may occur due to host-aggregate synchronisation problems. Specifically, the compute host may not be properly recognised by the Placement API as part of the aggregate, which leads to the scheduler being unable to locate valid hosts for allocation. > For SaaS users who do not have access to the DB, an alternative way to identify the affected host is to migrate an existing test VM to new or existing hosts.> > For Self-Hosted PCD this can be validated using the below steps: Validate if the resource mapping is completed by comparing the data of the existing nodes to the new node in the placement DB by following below steps 1. Log into the database ```bash $ kubectl exec -it deploy/mysqld-exporter -n -c mysqld-exporter -- mysql resmgr -u root -p ``` 2. Switch to the placement DB ```sql MySQL [resmgr]> use placement; ``` 3. Check the resource provider mapping is created in the tables *resource\_provider\_aggregates* and *resource\_providers*. If the resource details are not mapped correctly, there will be a discrepancy observed with the *created\_at* and *updated\_at* timestamps in the below two tables with the timestamp when the node was onboarded. ```sql MySQL [placement]> select * from resource_provider_aggregates; +---------------------+------------+----------------------+--------------+ | created_at | updated_at | resource_provider_id | aggregate_id | +---------------------+------------+----------------------+--------------+ | 2025-06-07 19:15:45 | NULL | 144 | 3 | | 2025-06-07 19:15:10 | NULL | 151 | 3 | | 2025-06-09 22:45:13 | NULL | 4671 | 3 | | 2025-06-09 23:33:38 | NULL | 4672 | 3 | | 2025-06-09 23:51:00 | NULL | 4673 | 3 | +---------------------+------------+----------------------+--------------+ ``` ```sql MySQL [placement]> select * from resource_providers; +---------------------+---------------------+------+---------------------------+---------------------+------------+------------------+--------------------+ | created_at | updated_at | id | uuid | name | generation | root_provider_id | parent_provider_id | +---------------------+---------------------+------+---------------------------+---------------------+------------+------------------+--------------------+ | 2025-05-28 12:42:40 | 2025-06-05 17:23:20 | 144 | [resource_provider_UUID1] | [HOST1.EXAMPLE.COM] | 432 | 144 | NULL | | 2025-05-28 12:59:01 | 2025-06-10 17:38:18 | 151 | [resource_provider_UUID2] | [HOST2.EXAMPLE.COM] | 711 | 151 | NULL | | 2025-06-09 08:16:12 | 2025-06-10 17:23:21 | 4671 | [resource_provider_UUID3] | [HOST3.EXAMPLE.COM] | 1058 | 4671 | NULL | | 2025-06-09 08:16:52 | 2025-06-10 17:38:18 | 4672 | [resource_provider_UUID4] | [HOST4.EXAMPLE.COM] | 951 | 4672 | NULL | | 2025-06-09 08:55:40 | 2025-06-10 17:23:20 | 4673 | [resource_provider_UUID5] | [HOST5.EXAMPLE.COM] | 434 | 4673 | NULL | +---------------------+---------------------+------+---------------------------+---------------------+------------+------------------+--------------------+ ``` In the above sample output, the *updated\_at* timestamp (2025-06-05 17:23:20) in *resource\_providers* table for id "144" is older than the *created\_at* (2025-06-07 19:15:45) timestamp from the *resource\_provider\_aggregates* which highlights the discrepancy. In this case, *host1.example.com* is affected by this issue. ## Resolution There are 2 ways of resolving this issue. **Option 1:** This is a preferred and much easier way to resolve this issue. The issue can be resolve by running nova-manage placement sync\_aggregates from the nova-api-osapi pod as below: ```bash $ kubectl exec -it deploy/nova-api-osapi -n -- nova-manage placement sync_aggregates ``` **Option 2:** The issue could be resolved by removing the problematic compute host from its host aggregate and re-adding it. This action triggers Nova to re-notify placement and successfully schedule VMs to the host. Steps: 1. List All Aggregates ```bash $ openstack aggregate list ``` 2. Inspect a Specific Aggregate and check if the problematic host is part of any aggregate: ```bash $ openstack aggregate show ``` 3. Remove the Host from Aggregate ```bash $ openstack aggregate remove host ``` 4. Add the Host Back to the Aggregate ```bash $ openstack aggregate add host ``` This action re-registers the host within the aggregate and updates Placement API visibility. ## Additional Information Ensure that the nova-compute service is running and registered: ```bash $ openstack compute service list --service nova-compute ``` Check for any errors in the ostackhost log on the hosts: ```bash /var/log/pf9/ostackhost.log ``` > For Self-Hosted PCD users, check the below pod logs: ```bash $ kubectl logs -n $ kubectl logs -n $ kubectl logs -n $ kubectl logs -n $ kubectl logs -n ``` # How to Enable Debug Logging? Debug logging provides detailed service-level information that helps you identify the root causes of issues when performing operations such as creating VMs, attaching volumes, etc. Follow these steps to enable debug logging for Compute and Persistent Storage services. ## Compute Edit the `nova_override.conf` file: Path: `/opt/pf9/etc/nova/conf.d/nova_override.conf` If the `[DEFAULT]` section does not exist, create it and add the following: ```bash [DEFAULT] debug = true ``` Restart the Compute service: ```bash systemctl restart pf9-ostackhost ``` ### Persistent Storage Edit the `cinder_override.conf`file: Path: `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf` If the `[DEFAULT]` section does not exist, create it and add the following: ```bash [DEFAULT] debug = true ``` Restart the Storage service: ```bash systemctl restart pf9-cindervolume-base ``` ### Networking Edit the `neutron.conf` file: Path: `/opt/pf9/etc/neutron/conf.d/neutron.conf` If the `[DEFAULT]` section does not exist, create it and add the following: ```bash [DEFAULT] debug = true ``` Restart the Neutron service ```bash systemctl restart pf9-ostackhost ``` ## Reverting Debug Logging Debug logging generates large log files that may consume significant disk space. Once troubleshooting is complete, revert the changes to preserve storage. Open the respective configuration file and remove the `debug = true` line or set it to `debug = false`. Restart the service: ```bash systemctl restart pf9-ostackhost ``` # Unable to Deploy Windows VM Using ISO With AMD-Based CPU ## Problem Windows VM using ISO is failing with the following error in the PCD UI on hypervisor with AMD-based CPU. ```ruby Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance [UUID]. Traceback (most recent call last): File "/var/lib/openstack/lib/python3.10/site-packages/nova/conductor/manager.py", line 705, in build_instances raise exception.MaxRetriesExceeded(reason=msg) nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance [UUID]. ``` ## Environment * Private Cloud Director - Virtualisation: v2025.4 and Higher. * Self-Hosted Private Cloud Director - Virtualisation: v2025.4 and Higher. * Component - Compute Service ## Cause The failure occurs because the guest XML generated during instance creation is attempting to enable a feature called Hyper-V Enlightened VMCS (hv-evmcs). This feature is specific to Intel processors and is not supported on AMD CPUs. ```ruby File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags\n raise libvirtError(\'virDomainCreateWithFlags() failed\')\n', 'libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: [TIMESTAMP] qemu-system-x86_64: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel\n' ``` ## Resolution To prevent the use of `hv-evmcs` when launching instances on AMD-based hosts, update the image metadata to disable Hyper-V enlightenments: ```bash openstack image set --property hw_machine_type=q35 openstack image set --property hw_cpu_hyperv=off ``` # Unable to Create Virtual Machine on Compute Host – Host Not Mapped to Any Cell ## Problem Virtual Machine creation fails with the following error: ```bash Host '[HOST_UUID]' is not mapped to any cell ``` ## Environment * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher > For Private Cloud Director (SaaS) users, if you encounter the same issue, please contact [Platform9 Support](https://support.platform9.com/hc/en-us). ## Cause The compute host was not properly mapped to any Nova cell. This can happen if: * The host was added without triggering automatic cell discovery. * The host was removed or modified in a way that removed its mapping from the cell database. Without this mapping, Nova is unable to schedule workloads on the host, even if it appears active and functional. ## Diagnostics 1. Run the following command from the *nova-scheduler* pod on Management Plane to see if the host is linked to a cell: ```bash $ nova-manage cell_v2 list_hosts +--------------+--------------+---------------------+ | Cell Name | Cell UUID | Hostname | +--------------+--------------+---------------------+ | [CELL_NAME] | [CELL_UUID] | [HOST1.EXAMPLE.COM] | | [CELL_NAME] | [CELL_UUID] | [HOST2.EXAMPLE.COM] | +--------------+--------------+---------------------+ ``` 2. The compute service on the host still might be visible and enabled: ```bash $ openstack compute service list --host +------------------------+--------------+---------------------+-----------------+---------+-------+--------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +------------------------+--------------+---------------------+---------------------------+-------+--------------+ | [COMPUTE_SERVICE_UUID] | nova-compute | [HOST3.EXAMPLE.COM] | [CLUSTER_NAME] | enabled | up | [TIME_STAMP] | +------------------------+--------------+--------------+------------------------+---------+-------+--------------+ ``` ## Resolution 1. Run the following command from *nova-scheduler* pod on Management Plane to find any unmapped compute hosts and create their cell mappings: ```bash $ nova-manage cell_v2 discover_hosts --verbose --by-service Found [CELL_MAPPINGS_COUNT] cell mappings. Getting computes from cell '[CELL_NAME]': [CELL_UUID] Creating host mapping for service [HOST3.EXAMPLE.COM] Found [MISSING_COMPUTES_COUNT] unmapped computes in cell: [CELL_UUID] ``` ## Validation 1. Confirm the host is mapped to a cell. ```bash $ nova-manage cell_v2 list_hosts +--------------+-------------+---------------------+ | Cell Name | Cell UUID | Hostname | +--------------+-------------+---------------------+ | [CELL_NAME] | [CELL_UUID] | [HOST1.EXAMPLE.COM] | | [CELL_NAME] | [CELL_UUID] | [HOST2.EXAMPLE.COM] | | [CELL_NAME] | [CELL_UUID] | [HOST3.EXAMPLE.COM] | +--------------+-------------+---------------------+ ``` 2. Verify the compute service is active ```bash $ openstack compute service list --host +------------------------+--------------+---------------------+-----------------+---------+-------+--------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +------------------------+--------------+---------------------+---------------------------+-------+--------------+ | [COMPUTE_SERVICE_UUID] | nova-compute | [HOST3.EXAMPLE.COM] | [CLUSTER_NAME] | enabled | up | [TIME_STAMP] | +------------------------+--------------+--------------+------------------------+---------+-------+--------------+ ``` 3. Attempt to launch a virtual machine and confirm it succeeds. # VM Creation Failure Due to Image Role Host and Stale OVN NetNS Issue ## Problem VM creation was failing at the volume-from-image stage due to the image role host being in a failed state. The failure was linked to a stale OVN metadata network namespace, which caused the `pf9-neutron-ovn-metadata-agent` to crash, impacting Glance functionality. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component: Networking and Image Service ## Cause The root cause was traced to the Image role host being in a failed state. Additionally, a stale/corrupted OVN metadata namespace caused the `pf9-neutron-ovn-metadata-agent` service to fail, which contributed to network namespace issues affecting Image service and possibly volume creation. ## Diagnostics * Image service backend was not visible in the `openstack volume backend pool list` command output. * `pf9-neutron-ovn-metadata-agent` was in a failed state with the following critical log in `/var/log/pf9/pf9-neutron-ovn-metadata-agent.log`: ```bash [TimeStamp] CRITICAL neutron [-] Unhandled error: OSError: [Errno 22] failed to open netns ``` * Stale/invalid NETNS entry found under `/var/run/netns`: {% code title="Affected Host:" %} ```bash $ ip netns list ovnmeta-[ovnnetns-id-1] (id: 0) ovnmeta-[ovnnetns-id-2] $ ls -l /var/run/netns total 0 -r--r--r-- 1 root root 0 [Time Stamp] ovnmeta-[ovnnetns-id-1] ---------- 1 root root 0 [Time Stamp] ovnmeta-[ovnnetns-id-2] ``` {% endcode %} * The invalid namespace had no permissions and was inaccessible.. ## Resolution * Identified that `pf9-neutron-ovn-metadata-agent` was in a failed state. * Found a stale NETNS entry with invalid permissions: ```bash ---------- 1 root root 0 Jun [Time Stamp] ovnmeta-[ovnnetns-id-2] ``` * Deleted the stale netns: ```bash $ sudo ip netns delete ovnmeta- ``` * Restarted the metadata agent: ```bash $ sudo systemctl restart pf9-neutron-ovn-metadata-agent ``` * Restarted the `glance-api` service to ensure backend availability: ```bash $ sudo systemctl restart pf9-glance-api ``` ## Validation * Post-remediation, `pf9-neutron-ovn-metadata-agent` started successfully without errors. * Image service backend was registered properly. * VM creation using image-backed volume was tested and succeeded. # Unable to Delete Stale Compute Service ## Problem * Unable to delete stale compute service with the below error. ```ruby $ openstack compute service delete [COMPUTE_SERVICE_UUID] Failed to delete compute service with ID '[COMPUTE_SERVICE_UUID]': HttpException: 500: Server Error for url: https://[PCD_FQDN]/nova/v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID], Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. 1 of 1 compute services failed to delete. ``` ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Compute Service ## Cause * Forcefully removing a host from PCD, resulting in incomplete cleanup of resources on the management plane. ## Diagnostics * Compute Service List: ```bash $ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+---------------------+---------+--------+ | ID | Binary | Host | Status | State | +---------------------------+----------------+---------------------+----------+-------+ | [COMPUTE_SERVICE_UUID_1] | nova-scheduler | nova-scheduler | enabled | up | | [COMPUTE_SERVICE_UUID_2] | nova-conductor | nova-conductor | enabled | up | | [COMPUTE_SERVICE_UUID_3] | nova-compute | [HOST1.EXAMPLE.COM] | enabled | up | +---------------------------+----------------+---------------------+----------+-------+ ``` * Resource Provider List: ```bash $ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+ | uuid | name | +--------------------------------------+---------------------+ | [RESOURCE_PROVIDER_UUID_1] | [HOST1.EXAMPLE.COM] | | [RESOURCE_PROVIDER_UUID_2] | [HOST2.EXAMPLE.COM] | +--------------------------------------+---------------------+ ``` * Nova API Pod Logs: * First grep the Nova API logs with compute service UUID and Note the Deletion Request ID. {% code title="nova-api logs:" %} ```ruby $ kubectl logs -n | grep INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[åçPROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756 ``` {% endcode %} Now grep the Nova API logs with Deletion Request ID and Note the Missing Resource Provider UUID. {% code title="nova-api logs:" %} ```ruby $ kubectl logs -n | grep Defaulted container "nova-osapi" out of: nova-osapi, socat18774, init (init) ERROR nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] Unexpected exception in API method: ValueError: No such provider [MISSING_RESOURCE_PROVIDER_UUID] INFO nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_IDD] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756 /var/lib/openstack/lib/python3.10/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: unknown. To ensure interoperability, identifiers should be a valid uuid. ``` {% endcode %} * Based on above logs, as part of compute service deletion Nova is trying to delete the associated resource provider in Placement, but it's unable to locate it in the ProviderTree. ## Resolution * Created a dummy Resource Provider with the same UUID that Nova is looking for during compute service deletion. ```bash $ openstack resource provider create --uuid= temp_provider Expecting value: line 1 column 1 (char 0) ``` * Verify if the dummy Resource Provider with the same UUID is created. ```bash $ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+ | uuid | name | +--------------------------------------+---------------------+ | [RESOURCE_PROVIDER_UUID_1] | [HOST1.EXAMPLE.COM] | | [RESOURCE_PROVIDER_UUID_1] | [HOST2.EXAMPLE.COM] | | [MISSING_RESOURCE_PROVIDER_UUID1] | temp_provider | +--------------------------------------+---------------------+ ``` * Now, Delete the Compute service. ```ruby $ openstack compute service delete ``` ## Validation * Check the Complete Services list. ```bash $ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+-------------------+---------+--------+ | ID | Binary | Host | Status | State | +---------------------------+----------------+-------------------+----------+-------+ | [COMPUTE_SERVICE_UUID_1] | nova-scheduler | nova-scheduler | enabled | up | | [COMPUTE_SERVICE_UUID_2] | nova-conductor | nova-conductor | enabled | up | +---------------------------+----------------+-------------------+----------+-------+ ``` ## Additional Information * Please consult our [documentation ](https://platform9.com/docs/private-cloud-director/private-cloud-director/add-hosts-Virtualized-Cluster#remove-a-host)for the recommended procedure for offboarding a host from PCD. # Live Migration Failure Due to CPU Model Mismatch ## Problem Instances fail to live migrate between hypervisors, with errors `Unacceptable CPU info: CPU doesn't have compatibility` indicating that no suitable host is found, even though resources are available. ## Environment * Private Cloud Director Virtualisation - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualisation - v2025.4 and Higher * Component: Compute Service ## Cause The issue was caused by a [CPU model ](https://platform9.com/docs/private-cloud-director/private-cloud-director/cpu-model)mismatch between the source and destination hypervisors. The source hypervisor was running a different CPU model compared to other compute nodes. Nova’s CPU compatibility checks during live migration failed because the CPU feature set did not match across the involved hosts. When Nova detects incompatible CPU models, it will reject the migration with a "**No valid host found"** error, even if other resources (CPU, RAM, Disk) are sufficient. ## Diagnostics 1. Verify the [supported CPU Model](https://platform9.com/docs/private-cloud-director/private-cloud-director/supported-cpu-models-list) from the underlying hypervisor: ```ruby $ virsh cpu-models x86_64 ... SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Haswell Haswell-IBRS ``` 2. Check the current **cpu\_models** on the source hypervisor from the file `/opt/pf9/etc/nova/conf.d/nova_override.conf` , and check whether it is same on other hypervisor hosts: ```bash $ cat /opt/pf9/etc/nova/conf.d/nova_override.conf [libvirt] cpu_mode = custom cpu_models = SandyBridge-IBRS ## Example CPU Model ``` 3. Migration attempts from any source hypervisor to a destination hypervisor with a different CPU model configuration failed, displaying the following error in `/var/log/pf9/ostackhost.log` on the destination host: ```bash nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. ``` ## Resolution 1. Ensure the source and destination hypervisor nodes have the supported CPU model `/opt/pf9/etc/nova/conf.d/nova_override.conf`. And this CPU model should be same as configured on other hypervisors. 2. Restart `ostackhost` service on hypervisor where the `cpu_model` is updated: ```bash $ sudo systemctl restart pf9-ostackhost ``` ## Validation 1. Re-attempt migration of VMs hosted on the source hypervisor to another hypervisor — Check if the migration is successful. 2. Check the `/var/log/pf9/ostackhost.log` logs to ensure no CPU mismatch errors were reported. # VM Console Access Failures Related to noVNC Service ## Problem > VM console access fails with a *“site can’t be reached”* error when attempting to open the console. Attempts to connect to the console proxy port (6080), including telnet or direct connection tests, also time out. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component - novncproxy service ## Cause The issue is often caused by the novncproxy service on the affected hypervisor getting into a bad state due to: * Stale TCP connections in the CLOSE\_WAIT state, indicating the service did not properly close old client sessions. * Possible stale processes or token mappings, leading to connection failures. * This condition prevents the service from establishing new console connections for VMs on that hypervisor. ## Diagnostics When VM console access fails (e.g., *“site can’t be reached”* or "*timeout*"), perform the following checks: 1. Check service status from affected hosts: ```bash $ systemctl status pf9-novncproxy.service ``` Ensure the service is active and running without any errors. 2. Validate time synchronisation ```bash $ timedatectl status ``` Look for “System clock synchronized: yes” → means NTP/chrony is active. Confirm the local time shown in the timedatectl status output matches the expected current time. If the host clock is out of sync, it can cause authentication token mismatches during VM console access. Correcting the time and ensuring synchronization with NTP/chrony is essential before retrying console access. 3. Test local connectivity from affected host to console port: ```bash $ curl -vk http://127.0.0.1:6080/ $ curl -vk https://:6080/ ``` > If you are seeing a timeout here, it indicates the console proxy service is not responding to connections. This could be due to firewall rules, routing issues, SSL problems, or the service being in a stale state. In this case, continue with reviewing the novncproxy logs and other diagnostics to narrow down the cause. 4. Review novncproxy logs: ```bash $ less /var/log/pf9/novncproxy.log ``` > Review the noVNC logs to confirm whether new entries are being written when you attempt console access. If the logs stop updating, this can indicate the service is stuck in a stale state. In this case, continue by checking for stale or blocked connections. 5. Check for stale or blocked connections: ```bash $ lsof -i :6080 ``` If multiple CLOSE\_WAIT entries are observed for pf9-novncproxy, it indicates that remote connections were closed but the service did not release those sockets. Such stale connections can accumulate and eventually block new console sessions. ## Resolution Depending on the findings in the Diagnostics section, use the corresponding resolution: 1. If service is in-active or unhealthy ```bash $ sudo systemctl restart pf9-novncproxy.service ``` Restarts the noVNC proxy and re-establishes connections. 2. If stale TCP connections (CLOSE\_WAIT) are found. Then restart the novncproxy service (as above) to clear the stale sessions. 3. If time drift is detected as, correct the time using your NTP/chrony configuration and restart the service: ```bash $ sudo systemctl restart chronyd # or ntpd _ timesyncd depending on setup $ timedatectl status $ ntpq -p 2>/dev/null || chronyc sources $ sudo systemctl restart pf9-novncproxy.service ``` > These commands restart the system’s NTP/chrony service to re-sync the clock, check the current time status and offset, verify synchronization sources, and then restart the noVNC proxy service to ensure fresh tokens are generated using the corrected system time. > > (Optional): If none of the above steps resolve the issue, remove the hypervisor from the cluster and decommission it (removing all roles). Then, add the node back into PCD and re-assign the roles. This re-provisions the host configuration and can clear any underlying state inconsistencies. ## Additional Information Additionally, reviewing system logs (e.g., `/var/log/syslog` or `/var/log/messages`) and `/var/log/pf9/pf9-ostackhost.log` can provide deeper insight into underlying issues affecting console connectivity. These logs may reveal host-level errors, service crashes, or dependency failures that are not visible in the noVNC logs, and can be valuable for further analysis before taking corrective action. # VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service ## Problem Virtual machines configured with the `hw:numa_nodes=1` flavor extra spec shut down unexpectedly without any user-initiated action. The Linux kernel OOM killer terminates the QEMU process running the VM when the assigned NUMA memory node is exhausted — even when the host has hundreds of gigabytes of available RAM on other NUMA nodes. In OpenStack, the affected VM transitions to an error or stopped state. The shutdown is visible in the Compute Host syslog as an OOM kill event with `constraint=CONSTRAINT_MEMORY_POLICY`. ## Environment * Private Cloud Director Virtualization — version **v2026.4-89** and below. * Self-Hosted Private Cloud Director Virtualization — version **v2026.4-89** and below. * Component: Compute Service ## Cause When a VM is created with `hw:numa_nodes=1`, the Compute Service generates a libvirt domain XML that pins the QEMU process to a specific NUMA memory node with a policy of `strict`. Under strict policy, the QEMU process is restricted exclusively to the assigned NUMA node and cannot access RAM on any other node — even when the host has ample memory available elsewhere. When the assigned node's memory is exhausted, the Linux kernel OOM killer terminates the QEMU process, shutting down the VM immediately. The strict policy is hardcoded in the Compute Service libvirt driver (`designer.py`). There is currently no flavor extra spec, image property, or Compute Service configuration option to override this default behavior. ## Diagnostics The following steps confirm whether the VM was shut down by a NUMA-policy OOM kill and not by general host memory exhaustion or another cause. Run these steps on the Compute Host where the VM was running at the time of the unexpected shutdown. {% stepper %} {% step %} **Step 1 — Check the Compute Host Syslog for an OOM Kill with NUMA Policy Constraint** Search the syslog on the affected Compute Host for an OOM kill event. The keyword `CONSTRAINT_MEMORY_POLICY` confirms the kill was caused by the strict NUMA memory policy and not by general host memory exhaustion. {% code title="Compute Host" %} ```bash $ grep -i "CONSTRAINT_MEMORY_POLICY" /var/log/syslog ``` {% endcode %} {% code title="Compute Host - /var/log/syslog." overflow="wrap" %} ``` [TIMESTAMP] [HOSTNAME] kernel: oom-kill:constraint=CONSTRAINT_MEMORY_POLICY,nodemask=[NUMA_NODE],cpuset=vcpu[CPU_ID],mems_allowed=[NUMA_NODE],global_oom,task=qemu-system-x86,pid=[PID],uid=[UID] [TIMESTAMP] [HOSTNAME] kernel: Out of memory: Killed process [PID] (qemu-system-x86) total-vm:[VM_SIZE]kB, anon-rss:[RSS_SIZE]kB, file-rss:[FILE_RSS]kB, shmem-rss:0kB ``` {% endcode %} {% endstep %} {% step %} **Step 2 — Confirm NUMA Node Memory State** Check the current memory state of each NUMA node on the affected Compute Host to confirm the assigned node was exhausted while other nodes had free memory available. {% code title="Compute Host" %} ```bash $ cat /sys/devices/system/node/node0/meminfo $ cat /sys/devices/system/node/node1/meminfo ``` {% endcode %} {% code title="Sample Output" %} ``` Node 1 MemTotal: 1056888716 kB Node 1 MemFree: 529672 kB Node 1 MemUsed: 1056359044 kB ``` {% endcode %} A NUMA node with `MemFree` near zero while other nodes show substantial free memory confirms a NUMA-policy exhaustion event rather than host-level memory exhaustion. {% endstep %} {% step %} **Step 3 — Confirm the VM XML Has a Strict NUMA Memory Policy** Retrieve the libvirt domain XML for the affected VM and check the `numatune` section. The value `mode='strict'` confirms the VM was running with strict NUMA memory policy. To find the libvirt domain name, run `virsh list --all` on the Compute Host and look for the `instance-XXXXXXXX` entry corresponding to the VM UUID. {% code title="Compute Host" %} ```bash $ virsh dumpxml | grep -A5 "numatune" ``` {% endcode %} {% code title="Sample Output" %} ```xml ``` {% endcode %} {% endstep %} {% step %} **Step 4 — Confirm the VM Flavor Has hw:numa\_nodes Set** Identify the flavor used by the affected VM and confirm the `hw:numa_nodes` extra spec is present. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c flavor ``` {% endcode %} {% code title="Sample Output" %} ``` +--------+-------------------+ | Field | Value | +--------+-------------------+ | flavor | [FLAVOR_NAME] | +--------+-------------------+ ``` {% endcode %} {% code title="OpenStack CLI" %} ```bash $ openstack flavor show -c properties ``` {% endcode %} {% code title="Sample Output" %} ``` +------------+-------------------+ | Field | Value | +------------+-------------------+ | properties | hw:numa_nodes='1' | +------------+-------------------+ ``` {% endcode %} {% endstep %} {% endstepper %} If all checks above match — `CONSTRAINT_MEMORY_POLICY` in syslog, `mode='strict'` in the VM XML, and `hw:numa_nodes` set on the flavor — proceed to the [Workaround](#workaround) below. ## Resolution A permanent fix is tracked in **PCD-8069**. The fix introduces a new flavor extra spec `hw:numa_mempolicy` that allows operators to configure the NUMA memory policy (`preferred`, `strict`, or `interleave`) at the flavor level. And the corresponding changes will pushed in to the files `/opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/designer.py` and `/opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py` . Once delivered, the Compute Service will read this extra spec when generating the libvirt domain XML instead of defaulting to `strict`. Until the fix is released, use the [Workaround](#workaround) below. ## Workaround The suggested workaround provides immediate relief with no code changes but is not persistent across Nova-level operations. {% hint style="warning" %} The workaround suggested require a brief VM shutdown and restart. There is no zero-downtime path to change the NUMA memory policy of a running VM. OpenStack `server reboot` (hard or soft), `server stop/start`, live migration, and cold migration were tested — none update the NUMA memory policy without the code fix in place. Coordinate with the VM owner before proceeding. {% endhint %} ### **Edit the VM XML Directly (Immediate, Not Persistent)** This method changes the NUMA memory mode for a specific VM by editing the libvirt domain XML directly on the Compute Host. No changes to Compute Host software files are required. **Persistence limitations:** The change is overwritten by the Compute Service whenever any of the following OpenStack-level operations regenerate the domain XML — `openstack server stop/start`, `openstack server reboot` (hard or soft), `openstack server rebuild`, and cold migration. The change persists across `pf9-ostackhost` service restart and live migration. Apply these steps to critical VMs as an immediate measure while planning for a persistent resolution in the patch release. {% stepper %} {% step %} **Step 1 — Edit the VM XML** On the Compute Host where the VM is running, open the VM's libvirt XML for editing. {% code title="Compute Host" %} ```bash $ virsh edit ``` {% endcode %} In the `numatune` section, change every occurrence of `mode='strict'` to `mode='preferred'`: ```xml ``` Save and exit the editor. The `virsh edit` command validates the XML before saving — if an error appears, re-check the syntax and save again. {% endstep %} {% step %} **Step 2 — Shut Down the VM** Shut down the VM. A restart is required for the updated memory mode to take effect. {% code title="Compute Host" %} ```bash $ virsh shutdown ``` {% endcode %} Wait for the VM to reach the shut-off state before proceeding. {% code title="Compute Host" %} ```bash $ virsh domstate ``` {% endcode %} {% code title="Sample Output" %} ``` shut off ``` {% endcode %} {% endstep %} {% step %} **Step 3 — Start the VM** Start the VM to apply the updated domain XML. {% code title="Compute Host" %} ```bash $ virsh start ``` {% endcode %} {% endstep %} {% step %} **Step 4 — Verify the Memory Policy Is Now Preferred** Confirm the `numatune` section in the running VM's XML shows `mode='preferred'`. {% code title="Compute Host" %} ```bash $ virsh dumpxml | grep -A5 "numatune" ``` {% endcode %} {% code title="Sample Output" %} ```xml ``` {% endcode %} Verify the VM is running and accessible. The QEMU process now has access to memory on all NUMA nodes when the assigned node is under pressure. {% endstep %} {% endstepper %} {% hint style="warning" %} **Reapplication required after Nova-level operations:** The XML change made in the workaround is overwritten whenever OpenStack regenerates the domain XML. If `openstack server stop/start`, `openstack server reboot`, `openstack server rebuild`, or cold migration is performed on the VM, re-apply workaround until the permanent fix is available. {% endhint %} ## Additional Information This issue is tracked in **PCD-8069**. The engineering fix introduces a new `hw:numa_mempolicy` flavor extra spec that allows the NUMA memory policy to be configured at the flavor level. The fix modifies `designer.py` and `driver.py` in the Compute Service libvirt driver so the policy is read dynamically from the flavor instead of always defaulting to `strict`. The value is stored in `nova.instance_extra` (`InstanceNUMATopology`) so it persists across reboots and live migrations once the fix is applied and the VM is resized to the updated flavor. The version having the fix is not finalised yet. For more details on the ETA of the fix release, please contact Platform9 Support team. # VM Migration Failure Which Results the VM in "ERROR" State ## Problem Post live migration of VM, the **Nova API/DB** shows the VM status as **ERROR**, although the VM continues running successfully on the target hypervisor. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Componen: Compute Service ## Cause This issue occurs due to inconsistencies between the **Nova database** and the actual hypervisor state during/after migration. Different underlying issues can cause this problem. Below are the common causes: * Stale file locks on storage volumes. * Database state mismatch between Nova conductor and compute host. * Scheduler exceptions (e.g., `NoValidHost`) after migration completes. ## Diagnostics 1. **Stale file locks on storage volumes** * Migration fails with volume/lock errors. * The `/var/log/pf9/ostackhost.log` from the source compute host: ```bash nova.virt.libvirt.volume.mount [REQUEST_ID service] [instance: INSTANCE_ID] Request to remove attachment (volume-[VOLUME_ID] from [MOUNTPOINT], but we don't think it's in use. ERROR nova.virt.libvirt.driver [REQUEST_ID service] [instance: [INSTANCE_ID]] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: [TIMESTAMP] qemu-system-x86_64: -device virtio-blk-pci,..... Failed to get shared "write" lock ``` * Confirm Cinder volume status: ```bash $ openstack volume show ``` 2. **Database state mismatch between Nova and hypervisor** * The `openstack server show` reports **ERROR** or wrong hypervisor, but VM is running on **a different host**. * The `/var/log/pf9/ostackhost.log` from the source compute host: ```bash nova.exception_Remote.UnexpectedTaskStateError_Remote: Conflict updating instance [INSTANCE_ID] Expected: {\'task_state\': [\'migrating\']}. Actual: {\'task_state\': None}\nTraceback (most recent call last):\n\n File "/var/lib/openstack/lib/python3.10/site-packages/nova/db/main/api.py", line 2368, in _instance_update\n update_on_match(compare, raise NoRowsMatched("Zero rows matched for %d attempts" attempts)\n\noslo_db.sqlalchemy.update_match.NoRowsMatched: Zero rows matched for 3 attempts [TIMESTAMP] TRACE nova.compute.manager [instance: [INSTANCE_ID]] nova.exception.UnexpectedTaskStateError: Conflict updating instance [INSTANCE_ID]. Expected: {'task_state': ['migrating']}. Actual: {'task_state': None} ``` * Check Nova DB entry vs actual hypervisor: ```bash $ openstack server show -c OS-EXT-SRV-ATTR:hypervisor_hostname ``` * Verify if the VM is running on the above mentioned hypervisor and its state: ```bash $ virsh list --all | grep ``` > For Self-Hosted Private Cloud Director only 3. **Scheduler exceptions (e.g., `NoValidHost`) after migration** * Migration logs show `NoValidHost`, but VM still boots/migrates at the hypervisor level. * The nova-scheduler logs : ```bash $ kubectl logs -n NoValidHost(reason="")\\n\\nnova.exception.NoValidHost: No valid host was found. ``` ## Resolution To restore consistency and recover the VM: * Set the VM state to Active ```bash $ openstack server set --state active ``` * Start the VM: ```bash $ openstack server start ``` ## Validation * Confirm the VM is now in **ACTIVE** state: ```bash $ openstack server show -c status | status | ACTIVE | ``` * Ensure it is hosted correctly: ```bash $ openstack server show -c OS-EXT-SRV-ATTR:hypervisor_hostname ``` ## Additional Information * This procedure is **safe**: it resets Nova’s recorded state without modifying VM disk or memory. * Always validate using `virsh` command that the VM is running before resetting state. * For frequent issues, investigate root causes in **Nova conductor, scheduler, and Cinder logs**. # Instance Creation Failing due to Failed NFS Mount ## Problem * When attempting to create a VM on a compute host, the following error is observed in the UI or through the CLI: ```ruby Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance [INSTANCE_ID]. Last exception: Unexpected error while running command. Command: mount -t nfs -o vers=3,proto=tcp,lookupcache=pos,nolock,noacl [NFS_SERVER_IP] Traceback (most recent call last): File "/var/lib/openstack/lib/python3.10/site-packages/nova/conductor/manager.py", line 690, in build_instances scheduler_utils.populate_retry( File "/var/lib/openstack/lib/python3.10/site-packages/nova/scheduler/utils.py", line 1017, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance [INSTANCE_ID]. Last exception: Unexpected error while running command. Command: mount -t nfs -o vers=3,proto=tcp,lookupcache=pos,nolock,noacl [NFS_SERVER_IP]:/nfs/share /local/mount/point Exit code: 32 Stdout: '' Stderr: 'mount: /local/mount/point: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program.\n' ``` * VM migration fails due to an NFS mount error on the destination host. During the migration process, the destination host is unable to mount the required NFS share. ## Environment * Platform9 Private Cloud Director - v2025.4 and Higher * Self Hosted Private Cloud Director Virtualization – v2025.4 and Higher * Component - Compute Service ## Cause * The failure occurred because the `nfs-common` package *(which provides the mount.nfs helper utility)* was not installed or not properly configured on the destination host. As a result, the system was unable to execute the mount command for the NFS share during the VM migration process. This led to a failure in mounting the NFS volume required for shared storage, preventing the migration from completing successfully. ## Diagnostics 1. NFS Volume is not getting created in hosts with the aforementioned error. 2. Attempt manual NFS mount on the destination host. ```bash sudo mount -t nfs -o vers=3 [NFS_SERVER_IP]:/nfs/share /local/mount/point mount: /local/mount/point: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program. ``` 3. Check if the required package is installed. {% tabs %} {% tab title="Ubuntu" %} ```bash dpkg -l | grep nfs-common ``` {% endtab %} {% tab title="Platform9 OS" %} ```bash rpm -qa | grep nfs-utils ``` {% endtab %} {% endtabs %} ## Resolution 1. Manually install the NFS client package on the affected host. {% tabs %} {% tab title="Ubuntu" %} ```bash sudo apt install -y nfs-common ``` {% endtab %} {% tab title="Platform9 OS" %} ```bash sudo dnf install -y nfs-utils ``` {% endtab %} {% endtabs %} {% hint style="info" %} On Platform9 OS, install only packages provided by the Platform9 validated package repository. Adding a community Rocky Linux, EPEL, or upstream project repository puts the host in an unsupported configuration. For details, see [Host Operating System Support Matrix](https://platform9.com/docs/private-cloud-director/getting-started/host-operating-system-support-matrix). {% endhint %} ## Validation 1. Check if the the volumes are getting properly mounted in the affected hosts. ```bash $ sudo mount | grep "" [NFS_SERVER_IP]:/nfs/share on /local/mount/point/[UUID] type nfs (rw,vers=3,proto=tcp,...) ``` 2. If these steps prove insufficient to resolve the issue, reach out to the Platform9 Support Team for additional assistance. # VM Creation Fails Due to AIO (Asynchronous I/O) limit on Hypervisor ## Problem VM creation failed due to the following error on the hypervisor: ```ruby Unable to use native AIO: failed to create linux AIO context: Resource temporarily unavailable ``` ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher * Component: Compute Service ## Cause The **asynchronous I/O (AIO) limit** on the hypervisor is set too low, preventing libvirt/QEMU from allocating sufficient I/O handles for new VM disk operations. ## Diagnostics 1. Check current AIO limit: ```bash $ sysctl fs.aio-max-nr fs.aio-max-nr = 65536 # example output ``` 2. Review libvirt logs: ```bash $ journalctl -u libvirtd | grep aio #example error: Unable to use native AIO: failed to create linux AIO context: Resource temporarily unavailable ``` ## Resolution Increase the AIO limit: * To increase the limit one time (this change will not persist a node reboot): ```bash # 2097152 should be a safe value to start with and can be monitored against aio-nr for a longer term $ sudo sysctl -w fs.aio-max-nr=2097152 ``` * To increase the limit permanently (this change will persist a node reboot): ```bash # 2097152 should be a safe value to start with and can be monitored against aio-nr for a longer term $ vi /etc/sysctl.conf fs.aio-max-nr = 2097152 ``` ## Validation * Confirm updated value: ```bash $ sysctl fs.aio-max-nr fs.aio-max-nr = 2097152 # Expected output ``` * Re-attempt VM creation and verify it is active and Running. ## Additional Information * The **AIO (Asynchronous I/O) limit** is controlled by the Linux kernel parameter `fs.aio-max-nr`, defines the maximum number of asynchronous I/O requests that can exist system-wide at any given time. * If the current number of I/O handles `/proc/sys/fs/aio-nr` comes too close to `/proc/sys/fs/aio-max-nr` their is high risk of encountering this failure. The parameter values can fluctuate based on several factors, including the number of LUNs, the number of VMs, whether multipath is enabled, and other relevant configurations. These variations directly impact the number of asynchronous I/O operations. * QEMU/libvirt uses Linux AIO to perform disk I/O operations efficiently for VMs. # Compute Nodes Showing as Offline in PCD UI # Windows VM Performance Degradation on PCD ### Problem Windows 24H2 and Windows Server 2025 VMs running on PCD show noticeably slower performance compared to similar VMs running on other KVM-based platforms such as Proxmox. ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: Compute Service ### Cause The performance degradation was traced to **inconsistent CPU model exposure and topology configuration** across compute nodes. When libvirt launches a Windows VM, it attempts to match the CPU model defined in its XML configuration. If the match is **partial**, the guest OS receives fewer CPU instruction flags, leading to reduced optimization and slower performance. Additionally, default VM topology settings (e.g., `sockets=4 cores=1 threads=1`) can cause Windows to detect multiple single-core CPUs instead of a single multi-core processor. This results in: * Higher inter-socket latency * Reduced cache efficiency * Suboptimal thread scheduling by the Windows kernel ### Diagnostics **1.** Check CPU configuration for the affected VM from the hosted Compute Node: ```bash $ sudo virsh dumpxml | grep -A10 -i cpu ``` Look for: * `check='partial'` → Indicates incomplete CPU feature exposure * `fallback='allow'` → Allows degraded CPU model matching
2. Compare CPU settings across compute nodes: ``` $ sudo grep -i cpu /opt/pf9/etc/nova/conf.d/*.conf ``` Confirm consistent entries such as: {% code title="Expected Output" %} ```bash cpu_mode = custom cpu_models = EPYC-Rome ``` {% endcode %} 3. Verify host hardware consistency: ``` $ sudo lscpu ``` Ensure that all compute nodes have the same CPU model and features.\ \ 4\. Verify no third-party application such as an anti-virus or hardening application is contributing towards slowness. ### Resolution **1.** Define explicit CPU topology in the flavor: ```bash $ openstack flavor set \ --property hw:cpu_sockets=1 \ --property hw:cpu_cores=4 \ --property hw:cpu_threads=1 ``` This configuration ensures Windows detects a single multi-core processor, improving scheduling and cache utilization.
2. Use strict CPU model matching in libvirt domain XML: ```xml EPYC-Rome ``` This prevents libvirt from falling back to a partially matched CPU configuration.
3. Guest OS optimizations: * Verify that the DiagTrack (Connected User Experiences and Telemetry) service is enabled. Disabling it may negatively impact performance. * Review Virtualization-Based Security (VBS) settings — in some cases, disabling VBS may reduce overhead and improve responsiveness.
4. Optional graphics optimization (for GUI workloads)**:** You may test the following flavor or image properties: * `hw_video_model=virtio` or `hw_video_model=qxl` * `hw_graphics_type=vnc` or `hw_graphics_type=spice` ### Additional Information * Using strict CPU model matching and explicit topology can yield **10–15% performance improvement** in CPU-bound Windows workloads. * Linux guests are generally unaffected, as they handle partial CPU model exposure more gracefully. # Volume Creation Failure Due to Image Format Mismatch Resolve volume creation errors in OpenStack caused by image format mismatches. Learn how to diagnose, correct metadata discrepancies, and prevent orphaned files on Tintri/NFS backends for seamless pri # Missing \[barbican] Section in nova.conf Causes Host Upgrade Convergence Failure During PCD Upgrade ## Problem One or more hosts fail to converge during a Private Cloud Director (PCD) host upgrade and remain offline or stuck in a failed upgrade state in the Management Plane. Observed symptoms may include: * Host upgrade failure during reconciliation. * Host shown offline/unreachable in the Management Plane. * Upgrade unable to proceed for the affected node. * Repeated `pf9-ostackhost:get_config failed: 'barbican'` errors in hostagent logs. * Missing `[barbican]` section in `nova.conf`. Example log entries in `/var/log/pf9/hostagent.log`: ```bash pf9_app.py ERROR - pf9-ostackhost:get_config failed: 'barbican'session.py ERROR - Bad message, app config or reading current app config ``` ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher ## Cause This issue is caused by a reconciliation sequencing issue during the upgrade process. During the upgrade, the reconciliation process fails before the `[barbican]` configuration section is written into the `nova.conf` file. As a result, the `pf9-ostackhost` application fails repeatedly during configuration reconciliation and the host does not converge successfully. ## Diagnostics * Check hostagent logs for Barbican reconciliation failures ```shellscript grep -i -C5 barbican /var/log/pf9/pf9-hostagent.log ``` Expected error: ``` pf9-ostackhost:get_config failed: 'barbican' ``` * Verify whether the `[barbican]` section exists ``` grep -C5 "\[barbican\]" /opt/pf9/etc/nova/conf.d/nova.conf ``` If no output is returned, the section is missing. ## Resolution * Step 1: Copy the `[barbican]` section from a healthy node From a healthy upgraded node, copy the complete `[barbican]` section from: ```bash /opt/pf9/etc/nova/conf.d/nova.conf ``` Example: ```bash [barbican] barbican_region_name = [REGION_NAME] auth_url = http://localhost:8158/keystone/v3 barbican_endpoint_type = admin ``` #### Step 2: Paste the `[barbican]` section into the affected node Add the copied section into: ```bash /opt/pf9/etc/nova/conf.d/nova.conf ``` #### Step 3: Verify whether the host converges automatically Monitor the hostagent logs: ```bash tail -f /var/log/pf9/pf9-hostagent.log ``` Check whether the host becomes healthy in the Management Plane. #### Step 4: Restart services if the host does not converge automatically Restart the following services on the affected host: ```bash systemctl restart pf9-hostagent systemctl restart pf9-ostackhost ``` #### Step 5: Verify host convergence Confirm: * Host becomes online in the Management Plane. * Upgrade stage completes successfully. * Reconciliation errors no longer appear in logs. ## Validation Validate the following after remediation: * Host status becomes healthy/online. * Upgrade workflow completes successfully. * `/var/log/pf9/hostagent.log` no longer reports: * `pf9-ostackhost:get_config failed: 'barbican'` * `[barbican]` section exists in `nova.conf`. # Volume Migration Fails with Cannot Acquire State Change Lock Error ## Problem Block Storage volume migration via `openstack volume migrate` fails and the volume transitions to `error` state. The failure occurs during the final swap phase after the data copy has already completed. The Compute Service log on the source compute host shows: {% code title="/var/log/pf9/ostackhost.log" %} ``` Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) ``` {% endcode %} The migration appears to complete the data copy successfully but then fails at the point where the Compute Service signals the volume attachment swap. ## Environment * Private Cloud Director Virtualization - **All versions** * Self-Hosted Private Cloud Director Virtualization - **All versions** * Component: Block Storage Service, Compute Service ## Cause When a live volume migration runs, the Block Storage Service issues a `swap_volume` call to the Compute Service. The Compute Service responds by executing a `_swap_volume` operation in the Compute Service libvirt driver, which calls `blockJobInfo()` periodically to check whether the block copy is complete. libvirt holds a **global per-domain state lock** during the execution of `remoteDispatchConnectGetAllDomainStats` — a routine that iterates through every running VM on the hypervisor to collect statistics. This routine is triggered by the Compute Service `get_diagnostics` API call and by other monitoring clients that poll the hypervisor for VM state. If `blockJobInfo()` attempts to acquire this same lock while `remoteDispatchConnectGetAllDomainStats` is still running, the call waits up to 30 seconds. If the stats collection does not finish within 30 seconds, `blockJobInfo()` raises a `libvirtError`. Nova's original code treats this as a fatal error and marks the migration as `error` state, **even though the underlying block copy completed successfully**. The issue is more likely to occur when: * The hypervisor is running a large number of VMs (more VMs = longer stats collection cycle = longer lock hold time) * Monitoring tooling or automation is polling `get_diagnostics` at high frequency * The storage backend is under elevated I/O load, which slows the stats collection cycle ## Diagnostics {% stepper %} {% step %} #### Step 1 — Check the failed volume state and migration status Retrieve the volume details to confirm the migration failed and identify the source compute host. {% code title="OpenStack CLI" %} ```bash $ openstack volume show ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+----------------------------------------------------------+ | Field | Value | +--------------------------------------+----------------------------------------------------------+ | os-vol-mig-status-attr:migstat | error | | os-vol-host-attr:host | [CINDER_HOST]@[BACKEND]#[NFS_PATH] | | status | in-use | | attachments | [{'server_id': '[VM_UUID]', ...}] | +--------------------------------------+----------------------------------------------------------+ ``` {% endcode %} Note the `server_id` from the `attachments` field — this is the VM UUID whose compute host will have the relevant logs. {% endstep %} {% step %} #### Step 2 — Identify the source compute host Retrieve the hypervisor where the attached VM is running. {% code title="OpenStack CLI" %} ```bash $ openstack server show | grep "OS-EXT-SRV-ATTR:host" ``` {% endcode %} {% code title="Sample Output" %} ```bash | OS-EXT-SRV-ATTR:host | [COMPUTE_HOST] | ``` {% endcode %} SSH into `[COMPUTE_HOST]` to check the Hostagent log in [Step 3](#step-3--confirm-the-lock-contention-error-in-the-hostagent-log). {% endstep %} {% step %} #### Step 3 — Confirm the lock contention error in the Hostagent log Search the Hostagent log on the source compute host for the failed VM UUID. Confirm the `cannot acquire state change lock` error is present. {% code title="Compute Host" %} ```bash $ sudo grep -i "" /var/log/pf9/ostackhost.log | grep -i "state change lock\|VolumeRebaseFailed" $ sudo zcat /var/log/pf9/ostackhost.log.*.gz | grep -i "" | grep -i "state change lock\|VolumeRebaseFailed" ``` {% endcode %} {% code title="Sample Output" %} ```bash [TIMESTAMP] ERROR nova.compute.manager [REQ_UUID] [VM_UUID] Failed to swap volume [OLD_VOLUME_UUID] for [NEW_VOLUME_UUID]: nova.exception.VolumeRebaseFailed: Volume rebase failed: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) ``` {% endcode %} The presence of `remoteDispatchConnectGetAllDomainStats` in the error confirms libvirt lock contention as the root cause. The block copy itself completed — the failure occurred only during the status check. {% endstep %} {% step %} #### Step 4 — Verify the volume can be retried after resetting its migration state Before applying the workaround, reset the failed volume to `available` or `in-use` so it can be migrated again. A volume stuck in `error` migration state must be reset before it accepts a new migration request. {% hint style="info" %} For full details on resetting volumes from various stuck states, refer to [How To Reset Volume States](https://platform9.com/kb/pcd/storage/how-to-reset-volume-states). {% endhint %} {% code title="OpenStack CLI" %} ```bash $ openstack volume set --state in-use $ openstack volume show | grep "migstat\|status" ``` {% endcode %} {% code title="Sample Output" %} ```bash | os-vol-mig-status-attr:migstat | None | | status | in-use | ``` {% endcode %} {% endstep %} {% endstepper %} ## Workaround The workaround patches the Compute Service libvirt driver on each compute host to handle lock contention gracefully. Instead of treating the 30-second timeout as a fatal error, the patched code catches the `libvirtError` and retries the status check after a short wait. Once `remoteDispatchConnectGetAllDomainStats` finishes and releases the lock, the retry succeeds and the migration completes normally. {% hint style="warning" %} Apply this patch to **every compute host** in the region where volume migration failures are occurring — not just the compute host where the last failure was observed. Any hypervisor running VMs with attached volumes can be the source host for a future migration. {% endhint %} ### Patch guest.py to Compute Hosts {% stepper %} {% step %} #### Step 1 — Back up the original guest.py on the compute host SSH into the compute host and create a timestamped backup of the original file before replacing it. {% code title="Compute Host" %} ```bash $ sudo cp /opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py \ /opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py.bak.$(date +%Y%m%d) ``` {% endcode %} Confirm the backup exists before continuing: {% code title="Compute Host" %} ```bash $ ls -lh /opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py* ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Patched guest.py in the compute host Patch the `guest.py` file in the compute host with the patch code that is highlighted below: 
    def is_job_complete(self):
        """Return True if the job is complete, False otherwise

        :returns: True if the job is complete, False otherwise
        :raises: libvirt.libvirtError on error fetching block job info
        """
        # NOTE(mdbooth): This method polls for block job completion. It returns
        # true if either we get a status which indicates completion, or there
        # is no longer a record of the job. Ideally this method and its
        # callers would be rewritten to consume libvirt events from the job.
        # This would provide a couple of advantages. Firstly, as it would no
        # longer be polling it would notice completion immediately rather than
        # at the next 0.5s check, and would also consume fewer resources.
        # Secondly, with the current method we only know that 'no job'
        # indicates completion. It does not necessarily indicate successful
        # completion: the job could have failed, or been cancelled. When
        # polling for block job info we have no way to detect this, so we
        # assume success.

        try:
            status = self.get_job_info()
        except libvirt.libvirtError as e:
            if 'cannot acquire state change lock' in str(e):
                # The domain lock is temporarily held by another libvirt
                # operation (e.g. virConnectGetAllDomainStats from
                # get_diagnostics). The block copy job is still running;
                # we just cannot query its status right now. Return False
                # so the caller retries after the next 0.5s sleep.
                LOG.warning('Transient libvirt lock contention polling block '
                            'job status for %(disk)s, will retry: %(err)s',
                            {'disk': self._disk, 'err': e})
                return False
            raise

        # If the job no longer exists, it is because it has completed
{% endstep %} {% step %} #### Step 3 — Restart the Hostagent and libvirt services Restart both services to load the patched file. {% hint style="warning" %} Restarting `pf9-ostackhost` briefly interrupts the Compute Service worker on this host. Confirm no critical VM operations are in flight before proceeding. Restarting `libvirtd` briefly disconnects libvirt from running VMs but does not affect VM uptime. {% endhint %} {% code title="Compute Host" %} ```bash $ sudo systemctl restart libvirtd $ sudo systemctl restart pf9-ostackhost ``` {% endcode %} {% endstep %} {% step %} #### Step 4 — Verify both services are running {% code title="Compute Host" %} ```bash $ sudo systemctl status libvirtd pf9-ostackhost ``` {% endcode %} {% code title="Sample Output" %} ```bash ● libvirtd.service - Virtualization daemon Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled) Active: active (running) since [TIMESTAMP]; [UPTIME] ago ● pf9-ostackhost.service - Platform9 OpenStack Host Agent Loaded: loaded (/lib/systemd/system/pf9-ostackhost.service; enabled) Active: active (running) since [TIMESTAMP]; [UPTIME] ago ``` {% endcode %} Repeat Steps 1–4 on every compute host in the affected region. {% endstep %} {% step %} #### Step 5 — Test a volume migration to confirm the fix After patching all compute hosts, retry a migration for one of the previously failing volumes. {% code title="OpenStack CLI" %} ```bash $ openstack volume migrate --host ``` {% endcode %} Monitor the migration status until it reaches `success` and the volume `migstat` field clears: {% code title="OpenStack CLI" %} ```bash $ openstack volume show | grep "migstat\|status" ``` {% endcode %} {% code title="Sample Output — migration in progress" %} ```bash | os-vol-mig-status-attr:migstat | migrating | | status | in-use | ``` {% endcode %} {% code title="Sample Output — migration complete" %} ```bash | os-vol-mig-status-attr:migstat | None | | os-vol-host-attr:host | [NEW_BACKEND] | | status | in-use | ``` {% endcode %} A successful migration with no `VolumeRebaseFailed` error in the Hostagent log confirms the patch is working. {% endstep %} {% endstepper %} ## Resolution This is a known issue that was reported as **PCD-7824.** The patch `guest.py` mentioned in this article is an interim workaround validated in production environments. Contact [Platform9 Support](https://support.platform9.com/) for the latest status on the official fix or for assistance applying the patch across a large number of compute hosts. ## Additional Information Please ensure that the changes are reapplied after each PCD upgrade, as they will be reverted during the upgrade process. This will be necessary until a permanent fix is included in a future release. # VM Creation Fails with Network Allocation Timeout ## Problem Virtual machine creation fails consistently on a freshly deployed or re-deployed PCD region. VMs enter `ERROR` state after approximately 300 seconds and never reach `ACTIVE`. The fault message reports that network allocation failed. Symptoms observed: * `openstack server show` reports `vm_state: error` with fault message `Failed to allocate the network(s), not rescheduling` * All VM builds on the affected region fail — the issue is not isolated to specific flavors, images, or networks * The compute host Hostagent log shows a 300-second timeout waiting for a `network-vif-plugged` event that never arrives ## Environment * Private Cloud Director Virtualization - **v2026-4.89 and later** * Self-Hosted Private Cloud Director Virtualization - **v2026-4.89 and later** * Component: Networking Service, Compute Service ## Cause The Networking Service uses a dedicated service credential to notify the Compute Service when a VM's network port has been successfully bound (the `network-vif-plugged` event). These credentials are stored in `neutron.conf` under the `[nova]` section. During a fresh region installation, a race condition can cause the Networking Service configuration to be written before the Compute Service has persisted its service password to the configuration store. The result is `neutron.conf` with `password = ` in the `[nova]` section, as shown below: 
[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
auth_version = v3
endpoint_type = internal
password = <no value>
project_domain_name = default
project_name = services
region_name = [REGION_NAME]
user_domain_name = default
username = nova
Every attempt by the Networking Service to notify the Compute Service of a port-bind event fails with an HTTP 401 Unauthorized from the Identity Service. The Compute Service waits 300 seconds for the `network-vif-plugged` notification that never arrives, then aborts the build. This issue is tracked under **PCD-7657**. ## Diagnostics {% stepper %} {% step %} #### Step 1 — Confirm the VM fault message Retrieve the fault from the failed VM. The fault message confirms the network allocation failure. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c fault -c status -c "OS-EXT-STS:vm_state" ``` {% endcode %} {% code title="Sample Output" %} ```bash +---------------------------+----------------------------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------------------------+ | OS-EXT-STS:vm_state | error | | fault | {'code': 500, 'message': 'Build of instance [VM_UUID] aborted: | | | Failed to allocate the network(s), not rescheduling.'} | | status | ERROR | +---------------------------+----------------------------------------------------------------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Check the compute host Hostagent log for the VIF plug timeout On the compute host, search the Hostagent log for the affected VM UUID to confirm the timeout pattern. {% code title="Compute Host" %} ```bash $ sudo grep -i "network-vif-plugged\|VirtualInterfaceCreateException\|BuildAbortException" \ /var/log/pf9/ostackhost.log | grep "" ``` {% endcode %} {% code title="Sample Output" %} ```bash [TIMESTAMP] WARNING nova.compute.manager [REQ_UUID] [VM_UUID] Timeout waiting for ['network-vif-plugged-[PORT_UUID]'] for instance with vm_state building and task_state spawning. Event states are: network-vif-plugged-[PORT_UUID]: timed out after 300.00 seconds [TIMESTAMP] ERROR nova.compute.manager [REQ_UUID] [VM_UUID] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed ``` {% endcode %} {% endstep %} {% step %} #### Step 3 — Confirm the Networking Service is failing Keystone authentication Check the Networking Service pod logs for HTTP 401 errors on Compute Service notification calls. This confirms the root cause. {% hint style="info" %} **Self-Hosted only:** These steps require `kubectl` access to the control plane namespace. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to have this confirmed and resolved on behalf of the environment. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl logs -n \ $(kubectl get po -n -l application=neutron -o jsonpath="{.items[0].metadata.name}") \ | grep -i "Unauthorized\|401\|nova" ``` {% endcode %} {% code title="Sample Output" %} ```bash [TIMESTAMP] ERROR neutron.notifiers.nova keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: [REQ_UUID]) ``` {% endcode %} {% endstep %} {% step %} #### Step 4 — Confirm the missing Nova password in neutron.conf Inspect the `neutron-etc` Secret to confirm `password = ` in the `[nova]` section. {% hint style="info" %} **Self-Hosted only:** This step requires `kubectl` access to the control plane namespace. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to have this checked on behalf of the environment. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl get secret neutron-etc -n \ -o go-template='{{ index .data "neutron.conf" }}' \ | base64 -d \ | grep -A 10 "\[nova\]" ``` {% endcode %} 
[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
password = <no value>
username = nova
`password = ` confirms the race condition occurred during region deployment. Proceed to the [Workaround](#workaround). {% endstep %} {% endstepper %} ## Workaround The fix requires injecting the correct Nova service password into the `neutron-etc` Secret and restarting the Networking Service pods. The correct password value is retrieved from the Compute Service configuration in the same region. {% hint style="info" %} **SaaS customers:** All steps below require `kubectl` access to the control plane namespace. Contact [Platform9 Support](https://support.platform9.com/) to have this workaround applied on behalf of the environment. {% endhint %} {% stepper %} {% step %} #### Step 1 — Back up the current neutron-etc Secret and deployment Before making any changes, back up both the Secret and the Networking Service deployment. {% code title="Control Plane Node" %} ```bash $ export NS="" $ kubectl get secret neutron-etc -n $NS -o yaml > neutron-etc-backup.yaml $ kubectl get deployment neutron-server -n $NS -o yaml > neutron-server-deploy-backup.yaml ``` {% endcode %} {% code title="Sample Output" %} ```bash (no output — backup files written to current directory) ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Retrieve the correct Nova service password Get the Compute Service password from the Compute Service configuration in the same namespace. {% code title="Control Plane Node" %} ```bash $ kubectl get secret nova-etc -n $NS \ -o go-template='{{ index .data "nova.conf" }}' \ | base64 -d \ | grep -A10 "\[keystone_authtoken\]" \ | grep "^password" ``` {% endcode %} {% code title="Sample Output" %} ```bash password = [NOVA_SERVICE_PASSWORD] ``` {% endcode %} Note the password value. This is the shared service credential the Networking Service must use to authenticate to the Identity Service as the `nova` service account. {% endstep %} {% step %} #### Step 3 — Extract and edit neutron.conf Decode `neutron.conf` from the Secret into a temporary file and set the correct password. {% code title="Control Plane Node" %} ```bash $ kubectl get secret neutron-etc -n $NS \ -o go-template='{{ index .data "neutron.conf" }}' \ | base64 -d > /tmp/neutron.conf.tmp $ vi /tmp/neutron.conf.tmp ``` {% endcode %} In the editor, locate the `[nova]` section and replace `password = ` with the password retrieved in Step 2: ``` [nova] password = ``` Save and exit the editor. {% endstep %} {% step %} #### Step 4 — Patch the neutron-etc Secret with the corrected configuration Re-encode the updated `neutron.conf` and patch the Secret in the cluster. {% code title="Control Plane Node" %} ```bash $ NEW_CONF=$(cat /tmp/neutron.conf.tmp | base64 | tr -d '\n') $ kubectl patch secret neutron-etc -n $NS \ -p="{\"data\":{\"neutron.conf\": \"$NEW_CONF\"}}" ``` {% endcode %} {% code title="Sample Output" %} ```bash secret/neutron-etc patched ``` {% endcode %} {% endstep %} {% step %} #### Step 5 — Restart the Networking Service pods Delete the existing Networking Service pods to trigger a rolling restart with the corrected configuration. {% hint style="warning" %} Restarting Networking Service pods briefly interrupts network event processing. Avoid doing this while VMs are being actively provisioned on the affected region. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl delete pods -n $NS -l application=neutron ``` {% endcode %} {% code title="Sample Output" %} ```bash pod "neutron-server-[POD_ID_1]" deleted pod "neutron-server-[POD_ID_2]" deleted ``` {% endcode %} Wait for the new pods to reach `Running` state: {% code title="Control Plane Node" %} ```bash $ kubectl get pods -n $NS -l application=neutron -w ``` {% endcode %} {% code title="Sample Output" %} ```bash NAME READY STATUS RESTARTS AGE neutron-server-[NEW_POD_ID_1] 1/1 Running 0 30s neutron-server-[NEW_POD_ID_2] 1/1 Running 0 28s ``` {% endcode %} {% endstep %} {% step %} #### Step 6 — Verify VM creation succeeds Attempt to provision a new VM on the affected region and confirm it reaches `ACTIVE` state. {% code title="OpenStack CLI" %} ```bash $ openstack server show | grep status ``` {% endcode %} {% code title="Sample Output" %} ```bash | status | ACTIVE | ``` {% endcode %} Confirm the Networking Service logs no longer show HTTP 401 errors on Compute Service notification calls. {% endstep %} {% endstepper %} ## Resolution A permanent fix for the race condition that causes this issue is planned to be included in the upcoming **2026.AUG** release. After upgrading to that release, the Networking Service configuration will always contain the correct Compute Service password regardless of the deployment order of dependent services. The fix for **PCD-7657** ensures that the Compute Service password is pre-seeded in the Networking Service configuration store before either chart is deployed, eliminating the ordering dependency. ## Additional Information **Related articles:** * [VM Creation Failed Due to Port Binding Failure](https://platform9.com/kb/pcd/networking/vm-creation-failed-due-to-port-binding-failure) — covers a similar symptom (VM creation failure from Networking Service issues) where the root cause is an OVN controller problem rather than a missing service credential. Review this article if the `[nova]` section in `neutron.conf` shows a valid password but VM creation still fails. # VM Fails to Start Due to Boot Volume Format Mismatch in Block Device Mapping ## Problem A virtual machine is active in the Private Cloud Director platform but fails to start or reboot. Every start and hard reboot attempt fails consistently. The VM console may display an "Instance not found" error. The Compute Service logs on the hypervisor (`/var/log/pf9/ostackhost.log`) record the following error for every start attempt: ``` libvirt.libvirtError: internal error: process exited while connecting to monitor: qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","driver":"qcow2", "file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format ``` The VM reports as `ACTIVE` in the platform, but no corresponding libvirt domain exists on the hypervisor. ## Environment * Private Cloud Director Virtualization - v2026.1 and Higher * Self-Hosted Private Cloud Director Virtualization - v2026.1 and Higher * Component: Compute Service * NFS-backed boot volume (Tintri NFS or equivalent) ## Cause The VM's boot volume is hosted on NFS storage. Under certain conditions — most commonly during a live migration — the on-disk format of the volume file can change from `qcow2` to `raw`. The `format` field inside the `connection_info` JSON column of the `block_device_mapping` table in the Compute Service database is not automatically updated when this format change occurs. When the Compute Service subsequently attempts to start the VM, it reads the stale `"format": "qcow2"` metadata from the database and instructs QEMU to open the disk as `qcow2` via libvirt. QEMU rejects the file because the actual on-disk format is `raw`, causing every start and reboot attempt to fail with `Image is not in qcow2 format`. ## Diagnostics The following checks confirm the VM is affected by this specific issue. Complete all steps before proceeding to [Workaround](#workaround). {% stepper %} {% step %} #### Step 1 — Identify the Boot Volume and Hypervisor Run the following command to retrieve the hypervisor hostname and boot volume UUID for the stuck VM. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c OS-EXT-SRV-ATTR:hypervisor_hostname -c volumes_attached -c status ``` {% endcode %} {% code title="Sample Output" %} ```bash +--------------------------------------+---------------------------------------------------+ | Field | Value | +--------------------------------------+---------------------------------------------------+ | OS-EXT-SRV-ATTR:hypervisor_hostname | [HYPERVISOR_HOSTNAME] | | status | ACTIVE | | volumes_attached | delete_on_termination='True', id='[VOLUME_UUID]' | +--------------------------------------+---------------------------------------------------+ ``` {% endcode %} Note the `[HYPERVISOR_HOSTNAME]` and the volume `id` (`[VOLUME_UUID]`) — both are required in subsequent steps. {% endstep %} {% step %} #### Step 2 — Confirm the Libvirt Domain Is Absent SSH to the hypervisor identified in Step 1 and verify no libvirt domain exists for the VM. {% code title="Compute Host" %} ```bash $ virsh list --all | grep ``` {% endcode %} If no output is returned, the libvirt domain is absent. The VM has no running or stopped guest on the hypervisor, even though the platform reports it as `ACTIVE`. {% endstep %} {% step %} #### Step 3 — Confirm the Actual On-Disk Volume Format On the same hypervisor, locate the boot volume file and inspect its format using `qemu-img`. First, locate the full path to the volume file: {% code title="Compute Host" %} ```bash $ ls /opt/pf9/data/state/mnt/*/volume- ``` {% endcode %} {% code title="Sample Output" %} ```bash /opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID] ``` {% endcode %} Then run `qemu-img info` on the full path returned above: {% code title="Compute Host" %} ```bash $ qemu-img info /opt/pf9/data/state/mnt//volume- ``` {% endcode %} {% code title="Sample Output" %} ```bash image: /opt/pf9/data/state/mnt/[MOUNT_UUID]/volume-[VOLUME_UUID] file format: raw virtual size: [DISK_SIZE] GiB ([DISK_BYTES] bytes) disk size: [USED_SIZE] GiB ``` {% endcode %} `file format: raw` confirms the on-disk format does not match the `qcow2` value stored in the database. {% endstep %} {% step %} #### Step 4 — Confirm the Stale Format Metadata in the Database {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} Query the `block_device_mapping` table in the Nova database to confirm the stale format entry. {% code title="Nova DB" %} ```sql MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format') FROM block_device_mapping WHERE instance_uuid = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------------------------------------+ | JSON_EXTRACT(connection_info,'$.data.format') | +-----------------------------------------------+ | "qcow2" | +-----------------------------------------------+ 1 row in set (0.002 sec) ``` {% endcode %} A result of `"qcow2"` when the actual on-disk format is `raw` (confirmed in Step 3) confirms the database metadata mismatch causing the boot failure. {% endstep %} {% endstepper %} If all checks above match — domain absent in Step 2, `file format: raw` in Step 3, and `"qcow2"` in Step 4 — proceed to [Workaround](#workaround). ## Resolution No permanent code fix exists for this issue. Applying the [Workaround](#workaround) below permanently corrects the stale format metadata in the Compute Service database and restores normal VM operation. ## Workaround The following procedure corrects the stale `format` value in the `block_device_mapping` table and restores the ability to start the VM. {% hint style="info" %} **Self-Hosted only (Steps 1–4):** Steps 1–4 require direct access to the Compute Service database on the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have the database correction applied on your behalf. Steps 5–7 use the OpenStack CLI and can be run from any machine with the OpenStack RC file sourced. {% endhint %} {% stepper %} {% step %} #### Step 1 — Back Up the Current connection\_info Row Before making any changes, record the existing `connection_info` value so the original row can be restored if needed. {% code title="Nova DB" %} ```sql MySQL [nova]> SELECT connection_info FROM block_device_mapping WHERE instance_uuid = '' AND deleted = 0; ``` {% endcode %} Save the full output to a safe location before proceeding. {% endstep %} {% step %} #### Step 2 — Verify a Single Boot Device Mapping Exists Confirm there is only one block device mapping with `boot_index = 0` for the VM before modifying any records. {% code title="Nova DB" %} ```sql MySQL [nova]> SELECT id, device_name, source_type, boot_index, volume_id, deleted FROM block_device_mapping WHERE instance_uuid = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +------+-------------+-------------+------------+--------------------+---------+ | id | device_name | source_type | boot_index | volume_id | deleted | +------+-------------+-------------+------------+--------------------+---------+ | [ID] | /dev/vda | image | 0 | [VOLUME_UUID] | 0 | +------+-------------+-------------+------------+--------------------+---------+ 1 row in set (0.001 sec) ``` {% endcode %} The output must show exactly one row with `boot_index = 0`. If multiple rows exist or the volume UUID does not match the boot volume identified in Diagnostics, stop and contact [Platform9 Support](https://support.platform9.com/) before proceeding. {% endstep %} {% step %} #### Step 3 — Correct the Volume Format in the Database {% hint style="warning" %} This step directly modifies the Compute Service database. Ensure the backup from Step 1 is saved before proceeding. {% endhint %} Update the `format` field in `connection_info` from `qcow2` to `raw`. {% code title="Nova DB" %} ```sql MySQL [nova]> UPDATE block_device_mapping SET connection_info = JSON_SET(connection_info, '$.data.format', 'raw') WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash Query OK, 1 row affected (0.008 sec) Rows matched: 1 Changed: 1 Warnings: 0 ``` {% endcode %} `Rows matched: 1 Changed: 1` confirms the update was applied to exactly one row. {% endstep %} {% step %} #### Step 4 — Verify the Format Was Updated Confirm the `format` field now reads `"raw"`. {% code title="Nova DB" %} ```sql MySQL [nova]> SELECT JSON_EXTRACT(connection_info, '$.data.format') FROM block_device_mapping WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` {% endcode %} {% code title="Sample Output" %} ```bash +-----------------------------------------------+ | JSON_EXTRACT(connection_info,'$.data.format') | +-----------------------------------------------+ | "raw" | +-----------------------------------------------+ 1 row in set (0.002 sec) ``` {% endcode %} The result must be `"raw"` before proceeding. {% endstep %} {% step %} #### Step 5 — Check VM Status and Reset State if Needed Check the current server status using the OpenStack CLI. {% code title="OpenStack CLI" %} ```bash $ openstack server show -f value -c status ``` {% endcode %} {% code title="Sample Output" %} ```bash ACTIVE ``` {% endcode %} If the status returns `ACTIVE`, proceed directly to Step 6. If the status returns `ERROR`, run the following command to reset the VM state before rebooting: {% code title="OpenStack CLI" %} ```bash $ openstack server set --state active ``` {% endcode %} {% endstep %} {% step %} #### Step 6 — Hard Reboot the VM Issue a hard reboot to restart the VM using the corrected volume format metadata. {% code title="OpenStack CLI" %} ```bash $ openstack server reboot --hard ``` {% endcode %} Allow up to 60 seconds for the reboot to complete before verifying. {% endstep %} {% step %} #### Step 7 — Verify the VM Is Running Confirm the VM is `ACTIVE` with power state `Running`. {% code title="OpenStack CLI" %} ```bash $ openstack server show -f value -c status -c OS-EXT-STS:power_state ``` {% endcode %} {% code title="Sample Output" %} ```bash 1 ACTIVE ``` {% endcode %} Power state `1` indicates the VM is running. Optionally, confirm the libvirt domain is now present on the hypervisor: {% code title="Compute Host" %} ```bash $ virsh list --all | grep ``` {% endcode %} {% code title="Sample Output" %} ```bash [DOMAIN_ID] [VM_UUID] running ``` {% endcode %} The VM is successfully restored when the status is `ACTIVE`, power state is `1`, and the libvirt domain appears as `running` on the hypervisor. {% hint style="warning" %} **Rollback:** If the VM does not start successfully after the hard reboot, restore the original `connection_info` value saved in Step 1 by running: ```sql MySQL [nova]> UPDATE block_device_mapping SET connection_info = '' WHERE instance_uuid = '' AND volume_id = '' AND deleted = 0; ``` Then contact [Platform9 Support](https://support.platform9.com/) for further assistance. {% endhint %} {% endstep %} {% endstepper %} ## Additional Information For issues where a volume creation operation fails due to an image format mismatch on NFS or Tintri storage, see [Volume Creation Failure Due to Image Format Mismatch](https://platform9.com/kb/pcd/compute/volume-creation-failure-due-to-image-format-mismatch). # VM Stuck in Migrating State After Live Migration ### Problem A virtual machine remains in MIGRATING state in the PCD UI indefinitely after a live migration attempt. No VM operations can be initiated on the instance, and the state does not transition to ACTIVE or ERROR on its own. Refreshing the PCD UI does not resolve the condition. ### Environment * Private Cloud Director Virtualization — v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization — v2025.10-180 and Higher * Component: Compute Service ## Cause A live migration interacts with three layers that must stay in sync: the `libvirt` driver on the source and destination hypervisors, the Nova database (`instances` and `migrations` tables), and the OpenStack Placement service (which tracks VCPU and MEMORY\_MB allocations per resource provider). When any layer falls out of sync, the VM can become permanently stuck in `task_state = migrating`. Two distinct failure modes produce this symptom: * **Migration accepted but failed before starting (VM still on source)** — the destination compute service accepted the migration but failed before memory transfer began. The migration record is in `accepted` status. An orphaned Placement allocation exists under the migration UUID pointing to the destination RP. * **Migration completed at hypervisor level but management plane desynced (VM on destination)** — the VM successfully transferred to the destination at the `libvirt` level, but the Nova conductor did not process the completion event. The migration record is stuck in `running` status. The Nova `instances` table still references the source host, and the Placement service holds a stale allocation under the migration UUID on the source RP while the instance UUID has zero allocation. ## Diagnostics **1. Get the libvirt domain name for the VM:** {% code title="OpenStack CLI" %} ```bash $ openstack server show -c OS-EXT-SRV-ATTR:instance_name ``` {% endcode %} {% code title="Sample Output" %} ``` +-----------------------------------------+-------------------+ | Field | Value | +-----------------------------------------+-------------------+ | OS-EXT-SRV-ATTR:instance_name | [INSTANCE_NAME] | +-----------------------------------------+-------------------+ ``` {% endcode %} **2. SSH into both the source and destination compute hosts and check whether the VM domain is running:** {% code title="Compute Host" %} ```bash $ sudo virsh list --all | grep ``` {% endcode %} {% code title="Sample Output (VM found on destination)" %} ``` 42 [VM_UUID] running ``` {% endcode %} **3. Confirm the migration record status:** {% hint style="info" %} **Self-Hosted only:** The Nova DB query below requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf. {% endhint %} Retrieve the database password and MySQL pod IP, then open an interactive session: {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS" ``` {% endcode %} {% code title="Nova DB" %} ```sql USE nova; SELECT id, uuid, status, source_compute, dest_compute, updated_at FROM migrations WHERE instance_uuid = '' ORDER BY id DESC LIMIT 5; ``` {% endcode %} Note the `uuid` value from the output — referred to as `` throughout the Workaround steps. Use the migration `status` and the `virsh` result to identify the correct method: | Migration `status` | VM location (virsh) | Method | | ------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `accepted` | Source host only | [Method 1 — Migration Failed Before Completing](#method-1--migration-failed-before-completing-vm-on-source) | | `running` | Destination host | [Method 2 — Migration Completed at Hypervisor Level](#method-2--migration-completed-at-hypervisor-level-vm-on-destination) | If the checks above match, proceed to the [Workaround](#workaround). ## Workaround Apply the method that matches the diagnostic results above. ### Method 1 — Migration Failed Before Completing (VM on Source) The migration record is in `accepted` status and the VM is still running on the source host. {% hint style="warning" %} Before proceeding, confirm that the VM domain is running **only** on the source host using `virsh list`. Applying Method 1 to a VM that has already completed migration to the destination will corrupt Placement state. {% endhint %} {% hint style="info" %} **Self-Hosted only:** Steps 1 and 3 require direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf. {% endhint %} {% stepper %} {% step %} #### Step 1 — Mark the Migration Record as Error Retrieve database credentials and open a Nova DB session: {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS" ``` {% endcode %} {% code title="Nova DB" %} ```sql USE nova; UPDATE migrations SET status = 'error', updated_at = NOW() WHERE uuid = ''; ``` {% endcode %} Verify: {% code title="Nova DB" %} ```sql SELECT id, uuid, status FROM migrations WHERE uuid = ''; ``` {% endcode %} {% code title="Sample Output" %} ``` +------+--------------+--------+ | id | uuid | status | +------+--------------+--------+ | [ID] | [VM_UUID] | error | +------+--------------+--------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Reset the VM Task State {% code title="OpenStack CLI" %} ```bash $ openstack server set --state active ``` {% endcode %} Verify: {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host ``` {% endcode %} {% code title="Sample Output" %} ``` +----------------------------+---------------+ | Field | Value | +----------------------------+---------------+ | OS-EXT-SRV-ATTR:host | [SOURCE_HOST] | | OS-EXT-STS:task_state | None | | status | ACTIVE | +----------------------------+---------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 3 — Delete the Orphaned Migration Placement Allocation The migration UUID holds a provisional Placement allocation created when the migration was accepted. Remove this orphaned allocation: {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation delete ``` {% endcode %} Verify the allocation is removed: {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation show ``` {% endcode %} An empty table confirms the orphaned allocation has been cleared. {% endstep %} {% step %} #### Step 4 — Verify Final State {% code title="OpenStack CLI" %} ```bash $ openstack server show | grep -E 'status|task_state|host' ``` {% endcode %} The VM should show `status = ACTIVE`, `task_state = None`, and `host` pointing to the source compute node. The VM is now fully recovered and available for operations. {% endstep %} {% endstepper %} ### Method 2 — Migration Completed at Hypervisor Level (VM on Destination) The migration record is stuck in `running` status. The VM is running on the destination hypervisor, but the Nova `instances` table still points to the source host and the Placement service holds a stale allocation for the migration UUID on the source RP while the instance UUID has zero allocation. {% hint style="danger" %} This procedure modifies the Nova database directly and performs surgery on Placement allocations. Take a Nova DB backup before proceeding. Performing steps out of order will result in ghost resource reservations on the source RP or an over-allocated destination RP. {% endhint %} {% hint style="info" %} **Self-Hosted only:** This entire procedure requires direct access to the control plane. SaaS customers cannot perform these steps independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on their behalf. {% endhint %} {% hint style="warning" %} Fix the `instances` table and Placement allocations **before** marking the migration as `completed` (Steps 4–7 must complete before Step 8). Marking the migration complete first triggers a Nova cleanup path that may remove allocations incorrectly. {% endhint %} {% stepper %} {% step %} #### Step 1 — Verify VM Accessibility on the Destination SSH into the destination compute host and confirm the VM domain is running: {% code title="Compute Host" %} ```bash $ sudo virsh list | grep ``` {% endcode %} Also confirm the VM is reachable over the network from the assigned IP before proceeding. {% endstep %} {% step %} #### Step 2 — Retrieve Database Credentials and Take a Nova Database Backup Retrieve the database password and MySQL pod IP. The `$DB_PASS` and `$IP` shell variables set here are reused in Steps 4 and 8. {% code title="Control Plane Node" %} ```bash $ DB_PASS=$(kubectl exec -i -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c \ "DB=\$(consul-dump-yaml --start-key customers//regions//db | awk '/dbserver:/ {print \$2}'); \ consul-dump-yaml --start-key customers//dbservers/\$DB | awk '/admin_pass:/ {print \$2}'") $ IP=$(kubectl get pods -n "" \ $(kubectl get po -n "" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \ --template '{{.status.podIP}}') ``` {% endcode %} Take a backup of the relevant Nova tables: {% code title="Control Plane Node" %} ```bash $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysqldump -u root -h $IP -p$DB_PASS nova instances migrations" \ > nova_backup_$(date +%Y%m%d_%H%M%S).sql ``` {% endcode %} {% endstep %} {% step %} #### Step 3 — Collect Required Values Collect all variable values needed for subsequent steps before making any changes. **Get the destination RP UUID** (this value is also used as the `node` field in the Nova `instances` table): {% code title="OpenStack CLI" %} ```bash $ openstack resource provider list | grep ``` {% endcode %} {% code title="Sample Output" %} ``` +--------------------------------------+-----------------+------------+ | uuid | name | generation | +--------------------------------------+-----------------+------------+ | [DEST_RP_UUID] | [DEST_HOSTNAME] | 42 | +--------------------------------------+-----------------+------------+ ``` {% endcode %} **Get the project ID, user ID, and flavor resource requirements for the VM:** {% code title="OpenStack CLI" %} ```bash $ openstack server show -c project_id -c user_id -c flavor ``` {% endcode %} Note the `vcpus` and `ram` (in MB) values from the flavor output — these are the `` and `` values used in Step 6. **Confirm the destination RP has sufficient remaining capacity:** {% code title="OpenStack CLI" %} ```bash $ openstack resource provider usage show $ openstack resource provider inventory list ``` {% endcode %} Verify the destination RP can accommodate the VM's VCPU and MEMORY\_MB requirements before proceeding. {% endstep %} {% step %} #### Step 4 — Update the Instance Record in Nova DB Update the `instances` table to point to the destination host. Use `` as the `node` value (the Nova `instances.node` field stores the compute node's RP UUID). Connect to the Nova DB using the credentials retrieved in Step 2: {% code title="Control Plane Node" %} ```bash $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS" ``` {% endcode %} {% code title="Nova DB" %} ```sql USE nova; UPDATE instances SET host = '', node = '', vm_state = 'active', power_state = 1, task_state = NULL, updated_at = NOW() WHERE uuid = ''; ``` {% endcode %} Verify: {% code title="Nova DB" %} ```sql SELECT uuid, host, node, vm_state, power_state, task_state FROM instances WHERE uuid = ''; ``` {% endcode %} {% endstep %} {% step %} #### Step 5 — Reset the VM State via OpenStack API {% code title="OpenStack CLI" %} ```bash $ openstack server set --state active ``` {% endcode %} Verify: {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host ``` {% endcode %} {% code title="Sample Output" %} ``` +----------------------------+----------------+ | Field | Value | +----------------------------+----------------+ | OS-EXT-SRV-ATTR:host | [DEST_HOSTNAME]| | OS-EXT-STS:task_state | None | | status | ACTIVE | +----------------------------+----------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 6 — Set the Instance Placement Allocation on the Destination RP Create the correct Placement allocation for the instance UUID on the destination RP. Include all resources tracked by Placement for the VM's flavor (at minimum VCPU and MEMORY\_MB; add DISK\_GB if local storage is tracked in the environment): {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation set \ --allocation rp=,MEMORY_MB=,VCPU= \ --project \ --user \ ``` {% endcode %} Verify the instance allocation is now on the destination RP: {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation show ``` {% endcode %} The output should show `` with the correct VCPU and MEMORY\_MB values. {% endstep %} {% step %} #### Step 7 — Delete the Stale Migration Placement Allocation Remove the migration consumer's orphaned allocation from the source RP: {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation delete ``` {% endcode %} Verify the allocation is removed: {% code title="OpenStack CLI" %} ```bash $ openstack resource provider allocation show ``` {% endcode %} An empty table confirms the stale source RP allocation has been cleared. {% endstep %} {% step %} #### Step 8 — Mark the Migration Record as Completed With the `instances` table and Placement state now consistent with the destination, mark the migration as `completed`. Connect to the Nova DB using the credentials retrieved in Step 2: {% code title="Control Plane Node" %} ```bash $ kubectl exec -it -n "" \ $(kubectl get po -n "" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \ -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS" ``` {% endcode %} {% code title="Nova DB" %} ```sql USE nova; UPDATE migrations SET status = 'completed', updated_at = NOW() WHERE uuid = ''; ``` {% endcode %} Verify: {% code title="Nova DB" %} ```sql SELECT id, uuid, status, dest_compute FROM migrations WHERE uuid = ''; ``` {% endcode %} {% code title="Sample Output" %} ``` +------+-----------+-----------+----------------+ | id | uuid | status | dest_compute | +------+-----------+-----------+----------------+ | [ID] | [VM_UUID] | completed | [DEST_HOSTNAME]| +------+-----------+-----------+----------------+ ``` {% endcode %} {% endstep %} {% step %} #### Step 9 — Final Verification Run all three checks to confirm a clean recovery: {% code title="OpenStack CLI" %} ```bash $ openstack server show | grep -E 'status|task_state|host|progress' ``` {% endcode %} {% code title="Sample Output" %} ``` +---------------------------+----------------+ | Field | Value | +---------------------------+----------------+ | OS-EXT-SRV-ATTR:host | [DEST_HOSTNAME]| | OS-EXT-STS:task_state | None | | progress | 0 | | status | ACTIVE | +---------------------------+----------------+ ``` {% endcode %} {% code title="OpenStack CLI" %} ```bash # Instance allocation — should show DEST_RP_UUID with correct VCPU and MEMORY_MB $ openstack resource provider allocation show # Migration allocation — should return an empty table $ openstack resource provider allocation show ``` {% endcode %} Expected results: * `status = ACTIVE`, `task_state = None`, `host = [DEST_HOSTNAME]`, `progress = 0` * Instance allocation references `[DEST_RP_UUID]` with correct resource values * Migration allocation returns an empty table If these steps prove insufficient to resolve the issue, reach out to the [Platform9 Support team](https://support.platform9.com/) for additional assistance. {% endstep %} {% endstepper %} ## Additional Information **Related Articles:** * [VM Migration Failure Which Results the VM in "ERROR" State](https://platform9.com/kb/pcd/compute/vm-migration-failure-which-results-the-vm-in-error-state) # Image Upload Fails due to Missing Endpoint ## Problem * Image upload is failing via CLI due to missing `admin` endpoint. ```bash $ echo $OS_INTERFACE admin $ openstack image create --insecure --container-format bare --disk-format qcow2 --file admin endpoint for image service in [REGION_NAME] region not found ``` ## Environment * Platform9 Cloud Director Virtualization - v2025.4 and Higher * Component - Image Library ## Cause * Image uploads greater than 1 Gi are recommended to be performed via CLI command. Also, adding an admin endpoint interface is a prerequisite before uploading the image via CLI. * The `admin` endpoint is missing from glance service catalog; that blocks any Glance administrative tasks. ```bash $ openstack catalog show image +-----------+---------------------------------------------------------------------------+ | Field | Value | +-----------+---------------------------------------------------------------------------+ | endpoints | [REGION_NAME] | | | public: https:__[FQDN]_glance_ | | | [REGION_NAME] | | | internal: http:__glance-api.[REGION_NAME].svc.cluster.local:9292_ | | id | [IMAGE_UUID] | | name | glance | | type | image | +-----------+---------------------------------------------------------------------------+ ``` * It can also be verified by listing endpoint information for a image service. ```bash $ openstack --insecure endpoint list --service image --region +------------------+---------------+--------------+--------------+-----------+-----------+---------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +------------------+---------------+--------------+--------------+-----------+-----------+---------------------------------------------------------+ | [ENDPOINT_UUID1] | [REGION_NAME] | glance | image | True | public | https:__[FQDN]_glance_ | | [ENDPOINT_UUID1] | [REGION_NAME] | glance | image | True | internal | http:__glance-api.[REGION_NAME].svc.cluster.local:9292_ | +----------------------------------+--------------+--------------+-----------+-----------+-----------+---------------------------------------------+ ``` ## Resolution * Manually creating admin endpoint for the image service will allow the images to be uploaded. ```bash $ openstack endpoint create --region image admin https:__:9494 ``` * Verfiy the newly created `admin` endpoint. ```bash $ openstack catalog show image +-----------+---------------------------------------------------------------------------+ | Field | Value | +-----------+---------------------------------------------------------------------------+ | endpoints | [REGION_NAME] | | | public: https:__[FQDN]_glance_ | | | [REGION_NAME] | | | admin: https:__[HOST_IP]:9494 | | | [REGION_NAME] | | | internal: http:__glance-api.[REGION_NAME].svc.cluster.local:9292_ | | id | [IMAGE_UUID] | | name | glance | | type | image | +-----------+---------------------------------------------------------------------------+ ``` ## Additional Information * When the `OS_INTERFACE` is set as `public`*\[default configuration]*, image upload fails with the below error. This is because, **nginx** placed in front of Glance API, blocks an image upload (or download) as the file size exceeds the configured limit *\[not more than 1Gi]*. ```bash HttpException: 413: Client Error for url: https:__[FQDN]_glance_v2_images_[IMAGE_UUID]_file, 413 Request Entity Too Large: nginx ``` # Image Creation is Failing for Volume Size More than 100GB ## Problem When converting large volumes (100GB–400GB or more) from the block device service into images in the image service, the operation may fail or result in an image being created with zero size. ## Environment * Private Cloud Director Virtualization - v2025.10 to v2026.1.427 * Self-Hosted Private Cloud Director Virtualization - v2025.10 to v2026.1.427 * Component - Image Service ## Cause The issue arises from a combination of buffering behavior and timeout limitations during the **volume-to-image** conversion process. When a large upload is initiated, the web server first buffers the incoming data temporarily before forwarding it to the image service. The image service, in turn, writes this data to a staging area before finally committing it to backend storage. Although moving the temporary buffering path to a network file system helps reduce local disk exhaustion, it does not eliminate the dependency on internal timeout limits for handling large uploads. One of the primary causes is that the default timeout configured between the block device service and the image service, typically around 600 seconds, is not sufficient for larger data transfers. As a result, the client connection may get interrupted during the upload process, leading to errors such as a disconnection while sending data. This ultimately results in incomplete uploads, where the image may be marked as successfully created but ends up having zero size due to the partial transfer. Repeated errors in `cindervolume-base.log`: {% code title="/var/log/pf9/cindervolume-base.log" overflow="wrap" %} ```bash INFO cinder.image.pf9_glance [req-[REQ_UUID] None [TENANT]] Exception https://[IMAGE_HOST_IP]:9494 calling 'upload' with args ('[IMAGE_UUID]', <_io.BufferedReader name='/opt/pf9/pf9-cindervolume-base/state/mnt/[UUID]/volume-[VOLUME_UUID]'>), {}: Error communicating with https://[IMAGE_HOST_IP]:9494/v2/images/[IMAGE_UUID]/file: HTTPSConnectionPool(host='[IMAGE_HOST_IP]', port=9494): Read timed out. (read timeout=600.0) ``` {% endcode %} ## Resolution The engineering team has worked on an enhancement that is available starting v2026.4.89 (April Release) and above versions, which was tracked via **PCD-5923.** If an upgrade is not a viable option, proceed with the [#workaround](#workaround "mention") ## Workaround {% hint style="info" %} Workaround changes will **NOT** persist across upgrades since they are overwritten during package updates and will need to be **reapplied** until upgraded to v2026.4.89 version {% endhint %} Update the web server proxy timeout settings to align with the block device service and image service timeout values, ensuring that the entire request flow can support longer upload durations without interruption. ### **Recommended Timeout Values by Volume Size** | **Volume Size** | **Expected Upload Time\*** | **Recommended Timeout** | **Safety Buffer** | | --------------- | -------------------------- | ----------------------- | ----------------- | | 10GB | 5-10 minutes | 1800s (30 min) | 3x buffer | | 50GB | 15-25 minutes | 3600s (1 hour) | 2.5x buffer | | 100GB | 30-45 minutes | 7200s (2 hours) | 2.5x buffer | | 200GB | 60-90 minutes | 14400s (4 hours) | 2.5x buffer | | 250GB | 75-110 minutes | 14400s (4 hours) | 2x buffer | | 500GB | 150-220 minutes | 21600s (6 hours) | 2x buffer | *\*Based on 100 MB/s network speed with 15% overhead* {% stepper %} {% step %} #### Backup the existing config Take Backup of existing configurations from Image Library Host {% code title="Image Library Host" overflow="wrap" %} ```bash $ cp /etc/nginx/conf.d/glance-ssl.conf /etc/nginx/conf.d/glance-ssl.conf.bkp ``` {% endcode %} Take Backup of existing configuration from Persistent Storage Host {% code title="Persistent Storage Host " overflow="wrap" %} ```bash $ cp /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py \ /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py.bkp ``` {% endcode %} {% endstep %} {% step %} #### Increase image upload timeout On the Persistent storage host, check if a `cinder_override.conf` is present. If present update the file to add the timeout. If not create a new file and add the timeout values. {% code title="/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder\_override.conf" overflow="wrap" %} ```bash # cinder_override.conf should have ownership set to pf9:pf9group [DEFAULT] glance_request_timeout = 14400 ``` {% endcode %} {% endstep %} {% step %} #### Redirect Nginx temporary buffer to shared storage On the host with Image library role, update 
location / {
    client_body_temp_path /var/lib/glance/os_glance_staging_store 1 2;
    client_max_body_size 0;
    proxy_connect_timeout 300;     # 5 minutes
    proxy_send_timeout 14400;      # 4 hours (match Cinder) <--Add the timeout same as above
    proxy_read_timeout 14400;      # 4 hours (match Cinder) <-
    send_timeout 14400;            # 4 hours (match Cinder) <-
    proxy_pass http://127.0.0.1:9393;
}
{% endstep %} {% step %} #### Add timeout parameter in pf9\_glance.py Update `pf9_glance.py` file on Persistent Storage host. Apply this change on one Persistent Storage host first, confirm the service is healthy, then repeat on the remaining hosts {% code title="" %} ```py sudo python3 - <<'PYEOF' p = "/opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py" lines = open(p).read().split("\n") out, done, i = [], False, 0 while i < len(lines): s = lines[i] if not done and s.rstrip().endswith("glanceclient.Client(str(version), endpoint,"): ind = s[:len(s) - len(s.lstrip(" "))] # exact leading spaces of the original line out += [ ind + "client_kwargs = {", ind + " 'token': str(context.auth_token),", ind + " 'insecure': CONF.glance_api_insecure", ind + "}", ind + "timeout = getattr(CONF, 'glance_request_timeout', None)", ind + "if timeout is not None:", ind + " client_kwargs['timeout'] = timeout", "", ind + "client = glanceclient.Client(str(version), endpoint, **client_kwargs)", ] i += 2 # skip the original 2-line statement done = True continue out.append(s) i += 1 assert done, "original client line not found - file may differ, no change made" open(p, "w").write("\n".join(out)) print("patched") PYEOF ``` {% endcode %} Post applying above changes, restart the service {% code title="Persistent Storage Host" %} ```bash $ systemctl restart pf9-cindervolume-base.service $ systemctl reload nginx ``` {% endcode %} {% endstep %} {% endstepper %} ## **Additional Information** Nginx temporary buffering requires sufficient disk space, and it is recommended to use an NFS-backed path to prevent exhaustion of the local filesystem during large uploads. The image service staging area will continue to generate temporary files throughout the upload process, which is expected behavior, particularly for large volume-to-image operations. # How to Enable Image Cache in Block Storage ## Problem Creating volumes from images can be slow, especially when the same image is used repeatedly. This can lead to increased I/O load and longer provisioning times. Enabling image caching in PCD's Block Storage (Cinder) service can significantly improve performance by storing a cached copy of frequently used images. ## Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Private Cloud Director Kubernetes – v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher ## Procedure 1. Edit Cinder Backend Configuration: On the host where the Persistent Storage (or Cinder Volume) role is enabled, add the following parameters in the `[DEFULT]` section of the `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` ```bash image_volume_cache_enabled image_volume_cache_max_count image_volume_cache_max_size_gb ``` > Adjust `image_volume_cache_max_count` and `image_volume_cache_max_size_gb` based on the available disk space and use case. For Example: ```bash image_volume_cache_enabled = True image_volume_cache_max_count = 50 image_volume_cache_max_size_gb = 200 ``` > If NFS is configured as the backend for block storage, then do not enable image caching. Set `image_volume_cache_enabled = False` to avoid issues related to root disk sizing. (Known issue PCD-2623) 2. After updating the configuration, restart the `pf9-cindervolume-base` service for the changes to take effect: ```bash $ sudo systemctl restart pf9-cindervolume-base ``` ## Validation **Check if Image Cache is Enabled:** Verify the effective configuration using the following command on the cinder node: ```bash $ grep image_volume_cache _opt_pf9_etc_pf9-cindervolume-base_conf.d_cinder.conf ``` ## Additional Information What Each Parameter Means: * `image_volume_cache_enabled`: Enables or disables the image caching mechanism. * `image_volume_cache_max_count`: Limits the number of images to cache. If the limit is reached, the least recently used (LRU) image will be evicted. * `image_volume_cache_max_size_gb`: Limits the total size (in GB) of all cached images. The LRU policy also applies here when the limit is hit. # VM Creation Fails With ImageUnacceptable Error Resolve VM creation errors in Private Cloud Director due to QCOW2 image corruption. Learn how to diagnose and fix issues with image conversion to RAW, ensuring successful instance boots. Get tips on v --- [Next Page](https://platform9.com/kb/llms-full.txt/1)