> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/generic/hotplug-memory-reporting-mismatch-in-windows-guest-os-1.md).

# Hostagent Intermittently Fails to Converge with Virsh Domcapabilities Failures

## Problem

After performing a host upgrade or re-onboarding (decommission + prep-node) of a compute host in Platform9 PCD, the hostagent intermittently fails to converge. The host may appear **Online** in the PCD UI, but the hostagent log continuously reports errors related to `virsh domcapabilities` and `fetch_gpu_info.py`.

## Environment

* Private Cloud Director Virtualization — v2025.10-180
* Self-Hosted Private Cloud Director Virtualization — v2025.10-180
* Component**:** Hostagent, GPU.

## Diagnostics

* Check for virsh domcapabilities failure and converging failures in `hostagent.log` 

  <pre class="language-bash" data-title="/var/log/pf9/hostagent.log" data-overflow="wrap"><code class="lang-bash">sysinfo.py ERROR - Failed to get supported CPU models:
    Command '['virsh', 'domcapabilities']' returned non-zero exit status 1.
  Traceback (most recent call last):
    File ".../bbslave/sysinfo.py", in get_supported_cpu_models
      result = subprocess.run(['virsh', 'domcapabilities'],
  subprocess.CalledProcessError: Command '['virsh', 'domcapabilities']'
    returned non-zero exit status 1.
  </code></pre>
* Check for `fetch_gpu_info.py` extension failure

  <pre class="language-bash" data-title="/var/log/pf9/hostagent.log" data-overflow="wrap"><code class="lang-bash">session.py ERROR - /opt/pf9/hostagent/extensions/fetch_gpu_info.py command failed:
    Command '['/opt/pf9/hostagent/extensions/fetch_gpu_info.py']'
    returned non-zero exit status 1.
  </code></pre>

#### Affected Scenarios:

This issue has been observed in the following scenarios:

* Host upgrade (in-place package upgrade of `pf9-ostackhost`)
* Host re-onboarding via `pcdctl decommission-node -f` followed by `pcdctl prep-node`

## Cause

The `/opt/pf9/home` directory is incorrectly owned by `root:root` instead of `pf9:pf9group` after a host upgrade or re-onboarding.

The hostagent runs `virsh domcapabilities` under the `pf9` user context. When `/opt/pf9/home` is owned by `root`, the `pf9` user cannot access its home directory, causing `virsh` to fail with a non-zero exit code.

The root cause is in the Platform9 package installation script `/opt/pf9/hostagent/bin/pf9-apt`. The package manager runs as `root` and extracts package files with root ownership. When the `pf9-ostackhost` package is installed or upgraded, `/opt/pf9/home` is extracted/created with `root:root` ownership instead of `pf9:pf9group`.

This is confirmed by the directory listing post-upgrade:

```bash
$ ls -lart /opt/pf9/
drwxr-xr-x  3 root root     4096 [TIMESTAMP] home            ← INCORRECT (should be pf9:pf9group)
drwxr-xr-x  2 root root     4096 [TIMESTAMP] pf9-ostackhost   
drwxr-xr-x  8 pf9  pf9group 4096 [TIMESTAMP] venv            
drwxr-xr-x  2 pf9  pf9group 4096 [TIMESTAMP] cache           
```

## Resolution

This is a known issue identified in PCD v2025.10-180, and the fix is to ensure the package installation/upgrade process correctly sets `pf9:pf9group` ownership to `/opt/pf9/home` during package extraction is now incorporated into **PCD v2026.1-260** and above versions.

## Workaround

If upgrading immediately is not possible, apply the following immediate workaround on each affected compute host.

{% stepper %}
{% step %}

#### Fix directory ownership

```bash
chown pf9:pf9group /opt/pf9/home
```

{% endstep %}

{% step %}

#### Verify the ownership fix

```bash
ls -la /opt/pf9/ | grep home

##Expected output:
drwxr-xr-x  3 pf9  pf9group  4096 [TIMESTAMP] home
```

{% endstep %}

{% step %}

#### Verify virsh works under the pf9 user

```bash
sudo -u pf9 virsh domcapabilities
```

Expected: XML output returned without errors.
{% endstep %}

{% step %}

#### Check hostagent converge recovers automatically

```bash
tail -f /var/log/pf9/hostagent.log | grep -i "converge\|error"
```

Expected: `--- Converging ---` followed by no `virsh` or `fetch_gpu_info` errors.

In most cases, **no service restart is required**. The hostagent will automatically retry to converge and succeed once the ownership is corrected.
{% endstep %}

{% step %}

#### (If needed) Restart hostagent manually

If the hostagent does not recover automatically within 2–3 minutes:

```bash
sudo systemctl restart pf9-hostagent
sudo systemctl status pf9-hostagent
```

{% endstep %}
{% endstepper %}

## Validation

After applying the workaround or upgrade, validate in the following order:

{% stepper %}
{% step %}

#### Confirm correct ownership

```bash
ls -la /opt/pf9/ | grep home

##Expected file ownership: pf9 pf9group
```

{% endstep %}

{% step %}

#### Confirm virsh works under the pf9 user

<pre class="language-bash"><code class="lang-bash">sudo -u pf9 virsh domcapabilities | head -5
<strong>
</strong><strong>## Expected XML output:
</strong>&#x3C;domainCapabilities>
  &#x3C;path>/usr/bin/qemu-system-x86_64&#x3C;/path>
  &#x3C;domain>kvm&#x3C;/domain>
  &#x3C;machine>pc-i440fx-focal&#x3C;/machine>
  &#x3C;arch>x86_64&#x3C;/arch>
</code></pre>

{% endstep %}

{% step %}

#### Confirm hostagent is converging cleanly

```bash
tail -30 /var/log/pf9/hostagent.log
```

Expected: `--- Converging ---` with no `virsh` or `fetch_gpu_info` errors
{% endstep %}

{% step %}

#### Confirm the host is Online in the PCD UI

Navigate to: **Infrastructure → Hosts**

Expected: Host status = **Online / Applied**
{% endstep %}
{% endstepper %}

## Additional Information

* **GPU Hosts:** This issue has a higher impact on GPU compute hosts. In addition to the `virsh domcapabilities` failure, the `fetch_gpu_info.py` extension also fails, which can prevent GPU resource reporting to the hostagent and affect GPU VM scheduling. Apply the ownership fix on **all GPU hosts** after any upgrade or re-onboarding.
* **Re-onboarding:** If a host was decommissioned (`pcdctl decommission-node -f`) and re-onboarded (`pcdctl prep-node`), the ownership issue may reoccur. Always verify `/opt/pf9/home` ownership after re-onboarding and apply the `chown` fix if needed.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/generic/hotplug-memory-reporting-mismatch-in-windows-guest-os-1.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.