> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/storage/netapp-nfs-block-storage-service-get_volume_stats-high-latency.md).

# NetApp NFS Block Storage Service get\_volume\_stats High Latency

## Problem

The NetApp ONTAP Cluster-mode NFS driver's `get_volume_stats` operation consistently takes 30–40 seconds per stats cycle on Block Storage Service hosts. Warning log entries similar to the following appear repeatedly at approximately 60-second intervals:

{% code title="Sample Output" %}

```bash
WARNING cinder.volume.manager [req-[REQ_UUID] None None] The NetAppCmodeNfsDriver volume driver's get_volume_stats operation ran for [DURATION] seconds. This may indicate a performance problem with the backend which can lead to instability.
```

{% endcode %}

Because the stats cycle runs synchronously in the Block Storage Service processing thread, volume create, snapshot, and other storage operations queue behind each active cycle. Operations that complete in 2–6 seconds on the storage backend appear to take 60–70 seconds from the user's perspective.

## Environment

* Private Cloud Director Virtualization - All current versions
* Self-Hosted Private Cloud Director Virtualization - All current versions
* Component: Block Storage Service, NetApp ONTAP Cluster-mode NFS Driver

## Cause

The NetApp ONTAP Cluster-mode NFS driver uses Python's `HTTPBasicAuthHandler` for all ONTAP management API calls. This handler is stateless — no credential caching and no session reuse between calls. Each management API call triggers a `401 Unauthorized` challenge from ONTAP, after which ONTAP validates the credentials against Active Directory (AD) before returning a response. Additionally, `HTTPBasicAuthHandler` does not pre-send the `Authorization` header — the handler waits for the 401 challenge before attaching credentials — resulting in two HTTP round-trips per API call.

The `_update_volume_stats` path fans out approximately `2N + 3` API calls per cycle, where N is the number of FlexVols or NFS shares managed by the backend. When the configured NetApp service account is AD-backed, each call incurs the AD domain controller lookup latency (\~350–400 ms) on top of normal ONTAP processing time (\~10–50 ms). A backend with 47 FlexVols makes approximately 97 API calls per cycle — at \~390 ms per call, this produces a 37–40-second stats cycle.

The stats cycle runs synchronously in the Block Storage Service processing thread. A 37–40-second blocking operation on the default 60-second `periodic_interval` starves the thread and prevents RPC messages — volume create, snapshot, clone, and attach requests — from being processed during that window. 

## Diagnostics

{% stepper %}
{% step %}

#### Step 1 — Check for Warning Log Entries

Check the Block Storage Service logs on each Persistent Storage Host for the `get_volume_stats` warning.

{% code title="Persistent Storage Host - /var/log/pf9/cindervolume-base.log" %}

```bash
$ grep -i "get_volume_stats" /var/log/pf9/cindervolume-base.log
```

{% endcode %}

{% code title="Sample Output" %}

```bash
[TIMESTAMP] WARNING cinder.volume.manager [req-[REQ_UUID] None None] The NetAppCmodeNfsDriver volume driver's get_volume_stats operation ran for [DURATION] seconds. This may indicate a performance problem with the backend which can lead to instability.
```

{% endcode %}

Repeated warning entries at approximately 60-second intervals confirm the issue is active.
{% endstep %}

{% step %}

#### Step 2 — Identify the Service Account Authentication Method

Read the `netapp_login` value from the active Block Storage Service configuration on each Persistent Storage Host.

{% code title="Persistent Storage Host" %}

```bash
$ sudo grep 'netapp_login' /opt/pf9/etc/pf9-cindervolume-base/cinder.conf
```

{% endcode %}

{% code title="Sample Output" %}

```bash
netapp_login = [DOMAIN]\[SERVICE_ACCOUNT]
```

{% endcode %}

A login value in the format `DOMAIN\username` or `username@domain.com` indicates an AD-backed account. To confirm the account type, ask the NetApp administrator to run the following command on the ONTAP cluster: `security login show -vserver <SVM_NAME> -user-or-group-name <SERVICE_ACCOUNT>`. An `Authentication Method` value of `domain` confirms the account is AD-backed and this issue applies.
{% endstep %}
{% endstepper %}

If both checks above match, proceed to [Workaround](#workaround).

## Resolution

The upstream fix (Cinder bug #2128652, merged in Cinder 28.0/Gazpacho) will be available in PCD once the upstream Cinder version is adopted. We are tracking this under `PCD-7975`. In the meantime, use the [Workaround](#workaround) below.

## Workaround

Switching the NetApp service account from an AD-backed account to a local ONTAP account eliminates the AD lookup overhead per API call. This reduces `get_volume_stats` cycle time from 37–40 seconds to approximately 2 seconds on typical deployments.

{% hint style="info" %}
Before proceeding, ask the NetApp administrator to create a local ONTAP account on the relevant SVM with `ontapi` application access and `password` authentication: `security login create -vserver <SVM_NAME> -user-or-group-name <LOCAL_ONTAP_USER> -application ontapi -authmethod password`. Note the account name and password before continuing.
{% endhint %}

{% hint style="info" %}
**SaaS customers:** The configuration override file (Method 1) is not available for SaaS deployments. Use [#method-2-apply-via-cluster-blueprint](#method-2-apply-via-cluster-blueprint "mention") to apply the credential change, or contact [Platform9 Support](https://support.platform9.com/) for assistance.
{% endhint %}

Apply the new credentials using one of the following methods. Method 1 is a quick validation approach and does not persist across blueprint reapplication. Method 2 is the permanent configuration change.

### Method 1 — Apply via Configuration Override File

{% stepper %}
{% step %}

#### Step 1 — Edit the Configuration Override File

On each Persistent Storage Host, edit or create the configuration override file at `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf`. Add the local ONTAP account credentials under each backend section. Backend section names match those defined in `/opt/pf9/etc/pf9-cindervolume-base/cinder.conf`.

{% code title="Persistent Storage Host" %}

```bash
$ sudo vi /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf
```

{% endcode %}

{% code title="Persistent Storage Host" %}

```ini
[<BACKEND_SECTION_1>]
netapp_login = <LOCAL_ONTAP_USER>
netapp_password = <LOCAL_ONTAP_PASSWORD>

[<BACKEND_SECTION_2>]
netapp_login = <LOCAL_ONTAP_USER>
netapp_password = <LOCAL_ONTAP_PASSWORD>
```

{% endcode %}

Add one stanza per backend section. Omit sections that do not use an AD-backed account.
{% endstep %}

{% step %}

#### Step 2 — Restart the Block Storage Service

{% hint style="warning" %}
Restarting the Block Storage Service briefly interrupts volume operation processing. In-flight operations resume after the service restarts.
{% endhint %}

{% code title="Persistent Storage Host" %}

```bash
$ sudo systemctl restart pf9-cindervolume-base
```

{% endcode %}

Repeat Step 1 and Step 2 on all Persistent Storage Hosts in the affected region.
{% endstep %}

{% step %}

#### Step 3 — Verify the Warning Entries Stop

Monitor the Block Storage Service logs for 5–10 minutes after restart. No `get_volume_stats` warning entries should appear.

{% code title="Persistent Storage Host" %}

```bash
$ sudo journalctl -u pf9-cindervolume-base -f | grep -i "get_volume_stats"
```

{% endcode %}

Confirm all Block Storage Service backends are in `up` state:

{% code title="OpenStack CLI" %}

```bash
$ openstack volume service list
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+-----------------+---------------------------+------+---------+-------+----------------------------+
| Binary          | Host                      | Zone | Status  | State | Updated At                 |
+-----------------+---------------------------+------+---------+-------+----------------------------+
| cinder-volume   | [HOSTNAME]@[BACKEND_NAME] | nova | enabled | up    | [TIMESTAMP]                |
+-----------------+---------------------------+------+---------+-------+----------------------------+
```

{% endcode %}

All `State` values must show `up` and all `Status` values must show `enabled`.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
The configuration override file does not persist through blueprint reapplication. After validating with Method 1, promote the change to the cluster blueprint using Method 2 and remove the override entries from `cinder_override.conf`.
{% endhint %}

### Method 2 — Apply via Cluster Blueprint

{% stepper %}
{% step %}

#### Step 1 — Update the NetApp Backend Credentials in the Blueprint

In the PCD management interface, open the cluster and navigate to the cluster blueprint. Locate the NetApp NFS backend configuration and update the `netapp_login` and `netapp_password` fields with the local ONTAP account credentials.

Save and apply the blueprint. PCD deploys the updated configuration to all Persistent Storage Hosts and restarts the Block Storage Service automatically.
{% endstep %}

{% step %}

#### Step 2 — Verify the Configuration Is Applied

On each Persistent Storage Host, confirm the local ONTAP account name is now in the active configuration.

{% code title="Persistent Storage Host" %}

```bash
$ sudo grep 'netapp_login' /opt/pf9/etc/pf9-cindervolume-base/cinder.conf
```

{% endcode %}

{% code title="Sample Output" %}

```bash
netapp_login = [LOCAL_ONTAP_USER]
```

{% endcode %}

The output must show the local ONTAP account name with no domain prefix.
{% endstep %}

{% step %}

#### Step 3 — Verify the Warning Entries Stop

Monitor the Block Storage Service logs for 5–10 minutes after blueprint application. No `get_volume_stats` warning entries should appear.

{% code title="Persistent Storage Host" %}

```bash
$ sudo journalctl -u pf9-cindervolume-base -f | grep -i "get_volume_stats"
```

{% endcode %}

Confirm all Block Storage Service backends are in `up` state:

{% code title="OpenStack CLI" %}

```bash
$ openstack volume service list
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+-----------------+---------------------------+------+---------+-------+----------------------------+
| Binary          | Host                      | Zone | Status  | State | Updated At                 |
+-----------------+---------------------------+------+---------+-------+----------------------------+
| cinder-volume   | [HOSTNAME]@[BACKEND_NAME] | nova | enabled | up    | [TIMESTAMP]                |
+-----------------+---------------------------+------+---------+-------+----------------------------+
```

{% endcode %}

All `State` values must show `up` and all `Status` values must show `enabled`.
{% endstep %}
{% endstepper %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://platform9.com/kb/pcd/storage/netapp-nfs-block-storage-service-get_volume_stats-high-latency.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.