90 days. Unlimited cores. Zero commitment | The VMware Exit built for service
Learn More
Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

NetApp NFS Block Storage Service get_volume_stats High Latency

Problem

The NetApp ONTAP Cluster-mode NFS driver's get_volume_stats operation consistently takes 30–40 seconds per stats cycle on Block Storage Service hosts. Warning log entries similar to the following appear repeatedly at approximately 60-second intervals:

Sample Output
WARNING cinder.volume.manager [req-[REQ_UUID] None None] The NetAppCmodeNfsDriver volume driver's get_volume_stats operation ran for [DURATION] seconds. This may indicate a performance problem with the backend which can lead to instability.

Because the stats cycle runs synchronously in the Block Storage Service processing thread, volume create, snapshot, and other storage operations queue behind each active cycle. Operations that complete in 2–6 seconds on the storage backend appear to take 60–70 seconds from the user's perspective.

Environment

  • Private Cloud Director Virtualization - All current versions

  • Self-Hosted Private Cloud Director Virtualization - All current versions

  • Component: Block Storage Service, NetApp ONTAP Cluster-mode NFS Driver

Cause

The NetApp ONTAP Cluster-mode NFS driver uses Python's HTTPBasicAuthHandler for all ONTAP management API calls. This handler is stateless — no credential caching and no session reuse between calls. Each management API call triggers a 401 Unauthorized challenge from ONTAP, after which ONTAP validates the credentials against Active Directory (AD) before returning a response. Additionally, HTTPBasicAuthHandler does not pre-send the Authorization header — the handler waits for the 401 challenge before attaching credentials — resulting in two HTTP round-trips per API call.

The _update_volume_stats path fans out approximately 2N + 3 API calls per cycle, where N is the number of FlexVols or NFS shares managed by the backend. When the configured NetApp service account is AD-backed, each call incurs the AD domain controller lookup latency (~350–400 ms) on top of normal ONTAP processing time (~10–50 ms). A backend with 47 FlexVols makes approximately 97 API calls per cycle — at ~390 ms per call, this produces a 37–40-second stats cycle.

The stats cycle runs synchronously in the Block Storage Service processing thread. A 37–40-second blocking operation on the default 60-second periodic_interval starves the thread and prevents RPC messages — volume create, snapshot, clone, and attach requests — from being processed during that window.

Diagnostics

1

Step 1 — Check for Warning Log Entries

Check the Block Storage Service logs on each Persistent Storage Host for the get_volume_stats warning.

Repeated warning entries at approximately 60-second intervals confirm the issue is active.

2

Step 2 — Identify the Service Account Authentication Method

Read the netapp_login value from the active Block Storage Service configuration on each Persistent Storage Host.

A login value in the format DOMAIN\username or username@domain.com indicates an AD-backed account. To confirm the account type, ask the NetApp administrator to run the following command on the ONTAP cluster: security login show -vserver <SVM_NAME> -user-or-group-name <SERVICE_ACCOUNT>. An Authentication Method value of domain confirms the account is AD-backed and this issue applies.

If both checks above match, proceed to Workaround.

Resolution

The upstream fix (Cinder bug #2128652, merged in Cinder 28.0/Gazpacho) will be available in PCD once the upstream Cinder version is adopted. We are tracking this under PCD-7975. In the meantime, use the Workaround below.

Workaround

Switching the NetApp service account from an AD-backed account to a local ONTAP account eliminates the AD lookup overhead per API call. This reduces get_volume_stats cycle time from 37–40 seconds to approximately 2 seconds on typical deployments.

Before proceeding, ask the NetApp administrator to create a local ONTAP account on the relevant SVM with ontapi application access and password authentication: security login create -vserver <SVM_NAME> -user-or-group-name <LOCAL_ONTAP_USER> -application ontapi -authmethod password. Note the account name and password before continuing.

SaaS customers: The configuration override file (Method 1) is not available for SaaS deployments. Use Method 2 — Apply via Cluster Blueprint to apply the credential change, or contact Platform9 Support for assistance.

Apply the new credentials using one of the following methods. Method 1 is a quick validation approach and does not persist across blueprint reapplication. Method 2 is the permanent configuration change.

Method 1 — Apply via Configuration Override File

1

Step 1 — Edit the Configuration Override File

On each Persistent Storage Host, edit or create the configuration override file at /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf. Add the local ONTAP account credentials under each backend section. Backend section names match those defined in /opt/pf9/etc/pf9-cindervolume-base/cinder.conf.

Add one stanza per backend section. Omit sections that do not use an AD-backed account.

2

Step 2 — Restart the Block Storage Service

Restarting the Block Storage Service briefly interrupts volume operation processing. In-flight operations resume after the service restarts.

Repeat Step 1 and Step 2 on all Persistent Storage Hosts in the affected region.

3

Step 3 — Verify the Warning Entries Stop

Monitor the Block Storage Service logs for 5–10 minutes after restart. No get_volume_stats warning entries should appear.

Confirm all Block Storage Service backends are in up state:

All State values must show up and all Status values must show enabled.

The configuration override file does not persist through blueprint reapplication. After validating with Method 1, promote the change to the cluster blueprint using Method 2 and remove the override entries from cinder_override.conf.

Method 2 — Apply via Cluster Blueprint

1

Step 1 — Update the NetApp Backend Credentials in the Blueprint

In the PCD management interface, open the cluster and navigate to the cluster blueprint. Locate the NetApp NFS backend configuration and update the netapp_login and netapp_password fields with the local ONTAP account credentials.

Save and apply the blueprint. PCD deploys the updated configuration to all Persistent Storage Hosts and restarts the Block Storage Service automatically.

2

Step 2 — Verify the Configuration Is Applied

On each Persistent Storage Host, confirm the local ONTAP account name is now in the active configuration.

The output must show the local ONTAP account name with no domain prefix.

3

Step 3 — Verify the Warning Entries Stop

Monitor the Block Storage Service logs for 5–10 minutes after blueprint application. No get_volume_stats warning entries should appear.

Confirm all Block Storage Service backends are in up state:

All State values must show up and all Status values must show enabled.

PreviousHow to Migrate the NFS Filer Backing an NFS BackendNextVolume Snapshots Stuck in Error Deleting on Tintri VMstore

Last updated