Cinder Volume Operations Failing With Vmstore “NotFound” Error on Tintri Backend

Problem

Volume operations in PCD Storage Service fail when using the Tintri (vmstore) backend.

Users may observe:

• Volume stuck in creating state and eventually failing.
• Failures during operations such as:

  • Volume create
  • Volume clone
  • Other CRUD operations
• PCD UI throws error like:

  Could not find any available weighted backend

• On the persistent storage host, cindervolume-base logs reports:

  VmstoreException: Volume driver reported an error: No details 
  (typeId: VmstoreError, code: NotFound)

Environment

• Private Cloud Director Virtualization
• Self-Hosted Private Cloud Director Virtualization
• Component - Storage Service, Tintri
• Tintri - v3.0.5 and below

Resolution

• The Tintri Driver needs to be upgraded to the version 3.0.6 and above, followed by a rollout restart of the cinder-scheduler workload for the respective PCD region.

  • Customers need to apply the Tintri driver upgrade patch to the Persistent Storage Nodes. Reach out to the Tintri Support Team to implement this patch.
  • Once the Tintri driver is upgraded, reach out to the Platform9 Team about a rollout restart of cinder-scheduler workload in the affected region.

Only Self-hosted PCD customers can run the command to restart the cinder-scheduler:

$ kubectl rollout restart deployment/cinder-scheduler -n $REGION_NAMESPACE

Additional Information

• Post PCD upgrade, customers need to upgrade the Tintri driver to 3.0.6 or above versions.