Volume Snapshots Stuck in Error Deleting on Tintri VMstore

Problem

One or more Block Storage volume snapshots are stuck in error_deleting status on a Tintri VMstore-backed deployment and cannot be deleted via the PCD UI or CLI. Deletion attempts fail silently — no explicit error is surfaced — and the snapshot reverts to error_deleting after each attempt.

Running openstack volume snapshot list shows one or more snapshots in error_deleting status:

OpenStack CLI

$ openstack volume snapshot list --all-projects | grep error_deleting

Sample Output — Snapshots Stuck in Error Deleting

| [SNAPSHOT_UUID_1] | [SNAPSHOT_NAME_1] | [VOLUME_NAME_1] | error_deleting | [SIZE] GiB |
| [SNAPSHOT_UUID_2] | [SNAPSHOT_NAME_2] | [VOLUME_NAME_2] | error_deleting | [SIZE] GiB |

Attempting to delete the snapshot via CLI produces no error output, but the snapshot returns to error_deleting:

OpenStack CLI

$ openstack volume snapshot delete <SNAPSHOT_UUID>

Environment

• Private Cloud Director Virtualization - All versions

• Self-Hosted Private Cloud Director Virtualization - All versions

• Component: Block Storage Service, Tintri VMstore

Cause

Tintri VMstore enforces referential integrity on snapshot objects. When a volume is created by cloning a snapshot — either explicitly or as a result of volume creation from an image that was snapshot-backed — Tintri increments the cloneReferenceCount on the source snapshot. Tintri does not permit deletion of a snapshot whose cloneReferenceCount is greater than zero, even when the snapshot is flagged as isOrphaned: true on the Tintri side.

When the Block Storage Service issues a delete call to the Tintri API, the API silently blocks the operation rather than returning an explicit error. The Block Storage Service receives no failure signal, releases the coordination lock, and the snapshot transitions back to error_deleting. This cycle repeats on every subsequent delete attempt.

Block Storage Service force-delete options do not bypass this constraint — the block is enforced at the Tintri storage layer, not within the Block Storage Service itself.

Diagnostics

1

Step 1 — Confirm the snapshot is stuck in error_deleting

OpenStack CLI

$ openstack volume snapshot show <SNAPSHOT_UUID> -c status -c volume_id -c name

Sample Output — Snapshot Stuck in Error Deleting

+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| name      | [SNAPSHOT_NAME]                      |
| status    | error_deleting                       |
| volume_id | [SOURCE_VOLUME_UUID]                 |
+-----------+--------------------------------------+

2

Step 2 — Identify the Block Storage host and backend for the source volume

OpenStack CLI

$ openstack volume show <SOURCE_VOLUME_UUID> -c status -c "os-vol-host-attr:host" -c type

Sample Output — Volume on Tintri Backend

+----------------------+----------------------------------------------------+
| Field                | Value                                              |
+----------------------+----------------------------------------------------+
| os-vol-host-attr:host| [BLOCK_STORAGE_HOST_UUID]@[BACKEND_NAME]#[POOL_PATH] |
| status               | available                                          |
| type                 | [VOLUME_TYPE]                                      |
+----------------------+----------------------------------------------------+

Confirm the backend name contains a Tintri-specific path (e.g., /tintri/). If the host shows a different backend, this workaround may not apply.

3

Step 3 — Inspect the Block Storage pod log for repeated delete cycles

Identify the Block Storage volume pod name, then grep the logs for the snapshot UUID to confirm repeated delete lock cycles with no state change.

Self-Hosted only: This step requires access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.

Control Plane Node

$ kubectl get pods -n <REGION_NS> | grep cinder-api
$ kubectl logs -n <REGION_NS> <CINDER_API_POD> | grep <SNAPSHOT_UUID>

Sample Output — Block Storage Log Showing Repeated Delete Attempts

INFO cinder.volume.drivers.vmstore.nfs [...] Deleting snapshot snapshot-[SNAPSHOT_UUID]
DEBUG cinder.volume.drivers.vmstore.api [...] Start request hook on session request GET https://[TINTRI_APPLIANCE_FQDN]:443/api/v310/snapshot?contain=snapshot-[SNAPSHOT_UUID] None and session response 401 {"code":"ERR-API-0104","message":"Invalid session."}
DEBUG cinder.volume.drivers.vmstore.api [...] Getting list of VolumeSnapshot with path: snapshot?contain=snapshot-[SNAPSHOT_UUID]
DEBUG cinder.coordination [...] Lock "cinder-[SNAPSHOT_UUID]-delete_snapshot" released by "delete_snapshot" :: held [DURATION]s

The ERR-API-0104 Invalid session response is a normal Tintri session expiry that auto-recovers — it does not indicate an authentication failure. The delete cycle silently fails after re-authentication succeeds, with no ERROR line in the log.

4

Step 4 — Verify the clone reference count on the Tintri backend

Log in to the Tintri VMstore management UI or query the Tintri API directly from the Block Storage host to confirm cloneReferenceCount is blocking deletion.

Persistent Storage Host

$ curl -sk -H "Content-Type: application/json" \
  https://<TINTRI_HOST>/api/v310/snapshot/<TINTRI_SNAPSHOT_UUID> \
  | python3 -m json.tool | grep -E "cloneReferenceCount|isOrphaned|description"

Sample Output — Snapshot Blocked by Clone Reference

"cloneReferenceCount": 1,
"description": "snapshot-[SNAPSHOT_UUID]",
"isOrphaned": true,

A cloneReferenceCount value greater than 0 confirms that the Tintri backend is blocking deletion.

If all checks above match, proceed to Workaround.

Resolution

No permanent fix exists within the Block Storage Service for this behavior. The constraint is enforced at the Tintri VMstore storage layer and cannot be bypassed through Block Storage Service configuration. Use the Workaround below to release the clone reference before deleting the snapshot.

Workaround

The following steps release the clone reference on the Tintri backend, allowing the snapshot to be deleted. The order of these steps is critical — deleting the snapshot from PCD before the reference is released at the storage layer will return the snapshot to error_deleting.

Complete all steps in the order listed. Skipping or reordering steps will cause the snapshot to return to error_deleting.

1

Step 1 — Migrate the source volume to a different backend

Identify the source volume UUID from the snapshot details confirmed in Step 1 of the Diagnostics section. Migrate the volume off the Tintri backend to any other available backend. This causes Tintri to release the clone reference on the snapshot.

OpenStack CLI

$ openstack volume migrate --host <DESTINATION_BACKEND_HOST> <SOURCE_VOLUME_UUID>

Sample Output — Migration Initiated

+----------------------------+-----------+
| Field                      | Value     |
+----------------------------+-----------+
| migration_status           | migrating |
| status                     | available |
+----------------------------+-----------+

<DESTINATION_BACKEND_HOST> must be a valid backend host string in the format <BLOCK_STORAGE_HOST_UUID>@<BACKEND_NAME>#<POOL_PATH>. Run openstack volume backend pool list to find available backends.

Monitor the volume status until migration_status returns to None and status is available before proceeding to the next step.

If the source volume has already been deleted in PCD (the snapshot list shows None in the volume column), the migration step is not applicable. Skip to Additional Information for the alternative approach.

2

Step 2 — Wait for the clone reference count to drop to zero

Log in to the Tintri VMstore management UI for the relevant appliance and navigate to the snapshot list for the affected VM. Wait until the snapshot entry shows cloneReferenceCount: 0. Do not proceed to Step 3 until this is confirmed.

Alternatively, poll the Tintri API from the Block Storage host:

Persistent Storage Host

$ curl -sk -H "Content-Type: application/json" \
  https://<TINTRI_HOST>/api/v310/snapshot/<TINTRI_SNAPSHOT_UUID> \
  | python3 -m json.tool | grep cloneReferenceCount

Sample Output — Reference Released

"cloneReferenceCount": 0,

3

Step 3 — Delete the snapshot object from the Tintri VMstore UI

After confirming cloneReferenceCount is 0, delete the snapshot directly from the Tintri VMstore management UI. This removes the underlying storage object before the Block Storage Service attempts deletion.

Navigate to the snapshot list in the Tintri UI, locate the snapshot by its description (snapshot-<SNAPSHOT_UUID>), and delete it.

Delete the snapshot from the Tintri UI before proceeding to Step 4. Deleting the PCD snapshot first will return the snapshot to error_deleting.

4

Step 4 — Delete the snapshot from PCD and verify removal

With the underlying Tintri object removed, the Block Storage Service delete call will succeed because no storage object remains to block it.

OpenStack CLI

$ openstack volume snapshot delete <SNAPSHOT_UUID>

Confirm the snapshot no longer appears in the list:

OpenStack CLI

$ openstack volume snapshot show <SNAPSHOT_UUID>

Sample Output — Snapshot Successfully Deleted

No snapshot with a name or ID of '[SNAPSHOT_UUID]' exists.

The snapshot has been successfully removed from both the Tintri backend and PCD.

Additional Information

• The cloneReferenceCount field on a Tintri snapshot reflects the number of volumes created by cloning that snapshot. This count must reach 0 before Tintri permits snapshot deletion — either via the Tintri UI or via a Block Storage Service driver call.

• The isOrphaned: true flag on the Tintri snapshot indicates that Tintri has lost the association between the snapshot and its parent VM. This flag does not remove the clone reference block.

• Deleting the PCD snapshot before clearing the Tintri storage object will return the snapshot to error_deleting. The order in the Workaround section must be followed exactly.

• If the source volume has already been deleted in PCD but the snapshot remains stuck, the volume migration step is not applicable. Work directly with the Tintri support team to manually remove the clone reference from the VMstore before attempting PCD deletion.

• In some cases, the Block Storage log may contain a Tintri ERR-API-3107 error with the message Cannot delete the shared snapshot because it has younger or older siblings. This indicates the snapshot has sibling dependencies on the Tintri side in addition to the clone reference. Engage the Tintri support team to identify and remove all sibling and clone references before reattempting deletion.

• Platform9 is engaged with Tintri Engineering to address this behavior and provide a more direct resolution path in a future release.

• If the standard workaround does not resolve the issue, contact Platform9 Support for assistance. A Block Storage database soft-delete is available as a last resort and must only be performed under Platform9 guidance.