> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd-ts/storage/troubleshooting-volume-attachment-and-detachment-issues.md). # Troubleshooting Volume Attachment & Detachment Issues ### Problem This guide provides step-by-step instructions for troubleshooting and resolving issues that arise when a volume attachment or detachment fails. Symptoms typically include volumes stuck in `attaching` or `detaching` states, or failing with specific hypervisor faults. ### Environment * Private Cloud Director Virtualization - v2025.4 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component - PCD Storage Service ### Volume Attachment Protocols The method by which a volume is attached depends entirely on your backend storage configuration. To check the list of the supported backend volume drivers Refer the following[ link](https://docs.openstack.org/cinder/2023.1/configuration/block-storage/drivers/). ### Deep Dive The Private Cloud Director volume attach/detach process is a highly coordinated handshake between the Compute service (Nova) and the Block Storage service (Cinder). Nova acts as the primary orchestrator, asking Cinder for connection details, physically mapping the storage to the compute node using a library called `os-brick`, and finally hot-plugging it into the running VM via Libvirt. {% hint style="info" %} Note: The `kubectl` logs shown in the API steps below can only be reviewed in Self-Hosted Private Cloud Director environments. {% endhint %} ### Step 1: User Request & API Validation A user submits a request to attach a volume to a specific VM. The `nova-api` validates the request, checks if the VM is in a valid state (`ACTIVE` or `SHUTOFF`), and checks if the volume is `available`. Cinder then changes the volume status to `attaching` to lock it. {% code title="Sample Log (nova-api-osapi)" %} ```ruby $ kubectl logs deployment/nova-api-osapi -n | grep "POST /v2.1" INFO nova.osapi_compute.wsgi.server [None [REQ_ID] [USER_ID] [TENANT_ID] - - default default] [IP] "POST /v2.1/[tenant_id]/servers/[VM_UUID]/os-volume_attachments HTTP/1.1" status: 202 len: [.] time: [.] ``` {% endcode %} ### Step 2: Connection Initialization (Cinder) Nova Compute asks Cinder to "initialize the connection." Cinder communicates with the storage backend to allow the compute node to access the volume (and returns these connection details to Nova. {% code title="Sample Log (cinder-api.logs):" %} ```ruby $ kubectl logs deployment/cinder-api -n | grep INFO cinder.api.v3.attachments [None [REQ_ID] [USER_ID] [TENANT_ID] - - default default] Create volume attachment for volume [VOLUME_UUID] ``` {% endcode %} ### Step 3: Host-Level Storage Mapping The `pf9-ostackhost` service on the target compute node receives the connection details. Using the `os-brick` library, the compute node discovers the physical volume (e.g., logging into the iSCSI target and scanning for new SCSI devices). {% code title="Sample Log (ostackhost.log):" %} ```ruby $ cat /var/log/pf9/ostackhost.log | grep INFO nova.virt.libvirt.volume.iscsi [REQ_ID] Connecting to iSCSI target [TARGET_IQN] INFO os_brick.initiator.connectors.iscsi [REQ_ID] Found iSCSI device at /dev/disk/by-path/ip-[IP]-iscsi-[IQN]-lun-[ID] ``` {% endcode %} ### Step 4: Hypervisor Hot-Plug & Status Update Nova generates the XML for the disk and calls Libvirt to dynamically hot-plug the block device to the running QEMU process. Once confirmed, Nova tells Cinder to finalize the attachment, moving the volume status to `in-use`. {% code title="Sample Log (ostackhost.log):" %} ```ruby $ cat /var/log/pf9/ostackhost.log | grep INFO nova.compute.manager [REQ_ID] Attaching volume [VOLUME_UUID] to instance [VM_UUID] at /dev/vdX INFO nova.virt.libvirt.guest [REQ_ID] Successfully attached device to instance [VM_UUID] ``` {% endcode %} ### Procedure: Diagnostic Commands ### 1. Get the Volume and Attachment Status Check if the volume is stuck in a transitional state or if it threw a specific fault. ```shellscript $ openstack volume show $ openstack server show ``` ### 2. Validate Compute and Volume Service Health Since attach/detach operations depend on the compute node physically mapping the storage, ensure that the compute service hosting the VM is healthy and that the block storage service running on the storage node is also healthy. ```shellscript $ openstack compute service list $ openstack volume service list ``` ### 3. Review API Rejections (Management Plane) Check if Nova or Cinder API rejected the request before it reached the compute node. ```shellscript $ kubectl get pods -o wide -n | grep -E "nova|cinder" $ kubectl logs -n | grep $ kubectl logs -n | grep ``` ### 4. Check the Hypervisor (Libvirt) State Verify if the hypervisor actually sees the disk attached to the VM at the system level. ```shellscript # On the Compute Node hosting the VM: $ sudo virsh domblklist ``` ### 5. Validate OS-Level Block Devices If using iSCSI or NFS, verify that the compute node's operating system successfully discovered the disk. ```shellscript # On the Compute Node hosting the VM: $ lsblk $ multipath -ll $ iscsiadm -m session ``` ### 6. Check the Compute Node Logs The vast majority of attach/detach failures happen during the `os-brick` mapping phase on the compute host. ```shellscript # On the Compute Node hosting the VM: $ grep /var/log/pf9/ostackhost.log $ grep /var/log/libvirt/libvirtd.log ``` ### 7. Resetting a Stuck Volume State (Admin Only) If a process completely times out and the volume is permanently stuck in `attaching` or `detaching` despite no active tasks running, you may need to manually reset the state. ```shellscript # To force a volume back to available (if attach failed): $ openstack volume set --state available # To force a volume back to in-use (if detach failed): $ openstack volume set --state in-use ``` ### Most Common Causes * Guest OS Holding the Disk (Detach Failure): The most common detach issue. The virtual machine's operating system is actively reading/writing to the disk (or it is mounted in `/etc/fstab`). Libvirt will refuse to hot-unplug the disk to prevent data corruption. * Stale iSCSI Sessions / Multipath Issues: The compute node has leftover, broken iSCSI sessions from previous operations, causing `os-brick` to hang or map the wrong device path during the attachment phase. * Storage Network Connectivity: The compute node lacks the network routing required to reach the storage data network (e.g., the iSCSI target IPs or the Ceph public network). * Libvirt/QEMU Timeout: The storage backend responds too slowly, causing the hypervisor's hot-plug operation to time out, leaving the volume locked in an `attaching` state. * Missing Packages: The compute node is missing required backend tools (like `sysfsutils`, `multipath-tools`, `open-iscsi`, or `ceph-common`), preventing it from successfully mapping the volume to the host. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd-ts/storage/troubleshooting-volume-attachment-and-detachment-issues.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.