> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/generic/virsh-commands-run-infinitely-without-output-vm-console-inaccessible.md). # Virsh Commands run Infinitely Without Output / VM Console Inaccessible ## Problem * VM console inaccessible for VMs running on specific host. * `virsh` commands on hypervisor run infinitely without an output. ## Environment * Private Cloud Director Virtualization - v2025.10 and Higher * Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher * Component: `libvirtd` running on hypervisors, SCSI Storage Backend. ## Cause * It has been observed in such cases that one or more stale multipath devices remain on the host for the VM/s that have been already deleted. * These stale multipath/s cause the `libvirtd` service to enter in an endless I/O loop while communicating with these failed multipath/s devices. * This issue is already reported with ID: **PCD-7407** to better handle such situation in PCD. ## Diagnostics * Check if the `virsh` commands are infinitely running without any output. * Check if the console of the VMs running on the specific/affected host is inaccessible with execution stuck infinitely. * Identify the failing QEMU `libvirt` processes using the command below: {% code title="Affected Host - Sample output " %} ```bash $ sudo ps -aux | grep libvirt | grep /usr/bin/qemu-system-x86_64 libvirt+ 239349 65.8 12.7 140438780 134239804 ? SLl [TIMESTAMP] 15079:01 /usr/bin/qemu-system-x86_64 -name guest=[VM_UUID],debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-72-[VM_UUID]/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.ms.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/[VM_UUID]_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-6.2,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram -accel kvm -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,erms=off,fsrm=off -global driver=cfi.pflash01,property=secure,value=on -m 131072 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":137438953472} -overcommit mem-lock=off -smp 16,sockets=16,dies=1,cores=1,threads=1 -uuid [VM_UUID] -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=5.6.0,serial=[VM_UUID],uuid=[VM_UUID],family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=39,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_ticket_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0 -device pcie-root-port,port=19,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x6 -device piix3-usb-uhci,id=usb,bus=pci.4,addr=0x1 -device virtio-scsi-pci,id=scsi0,bus=pci.3,addr=0x0 -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=[VM_UUID],drive=libvirt-2-format,id=ua-[VM_UUID],bootindex=1,write-cache=off,serial=[VM_UUID] -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=[VM_UUID],drive=libvirt-1-format,id=ua-[VM_UUID],write-cache=off,serial=[VM_UUID] -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=44 -device virtio-net-pci,host_mtu=1500,netdev=hostnet0,id=net0,mac=[VM_UUID],bus=pci.2,addr=0x0 -add-fd set=3,fd=40 -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc 0.0.0.0:0,audiodev=audio1 -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/[VM_UUID],display=off,bus=pci.5,addr=0x0 -device virtio-balloon-pci,id=balloon0,free-page-reporting=on,bus=pci.6,addr=0x0 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on libvirt+ 239345 61.8 15.8 140438783 134239808 ? SLl [TIMESTAMP] 15079:03 /usr/bin/qemu-system-x86_64 -name guest=[VM_UUID],debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-72-[VM_UUID]/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.ms.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/[VM_UUID]_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-6.2,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram -accel kvm -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,erms=off,fsrm=off -global driver=cfi.pflash01,property=secure,value=on -m 131072 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":137438953472} -overcommit mem-lock=off -smp 16,sockets=16,dies=1,cores=1,threads=1 -uuid [VM_UUID] -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=5.6.0,serial=[VM_UUID],uuid=[VM_UUID],family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=39,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_ticket_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0 -device pcie-root-port,port=19,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x6 -device piix3-usb-uhci,id=usb,bus=pci.4,addr=0x1 -device virtio-scsi-pci,id=scsi0,bus=pci.3,addr=0x0 -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=[VM_UUID],drive=libvirt-2-format,id=ua-[VM_UUID],bootindex=1,write-cache=off,serial=[VM_UUID] -blockdev {"driver":"file","filename":"/opt/pf9/data/state/mnt/[VM_UUID]/volume-[VM_UUID]","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=[VM_UUID],drive=libvirt-1-format,id=ua-[VM_UUID],write-cache=off,serial=[VM_UUID] -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=44 -device virtio-net-pci,host_mtu=1500,netdev=hostnet0,id=net0,mac=[VM_UUID],bus=pci.2,addr=0x0 -add-fd set=3,fd=40 -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc 0.0.0.0:0,audiodev=audio1 -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/[VM_UUID],display=off,bus=pci.5,addr=0x0 -device virtio-balloon-pci,id=balloon0,free-page-reporting=on,bus=pci.6,addr=0x0 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on ``` {% endcode %} * In this list, identify `guest=[VM_UUID]`. This UUID corresponds to the VM's UUID. * These processes are the actual VMs running on the host. * In this list, there is a high possibility, as observed in the such issues, that a process would exist for a VM that is already deleted. * That process will be a Zombie process `Z` and in `defunct` state. {% code title="Sample Output of Defunct Process" %} ```bash $ sudo ps -aux | grep libvirt | grep 1912566 libvirt+ 1912566 330 0.0 0 0 ? Zl [TIMESTAMP] 756698:25 [qemu-system-x86 LONG_CONTENT] ``` {% endcode %} * Identify `guest=[VM_UUID]` from this process. ## Resolution * Move the `.xml` file of this `VM_UUID` from `/etc/libvirt/qemu/.xml` to any other location like `/tmp` or user's home dir. * After moving the file, this defunct process should get killed automatically and that can be verified using the below command: {% code title="Returns Empty Output" %} ```bash $ sudo ps -aux | grep ``` {% endcode %} * Read the `.xml` of the VM UUID that was copied earlier and search for `` section. * In there, identify the line `` {% hint style="info" %} **Example:** The **disk mapper device name** will appear as `dm-11` \ `` {% endhint %} * Identify the WWN of that multipath using the command below: {% code title="Sample Output" %} ```bash $ multipath -ll | grep [WWN] [DISK_MAPPER_DEVICE_NAME] NETAPP, LUN C-Mode ``` {% endcode %} * Get full details of this multipath using: 
multipath -ll <WWN_UUID>
size=100G features='2 pg_init_retries 50' hwhandler='1 alua' wp=rw
-+- policy='service-time 0' prio=0 status=enabled
I '- 6:0:0:3 sda 8:112 failed faulty running
'-+- policy='service-time 0' prio=0 status=enabled
'- 7:0:0:3 sdb 8:128 failed faulty running
* Identify that there are `failed faulty` block devices for this multipath. `sda` and `sdb` are examples and the actual output will vary. * Delete the faulty block devices using the below command for each one: {% code title="" %} ```bash echo 1 | sudo tee /sys/block//device/delete ``` {% endcode %} * `libvirtd` service auto-recovers from here. ## Validation * Check if `virsh` commands are executing on the host. * Check if the VM Console is accessible. * On the affected host, check if the faulty multipath is flushed automatically and the below command should give empty response {% code title="Affected Host" %} ```bash $ multipath -ll ``` {% endcode %} * `systemctl status libvirtd` should display status as `Active` `Running`. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd/generic/virsh-commands-run-infinitely-without-output-vm-console-inaccessible.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.