Ctrlk

Virsh Commands run Infinitely Without Output / VM Console Inaccessible

Problem

  • VM console inaccessible for VMs running on specific host.

  • virsh commands on hypervisor run infinitely without an output.

Environment

  • Private Cloud Director Virtualization - v2025.10 and Higher

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

  • Component: libvirtd running on hypervisors, SCSI Storage Backend.

Cause

  • It has been observed in such cases that one or more stale multipath devices remain on the host for the VM/s that have been already deleted.

  • These stale multipath/s cause the libvirtd service to enter in an endless I/O loop while communicating with these failed multipath/s devices.

  • This issue is already reported with ID: PCD-7407 to better handle such situation in PCD.

Diagnostics

  • Check if the virsh commands are infinitely running without any output.

  • Check if the console of the VMs running on the specific/affected host is inaccessible with execution stuck infinitely.

  • Identify the failing QEMU libvirt processes using the command below:

  • In this list, identify guest=[VM_UUID]. This UUID corresponds to the VM's UUID.

  • These processes are the actual VMs running on the host.

  • In this list, there is a high possibility, as observed in the such issues, that a process would exist for a VM that is already deleted.

  • That process will be a Zombie process Z and in defunct state.

  • Identify guest=[VM_UUID] from this process.

Resolution

  • Move the .xml file of this VM_UUID from /etc/libvirt/qemu/<VM_UUID>.xml to any other location like /tmp or user's home dir.

  • After moving the file, this defunct process should get killed automatically and that can be verified using the below command:

  • Read the .xml of the VM UUID that was copied earlier and search for <disk> section.

  • In there, identify the line <source dev='/dev/dm-[NUMBER]'>

Example: The disk mapper device name will appear as dm-11 <source dev='/dev/dm-11'>

  • Identify the WWN of that multipath using the command below:

  • Get full details of this multipath using:

  • Identify that there are failed faulty block devices for this multipath. sda and sdb are examples and the actual output will vary.

  • Delete the faulty block devices using the below command for each one:

  • libvirtd service auto-recovers from here.

Validation

  • Check if virsh commands are executing on the host.

  • Check if the VM Console is accessible.

  • On the affected host, check if the faulty multipath is flushed automatically and the below command should give empty response

  • systemctl status libvirtd should display status as Active Running.

PreviousUnable to Create Virtualized ClusterNextHow To Manually Apply Roles On a Host Using API Calls

Last updated