VM Creation Fails Due to AIO (Asynchronous I/O) limit on Hypervisor

Problem

VM creation failed due to the following error on the hypervisor:

ostackhost.log
Copy

Environment

  • Private Cloud Director Virtualization - v2025.4 and Higher
  • Private Cloud Director Kubernetes – v2025.4 and Higher
  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
  • Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher

Cause

The asynchronous I/O (AIO) limit on the hypervisor is set too low, preventing libvirt/QEMU from allocating sufficient I/O handles for new VM disk operations.

Diagnostics

  1. Check current AIO limit:
command
Copy
  1. Review libvirt logs:
command
Copy

Resolution

Increase the AIO limit:

  • To increase the limit one time (this change will not persist a node reboot):
command
Copy
  • To increase the limit permanently (this change will persist a node reboot):
command
Copy

Validation

  • Confirm updated value:
command
Copy
  • Re-attempt VM creation and verify it is active and Running.
  • If these steps prove insufficient to resolve the issue, reach out to the Platform9 Support Team for additional assistance.

Additional Information

  • The AIO (Asynchronous I/O) limit is controlled by the Linux kernel parameter fs.aio-max-nr, defines the maximum number of asynchronous I/O requests that can exist system-wide at any given time.
  • If the current number of I/O handles /proc/sys/fs/aio-nr comes too close to /proc/sys/fs/aio-max-nr their is high risk of encountering this failure. The parameter values can fluctuate based on several factors, including the number of LUNs, the number of VMs, whether multipath is enabled, and other relevant configurations. These variations directly impact the number of asynchronous I/O operations.
  • QEMU/libvirt uses Linux AIO to perform disk I/O operations efficiently for VMs.
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard