VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service

Problem

Virtual machines configured with the hw:numa_nodes=1 flavor extra spec shut down unexpectedly without any user-initiated action. The Linux kernel OOM killer terminates the QEMU process running the VM when the assigned NUMA memory node is exhausted — even when the host has hundreds of gigabytes of available RAM on other NUMA nodes. In OpenStack, the affected VM transitions to an error or stopped state. The shutdown is visible in the Compute Host syslog as an OOM kill event with constraint=CONSTRAINT_MEMORY_POLICY.

Environment

• Private Cloud Director Virtualization — version v2026.4-89 and below.
• Self-Hosted Private Cloud Director Virtualization — version v2026.4-89 and below.
• Component: Compute Service

Cause

When a VM is created with hw:numa_nodes=1, the Compute Service generates a libvirt domain XML that pins the QEMU process to a specific NUMA memory node with a policy of strict. Under strict policy, the QEMU process is restricted exclusively to the assigned NUMA node and cannot access RAM on any other node — even when the host has ample memory available elsewhere. When the assigned node's memory is exhausted, the Linux kernel OOM killer terminates the QEMU process, shutting down the VM immediately.

The strict policy is hardcoded in the Compute Service libvirt driver (designer.py). There is currently no flavor extra spec, image property, or Compute Service configuration option to override this default behavior.

Diagnostics

The following steps confirm whether the VM was shut down by a NUMA-policy OOM kill and not by general host memory exhaustion or another cause. Run these steps on the Compute Host where the VM was running at the time of the unexpected shutdown.

1

Step 1 — Check the Compute Host Syslog for an OOM Kill with NUMA Policy Constraint

Search the syslog on the affected Compute Host for an OOM kill event. The keyword CONSTRAINT_MEMORY_POLICY confirms the kill was caused by the strict NUMA memory policy and not by general host memory exhaustion.

Compute Host

$ grep -i "CONSTRAINT_MEMORY_POLICY" /var/log/syslog

Compute Host - /var/log/syslog.

[TIMESTAMP] [HOSTNAME] kernel: oom-kill:constraint=CONSTRAINT_MEMORY_POLICY,nodemask=[NUMA_NODE],cpuset=vcpu[CPU_ID],mems_allowed=[NUMA_NODE],global_oom,task=qemu-system-x86,pid=[PID],uid=[UID]
[TIMESTAMP] [HOSTNAME] kernel: Out of memory: Killed process [PID] (qemu-system-x86) total-vm:[VM_SIZE]kB, anon-rss:[RSS_SIZE]kB, file-rss:[FILE_RSS]kB, shmem-rss:0kB

2

Step 2 — Confirm NUMA Node Memory State

Check the current memory state of each NUMA node on the affected Compute Host to confirm the assigned node was exhausted while other nodes had free memory available.

Compute Host

$ cat /sys/devices/system/node/node0/meminfo
$ cat /sys/devices/system/node/node1/meminfo

Sample Output

Node 1 MemTotal:       1056888716 kB
Node 1 MemFree:            529672 kB
Node 1 MemUsed:        1056359044 kB

A NUMA node with MemFree near zero while other nodes show substantial free memory confirms a NUMA-policy exhaustion event rather than host-level memory exhaustion.

3

Step 3 — Confirm the VM XML Has a Strict NUMA Memory Policy

Retrieve the libvirt domain XML for the affected VM and check the numatune section. The value mode='strict' confirms the VM was running with strict NUMA memory policy.

To find the libvirt domain name, run virsh list --all on the Compute Host and look for the instance-XXXXXXXX entry corresponding to the VM UUID.

Compute Host

$ virsh dumpxml <VM_LIBVIRT_NAME> | grep -A5 "numatune"

Sample Output

<numatune>
  <memory mode='strict' nodeset='1'/>
  <memnode cellid='0' mode='strict' nodeset='1'/>
</numatune>

4

Step 4 — Confirm the VM Flavor Has hw:numa_nodes Set

Identify the flavor used by the affected VM and confirm the hw:numa_nodes extra spec is present.

OpenStack CLI

$ openstack server show <VM_UUID> -c flavor

Sample Output

+--------+-------------------+
| Field  | Value             |
+--------+-------------------+
| flavor | [FLAVOR_NAME]     |
+--------+-------------------+

OpenStack CLI

$ openstack flavor show <FLAVOR_NAME> -c properties

Sample Output

+------------+-------------------+
| Field      | Value             |
+------------+-------------------+
| properties | hw:numa_nodes='1' |
+------------+-------------------+

If all checks above match — CONSTRAINT_MEMORY_POLICY in syslog, mode='strict' in the VM XML, and hw:numa_nodes set on the flavor — proceed to the Workaround below.

Resolution

A permanent fix is tracked in PCD-8069. The fix introduces a new flavor extra spec hw:numa_mempolicy that allows operators to configure the NUMA memory policy (preferred, strict, or interleave) at the flavor level. And the corresponding changes will pushed in to the files /opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/designer.py and /opt/pf9/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py .

Once delivered, the Compute Service will read this extra spec when generating the libvirt domain XML instead of defaulting to strict.

Until the fix is released, use the Workaround below.

Workaround

The suggested workaround provides immediate relief with no code changes but is not persistent across Nova-level operations.

The workaround suggested require a brief VM shutdown and restart. There is no zero-downtime path to change the NUMA memory policy of a running VM. OpenStack server reboot (hard or soft), server stop/start, live migration, and cold migration were tested — none update the NUMA memory policy without the code fix in place. Coordinate with the VM owner before proceeding.

Edit the VM XML Directly (Immediate, Not Persistent)

This method changes the NUMA memory mode for a specific VM by editing the libvirt domain XML directly on the Compute Host. No changes to Compute Host software files are required.

Persistence limitations: The change is overwritten by the Compute Service whenever any of the following OpenStack-level operations regenerate the domain XML — openstack server stop/start, openstack server reboot (hard or soft), openstack server rebuild, and cold migration. The change persists across pf9-ostackhost service restart and live migration.

Apply these steps to critical VMs as an immediate measure while planning for a persistent resolution in the patch release.

1

Step 1 — Edit the VM XML

On the Compute Host where the VM is running, open the VM's libvirt XML for editing.

Compute Host

$ virsh edit <VM_LIBVIRT_NAME>

In the numatune section, change every occurrence of mode='strict' to mode='preferred':

<!-- Before -->
<numatune>
  <memory mode='strict' nodeset='<NUMA_NODE_ID>'/>
  <memnode cellid='0' mode='strict' nodeset='<NUMA_NODE_ID>'/>
</numatune>

<!-- After -->
<numatune>
  <memory mode='preferred' nodeset='<NUMA_NODE_ID>'/>
  <memnode cellid='0' mode='preferred' nodeset='<NUMA_NODE_ID>'/>
</numatune>

Save and exit the editor. The virsh edit command validates the XML before saving — if an error appears, re-check the syntax and save again.

2

Step 2 — Shut Down the VM

Shut down the VM. A restart is required for the updated memory mode to take effect.

Compute Host

$ virsh shutdown <VM_LIBVIRT_NAME>

Wait for the VM to reach the shut-off state before proceeding.

Compute Host

$ virsh domstate <VM_LIBVIRT_NAME>

Sample Output

shut off

3

Step 3 — Start the VM

Start the VM to apply the updated domain XML.

Compute Host

$ virsh start <VM_LIBVIRT_NAME>

4

Step 4 — Verify the Memory Policy Is Now Preferred

Confirm the numatune section in the running VM's XML shows mode='preferred'.

Compute Host

$ virsh dumpxml <VM_LIBVIRT_NAME> | grep -A5 "numatune"

Sample Output

<numatune>
  <memory mode='preferred' nodeset='<NUMA_NODE_ID>'/>
  <memnode cellid='0' mode='preferred' nodeset='<NUMA_NODE_ID>'/>
</numatune>

Verify the VM is running and accessible. The QEMU process now has access to memory on all NUMA nodes when the assigned node is under pressure.

Reapplication required after Nova-level operations: The XML change made in the workaround is overwritten whenever OpenStack regenerates the domain XML. If openstack server stop/start, openstack server reboot, openstack server rebuild, or cold migration is performed on the VM, re-apply workaround until the permanent fix is available.

Additional Information

This issue is tracked in PCD-8069. The engineering fix introduces a new hw:numa_mempolicy flavor extra spec that allows the NUMA memory policy to be configured at the flavor level. The fix modifies designer.py and driver.py in the Compute Service libvirt driver so the policy is read dynamically from the flavor instead of always defaulting to strict. The value is stored in nova.instance_extra (InstanceNUMATopology) so it persists across reboots and live migrations once the fix is applied and the VM is resized to the updated flavor.

The version having the fix is not finalised yet. For more details on the ETA of the fix release, please contact Platform9 Support team.