DRR Service Does Not Trigger Live Migration for VMs With NUMA Node Memory Exhaustion
Problem
Virtual machines pinned to a specific NUMA node via the hw:numa_nodes=1 flavor extra spec are not automatically live-migrated by the DRR service when that NUMA node's memory is exhausted. The DRR service evaluates total available memory across the host when identifying candidates for live migration — it does not evaluate per-NUMA-node memory. When a single NUMA node is exhausted but the host has free memory available on other nodes, the DRR service considers the host within acceptable thresholds and does not trigger a migration. The affected VMs remain on the overloaded NUMA node and are vulnerable to kernel OOM kills that shut them down unexpectedly.
The same gap exists at scheduling time: the Placement service tracks host-level memory inventory and does not account for per-NUMA-node memory availability. New VMs may be scheduled to a host where the target NUMA node is already under pressure.
Environment
Private Cloud Director Virtualization — Version v2026.4-89 and below.
Self-Hosted Private Cloud Director Virtualization — Version v2026.4-89 and below.
Component: Watcher - DRR service
Cause
The DRR service and the Placement service do not model per-NUMA-node memory as a distinct resource dimension.
DRR service: When evaluating whether to live-migrate a VM, the DRR service calculates host-level memory utilization — the sum of memory used across all NUMA nodes divided by total host memory. When a single NUMA node is fully exhausted but the host has substantial free memory on other nodes, the host-level utilization ratio remains within configured thresholds and no live migration is triggered. The VM stays on the exhausted NUMA node.
Placement service: When scheduling new VMs, the Placement service checks whether the host has sufficient total memory to fulfill the flavor request. Per-NUMA-node memory availability is not tracked as a placement resource. The Placement service may return a host as a valid scheduling candidate even when the NUMA node the VM will be pinned to is already heavily loaded.
The combined effect is that VMs with hw:numa_nodes=1 that land on an overloaded NUMA node receive no automated relief — neither from the scheduler at creation time nor from the DRR service during operation.
Diagnostics
The following steps confirm the DRR service is running but not triggering live migration due to the NUMA memory evaluation gap — as opposed to the DRR service being down or misconfigured.
Step 1 — Confirm NUMA Node Is Exhausted While Total Host Memory Is Healthy
On the affected Compute Host, check memory availability per NUMA node and compare with total host memory. A scenario where one NUMA node shows near-zero free memory while overall host memory is healthy confirms the DRR evaluation gap.
If one NUMA node shows near-zero MemFree while free -h reports substantial total free memory, the DRR service would not identify this host as a rebalancing candidate — confirming this issue.
Step 2 — Verify the DRR Service Is Running
Confirm all DRR service pods are in a healthy running state. If the DRR service is down, the absence of live migration is caused by a service failure rather than this NUMA evaluation gap.
Self-Hosted only: This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on your behalf.
All three pods — watcher-api, watcher-decision-engine, and watcher-applier — must be in Running 2/2 state for the DRR service to be operational. If any pod is not running, contact Platform9 Support to investigate the DRR service failure separately.
Step 3 — Confirm No Live Migration Was Triggered for the Affected VM
Check the migration history for the affected VM to confirm the DRR service did not trigger a live migration before the OOM kill occurred.
An empty migration history for a VM that was OOM-killed on a NUMA-exhausted host — while the host had overall free memory — confirms the DRR service did not intervene.
If all checks above match — one NUMA node near-zero while total host memory is healthy, DRR service pods are running, and no live migration was triggered — proceed to the Workaround below.
Resolution
A fix for PCD-8111 is targeted for release in version 2026.AUG. After upgrading, the DRR service will account for per-NUMA-node memory when identifying rebalancing candidates, and the Placement service will factor NUMA node memory availability into scheduling decisions.
Until the fix is available, use the Workaround below.
Workaround
The DRR service cannot be configured to evaluate per-NUMA-node memory in affected versions. Two manual mitigations are available. Apply both methods together for the most effective interim protection.
Method 1 — Manually Monitor NUMA Node Memory and Live-Migrate VMs Proactively
Identify Compute Hosts where a NUMA node is approaching exhaustion and manually live-migrate VMs off that host before the OOM killer triggers. This directly addresses the DRR gap and moves VMs to hosts with healthier NUMA node memory.
Step 2 — Identify VMs Pinned to the Exhausted NUMA Node
List VMs running on the affected Compute Host, then confirm which ones are pinned to the exhausted node via their libvirt XML.
For each VM, check the NUMA node assignment.
VMs with nodeset='<EXHAUSTED_NUMA_NODE_ID>' are the live-migration candidates.
Step 3 — Live-Migrate Candidate VMs to a Less-Loaded Host
Initiate a live migration for each candidate VM. The Compute Service scheduler selects a destination host. The VM remains running throughout the migration.
Monitor migration status until the VM returns to ACTIVE on a new host.
Confirm the VM is ACTIVE on a new Compute Host. Verify the destination host has healthy NUMA node memory by repeating Step 1 on that host.
Method 2 — Apply the NUMA Memory Policy Fix to Prevent OOM Kills
Changing the NUMA memory policy from strict to preferred ensures VMs can access memory on other NUMA nodes when their assigned node is exhausted — preventing the kernel OOM killer from terminating them. This mitigates the consequence of the DRR gap without resolving the underlying evaluation limitation.
Method 2 protects VMs from OOM kills but does not cause the DRR service to rebalance overloaded hosts. Apply Method 1 alongside Method 2 for the most complete interim mitigation.
For full instructions, see VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service.
Additional Information
This issue is tracked in PCD-8111. The engineering fix updates the DRR service to evaluate per-NUMA-node memory availability as a distinct resource dimension when selecting live-migration candidates, and updates the Placement service to factor NUMA node memory availability into scheduling decisions for VMs using NUMA pinning.
Related Articles:
VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service — root cause of VMs being OOM-killed when a NUMA node is exhausted; workaround steps to change the NUMA memory policy from strict to preferred
Last updated