> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/generic/drr-service-does-not-trigger-live-migration-for-vms-with-numa-node-memory-exhaustion.md). # DRR Service Does Not Trigger Live Migration for VMs With NUMA Node Memory Exhaustion ## Problem Virtual machines pinned to a specific NUMA node via the `hw:numa_nodes=1` flavor extra spec are not automatically live-migrated by the DRR service when that NUMA node's memory is exhausted. The DRR service evaluates total available memory across the host when identifying candidates for live migration — it does not evaluate per-NUMA-node memory. When a single NUMA node is exhausted but the host has free memory available on other nodes, the DRR service considers the host within acceptable thresholds and does not trigger a migration. The affected VMs remain on the overloaded NUMA node and are vulnerable to kernel OOM kills that shut them down unexpectedly. The same gap exists at scheduling time: the Placement service tracks host-level memory inventory and does not account for per-NUMA-node memory availability. New VMs may be scheduled to a host where the target NUMA node is already under pressure. ## Environment * Private Cloud Director Virtualization — Version v2026.4-89 and below. * Self-Hosted Private Cloud Director Virtualization — Version v2026.4-89 and below. * Component: Watcher - DRR service ## Cause The DRR service and the Placement service do not model per-NUMA-node memory as a distinct resource dimension. **DRR service:** When evaluating whether to live-migrate a VM, the DRR service calculates host-level memory utilization — the sum of memory used across all NUMA nodes divided by total host memory. When a single NUMA node is fully exhausted but the host has substantial free memory on other nodes, the host-level utilization ratio remains within configured thresholds and no live migration is triggered. The VM stays on the exhausted NUMA node. **Placement service:** When scheduling new VMs, the Placement service checks whether the host has sufficient total memory to fulfill the flavor request. Per-NUMA-node memory availability is not tracked as a placement resource. The Placement service may return a host as a valid scheduling candidate even when the NUMA node the VM will be pinned to is already heavily loaded. The combined effect is that VMs with `hw:numa_nodes=1` that land on an overloaded NUMA node receive no automated relief — neither from the scheduler at creation time nor from the DRR service during operation. ## Diagnostics The following steps confirm the DRR service is running but not triggering live migration due to the NUMA memory evaluation gap — as opposed to the DRR service being down or misconfigured. {% stepper %} {% step %} #### Step 1 — Confirm NUMA Node Is Exhausted While Total Host Memory Is Healthy On the affected Compute Host, check memory availability per NUMA node and compare with total host memory. A scenario where one NUMA node shows near-zero free memory while overall host memory is healthy confirms the DRR evaluation gap. {% code title="Compute Host" overflow="wrap" %} ```bash $ for node in /sys/devices/system/node/node*/meminfo; do echo "=== $node ==="; grep -E "MemTotal|MemFree" "$node"; done ``` {% endcode %} {% code title="Sample Output" %} ``` === /sys/devices/system/node/node0/meminfo === Node 0 MemTotal: 1056888716 kB Node 0 MemFree: 504399044 kB === /sys/devices/system/node/node1/meminfo === Node 1 MemTotal: 1056888716 kB Node 1 MemFree: 529672 kB ``` {% endcode %} {% code title="Compute Host" %} ```bash $ free -h ``` {% endcode %} {% code title="Sample Output" %} ``` total used free shared buff/cache available Mem: [TOTAL] [USED] [FREE] [SHARE] [CACHE] [AVAIL] ``` {% endcode %} If one NUMA node shows near-zero `MemFree` while `free -h` reports substantial total free memory, the DRR service would not identify this host as a rebalancing candidate — confirming this issue. {% endstep %} {% step %} #### Step 2 — Verify the DRR Service Is Running Confirm all DRR service pods are in a healthy running state. If the DRR service is down, the absence of live migration is caused by a service failure rather than this NUMA evaluation gap. {% hint style="info" %} **Self-Hosted only:** This step requires direct access to the control plane. SaaS customers cannot perform this step independently — contact [Platform9 Support](https://support.platform9.com/) to have this completed on your behalf. {% endhint %} {% code title="Control Plane Node" %} ```bash $ kubectl get pods -n | grep -i watcher ``` {% endcode %} {% code title="Sample Output" %} ``` watcher-api-[POD_ID] 2/2 Running 0 [TIMESTAMP] watcher-decision-engine-[POD_ID] 2/2 Running 0 [TIMESTAMP] watcher-applier-[POD_ID] 2/2 Running 0 [TIMESTAMP] ``` {% endcode %} All three pods — `watcher-api`, `watcher-decision-engine`, and `watcher-applier` — must be in `Running 2/2` state for the DRR service to be operational. If any pod is not running, contact Platform9 Support to investigate the DRR service failure separately. {% endstep %} {% step %} #### Step 3 — Confirm No Live Migration Was Triggered for the Affected VM Check the migration history for the affected VM to confirm the DRR service did not trigger a live migration before the OOM kill occurred. {% code title="OpenStack CLI" %} ```bash $ openstack server migration list --server ``` {% endcode %} {% code title="Sample Output" %} ``` +----+-------------+-----------+------+--------+ | Id | Source Node | Dest Node | Type | Status | +----+-------------+-----------+------+--------+ (no entries) +----+-------------+-----------+------+--------+ ``` {% endcode %} An empty migration history for a VM that was OOM-killed on a NUMA-exhausted host — while the host had overall free memory — confirms the DRR service did not intervene. {% endstep %} {% endstepper %} If all checks above match — one NUMA node near-zero while total host memory is healthy, DRR service pods are running, and no live migration was triggered — proceed to the [Workaround](#workaround) below. ## Resolution A fix for **PCD-8111** is targeted for release in version 2026.AUG. After upgrading, the DRR service will account for per-NUMA-node memory when identifying rebalancing candidates, and the Placement service will factor NUMA node memory availability into scheduling decisions. Until the fix is available, use the [Workaround](#workaround) below. ## Workaround The DRR service cannot be configured to evaluate per-NUMA-node memory in affected versions. Two manual mitigations are available. Apply both methods together for the most effective interim protection. ### Method 1 — Manually Monitor NUMA Node Memory and Live-Migrate VMs Proactively Identify Compute Hosts where a NUMA node is approaching exhaustion and manually live-migrate VMs off that host before the OOM killer triggers. This directly addresses the DRR gap and moves VMs to hosts with healthier NUMA node memory. {% stepper %} {% step %} #### Step 1 — Check NUMA Node Memory Across Affected Compute Hosts Run the following on each Compute Host to identify which NUMA node is under pressure. A `MemFree` value below 5% of `MemTotal` on any NUMA node signals risk of OOM kills. {% code title="Compute Host" %} ```bash $ for node in /sys/devices/system/node/node*/meminfo; do echo "=== $node ==="; grep -E "MemTotal|MemFree" "$node"; done ``` {% endcode %} {% code title="Sample Output" %} ``` === /sys/devices/system/node/node0/meminfo === Node 0 MemTotal: 1056888716 kB Node 0 MemFree: 504399044 kB === /sys/devices/system/node/node1/meminfo === Node 1 MemTotal: 1056888716 kB Node 1 MemFree: 529672 kB ``` {% endcode %} {% endstep %} {% step %} #### Step 2 — Identify VMs Pinned to the Exhausted NUMA Node List VMs running on the affected Compute Host, then confirm which ones are pinned to the exhausted node via their libvirt XML. {% code title="OpenStack CLI" %} ```bash $ openstack server list --host -c ID -c Name -c Status ``` {% endcode %} For each VM, check the NUMA node assignment. {% code title="Compute Host" %} ```bash $ virsh dumpxml | grep -A5 "numatune" ``` {% endcode %} {% code title="Sample Output" %} ```xml ``` {% endcode %} VMs with `nodeset=''` are the live-migration candidates. {% endstep %} {% step %} #### Step 3 — Live-Migrate Candidate VMs to a Less-Loaded Host Initiate a live migration for each candidate VM. The Compute Service scheduler selects a destination host. The VM remains running throughout the migration. {% code title="OpenStack CLI" %} ```bash $ openstack server migrate --live-migration ``` {% endcode %} Monitor migration status until the VM returns to `ACTIVE` on a new host. {% code title="OpenStack CLI" %} ```bash $ openstack server show -c status -c "OS-EXT-SRV-ATTR:host" ``` {% endcode %} {% code title="Sample Output" %} ``` +---------------------------+--------------------+ | Field | Value | +---------------------------+--------------------+ | status | ACTIVE | | OS-EXT-SRV-ATTR:host | [NEW_COMPUTE_HOST] | +---------------------------+--------------------+ ``` {% endcode %} Confirm the VM is `ACTIVE` on a new Compute Host. Verify the destination host has healthy NUMA node memory by repeating Step 1 on that host. {% endstep %} {% endstepper %} ### Method 2 — Apply the NUMA Memory Policy Fix to Prevent OOM Kills Changing the NUMA memory policy from `strict` to `preferred` ensures VMs can access memory on other NUMA nodes when their assigned node is exhausted — preventing the kernel OOM killer from terminating them. This mitigates the consequence of the DRR gap without resolving the underlying evaluation limitation. {% hint style="info" %} Method 2 protects VMs from OOM kills but does not cause the DRR service to rebalance overloaded hosts. Apply Method 1 alongside Method 2 for the most complete interim mitigation. {% endhint %} For full instructions, see [VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service](https://platform9.com/kb/pcd/compute/vms-shut-down-unexpectedly-due-to-strict-numa-memory-policy-in-compute-service). ## Additional Information This issue is tracked in **PCD-8111**. The engineering fix updates the DRR service to evaluate per-NUMA-node memory availability as a distinct resource dimension when selecting live-migration candidates, and updates the Placement service to factor NUMA node memory availability into scheduling decisions for VMs using NUMA pinning. **Related Articles:** * [VMs Shut Down Unexpectedly Due to Strict NUMA Memory Policy in Compute Service](https://platform9.com/kb/pcd/compute/vms-shut-down-unexpectedly-due-to-strict-numa-memory-policy-in-compute-service) — root cause of VMs being OOM-killed when a NUMA node is exhausted; workaround steps to change the NUMA memory policy from strict to preferred --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd/generic/drr-service-does-not-trigger-live-migration-for-vms-with-numa-node-memory-exhaustion.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.