VM Creation Fails with Network Allocation Timeout

Problem

Virtual machine creation fails consistently on a freshly deployed or re-deployed PCD region. VMs enter ERROR state after approximately 300 seconds and never reach ACTIVE. The fault message reports that network allocation failed.

Symptoms observed:

• openstack server show reports vm_state: error with fault message Failed to allocate the network(s), not rescheduling
• All VM builds on the affected region fail — the issue is not isolated to specific flavors, images, or networks
• The compute host Hostagent log shows a 300-second timeout waiting for a network-vif-plugged event that never arrives

Environment

• Private Cloud Director Virtualization - v2026-4.89 and later
• Self-Hosted Private Cloud Director Virtualization - v2026-4.89 and later
• Component: Networking Service, Compute Service

Cause

The Networking Service uses a dedicated service credential to notify the Compute Service when a VM's network port has been successfully bound (the network-vif-plugged event). These credentials are stored in neutron.conf under the [nova] section.

During a fresh region installation, a race condition can cause the Networking Service configuration to be written before the Compute Service has persisted its service password to the configuration store. The result is neutron.conf with password = <no value> in the [nova] section, as shown below:

[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
auth_version = v3
endpoint_type = internal
password = <no value>
project_domain_name = default
project_name = services
region_name = [REGION_NAME]
user_domain_name = default
username = nova

Every attempt by the Networking Service to notify the Compute Service of a port-bind event fails with an HTTP 401 Unauthorized from the Identity Service. The Compute Service waits 300 seconds for the network-vif-plugged notification that never arrives, then aborts the build.

This issue is tracked under PCD-7657.

Diagnostics

1

Step 1 — Confirm the VM fault message

Retrieve the fault from the failed VM. The fault message confirms the network allocation failure.

OpenStack CLI

$ openstack server show <VM_UUID> -c fault -c status -c "OS-EXT-STS:vm_state"

Sample Output

+---------------------------+----------------------------------------------------------------------+
| Field                     | Value                                                                |
+---------------------------+----------------------------------------------------------------------+
| OS-EXT-STS:vm_state       | error                                                                |
| fault                     | {'code': 500, 'message': 'Build of instance [VM_UUID] aborted:      |
|                           | Failed to allocate the network(s), not rescheduling.'}              |
| status                    | ERROR                                                                |
+---------------------------+----------------------------------------------------------------------+

2

Step 2 — Check the compute host Hostagent log for the VIF plug timeout

On the compute host, search the Hostagent log for the affected VM UUID to confirm the timeout pattern.

Compute Host

$ sudo grep -i "network-vif-plugged\|VirtualInterfaceCreateException\|BuildAbortException" \
    /var/log/pf9/ostackhost.log | grep "<VM_UUID>"

Sample Output

[TIMESTAMP] WARNING nova.compute.manager [REQ_UUID] [VM_UUID] Timeout waiting for ['network-vif-plugged-[PORT_UUID]'] for instance with vm_state building and task_state spawning. Event states are: network-vif-plugged-[PORT_UUID]: timed out after 300.00 seconds
[TIMESTAMP] ERROR nova.compute.manager [REQ_UUID] [VM_UUID] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed

3

Step 3 — Confirm the Networking Service is failing Keystone authentication

Check the Networking Service pod logs for HTTP 401 errors on Compute Service notification calls. This confirms the root cause.

Self-Hosted only: These steps require kubectl access to the control plane namespace. SaaS customers cannot perform this step — contact Platform9 Support to have this confirmed and resolved on behalf of the environment.

Control Plane Node

$ kubectl logs -n <REGION_NS> \
    $(kubectl get po -n <REGION_NS> -l application=neutron -o jsonpath="{.items[0].metadata.name}") \
    | grep -i "Unauthorized\|401\|nova"

Sample Output

[TIMESTAMP] ERROR neutron.notifiers.nova keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: [REQ_UUID])

4

Step 4 — Confirm the missing Nova password in neutron.conf

Inspect the neutron-etc Secret to confirm password = <no value> in the [nova] section.

Self-Hosted only: This step requires kubectl access to the control plane namespace. SaaS customers cannot perform this step — contact Platform9 Support to have this checked on behalf of the environment.

Control Plane Node

$ kubectl get secret neutron-etc -n <REGION_NS> \
    -o go-template='{{ index .data "neutron.conf" }}' \
    | base64 -d \
    | grep -A 10 "\[nova\]"

Sample Output — confirms the issue

[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
password = <no value>
username = nova

password = <no value> confirms the race condition occurred during region deployment. Proceed to the Workaround.

Workaround

The fix requires injecting the correct Nova service password into the neutron-etc Secret and restarting the Networking Service pods. The correct password value is retrieved from the Compute Service configuration in the same region.

SaaS customers: All steps below require kubectl access to the control plane namespace. Contact Platform9 Support to have this workaround applied on behalf of the environment.

1

Step 1 — Back up the current neutron-etc Secret and deployment

Before making any changes, back up both the Secret and the Networking Service deployment.

Control Plane Node

$ export NS="<REGION_NS>"
$ kubectl get secret neutron-etc -n $NS -o yaml > neutron-etc-backup.yaml
$ kubectl get deployment neutron-server -n $NS -o yaml > neutron-server-deploy-backup.yaml

Sample Output

(no output — backup files written to current directory)

2

Step 2 — Retrieve the correct Nova service password

Get the Compute Service password from the Compute Service configuration in the same namespace.

Control Plane Node

$ kubectl get secret nova-etc -n $NS \
    -o go-template='{{ index .data "nova.conf" }}' \
    | base64 -d \
    | grep -A10 "\[keystone_authtoken\]" \
    | grep "^password"

Sample Output

password = [NOVA_SERVICE_PASSWORD]

Note the password value. This is the shared service credential the Networking Service must use to authenticate to the Identity Service as the nova service account.

3

Step 3 — Extract and edit neutron.conf

Decode neutron.conf from the Secret into a temporary file and set the correct password.

Control Plane Node

$ kubectl get secret neutron-etc -n $NS \
    -o go-template='{{ index .data "neutron.conf" }}' \
    | base64 -d > /tmp/neutron.conf.tmp
$ vi /tmp/neutron.conf.tmp

In the editor, locate the [nova] section and replace password = <no value> with the password retrieved in Step 2:

[nova]
password = <NOVA_SERVICE_PASSWORD>

Save and exit the editor.

4

Step 4 — Patch the neutron-etc Secret with the corrected configuration

Re-encode the updated neutron.conf and patch the Secret in the cluster.

Control Plane Node

$ NEW_CONF=$(cat /tmp/neutron.conf.tmp | base64 | tr -d '\n')
$ kubectl patch secret neutron-etc -n $NS \
    -p="{\"data\":{\"neutron.conf\": \"$NEW_CONF\"}}"

Sample Output

secret/neutron-etc patched

5

Step 5 — Restart the Networking Service pods

Delete the existing Networking Service pods to trigger a rolling restart with the corrected configuration.

Restarting Networking Service pods briefly interrupts network event processing. Avoid doing this while VMs are being actively provisioned on the affected region.

Control Plane Node

$ kubectl delete pods -n $NS -l application=neutron

Sample Output

pod "neutron-server-[POD_ID_1]" deleted
pod "neutron-server-[POD_ID_2]" deleted

Wait for the new pods to reach Running state:

Control Plane Node

$ kubectl get pods -n $NS -l application=neutron -w

Sample Output

NAME                              READY   STATUS    RESTARTS   AGE
neutron-server-[NEW_POD_ID_1]    1/1     Running   0          30s
neutron-server-[NEW_POD_ID_2]    1/1     Running   0          28s

6

Step 6 — Verify VM creation succeeds

Attempt to provision a new VM on the affected region and confirm it reaches ACTIVE state.

OpenStack CLI

$ openstack server show <VM_UUID> | grep status

Sample Output

| status | ACTIVE |

Confirm the Networking Service logs no longer show HTTP 401 errors on Compute Service notification calls.

Resolution

A permanent fix for the race condition that causes this issue is planned to be included in the upcoming 2026.AUG release. After upgrading to that release, the Networking Service configuration will always contain the correct Compute Service password regardless of the deployment order of dependent services.

The fix for PCD-7657 ensures that the Compute Service password is pre-seeded in the Networking Service configuration store before either chart is deployed, eliminating the ordering dependency.

Additional Information

Related articles:

• VM Creation Failed Due to Port Binding Failure — covers a similar symptom (VM creation failure from Networking Service issues) where the root cause is an OVN controller problem rather than a missing service credential. Review this article if the [nova] section in neutron.conf shows a valid password but VM creation still fails.