> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/compute/vm-creation-fails-with-network-allocation-timeout.md).

# VM Creation Fails with Network Allocation Timeout

## Problem

Virtual machine creation fails consistently on a freshly deployed or re-deployed PCD region. VMs enter `ERROR` state after approximately 300 seconds and never reach `ACTIVE`. The fault message reports that network allocation failed.

Symptoms observed:

* `openstack server show` reports `vm_state: error` with fault message `Failed to allocate the network(s), not rescheduling`
* All VM builds on the affected region fail — the issue is not isolated to specific flavors, images, or networks
* The compute host Hostagent log shows a 300-second timeout waiting for a `network-vif-plugged` event that never arrives

## Environment

* Private Cloud Director Virtualization - **v2026-4.89 and later**
* Self-Hosted Private Cloud Director Virtualization - **v2026-4.89 and later**
* Component: Networking Service, Compute Service

## Cause

The Networking Service uses a dedicated service credential to notify the Compute Service when a VM's network port has been successfully bound (the `network-vif-plugged` event). These credentials are stored in `neutron.conf` under the `[nova]` section.

During a fresh region installation, a race condition can cause the Networking Service configuration to be written before the Compute Service has persisted its service password to the configuration store. The result is `neutron.conf` with `password = <no value>` in the `[nova]` section, as shown below:

<pre class="language-bash"><code class="lang-bash">[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
auth_version = v3
endpoint_type = internal
<strong>password = &#x3C;no value>
</strong>project_domain_name = default
project_name = services
region_name = [REGION_NAME]
user_domain_name = default
username = nova
</code></pre>

Every attempt by the Networking Service to notify the Compute Service of a port-bind event fails with an HTTP 401 Unauthorized from the Identity Service. The Compute Service waits 300 seconds for the `network-vif-plugged` notification that never arrives, then aborts the build.

This issue is tracked under **PCD-7657**.

## Diagnostics

{% stepper %}
{% step %}

#### Step 1 — Confirm the VM fault message

Retrieve the fault from the failed VM. The fault message confirms the network allocation failure.

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> -c fault -c status -c "OS-EXT-STS:vm_state"
```

{% endcode %}

{% code title="Sample Output" %}

```bash
+---------------------------+----------------------------------------------------------------------+
| Field                     | Value                                                                |
+---------------------------+----------------------------------------------------------------------+
| OS-EXT-STS:vm_state       | error                                                                |
| fault                     | {'code': 500, 'message': 'Build of instance [VM_UUID] aborted:      |
|                           | Failed to allocate the network(s), not rescheduling.'}              |
| status                    | ERROR                                                                |
+---------------------------+----------------------------------------------------------------------+
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 2 — Check the compute host Hostagent log for the VIF plug timeout

On the compute host, search the Hostagent log for the affected VM UUID to confirm the timeout pattern.

{% code title="Compute Host" %}

```bash
$ sudo grep -i "network-vif-plugged\|VirtualInterfaceCreateException\|BuildAbortException" \
    /var/log/pf9/ostackhost.log | grep "<VM_UUID>"
```

{% endcode %}

{% code title="Sample Output" %}

```bash
[TIMESTAMP] WARNING nova.compute.manager [REQ_UUID] [VM_UUID] Timeout waiting for ['network-vif-plugged-[PORT_UUID]'] for instance with vm_state building and task_state spawning. Event states are: network-vif-plugged-[PORT_UUID]: timed out after 300.00 seconds
[TIMESTAMP] ERROR nova.compute.manager [REQ_UUID] [VM_UUID] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 3 — Confirm the Networking Service is failing Keystone authentication

Check the Networking Service pod logs for HTTP 401 errors on Compute Service notification calls. This confirms the root cause.

{% hint style="info" %}
**Self-Hosted only:** These steps require `kubectl` access to the control plane namespace. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to have this confirmed and resolved on behalf of the environment.
{% endhint %}

{% code title="Control Plane Node" %}

```bash
$ kubectl logs -n <REGION_NS> \
    $(kubectl get po -n <REGION_NS> -l application=neutron -o jsonpath="{.items[0].metadata.name}") \
    | grep -i "Unauthorized\|401\|nova"
```

{% endcode %}

{% code title="Sample Output" %}

```bash
[TIMESTAMP] ERROR neutron.notifiers.nova keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: [REQ_UUID])
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 4 — Confirm the missing Nova password in neutron.conf

Inspect the `neutron-etc` Secret to confirm `password = <no value>` in the `[nova]` section.

{% hint style="info" %}
**Self-Hosted only:** This step requires `kubectl` access to the control plane namespace. SaaS customers cannot perform this step — contact [Platform9 Support](https://support.platform9.com/) to have this checked on behalf of the environment.
{% endhint %}

{% code title="Control Plane Node" %}

```bash
$ kubectl get secret neutron-etc -n <REGION_NS> \
    -o go-template='{{ index .data "neutron.conf" }}' \
    | base64 -d \
    | grep -A 10 "\[nova\]"
```

{% endcode %}

<pre class="language-bash" data-title="Sample Output — confirms the issue"><code class="lang-bash">[nova]
auth_type = password
auth_url = http://keystone.[REGION].svc.cluster.local:5000/keystone/v3
<strong>password = &#x3C;no value>
</strong>username = nova
</code></pre>

`password = <no value>` confirms the race condition occurred during region deployment. Proceed to the [Workaround](#workaround).
{% endstep %}
{% endstepper %}

## Workaround

The fix requires injecting the correct Nova service password into the `neutron-etc` Secret and restarting the Networking Service pods. The correct password value is retrieved from the Compute Service configuration in the same region.

{% hint style="info" %}
**SaaS customers:** All steps below require `kubectl` access to the control plane namespace. Contact [Platform9 Support](https://support.platform9.com/) to have this workaround applied on behalf of the environment.
{% endhint %}

{% stepper %}
{% step %}

#### Step 1 — Back up the current neutron-etc Secret and deployment

Before making any changes, back up both the Secret and the Networking Service deployment.

{% code title="Control Plane Node" %}

```bash
$ export NS="<REGION_NS>"
$ kubectl get secret neutron-etc -n $NS -o yaml > neutron-etc-backup.yaml
$ kubectl get deployment neutron-server -n $NS -o yaml > neutron-server-deploy-backup.yaml
```

{% endcode %}

{% code title="Sample Output" %}

```bash
(no output — backup files written to current directory)
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 2 — Retrieve the correct Nova service password

Get the Compute Service password from the Compute Service configuration in the same namespace.

{% code title="Control Plane Node" %}

```bash
$ kubectl get secret nova-etc -n $NS \
    -o go-template='{{ index .data "nova.conf" }}' \
    | base64 -d \
    | grep -A10 "\[keystone_authtoken\]" \
    | grep "^password"
```

{% endcode %}

{% code title="Sample Output" %}

```bash
password = [NOVA_SERVICE_PASSWORD]
```

{% endcode %}

Note the password value. This is the shared service credential the Networking Service must use to authenticate to the Identity Service as the `nova` service account.
{% endstep %}

{% step %}

#### Step 3 — Extract and edit neutron.conf

Decode `neutron.conf` from the Secret into a temporary file and set the correct password.

{% code title="Control Plane Node" %}

```bash
$ kubectl get secret neutron-etc -n $NS \
    -o go-template='{{ index .data "neutron.conf" }}' \
    | base64 -d > /tmp/neutron.conf.tmp
$ vi /tmp/neutron.conf.tmp
```

{% endcode %}

In the editor, locate the `[nova]` section and replace `password = <no value>` with the password retrieved in Step 2:

```
[nova]
password = <NOVA_SERVICE_PASSWORD>
```

Save and exit the editor.
{% endstep %}

{% step %}

#### Step 4 — Patch the neutron-etc Secret with the corrected configuration

Re-encode the updated `neutron.conf` and patch the Secret in the cluster.

{% code title="Control Plane Node" %}

```bash
$ NEW_CONF=$(cat /tmp/neutron.conf.tmp | base64 | tr -d '\n')
$ kubectl patch secret neutron-etc -n $NS \
    -p="{\"data\":{\"neutron.conf\": \"$NEW_CONF\"}}"
```

{% endcode %}

{% code title="Sample Output" %}

```bash
secret/neutron-etc patched
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 5 — Restart the Networking Service pods

Delete the existing Networking Service pods to trigger a rolling restart with the corrected configuration.

{% hint style="warning" %}
Restarting Networking Service pods briefly interrupts network event processing. Avoid doing this while VMs are being actively provisioned on the affected region.
{% endhint %}

{% code title="Control Plane Node" %}

```bash
$ kubectl delete pods -n $NS -l application=neutron
```

{% endcode %}

{% code title="Sample Output" %}

```bash
pod "neutron-server-[POD_ID_1]" deleted
pod "neutron-server-[POD_ID_2]" deleted
```

{% endcode %}

Wait for the new pods to reach `Running` state:

{% code title="Control Plane Node" %}

```bash
$ kubectl get pods -n $NS -l application=neutron -w
```

{% endcode %}

{% code title="Sample Output" %}

```bash
NAME                              READY   STATUS    RESTARTS   AGE
neutron-server-[NEW_POD_ID_1]    1/1     Running   0          30s
neutron-server-[NEW_POD_ID_2]    1/1     Running   0          28s
```

{% endcode %}
{% endstep %}

{% step %}

#### Step 6 — Verify VM creation succeeds

Attempt to provision a new VM on the affected region and confirm it reaches `ACTIVE` state.

{% code title="OpenStack CLI" %}

```bash
$ openstack server show <VM_UUID> | grep status
```

{% endcode %}

{% code title="Sample Output" %}

```bash
| status | ACTIVE |
```

{% endcode %}

Confirm the Networking Service logs no longer show HTTP 401 errors on Compute Service notification calls.
{% endstep %}
{% endstepper %}

## Resolution

A permanent fix for the race condition that causes this issue is planned to be included in the upcoming **2026.AUG** release. After upgrading to that release, the Networking Service configuration will always contain the correct Compute Service password regardless of the deployment order of dependent services.

The fix for **PCD-7657** ensures that the Compute Service password is pre-seeded in the Networking Service configuration store before either chart is deployed, eliminating the ordering dependency.

## Additional Information

**Related articles:**

* [VM Creation Failed Due to Port Binding Failure](https://platform9.com/kb/pcd/networking/vm-creation-failed-due-to-port-binding-failure) — covers a similar symptom (VM creation failure from Networking Service issues) where the root cause is an OVN controller problem rather than a missing service credential. Review this article if the `[nova]` section in `neutron.conf` shows a valid password but VM creation still fails.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/compute/vm-creation-fails-with-network-allocation-timeout.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.