# VMs Unable to Retrieve Metadata From Cloud-Init

## Problem

Virtual machines in a specific subnet fail to reach metadata service on `169.254.169.254`. As a result, cloud-init did not apply user-data configurations such as password injection.

## Environment

* Private Cloud Director Virtualization - v2025.4 and Higher
* Private Cloud Director Kubernetes – v2025.4 and Higher
* Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
* Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher
* Component:
  * Networking ( Neutron/OVN)

## Cause

The root cause of the issue is a missing **network namespace** used for metadata routing. This network namespace is critical for OVN to route metadata requests from VMs to the metadata proxy service.

## Diagnostics

1. VMs failed to reach `169.254.169.254` via `ping` or `curl`.
2. Gateway IP was also unreachable from within the instance.
3. `ip netns ls` command on impacted hosts showed no namespaces.
4. Found no distributed Neutron port on the host.
5. `pf9-neutron-ovn-metadata-agent` service was confirmed to be listening on port `8775`.

## Resolution

1. Check if a distributed neutron port exists for the network:

```bash
$ openstack port list --network <NETWORK_ID> --device-owner network:distributed

#sample output:
+--------------+-------------+---------------+-----------------------------------------------------+--------+
| ID           | Name        | MAC Address   | Fixed IP Addresses                                  | Status |
+--------------+-------------+---------------+-----------------------------------------------------+--------+
| [PORT_UUID]  | [PORT_NAME] | [MAC_ADDRESS] | ip_address='[IP_ADDRESS]', subnet_id='[SUBNET_UUID]'| DOWN   |
+--------------+-------------+---------------+-----------------------------------------------------+--------+
```

2. If the distributed Neutron port is present, proceed to **Step 3**. If it is missing, manually create the distributed Neutron port using the command:

```bash
$ openstack port create --network <NETWORK_ID> --fixed-ip subnet=<SUBNET_ID>,ip-address=<IP_ADDRESS> --device-owner network:distributed metadata-proxy-port-<IP_ADDRESS>
  
+-------------------------+-----------------------------------------------------+
| Field                   | Value                            										|                   
+-------------------------+-----------------------------------------------------+
| admin_state_up          | UP                               								 		|     
| allowed_address_pairs   |                                  								 		|   
| binding_host_id         |                                  								 		|   
| binding_profile         |                                  								 		|   
| binding_vif_details     |                                  								 		|   
| binding_vif_type        | unbound                          								 		|   
| binding_vnic_type       | normal                           								 		|    
| data_plane_status       | None                             								 		|    
| device_id               |                                  								 		|   
| device_owner            | network:distributed             								 		|
| device_profile          | None                            								 		|    
| extra_dhcp_opts					|																									 		|
| .                       |                                      								|
| fixed_ips               | ip_address='[IP-Address]', subnet_id='[subnet-id]'  |                   |.                        |																											|
| hardware_offload_type   | None                                 								|
| security_group_ids      |                                     								|
| status                  | DOWN                                 								|
+-------------------------+-----------------------------------------------------+
```

> The \[IP-Address] is any free IP Address available in the DHCP pool of the subnet.

3. Restart pf9-neutron-ovn-metadata-agent on all impacted compute nodes.

{% code title="Affected Host:" %}

```bash
$ sudo systemctl restart pf9-neutron-ovn-metadata-agent
```

{% endcode %}

4. Verify that ip netns ls showed namespaces post-restart.

{% code title="Affected Host:" %}

```bash
$ ip netns list
ovnmeta-[ovnnetns-id-1] (id: 0)
```

{% endcode %}

5. Re-test metadata access from inside VMs using curl.
6. Spawn a new VM with Ubuntu image and confirm that cloud-init correctly applied the configured password and user-data.

> Cirros OS image do not have cloud init service for handling the user data injection. Use image with cloud init service ex: Ubuntu or Rocky Linux for validation

## Validation

* From within the VM, below *curl* command returns the expected user-data content :

{% code title="Getting meta\_data within the VM" %}

```bash
$ curl http://169.254.169.254/openstack/latest/meta_data.json
{"uuid":"[UUID]", availability_zone":"[AZ]", "hostname":"[hostname]", "name": "[name]","launch_index": 0, "random_seed": "[random-id]", "devices":[], "dedicated_cpus":[]}
```

{% endcode %}

{% code title="Getting user\_data within the VM" %}

```bash
$ curl http://169.254.169.254/openstack/latest/user_data
#cloud-config
password: [Password] 
chpasswd: { expire: False }
ssh_pwauth: True
manage_etc_hosts: true 
runcmd:
- ['sh' , '-c', 'echo "Hello World" › /tmp/helloworld.txt' ]
```

{% endcode %}

* Cloud-init logs inside the Ubuntu VM (*/var/log/cloud-init.log*) confirmed successful metadata retrieval and password configuration.
* VM SSH access was successful using credentials configured via user-data.

## Additional Information

* In OVN-based OpenStack environments, the distributed metadata port must be explicitly present for each subnet requiring metadata access.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pcd/networking/vms-unable-to-retrieve-metadata-from-cloud-init.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.