VMs Unable to Retrieve Metadata From Cloud-Init

Problem

Virtual machines in a specific subnet fail to reach metadata service on 169.254.169.254. As a result, cloud-init did not apply user-data configurations such as password injection.

Environment

• Private Cloud Director Virtualization - v2025.4 and Higher

• Private Cloud Director Kubernetes – v2025.4 and Higher

• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

• Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher

• Component:

  • Networking ( Neutron/OVN)

Cause

The root cause of the issue is a missing network namespace used for metadata routing. This network namespace is critical for OVN to route metadata requests from VMs to the metadata proxy service.

Diagnostics

1. VMs failed to reach 169.254.169.254 via ping or curl.

2. Gateway IP was also unreachable from within the instance.

3. ip netns ls command on impacted hosts showed no namespaces.

4. Found no distributed Neutron port on the host.

5. pf9-neutron-ovn-metadata-agent service was confirmed to be listening on port 8775.

Resolution

1. Check if a distributed neutron port exists for the network:

$ openstack port list --network <NETWORK_ID> --device-owner network:distributed

#sample output:
+--------------+-------------+---------------+-----------------------------------------------------+--------+
| ID           | Name        | MAC Address   | Fixed IP Addresses                                  | Status |
+--------------+-------------+---------------+-----------------------------------------------------+--------+
| [PORT_UUID]  | [PORT_NAME] | [MAC_ADDRESS] | ip_address='[IP_ADDRESS]', subnet_id='[SUBNET_UUID]'| DOWN   |
+--------------+-------------+---------------+-----------------------------------------------------+--------+

1. If the distributed Neutron port is present, proceed to Step 3. If it is missing, manually create the distributed Neutron port using the command:

$ openstack port create --network <NETWORK_ID> --fixed-ip subnet=<SUBNET_ID>,ip-address=<IP_ADDRESS> --device-owner network:distributed metadata-proxy-port-<IP_ADDRESS>
  
+-------------------------+-----------------------------------------------------+
| Field                   | Value                            										|                   
+-------------------------+-----------------------------------------------------+
| admin_state_up          | UP                               								 		|     
| allowed_address_pairs   |                                  								 		|   
| binding_host_id         |                                  								 		|   
| binding_profile         |                                  								 		|   
| binding_vif_details     |                                  								 		|   
| binding_vif_type        | unbound                          								 		|   
| binding_vnic_type       | normal                           								 		|    
| data_plane_status       | None                             								 		|    
| device_id               |                                  								 		|   
| device_owner            | network:distributed             								 		|
| device_profile          | None                            								 		|    
| extra_dhcp_opts					|																									 		|
| .                       |                                      								|
| fixed_ips               | ip_address='[IP-Address]', subnet_id='[subnet-id]'  |                   |.                        |																											|
| hardware_offload_type   | None                                 								|
| security_group_ids      |                                     								|
| status                  | DOWN                                 								|
+-------------------------+-----------------------------------------------------+

The [IP-Address] is any free IP Address available in the DHCP pool of the subnet.

1. Restart pf9-neutron-ovn-metadata-agent on all impacted compute nodes.

Affected Host:

$ sudo systemctl restart pf9-neutron-ovn-metadata-agent

1. Verify that ip netns ls showed namespaces post-restart.

Affected Host:

$ ip netns list
ovnmeta-[ovnnetns-id-1] (id: 0)

1. Re-test metadata access from inside VMs using curl.

2. Spawn a new VM with Ubuntu image and confirm that cloud-init correctly applied the configured password and user-data.

Cirros OS image do not have cloud init service for handling the user data injection. Use image with cloud init service ex: Ubuntu or Rocky Linux for validation

Validation

• From within the VM, below curl command returns the expected user-data content :

Getting meta_data within the VM

$ curl http://169.254.169.254/openstack/latest/meta_data.json
{"uuid":"[UUID]", availability_zone":"[AZ]", "hostname":"[hostname]", "name": "[name]","launch_index": 0, "random_seed": "[random-id]", "devices":[], "dedicated_cpus":[]}

Getting user_data within the VM

$ curl http://169.254.169.254/openstack/latest/user_data
#cloud-config
password: [Password] 
chpasswd: { expire: False }
ssh_pwauth: True
manage_etc_hosts: true 
runcmd:
- ['sh' , '-c', 'echo "Hello World" › /tmp/helloworld.txt' ]

• Cloud-init logs inside the Ubuntu VM (/var/log/cloud-init.log) confirmed successful metadata retrieval and password configuration.

• VM SSH access was successful using credentials configured via user-data.

Additional Information

• In OVN-based OpenStack environments, the distributed metadata port must be explicitly present for each subnet requiring metadata access.