VM Deployment Fails Due to Missing Image Data on the Image Library Shared Storage

Problem

VM creation in the region was failing with repeated retry errors due to missing image data in the Image Library’s shared storage.

UI Error

Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance <INSTANCE_ID>. Last exception: Failed to call glance method data with args ('<IMAGE_ID>',), {} on endpoints ['https://10.1
nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance <INSTANCE_ID>. Last exception: Failed to call glance method data with args ('<IMAGE_ID>',), {} on endpoints ['https://<ENDPOINT_IP1>:9494', 'https://<ENDPOINT_IP2>:9494', 'https://<ENDPOINT_IP3>:9494']

Environment

• Private Cloud Director Virtualization - All versions

• Self-Hosted Private Cloud Director Virtualization - All versions

• Component: Image library

Cause

1. The Shared Storage option in the Image Library was disabled.

2. Images were uploaded before the NFS-backed storage was in use.

3. Once shared storage was enabled or mounted, the previously stored image files were no longer present on the new backend.

4. Image services still reported these images as active in the database, but the underlying files were missing.

5. As a result, compute failed to fetch the image data required for instance creation.

Diagnostics

1. Check the Image Library backend directory (/var/lib/glance/images/glance )

   The directory existed but contained no actual image files:

   Image Library Host

   $ cd /var/lib/glance/images/glance
   $ ls -al
   total 12
   drwxr-xr-x 3 pf9 pf9group 4096 Nov 11 22:02 .
   drwxr-xr-x 3 pf9 pf9group 4096 Nov 11 22:02 ..
   drwxr-xr-x 2 pf9 pf9group 4096 Nov 11 22:02 snapshot

2. Verify NFS mount

   Image Library Host

   $ df -h
   <NFS_FQDN>:/<LOCATION>  2.0T  426G  1.6T  22% /var/lib/glance/images

3. Confirm connectivity to NFS server using nslookup and ping

   Image Library Host

   $ nslookup <NFS_FQDN>
   $ ping <IP>

4. Inspect Cluster Blueprint

   1. The Shared Storage option in the Image Library section of the cluster blueprint was found to be disabled. If shared storage is required for image hosting, this option must be enabled.

Resolution

1. Enable Shared Storage

   1. In the region’s Cluster Blueprint, under Image Library → Shared Storage

2. Upload a new image after enabling shared storage

   1. Previously uploaded images cannot be recovered because their data no longer exists on the NFS path.

3. Wait for the new image to reach active state.

4. Attempt VM creation using the newly uploaded image.

5. Verify that the new VM deploys successfully.

Validation

Perform the following checks to ensure the issue is resolved:

1. Confirm that new image files exist on NFS backend.

   Image Library Host

   $ ls -al /var/lib/glance/images

2. Check image status

   $ openstack image list --long

3. Create a test VM

   $ openstack server create --flavor m1.tiny --image <NEW_IMAGE_ID> --network <NETWORK> <INSTANCE_NAME>

Additional Information

This issue typically occurs when image uploads are performed before shared storage is fully configured or mounted.