Multiple Instance Creation Failed Due to Max Scheduling Attempts Exceeded for Cinder Volume

Problem

During the creation of multiple VM instances, certain deployments are successful, but a subset experiences failures attributed to an error identified within the Cinder scheduler.

Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource <VOLUME_UUID>: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource <VOLUME_UUID>

Environment

• Self-Hosted Private Cloud Director Virtualisation – v2025.4
• Private Cloud Director Virtualisation – v2025.4

Cause

The Cinder scheduler overloads due to multiple concurrent volume creation requests in the backend. The changes are now part of the product starting in PCD June Release.

Workaround

PART-1

1. Cinder pods use cinder.conf from cinder-etc secret, so we need to update the cinder-etc secret. Verify if secret is available in the namespace.

$ kubectl get secrets -n <REGION_NAMESPACE> | grep -i cinder-etc

2. Take backup of the secret.

$ kubectl get secrets -n <REGION_NAMESPACE> cinder-etc -o yaml > cinder-etc.secret.bk

3. Get the cinder.conf information from the secret

$ kubectl get secrets -n <REGION_NAMESPACE> cinder-etc -o json | jq -r '.data."cinder.conf"' | base64 -d > cinder.conf

4. Open cinder.conf file in a file editor and make the below changes in default section. As shown below:

[default]

scheduler_max_attempts = 10

osapi_volume_workers = 8

service_down_time = 180

report_interval = 60

5. Save the cinder.conf and encode the file using base64.

$ cat cinder.conf | base64 -w0

6. Copy the encoded value from above command and edit the secret replace the older cinder.conf content with new encoded value.

$ kubectl edit secrets -n <REGION_NAMESPACE> cinder-etc

7. Save the secret and verify if the new cinder.conf values are reflecting.

$ kubectl get secrets -n <REGION_NAMESPACE> cinder-etc -o json | jq -r '.data."cinder.conf"' | base64 -d

8. Now restart cinder-api and cinder-scheduler pod so that it will start utilising updated cinder.conf file.

$ kubectl get pods -n <REGION_NAMESPACE> | grep -i cinder

cinder-api-xxxxx-xxxxx                     2/2     Running           0               10s

cinder-scheduler-xxxxxxx-xxxxx             1/1     Running           0               10s

PART-2

1. On every host that has Persistent Storage role, open /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf file in a file editor and make the below changes in default section as shown below:

[default]

scheduler_max_attempts = 10

osapi_volume_workers = 8

service_down_time = 180

report_interval = 60

2. Restart the below service on every host that has Persistent Storage role:

$ systemctl restart pf9-cindervolume-base.service

Additional Information

Execute these commands carefully, ensuring no unintended characters are added to cinder.conf. If the cinder.conf file is corrupted, restore the cinder-etc secret from the backup file and restart the cinder-api and cinder-scheduler pods.