> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/self-hosted/multiple-instance-creation-failed-due-to-max-scheduling-attempts-exceeded-for-cinder-volume.md). # Multiple Instance Creation Failed Due to Max Scheduling Attempts Exceeded for Cinder Volume ## Problem During the creation of multiple VM instances, certain deployments are successful, but a subset experiences failures attributed to an error identified within the Cinder scheduler. ```ruby Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource : cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource ``` ## Environment * Self-Hosted Private Cloud Director Virtualisation – v2025.4 * Private Cloud Director Virtualisation – v2025.4 ## Cause The Cinder scheduler overloads due to multiple concurrent volume creation requests in the backend. The changes are now part of the product starting in PCD **June Release.** ## Workaround > The resolution for this issue is a two part process, which involves making changes on the Management Plane and on the hosts.> > SaaS customers should reach out to Platform9 Support Team to implement Part-1 of the Resolution. **PART-1** 1. Cinder pods use `cinder.conf` from `cinder-etc` secret, so we need to update the `cinder-etc` secret. Verify if secret is available in the namespace. ```bash $ kubectl get secrets -n | grep -i cinder-etc ``` 2. Take backup of the secret. ```bash $ kubectl get secrets -n cinder-etc -o yaml > cinder-etc.secret.bk ``` 3. Get the `cinder.conf` information from the secret ```bash $ kubectl get secrets -n cinder-etc -o json | jq -r '.data."cinder.conf"' | base64 -d > cinder.conf ``` 4. Open `cinder.conf` file in a file editor and make the below changes in **default** section. As shown below: ```bash [default] scheduler_max_attempts = 10 osapi_volume_workers = 8 service_down_time = 180 report_interval = 60 ``` > It is not recommended to increase the scheduler\_\_max\_\_attempts beyond 10. As there are multiple factors such as Storage backend network latency, Storage IOPS, etc. 5. Save the `cinder.conf` and encode the file using `base64`. ```bash $ cat cinder.conf | base64 -w0 ``` 6. Copy the encoded value from above command and edit the secret replace the older `cinder.conf` content with new encoded value. ```bash $ kubectl edit secrets -n cinder-etc ``` 7. Save the secret and verify if the new `cinder.conf` values are reflecting. ```bash $ kubectl get secrets -n cinder-etc -o json | jq -r '.data."cinder.conf"' | base64 -d ``` 8. Now restart `cinder-api` and `cinder-scheduler` pod so that it will start utilising updated `cinder.conf` file. ```bash $ kubectl get pods -n | grep -i cinder cinder-api-xxxxx-xxxxx 2_2 Running 0 10s cinder-scheduler-xxxxxxx-xxxxx 1_1 Running 0 10s ``` **PART-2** 1. On every host that has *Persistent Storage* role, open `/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf` file in a file editor and make the below changes in **default** section as shown below: ```bash [default] scheduler_max_attempts = 10 osapi_volume_workers = 8 service_down_time = 180 report_interval = 60 ``` 2. Restart the below service on every host that has *Persistent Storage* role: ```bash $ systemctl restart pf9-cindervolume-base.service ``` > The changes made will not persist through an upgrade. Therefore, it is important to reapply these steps immediately after the upgrade to ensure continued functionality. ## Additional Information Execute these commands carefully, ensuring no unintended characters are added to `cinder.conf`. If the `cinder.conf` file is corrupted, restore the cinder-etc secret from the backup file and restart the cinder-api and cinder-scheduler pods. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://platform9.com/kb/pcd/self-hosted/multiple-instance-creation-failed-due-to-max-scheduling-attempts-exceeded-for-cinder-volume.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.