Multiple Instance Creation Failed Due to Max Scheduling Attempts Exceeded for Cinder Volume

Problem

During the creation of multiple VM instances, certain deployments are successful, but a subset experiences failures attributed to an error identified within the Cinder scheduler.

Cinder Scheduler Logs
Copy

Environment

  • Self-Hosted Private Cloud Director Virtualisation – v2025.4
  • Private Cloud Director Virtualisation – v2025.4

Cause

The Cinder scheduler overloads due to multiple concurrent volume creation requests in the backend.

Resolution

The resolution for this issue is a two part process, which involves making changes on the Management Plane and on the hosts.

SaaS customers should reach out to Platform9 Support Team to implement Part-1 of the Resolution.

PART-1

  1. Cinder pods use cinder.conf from cinder-etc secret, so we need to update the cinder-etc secret. Verify if secret is available in the namespace.
Command
Copy
  1. Take backup of the secret.
Command
Copy
  1. Get the cinder.conf information from the secret
Command
Copy
  1. Open cinder.conf file in a file editor and make the below changes in default section. As shown below:
cinder.conf
Copy

It is not recommended to increase the scheduler__max__attempts beyond 10. As there are multiple factors such as Storage backend network latency, Storage IOPS, etc.

  1. Save the cinder.conf and encode the file using base64.
Encode Cinder.conf
Copy
  1. Copy the encoded value from above command and edit the secret replace the older cinder.conf content with new encoded value.
Edit Cinder.conf
Copy
  1. Save the secret and verify if the new cinder.conf values are reflecting.
Command
Copy
  1. Now restart cinder-api and cinder-scheduler pod so that it will start utilising updated cinder.conf file.
Command
Copy

PART-2

  1. On every host that has Persistent Storage role, open /opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf file in a file editor and make the below changes in default section as shown below:
/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder.conf
Copy
  1. Restart the below service on every host that has Persistent Storage role:
Command
Copy

The changes made will not persist through an upgrade. Therefore, it is important to reapply these steps immediately after the upgrade to ensure continued functionality.

Additional Information

Execute these commands carefully, ensuring no unintended characters are added to cinder.conf. If the cinder.conf file is corrupted, restore the cinder-etc secret from the backup file and restart the cinder-api and cinder-scheduler pods.

Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard