# Services Fail After Accidental Deletion of Regional MySQL Databases ## Problem In certain cases, the MySQL database for a space.vars.self\_hosted\_product\_name region may be accidentally deleted or corrupted. This can cause critical platform components to fail, resulting in an unusable environment and loss of region-specific data. ## Environment * Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher * Component - MySQL ## Cause Accidental deletion or corruption of the MySQL database for a specific region(s). ## Diagnostics When a regional MySQL database is deleted or corrupted, symptoms may include: * Components failing to start or repeatedly crashing. * Errors related to missing tables or database connections in component/pod logs. * The region appearing as unreachable or inactive in the control plane UI. To confirm database loss login to MySQL from Percona pod: {% tabs %} {% tab title="Bash" %} ```bash $ kubectl exec -it percona-db-pxc-db-pxc-0 -n -c pxc -- bash bash-4.4$ mysql -u root -p mysql>show databases; [!DATABASE MISSING!] ``` {% endtab %} {% endtabs %} If the region databases are missing from the list, restoration from the latest backup is required. ## Resolution As part of the space.vars.self\_hosted\_product\_name backup process, MySQL data — including users and permissions — is backed up automatically for each region \[[Backup guide](https://platform9.com/docs/private-cloud-director/private-cloud-director/backup-and-restore)]. To restore the database without performing a full region restore, follow the steps below: {% hint style="info" %} **IMPORTANT NOTE** Before proceeding with the MySQL restore process, MySQL root password needs to be retrieved from the Kubernetes secret in its corresponding region namespace using `kubectl get secret -n mysql -o jsonpath='{.data.pass}' | base64 -d` {% endhint %} 1. **Extract the Backup** Locate the backup archive obtained from the automated or manual backup process and extract it: {% tabs %} {% tab title="Bash" %} ```bash $ tar -xzvf backups/backup_20251113_181414.tar.gz backups/state_backup.yaml backups/kplane_values_backup.yaml backups/consul.snap backups/mysql_dump_Infra.sql backups/mysql_dump_LON.sql backups/ovn-north-backup-LON-ovn-ovsdb-nb-0.tar.gz backups/ovn-north-backup-LON-ovn-ovsdb-nb-1.tar.gz backups/ovn-north-backup-LON-ovn-ovsdb-nb-2.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-0.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-1.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-2.tar.gz backups/rabbitmq-backup-Infra.tar.gz backups/rabbitmq-backup-LON.tar.gz ``` {% endtab %} {% endtabs %} Within the extracted backup, locate the MySQL dump file for the affected region mysql\_dump\_\.sql For example, for the Infra region `backups/mysql_dump_Infra.sql` 2. **Scale Down Percona Replicas** To safely restore the database, first scale down the Percona PXC replicas for the region: {% tabs %} {% tab title="Bash" %} ```bash $ kubectl scale sts percona-db-pxc-db-pxc -n --replicas=0 statefulset.apps/percona-db-pxc-db-pxc scaled ``` {% endtab %} {% endtabs %} 3. **Delete Existing Percona PVCs** Delete the existing PersistentVolumeClaims (PVCs) associated with the Percona MySQL pods to remove the corrupted data: {% tabs %} {% tab title="Bash" %} ```bash $ kubectl get pvc -n | grep percona-db-pxc-db-pxc datadir-percona-db-pxc-db-pxc-0 Bound [PVC_UUID1] 96Gi RWO hostpath-csi 3d6h datadir-percona-db-pxc-db-pxc-1 Bound [PVC_UUID2] 96Gi RWO hostpath-csi 3d6h datadir-percona-db-pxc-db-pxc-2 Bound [PVC_UUID3] 96Gi RWO hostpath-csi 3d6h $ kubectl delete pvc datadir-percona-db-pxc-db-pxc-0 datadir-percona-db-pxc-db-pxc-1 datadir-percona-db-pxc-db-pxc-2 -n persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-0" deleted persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-1" deleted persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-2" deleted ``` {% endtab %} {% endtabs %} 4. **Scale Up Percona Replicas** Scale the replicas back up to recreate the database with new, empty volumes: {% tabs %} {% tab title="Bash" %} ```bash $ kubectl scale sts percona-db-pxc-db-pxc -n --replicas=3 statefulset.apps/percona-db-pxc-db-pxc scaled $ kubectl get pods -n | grep percona-db-pxc-db-pxc percona-db-pxc-db-pxc-0 3/3 Running 0 3m26s percona-db-pxc-db-pxc-1 3/3 Running 0 2m25s percona-db-pxc-db-pxc-2 3/3 Running 0 83s ``` {% endtab %} {% endtabs %} 5. **Restore the Database from Backup** Once the Percona pods are up and running, copy the regional MySQL dump into `percona-db-pxc-db-pxc-0` pod: {% tabs %} {% tab title="Example" %} ```bash $ kubectl cp backups/mysql_dump_Infra.sql percona-db-pxc-db-pxc-0:/tmp -n -c pxc $ kubectl exec -it percona-db-pxc-db-pxc-0 -n -c pxc -- bash bash-4.4$ ls /tmp/mysql_dump_Infra.sql /tmp/mysql_dump_Infra.sql ## No platform9 service bash-4.4$ mysql -u root -p mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec) ``` {% endtab %} {% endtabs %} Then, restore the dump using the MySQL client: {% tabs %} {% tab title="Bash" %} ```bash bash-4.4$ mysql -u root -p < && mysql -u root -p -e 'flush privileges' mysql: [Warning] Using a password on the command line interface can be insecure. mysql: [Warning] Using a password on the command line interface can be insecure. ``` {% endtab %} {% endtabs %} ## Validation * Log in to the database from percona pod and confirm that the expected databases and tables are present: {% tabs %} {% tab title="Bash" %} ```bash mysql> show databases; +--------------------+ | Database | +--------------------+ | alertmanager | | hagrid | | information_schema | | keystone | | mysql | | performance_schema | | preference_store | | resmgr | | sys | +--------------------+ 9 rows in set (0.00 sec) ``` {% endtab %} {% endtabs %} * Verify that services are running and the region is healthy: {% tabs %} {% tab title="Bash" %} ```bash $ /opt/pf9/airctl/airctl status --region --config /opt/pf9/airctl/conf/airctl-config.yaml ------------- deployment details --------------- fqdn: [FQDN] region: [REGION_NAME] deployment status: ready region health: ✅ Ready version: PCD 2025.10-3111 -------- region service status ---------- desired services: 30 ready services: 30 ``` {% endtab %} {% endtabs %} * Access the management plane UI and confirm normal operations , API responses, and workflows should function as expected. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://platform9.com/kb/pcd/self-hosted/services-fail-after-accidental-deletion-mysql-databases.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.