Services Fail After Accidental Deletion of Regional MySQL Databases

Problem

In certain cases, the MySQL database for a region may be accidentally deleted or corrupted. This can cause critical platform components to fail, resulting in an unusable environment and loss of region-specific data.

Environment

• Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher
• Component - MySQL

Cause

Accidental deletion or corruption of the MySQL database for a specific region(s).

Diagnostics

When a regional MySQL database is deleted or corrupted, symptoms may include:

• Components failing to start or repeatedly crashing.
• Errors related to missing tables or database connections in component/pod logs.
• The region appearing as unreachable or inactive in the control plane UI.

To confirm database loss login to MySQL from Percona pod:

Bash

$ kubectl exec -it percona-db-pxc-db-pxc-0   -n <REGION_NAME> -c pxc  -- bash 

bash-4.4$ mysql -u root -p<PASSWORD>

mysql>show databases;
[!DATABASE MISSING!]

If the region databases are missing from the list, restoration from the latest backup is required.

Resolution

As part of the backup process, MySQL data — including users and permissions — is backed up automatically for each region [Backup guide]. To restore the database without performing a full region restore, follow the steps below:

IMPORTANT NOTE

Before proceeding with the MySQL restore process, MySQL root password needs to be retrieved from the Kubernetes secret in its corresponding region namespace using

kubectl get secret -n <REGION_NAME> mysql -o jsonpath='{.data.pass}' | base64 -d

1. Extract the Backup

Locate the backup archive obtained from the automated or manual backup process and extract it:

Bash

$ tar -xzvf backups/backup_20251113_181414.tar.gz 
backups/state_backup.yaml
backups/kplane_values_backup.yaml
backups/consul.snap
backups/mysql_dump_Infra.sql
backups/mysql_dump_LON.sql
backups/ovn-north-backup-LON-ovn-ovsdb-nb-0.tar.gz
backups/ovn-north-backup-LON-ovn-ovsdb-nb-1.tar.gz
backups/ovn-north-backup-LON-ovn-ovsdb-nb-2.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-0.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-1.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-2.tar.gz
backups/rabbitmq-backup-Infra.tar.gz
backups/rabbitmq-backup-LON.tar.gz

Within the extracted backup, locate the MySQL dump file for the affected region mysql_dump_<REGION_NAME>.sql

For example, for the Infra region backups/mysql_dump_Infra.sql

1. Scale Down Percona Replicas

To safely restore the database, first scale down the Percona PXC replicas for the region:

Bash

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=0
statefulset.apps/percona-db-pxc-db-pxc scaled

1. Delete Existing Percona PVCs

Delete the existing PersistentVolumeClaims (PVCs) associated with the Percona MySQL pods to remove the corrupted data:

Bash

$ kubectl get pvc -n <REGION_NAME> | grep percona-db-pxc-db-pxc
datadir-percona-db-pxc-db-pxc-0   Bound    [PVC_UUID1]   96Gi   RWO     hostpath-csi   <unset>   3d6h
datadir-percona-db-pxc-db-pxc-1   Bound    [PVC_UUID2]   96Gi   RWO     hostpath-csi   <unset>   3d6h
datadir-percona-db-pxc-db-pxc-2   Bound    [PVC_UUID3]   96Gi   RWO     hostpath-csi   <unset>   3d6h

$ kubectl delete pvc datadir-percona-db-pxc-db-pxc-0 datadir-percona-db-pxc-db-pxc-1 datadir-percona-db-pxc-db-pxc-2 -n <REGION_NAME>
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-0" deleted
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-1" deleted
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-2" deleted

1. Scale Up Percona Replicas

Scale the replicas back up to recreate the database with new, empty volumes:

Bash

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=3
statefulset.apps/percona-db-pxc-db-pxc scaled

$ kubectl get pods -n <REGION_NAME> | grep percona-db-pxc-db-pxc
percona-db-pxc-db-pxc-0    3/3     Running     0    3m26s
percona-db-pxc-db-pxc-1    3/3     Running     0    2m25s
percona-db-pxc-db-pxc-2    3/3     Running     0    83s

1. Restore the Database from Backup

Once the Percona pods are up and running, copy the regional MySQL dump into percona-db-pxc-db-pxc-0 pod:

Example

$ kubectl cp backups/mysql_dump_Infra.sql percona-db-pxc-db-pxc-0:/tmp -n <REGION_NAME> -c pxc

$ kubectl exec -it percona-db-pxc-db-pxc-0  -n <REGION_NAME> -c pxc -- bash
bash-4.4$ ls /tmp/mysql_dump_Infra.sql 
/tmp/mysql_dump_Infra.sql

## No platform9 service
bash-4.4$ mysql -u root -p<PASSWORD>

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

Then, restore the dump using the MySQL client:

Bash

bash-4.4$ mysql -u root -p<PASSWORD> < <PATH_TO_MYSQL_BACKUP.sql> && mysql -u root -p<PASSWORD> -e 'flush privileges'
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.

Validation

• Log in to the database from percona pod and confirm that the expected databases and tables are present:

Bash

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| alertmanager       |
| hagrid             |
| information_schema |
| keystone           |
| mysql              |
| performance_schema |
| preference_store   |
| resmgr             |
| sys                |
+--------------------+
9 rows in set (0.00 sec)

• Verify that services are running and the region is healthy:

Bash

$ /opt/pf9/airctl/airctl status --region <REGION_NAME> --config /opt/pf9/airctl/conf/airctl-config.yaml 
------------- deployment details ---------------
fqdn:                [FQDN]
region:              [REGION_NAME]
deployment status:   ready
region health:       ✅ Ready
version:              PCD 2025.10-3111
-------- region service status ----------
desired services:     30
ready services:       30

• Access the management plane UI and confirm normal operations , API responses, and workflows should function as expected.