Backup and Restore Management Plane

This guide provides steps for backing up and restoring the self-hosted Private Cloud Director management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.

When restoring the management plane, ensure it's done on a Kubernetes cluster that is separate from the cluster where the backup was generated.

Prerequisites

System Requirements

Access to the Kubernetes management cluster
Installed and configured airctl binary
Valid airctl configuration file at /opt/pf9/airctl/conf/airctl-config.yaml
Root or sudo access to the management node

For S3 Backup Storage

AWS credentials with S3 bucket access
Existing S3 bucket for backup storage
AWS CLI configured (for verification purposes)

Important Considerations

The restoration process must be performed on a separate Kubernetes management cluster that is different from the management cluster where the backup was generated.

Manual Backup Procedure

Create a backup directory:

Bash
    
 
mkdir -p /tmp/backup-mgmt/
Copy

Execute the airctl backup command:

Execute the following commands as a non-root user.

Bash
    
 
airctl backup --outdir /tmp/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml  --verbose
Copy

Use --region <region_name> parameter if you intend to back up only a specific region. If not specified, all the regions will be included in the backup.

Verify backup contents:

Bash
    
 
tar tvf /tmp/backup-mgmt/backup.tar.gz
Copy

The backup archive should contain:

state_backup.yaml: System state configuration
kplane_values_backup.yaml: Kubernetes management cluster configuration
consul.snap: Consul snapshot
mysql_dump_Infra.sql: Infrastructure database backup
mysql_dump_Region1.sql: Region-specific database backup
ovn-north-backup & ovn-south-backup : Ovn database backup

Metrics service (gnocci) persistent data will not be backed up or restored as part of the above procedure. Therefore, in a full disaster recovery scenario, you must manually copy the metrics service (gnocchi) metrics data from the original storage class pcd-sc persistent volume.

Automated Backup Configuration

The Private Cloud Director management plane includes an automated backup system that protects your data and configuration. This system creates regular backups and can store them both locally and in Amazon S3.

Learn how to verify backup operations and configure S3 storage for your backups.

Understanding the backup system

When you install Private Cloud Director management plane, the system automatically sets up backup protection for you. During installation, it creates a cronjob called mgmt-plane-backup in the pf9-utils namespace that runs every hour to back up your system.

Your backups get stored in a dedicated storage area called mgmt-plane-backup-pvc on your Kubernetes cluster. This storage persists even if pods restart, keeping your backup data safe and accessible.

Step 1: Verify backup operations

You can check the status of your backup system at any time using kubectl commands.

Run the following command to view the backup cronjob status.

Bash
    
 
kubectl get cronjob mgmt-plane-backup -n pf9-utils
Copy

This displays when you ran the last backup, confirming your system is working properly.

Step 2: Check backup logs

To troubleshoot backup operations or verify successful completion, you can view the backup logs.

List the backup pods to find the most recent operation.

Bash
    
 
kubectl get po -n pf9-utils | grep mgmt-plane-backup
Copy

From the output, copy the pod name with the most recent timestamp.
View the logs for that specific pod.

Bash
    
 
kubectl logs mgmt-plane-backup-29167800-l294f -n pf9-utils
Copy

Replace mgmt-plane-backup-29167800-l294fwith your actual pod name from step 1.

The logs show detailed information about the backup operation, including any errors or success messages.

Step 3: Configure S3 backup storage

To enable storing backups in an S3 bucket, you need to configure S3 credentials in a secret named aws-credentials in thepf9-utils namespace.

Before you begin, consider the following points.

The Kubernetes secret should have appropriate access permissions for the mgmt-plane-backup cronjob in the pf9-utils namespace.
The backup system stores data in the PVC named mgmt-plane-backup-pvc on the Kubernetes cluster and will also upload to your configured S3 location.
The mgmt-plane-backupcronjob runs hourly to ensure regular system backups to both local storage and S3.

Run the following command to open and edit aws-credentials on the editor.

Bash
    
 
kubectl edit secret aws-credentials -n pf9-utils
Copy

On the aws-credentials edit section data:

Bash
    
​x
 
#Make the edits the yaml file​data:AWS_ACCESS_KEY_ID: "<YOUR_ACCESS_KEY>"​AWS_SECRET_ACCESS_KEY: "<YOUR_SECRET_KEY>"​AWS_REGION: "<YOUR_AWS_REGION>"​AWSS3 PATH: "s3://<YOUR_BUCKET_NAME/PATH/>"
Copy

Replace the placeholder values with your actual AWS credentials.

Placeholder	Replace with
`YOUR_ACCESS_KEY`	Your AWS access key ID
`YOUR_SECRET_KEY`	Your AWS secret access key
`YOUR_AWS_REGION`	Your AWS region (for example, us-west-2)
`YOUR_BUCKET_NAME/PATH/`	Your S3 bucket name and optional path

Save and close the editor.

Once configured, your backups will be stored both locally in the PVC and in your specified S3 bucket location. This provides enhanced data protection and allows for disaster recovery scenarios.

You have successfully configured automated backups for your Private Cloud Director management plane. Your system now creates regular backups and stores them securely both locally and in Amazon S3.

Manual Restore Procedure

When you need to restore your Private Cloud Director management plane, you can access backups stored locally in the PVC or from Amazon S3. This section walks you through both restoration methods.

Standard Restore (from local PVC)

To restore from local backups, you need to access the backup files stored in the mgmt-plane-backup-pvc. This process involves finding the PVC, locating the underlying storage, and mounting it to access the backup files.

Step 1: Locate the backup PVC

Run the following command to find the backup PVC in the pf9-utils namespace.

Bash
    
 
kubectl get pvc -n pf9-utils
Copy

The output displays your backup PVC details. Here is a sample example.

Example
    
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGEmgmt-plane-backup-pvc   Bound    pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632   100Gi      RWO            pcd-sc         <unset>                 34h
Copy

Step 2: Find the underlying persistent volume

Get the volume name for the backup PVC, by running the following command.

Bash
    
 
kubectl get pvc mgmt-plane-backup-pvc -n pf9-utils -o jsonpath='{.spec.volumeName}'
Copy

The output returns the persistent volume name. Here is an example.

Bash
    
 
pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
Copy

Step 3: Get the persistent volume configuration

Describe the persistent volume to find the NFS share information by running the following command.

Bash
    
 
kubectl describe pv pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
Copy

In the output, look for the Source: section with VolumeHandle field. This contains the NFS server and path information that you need for mounting.

Step 4: Install NFS utilities

Install the required NFS packages on your system by running the following command.

Bash
    
 
ubuntu@sample-vm:~$ sudo apt update && sudo apt -y install nfs-common
Copy

Create a local directory and mount the NFS share:

Bash
    
 
ubuntu@sample-vm:~$ mkdir -p /home/ubuntu/backupsubuntu@sample-vm:~$ sudo mount -t nfs 10.149.106.253:/mnt/gnocchi /home/ubuntu/backups
Copy

Replace 10.149.106.253:/mnt/gnocchi with the server and share path from your VolumeHandle field.

Step 6: Access the backup files

List the available backup files by running the following command

Bash
    
 
ubuntu@sample-vm:~$ ls backups/pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632
Copy

The backup files will display with timestamps. Here is an sample output.

Bash
    
 
backup_20250608_210055.tar.gz  backup_20250608_230057.tar.gz  backup_20250609_010057.tar.gzbackup_20250609_030054.tar.gz  backup_20250609_050055.tar.gz  backup_20250608_220057.tar.gz
Copy

Choose the backup file you want to restore and proceed with your restoration process. Ensure you are using the appropriate restoration commands for your specific backup file.

Restore from S3 Backup

Create and configure the /etc/default/airctl-backup file with required AWS parameters, making sure that AWS_S3_PATH points specifically to the backup file you want to restore, not just the S3 bucket:

Bash
    
 
AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY>AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>AWS_REGION=<YOUR_AWS_REGION>AWS_S3_PATH=s3://<YOUR_BUCKET_NAME/PATH/SPECIFIC_BACKUP_FILE>
Copy

Execute the S3 restore command:

Bash
    
 
airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
Copy

For complete disaster recovery, manually restore Gnocchi metrics data from the original pcd-sc persistent volume

Verification Steps

Check backup file integrity using MD5 checksum::

Bash
    
 
# Generate MD5 checksum for the backup filemd5sum <BACKUP_FILE>.tar.gz​# Optional: Compare with a pre-recorded checksum# You can save the MD5 checksum when initially creating the backupmd5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt​# Later, verify the backup file matches the original checksummd5sum -c backup-checksum.txt
Copy

Verify S3 uploads (if configured):

Bash
    
 
aws s3 ls s3://<BUCKET_NAME>/<BACKUP_PATH>
Copy

Monitor restore progress:

Bash
    
 
kubectl logs -f <RESTORE_POD_NAME>
Copy

Common Issues

If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload
Restore operations may take significant time depending on data volume
Services may take additional time to start after restore completion

Last updated on Jul 25, 2025

Was this page helpful?

Backup and Restore Management Plane

System Requirements

For S3 Backup Storage

Step 1: Verify backup operations

Step 2: Check backup logs

Step 3: Configure S3 backup storage

Step 1: Locate the backup PVC

Step 2: Find the underlying persistent volume

Step 3: Get the persistent volume configuration

Step 4: Install NFS utilities

Step 5: Mount the NFS share

Step 6: Access the backup files