Backup and Restore Management Plane

This guide provides steps for backing up and restoring the self-hosted Private Cloud Director management plane in disaster recovery scenarios. The procedures include both manual and automated backup methods, as well as manual restoration process.

Prerequisites

System Requirements

• Access to the Kubernetes management cluster
• Installed and configured airctl binary
• Valid airctl configuration file at /opt/pf9/airctl/conf/airctl-config.yaml
• Root or sudo access to the management node

For S3 Backup Storage

• AWS credentials with S3 bucket access
• Existing S3 bucket for backup storage
• AWS CLI configured (for verification purposes)

Important Considerations

1. The restoration process must be performed on a separate Kubernetes management cluster that is different from the management cluster where the backup was generated.

Manual Backup Procedure

1. Create a backup directory:

mkdir -p /tmp/backup-mgmt/

2. Execute the airctl backup command:

airctl backup --outdir /tmp/backup-mgmt/ --config /opt/pf9/airctl/conf/airctl-config.yaml  --verbose

3. Verify backup contents:

tar tvf /tmp/backup-mgmt/backup.tar.gz

The backup archive should contain:

• state_backup.yaml: System state configuration
• kplane_values_backup.yaml: Kubernetes management cluster configuration
• consul.snap: Consul snapshot
• mysql_dump_Infra.sql: Infrastructure database backup
• mysql_dump_Region1.sql: Region-specific database backup
• ovn-north-backup & ovn-south-backup : Ovn database backup

Automated Backup Configuration

The Private Cloud Director management plane includes an automated backup system that protects your data and configuration. This system creates regular backups and can store them both locally and in Amazon S3.

Learn how to verify backup operations and configure S3 storage for your backups.

Understanding the backup system

When you install Private Cloud Director management plane, the system automatically sets up backup protection for you. During installation, it creates a cronjob called mgmt-plane-backup in the pf9-utils namespace that runs every hour to back up your system.

Your backups get stored in a dedicated storage area called mgmt-plane-backup-pvc on your Kubernetes cluster. This storage persists even if pods restart, keeping your backup data safe and accessible.

Step 1: Verify backup operations

You can check the status of your backup system at any time using kubectl commands.

1. Run the following command to view the backup cronjob status.

kubectl get cronjob mgmt-plane-backup -n pf9-utils

This displays when you ran the last backup, confirming your system is working properly.

Step 2: Check backup logs

To troubleshoot backup operations or verify successful completion, you can view the backup logs.

1. List the backup pods to find the most recent operation.

kubectl get po -n pf9-utils | grep mgmt-plane-backup

2. From the output, copy the pod name with the most recent timestamp.
3. View the logs for that specific pod.

kubectl logs mgmt-plane-backup-29167800-l294f -n pf9-utils

Replace mgmt-plane-backup-29167800-l294fwith your actual pod name from step 1.

The logs show detailed information about the backup operation, including any errors or success messages.

Step 3: Configure S3 backup storage

To enable storing backups in an S3 bucket, you need to configure S3 credentials in a secret named aws-credentials in thepf9-utils namespace.

Before you begin, consider the following points.

• The backup system stores data in the PVC named mgmt-plane-backup-pvc on the Kubernetes cluster and will also upload to your configured S3 location.
• The mgmt-plane-backupcronjob runs hourly to ensure regular system backups to both local storage and S3.

1. Run the following command to open and edit aws-credentials on the editor.

kubectl edit secret aws-credentials -n pf9-utils

2. On the aws-credentials edit section data:

# Make the edits to the yaml file

​

data:

  AWS_ACCESS_KEY_ID: "<YOUR_ACCESS_KEY>"

  AWS_SECRET_ACCESS_KEY: "<YOUR_SECRET_KEY>"

  AWS_REGION: "<YOUR_AWS_REGION>"

  AWS_S3_PATH: "s3://<YOUR_BUCKET_NAME/PATH/>"

3. Replace the placeholder values with your actual AWS credentials.

3. Save and close the editor.

Once configured, your backups will be stored both locally in the PVC and in your specified S3 bucket location. This provides enhanced data protection and allows for disaster recovery scenarios.

You have successfully configured automated backups for your Private Cloud Director management plane. Your system now creates regular backups and stores them securely both locally and in Amazon S3.

Manual Restore Procedure

When you need to restore your Private Cloud Director management plane, you can access backups stored locally in the PVC or from Amazon S3. This section walks you through both restoration methods.

Standard Restore (from local PVC)

To restore from local backups, you need to access the backup files stored in the mgmt-plane-backup-pvc. This process involves finding the PVC, locating the underlying storage, and mounting it to access the backup files.

Step 1: Locate the backup PVC

Run the following command to find the backup PVC in the pf9-utils namespace.

kubectl get pvc -n pf9-utils

The output displays your backup PVC details. Here is a sample example.

NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE

mgmt-plane-backup-pvc   Bound    pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632   100Gi      RWO            pcd-sc         <unset>                 34h

Step 2: Find the underlying persistent volume

Get the volume name for the backup PVC, by running the following command.

kubectl get pvc mgmt-plane-backup-pvc -n pf9-utils -o jsonpath='{.spec.volumeName}'

The output returns the persistent volume name. Here is an example.

pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632

Step 3: Get the persistent volume configuration

Describe the persistent volume to find the NFS share information by running the following command.

kubectl describe pv pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632

In the output, look for the Source: section with VolumeHandle field. This contains the NFS server and path information that you need for mounting.

Step 4: Install NFS utilities

Install the required NFS packages on your system by running the following command.

ubuntu@sample-vm:~$ sudo apt update && sudo apt -y install nfs-common

Step 5: Mount the NFS share

Create a local directory and mount the NFS share:

ubuntu@sample-vm:~$ mkdir -p /home/ubuntu/backups

ubuntu@sample-vm:~$ sudo mount -t nfs 10.149.106.253:/mnt/gnocchi /home/ubuntu/backups

Replace 10.149.106.253:/mnt/gnocchi with the server and share path from your VolumeHandle field.

Step 6: Access the backup files

List the available backup files by running the following command

ubuntu@sample-vm:~$ ls backups/pvc-cd010ade-87a2-4af1-bfce-ab7ce1308632

The backup files will display with timestamps. Here is a sample output.

backup_20250608_210055.tar.gz  backup_20250608_230057.tar.gz  backup_20250609_010057.tar.gz

backup_20250609_030054.tar.gz  backup_20250609_050055.tar.gz  backup_20250608_220057.tar.gz

Choose the backup file you want to restore based on the timestamp that matches your desired restore point.

Step 7: Execute the restore command

Extract and specify the backup directory path, then run the restore command:

airctl restore --backup-dir <backup_dir_path> --config /opt/pf9/airctl/conf/airctl-config.yaml

Replace <backup_dir_path>with the path to your extracted backup directory.

Restore from S3 Backup

Create and configure the /etc/default/airctl-backup file with required AWS parameters, making sure that AWS_S3_PATH points specifically to the backup file you want to restore, not just the S3 bucket:

AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY>

AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>

AWS_REGION=<YOUR_AWS_REGION>

AWS_S3_PATH=s3://<YOUR_BUCKET_NAME/PATH/SPECIFIC_BACKUP_FILE>

Execute the S3 restore command:

airctl restore --s3backup --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose

Verification Steps

Check backup file integrity using MD5 checksum::

# Generate MD5 checksum for the backup file

md5sum <BACKUP_FILE>.tar.gz

​

# Optional: Compare with a pre-recorded checksum

# You can save the MD5 checksum when initially creating the backup

md5sum /root/backup-mgmt/backup.tar.gz > backup-checksum.txt

​

# Later, verify the backup file matches the original checksum

md5sum -c backup-checksum.txt

Verify S3 uploads (if configured):

aws s3 ls s3://<BUCKET_NAME>/<BACKUP_PATH>

Monitor restore progress:

kubectl logs -f <RESTORE_POD_NAME>

Common Issues

• If AWS credentials are not properly configured, automated S3 backups will continue locally but skip S3 upload.
• Restore operations may take significant time depending on data volume.
• Services may take additional time to start after restore completion.