Upgrade Guide: PEC 5.6 to SMCP 5.9.1

Prerequisites

Before beginning the upgrade process, ensure the following requirements are met:

1. Ensure that the Minimum Requirements are met.
2. Minimum Kubernetes version for Workload Cluster should be v1.22.

Upgrade Procedure

1. Backup Current System

Follow the Backup Guide to back up the 5.6 management plane. Before proceeding, verify that the backup was created successfully.

2. Take etcd Snapshot

Before making any changes to the system, it's crucial to take a snapshot of the etcd cluster. This provides an additional recovery point if needed.

a. Check the etcd cluster status and identify the leader node:

sudo /opt/pf9/pf9-kube/bin/etcdctl endpoint status --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

Make a note of which node is the leader.

b. Take the etcd cluster backup:

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot save <path/backupfile.db> --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --endpoints=https://<master_IP>:4001

Replace <path/backupfile.db> with the desired path and filename for your backup and <master_IP> with the IP address of the master node.

c. Validate the etcd backup:

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot status --write-out=table <path/backupfile.db>

Ensure that the snapshot status shows as valid before proceeding.

3. Destroy Old Cluster

Remove the existing 1.22 based nodelet cluster:

airctl stop

# Stops all running services managed by airctl

​

airctl unconfigure-du --force 

# Removes DU configuration and forcefully cleans up resources

​

airctl advanced-ddu delete-mgmt

# Deletes the management plane components

4. Clean Up Configuration

On all master nodes, clean up the certificate directory:

rm -rf /etc/nodelet/airctl-mgmt/certs/*

# This command removes all files in the specified directory

Verify the directory is empty before proceeding.

5. Download and Install New Artifacts

To obtain and install the SMCP 5.9.1 artifacts:

a. Download the download script:

wget https://raw.githubusercontent.com/platform9/support-locker/master/edge-stack/download.sh

b. Make the script executable:

chmod u+x download.sh

c. Run the download script:

./download.sh <PF9-SHARED-SECRET> v-5.9.1-<airctl_version>

Replace <PF9-SHARED-SECRET> with your Platform9 shared secret.

d. After downloading the artifacts, verify their integrity using the following command:

curl --user-agent <secret> https://pf9-airctl.s3-accelerate.amazonaws.com/<version>/md5sum.txt | md5sum -c -

Replace <secret> with the same one used to download the airctl build from S3 and <version> with the SMCP version you're installing (e.g., v5.9.1).

e. Run the install.sh script to extract and place the artifacts under /opt/pf9/airctl directory.

bash install.sh v-5.9.1-<airctl_version>

Refer to the SMCP 5.9 release notes for the exact airctl version used by the install.sh script.

For example, when upgrading to SMCP v5.9.1 :

bash install.sh v-5.9.1-3097398

6. Configure New Management Cluster

Update configuration files before deployment:

1. Edit airctl-config.yaml:

   1. Update hostAgentRepo to use SMCP 5.9.1 images.
   2. Update dockerRepo to use SMCP 5.9.1 images.

2. Edit nodelet-bootstrap.yaml:

   1. Update systemImages to use SMCP 5.9.1 image files
   2. Set userImages to point to PEC 5.6.7 image files
   3. Add isAirgapped: true for offline installations

Example airctl-config.yaml and nodelet-bootstrap.yaml snippet:

hostAgentRepo: /opt/pf9/airctl/hostagent-8-v-5.9.1-<build_number>.tar.gz

dockerRepo: /opt/pf9/airctl/docker-8-v-5.9.1-<build_number>.tar.gz

isAirgapped: true

systemImages:

- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.9.1-<build_number>.tar.gz

- /opt/pf9/airctl/imgs/nodelet-imgs-v-5.9.1-<build_number>.tar.gz

userImages:

- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.6.7-<build_number>.tar.gz

7. Deploy New Management Cluster

Deploy the new 1.25 based nodelet management cluster:

airctl advanced-ddu create-mgmt --verbose

# This command creates and deploys the new management plane components

Initialize the management plane:

sudo airctl init

Monitor the deployment progress and check for any errors.

8. Restore from Backup

Restore the PEC 5.6 backup by following the restore guide for the 5.9.1 management plane on top of the 1.25 nodelet cluster.

9. Verify DU Status

Check that the DU is in a ready state:

airctl status

# This command displays the current status of the DU

Wait until all components show as ready before proceeding.

10. Upgrade DU Components

To upgrade the DU components to5.9.1, follow the upgrade guide.

After upgrade, verify DU status again:

airctl status

11. Verify Upgrade in GUI

Log in to the management UI and confirm the version is displayed as 5.9.1.

12. Push New Kubernetes Images

Push images for Kubernetes 1.24, 1.25, and 1.26 to your private registry (if applicable):

sudo airctl advanced-du push-images --img-tar [tar file name]

To push all new images at once:

ls -1 /opt/pf9/airctl/imgs/k8s* | xargs -I{} sh -c "sudo /opt/pf9/airctl/airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-du push-images --img-tar {}"

13. Upgrade Host Components

Upgrade the host-side components:

airctl configure-hosts --reset-pf9-managed-docker-conf --skip-docker-install --skip-docker-img-import --verbose

# This command configures hosts with new settings

​

airctl hosts-status

# Check the status of hosts before proceeding

​

airctl upgrade-hosts --verbose

# This command upgrades the host components

14. Verify Status and Version

After upgrading, verify management plane services are ready. Run the following command to verify.

airctl status

# Check the status of management plane

​

airctl version

# The output should be: v-5.9.1-3097398

​

kubectl get nodes -o wide

# Ensure all the nodes are in Ready state

At this point all the components have been upgraded to SMCP 5.9.1 (LTS3).

Now, proceed with the actual workload cluster upgrades either using the GUI or API.