How To Re-generate Certificates If Hostagent Certificates are Expired

Problem

• Hostagent certificate and other components certificates are expired and all nodes are marked as disconnected and unable to perform upgrades.

• Performing Higher Version Upgrade is not re-generating hostagent certificates even when the certificates are not expired.

Environment

• Platform9 Edge Cloud - v5.3 and Higher

• Airctl

• Hostagent Certificate

Procedure

Note:

There is product bug where the hostagent certificates are not being regenerated during same/higher version upgrades on up to patch 13. Please follow the steps mentioned in this KB if running a deployment on patch 13.

Starting patch 14 (v-5.3.0-2710638), these certificates will be regenerated during both - a same version upgrade, as well as a higher version upgrade.

• Login/SSH to Deployment Unit Host (DU Host).

• Start MongoDB container and exec into it.

DU Host

# docker start airctl-mongo
# docker exec -it airctl-mongo bash

• Now inside airctl-mongo run the below commands:

bash$ mongo
    > use pf9
    switched to pf9 DB
    > db.secrets.getIndexes()

Warning

If multiple indexes are seen with the db.secrets.getIndexes() output then skip below step to create a unique tag index. If getIndexes() output has only one index then create a unique tag index and verify using below commands.

Inside airctl-mongo container

> db.secrets.createIndex({"tag": 1}, {unique:true})
> db.secrets.getIndexes()

• Now open another terminal for Deployment Unit Host and start the pf9deployExec container using the below command and exec into it to export the mentioned env variables:

Start & Exec in pf9deployExec container on DU Host

# docker run --name pf9deployExec -d  -v /opt/pf9/airctl:/airctl -v /opt/pf9/airctl/ansible-stack:/ansible-stack --network host pf9-deploy:latest sleep 1000000
# docker exec -it pf9deployExec bash

bash$ export PF9_ANSIBLE_DIR=/ansible-stack
bash$ export PF9DEPLOY_CONF_FILE=/airctl/conf/pf9deploy.ini
bash$ export DU_FQDN=<your DU_FQDN>
bash$ export SHORTNAME=<shortname>
bash$ export PF9DEPLOY_CRYPT_BACKEND=null

bash$ /app/deployutil.py check-certs --shortname $SHORTNAME --debug

Error

If the check-certs command is failing with an error pf9deploy.db.exceptions.ModelNotFound: {'shortname': '<SHORTNAME>'} . The workaround step is to perform the same version upgrade of the Management plane.

• For Management plane versions patch 14 (v-5.3.0-2710638) and higher, the hostagent certificate will automatically get regenerated and needs to be copied to the hosts.

• For versions lower than patch 14 (v-5.3.0-2710638) follow the steps mentioned in this document to generate a new hostagent certificate and copy it to the hosts manually.

• The above check-certs would show the certs version and denote if it's expiring.

• Now generate certs and check using:

Inside pf9deploy container

bash$ /app/deployutil.py generate-certs --fqdn $DU_FQDN --debug
bash$ /app/deployutil.py check-certs --shortname $SHORTNAME --debug

• Now dump the MongoDB data using airctl:

• For DU version 5.3

DU Host

# /opt/pf9/airctl/airctl  advanced-du save-mongo --config /opt/pf9/airctl/conf/airctl-config.yaml

• For DU version 5.1

DU Host

# /opt/pf9/airctl/airctl  save-mongo --config /opt/pf9/airctl/conf/airctl-config.yaml

• Once the new certs are generated, you stop the pf9deployExec and airctl-mongo containers and delete the pf9deployExe container using:

DU Host

# docker stop pf9deployExec
# docker stop airctl-mongo
# docker rm pf9deployExec
# docker ps -a

IMPORTANT

When performing a SAME VERSION upgrade, ensure to use a pristine QCOW2 image when running the upgrade command.

• Now we can start with the Same Version or Higher Version upgrade using the steps mentioned in the Upgrade documentation. Only follow steps in that page till the Upgrade DU section section.

• If the Upgrade operation above fails with the below Ansible Log Trace, refer this Solution to troubleshoot the upgrade.

pf9deply.log

TASK [pf9-configure : Wait for resmgr service] *********************************
 2025-02-03T23:32:44.466-0800 info INFO:pf9deploy.server.util.shell:| Tuesday 04 February 2025  07:02:42 +0000 (0:00:01.849)       0:12:17.577 ******
 2025-02-03T23:32:44.466-0800 info INFO:pf9deploy.server.util.shell:| fatal: [airctl-1.pf9.localnet]: FAILED! => {"changed": false, "elapsed": 1800, "msg": "Timeout when waiting for 127.0.0.1:8083"}

• Final Step: Perform host upgrade using steps mentioned in Configure Host and HostAgent Upgrade.

  • On upgrading the DU to patch14 - v-5.3.0-2710638 (or higher) from a previous patch (up to LTS1 patch 13), host agent certificates generated on the DU using the process described in this page will be automatically copied over to the hosts as part of the host agent upgrade - and the host certificates will be renewed.