Management Plane Upgrade Fails From Airctl Version 5.0.0-1468184 to 5.1.0-1488368 as Host was Tainte

Problem

Management Plane Upgrade Fails From Airctl Version 5.0.0-1468184 to 5.1.0-1488368.

None

[centos@deployment-host ~]$ airctl get-du-version
DU Version: 5.0.0-1468184-platform9-v5.0

None

[centos@deployment-host ~]$ /opt/pf9/airctl/airctl upgrade --disk /opt/pf9/airctl/du-v-5.1.0-1488368.qcow2 --config /opt/pf9/airctl/conf/airctl-config.yaml
rollback state directory: /tmp/airctl_backup_771265372
--- backing up management plane ---
backup saved to /tmp/airctl_backup_771265372/backup.tar.gz
--- deleting configuration state ---
--- moving old state file to /tmp/airctl_backup_771265372/state.yaml ---
--- saving VM XML to /tmp/airctl_backup_771265372/domain.xml ---
--- saving network XML to /tmp/airctl_backup_771265372/net.xml ---
--- deleting management plane ---
old disk image (will NOT be deleted): /opt/pf9/airctl/du-v-5.0.0-1468184.qcow2
old ISO image (will NOT be deleted): /home/centos/.airctl/cloud-init-2021-03-22T11:55:32-05:00.iso
done
--- creating new management plane ---
--- restoring from backup ---
starting management plane ...
preparing configuration database for restore operation ...
restoring configuration database ...
preparing management plane for restore operation ...
fatal error: failed to prepare for restore operation: configure command returned status 1, log file: /home/centos/airctl-logs/pf9deploy-prepare-restore.log

airctl.log

2021-04-13T12:57:40.728-0500    INFO    waiting for container pf9deploy to finish
2021-04-13T13:08:36.951-0500    ERROR    fatal error:failed to prepare for restore operation: configure command returned status 1, log file: /home/centos/airctl-logs/pf9deploy-prepare-restore.log

pf9deploy-prepare-restore.log

INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
DEBUG:pf9deploy.server.util.ssh:<retry> Task is still not ready after 0 seconds
INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
DEBUG:pf9deploy.server.util.ssh:<retry> Task is still not ready after 15 seconds
...
INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
Traceback (most recent call last):
File "/airctl/scripts/run_playbook.py", line 77, in <module>
  op.commit()
File "/app/pf9deploy/server/operation/__init__.py", line 212, in commit
  ret_value = self.cleanse_data(self._commit())
File "/airctl/scripts/run_playbook.py", line 51, in _commit
  check_ssh(du.fqdn, key_file=private_key)
File "/app/pf9deploy/server/util/retry.py", line 40, in newfun
  raise RuntimeError("Task timed out after %d seconds" % max_wait)
RuntimeError: Task timed out after 600 seconds

Environment

• Platform9 Edge Cloud - v5.1

• Airctl

Cause

Deployment Unit image du-v-5.1.0-1488368.qcow2 under the directory /opt/pf9/airctl on the deployment host was tainted when a previous upgrade attempt had failed.

Resolution

In this case, the user will need to perform an airctl restore using a pristine image (freshly copied from S3 download), instead of using the tainted Deployment Unit image. Once that is done, re-upgrade using the pristine image.

1. Remove the tainted DU image from /opt/pf9/airctl.

2. Download DU image.

None

$ curl --user-agent [secret user agent] https://s3-us-west-1.amazonaws.com/pf9-airctl/v-5.1.0-1488368/index.txt

None

$ bash ./install.sh v-5.1.0-1488368

1. Rollback/Restore to the previous state.

None

$ ./airctl rollback --dir [rollback_dir]

1. Upgrade to the desired version 5.1.0-1488368.

None

$ /opt/pf9/airctl/airctl upgrade --disk /opt/pf9/airctl/du-v-5.1.0-1488368.qcow2 --config /opt/pf9/airctl/conf/airctl-config.yaml

Note: Connect with Platform9 support to get secret user agent value.