# Management Plane Upgrade Fails From Airctl Version 5.0.0-1468184 to 5.1.0-1488368 as Host was Tainte

## Problem

Management Plane Upgrade Fails From Airctl Version 5.0.0-1468184 to 5.1.0-1488368.

{% tabs %}
{% tab title="None" %}

```none
[centos@deployment-host ~]$ airctl get-du-version
DU Version: 5.0.0-1468184-platform9-v5.0
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="None" %}

```none
[centos@deployment-host ~]$ /opt/pf9/airctl/airctl upgrade --disk /opt/pf9/airctl/du-v-5.1.0-1488368.qcow2 --config /opt/pf9/airctl/conf/airctl-config.yaml
rollback state directory: /tmp/airctl_backup_771265372
--- backing up management plane ---
backup saved to /tmp/airctl_backup_771265372/backup.tar.gz
--- deleting configuration state ---
--- moving old state file to /tmp/airctl_backup_771265372/state.yaml ---
--- saving VM XML to /tmp/airctl_backup_771265372/domain.xml ---
--- saving network XML to /tmp/airctl_backup_771265372/net.xml ---
--- deleting management plane ---
old disk image (will NOT be deleted): /opt/pf9/airctl/du-v-5.0.0-1468184.qcow2
old ISO image (will NOT be deleted): /home/centos/.airctl/cloud-init-2021-03-22T11:55:32-05:00.iso
done
--- creating new management plane ---
--- restoring from backup ---
starting management plane ...
preparing configuration database for restore operation ...
restoring configuration database ...
preparing management plane for restore operation ...
fatal error: failed to prepare for restore operation: configure command returned status 1, log file: /home/centos/airctl-logs/pf9deploy-prepare-restore.log
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="airctl.log" %}

```none
2021-04-13T12:57:40.728-0500    INFO    waiting for container pf9deploy to finish
2021-04-13T13:08:36.951-0500    ERROR    fatal error:failed to prepare for restore operation: configure command returned status 1, log file: /home/centos/airctl-logs/pf9deploy-prepare-restore.log
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="pf9deploy-prepare-restore.log" %}

```none
INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
DEBUG:pf9deploy.server.util.ssh:<retry> Task is still not ready after 0 seconds
INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
DEBUG:pf9deploy.server.util.ssh:<retry> Task is still not ready after 15 seconds
...
INFO:pf9deploy.server.util.ssh:check_ssh: SSH Connection failed. Reason: [Errno None] Unable to connect to port 22 on 192.168.120.254, retrying...
Traceback (most recent call last):
File "/airctl/scripts/run_playbook.py", line 77, in <module>
  op.commit()
File "/app/pf9deploy/server/operation/__init__.py", line 212, in commit
  ret_value = self.cleanse_data(self._commit())
File "/airctl/scripts/run_playbook.py", line 51, in _commit
  check_ssh(du.fqdn, key_file=private_key)
File "/app/pf9deploy/server/util/retry.py", line 40, in newfun
  raise RuntimeError("Task timed out after %d seconds" % max_wait)
RuntimeError: Task timed out after 600 seconds
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Edge Cloud - v5.1
* Airctl

## Cause

Deployment Unit image *du-v-5.1.0-1488368.qcow2* under the directory */opt/pf9/airctl* on the deployment host was tainted when a previous upgrade attempt had failed.

## Resolution

In this case, the user will need to perform an airctl restore using a pristine image (freshly copied from S3 download), instead of using the tainted Deployment Unit image. Once that is done, re-upgrade using the pristine image.

1. Remove the tainted DU image from */opt/pf9/airctl.*
2. Download DU image.

{% tabs %}
{% tab title="None" %}

```none
$ curl --user-agent [secret user agent] https://s3-us-west-1.amazonaws.com/pf9-airctl/v-5.1.0-1488368/index.txt
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="None" %}

```none
$ bash ./install.sh v-5.1.0-1488368
```

{% endtab %}
{% endtabs %}

3. Rollback/Restore to the previous state.

{% tabs %}
{% tab title="None" %}

```none
$ ./airctl rollback --dir [rollback_dir]
```

{% endtab %}
{% endtabs %}

4. Upgrade to the desired version 5.1.0-1488368.

{% tabs %}
{% tab title="None" %}

```none
$ /opt/pf9/airctl/airctl upgrade --disk /opt/pf9/airctl/du-v-5.1.0-1488368.qcow2 --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

**Note**: Connect with Platform9 support to get secret user agent value.