> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/smcp/solution/while-upgrade-from-lts2-patch2-to-scmp--management-plane-cluster.md).

# While Restoring LTS2-Patch2 On SMCP, Management Plane Cluster Backup-Restore Process Fails.

## Problem

During the restoration process of LTS2-patch2 \[v-5.6.7-2624593] to SMCP, the restore step is failing with below error:

{% tabs %}
{% tab title="Backup Restore error" %}

```javascript
# airctl restore --backupdir /root/ --config /opt/pf9/airctl/conf/airctl-config.yaml --verbose
...
2023-09-22T14:01:25.353Z        info    restoring mysql
2023-09-22T14:01:25.353Z        info    state file does not contain SSH user
▀  Starting vault... (6m28s)2023-09-22T14:01:25.456Z    debug   found pod percona-db-pxc-db-pxc-0
  ERROR   setting up kplane...
2023-09-22T14:01:34.475Z        error   failed to install kplane components: failed to install helm chart /sbin/helm install kplane-usermgr /opt/pf9/airctl/conf/helm_charts/kplane-components-0.3.4.tgz -f /opt/pf9/airctl/conf/kplane_values.yaml -f /opt/pf9/airctl/conf/secrets.yaml: exit status 1 - Error: INSTALLATION FAILED: execution error at (kplane-components/templates/required.yaml:17:5): consul_fallback_token is required from values.yaml
2023-09-22T14:01:34.475Z        fatal   error: failed to install helm chart /sbin/helm install kplane-usermgr /opt/pf9/airctl/conf/helm_charts/kplane-components-0.3.4.tgz -f /opt/pf9/airctl/conf/kplane_values.yaml -f /opt/pf9/airctl/conf/secrets.yaml: exit status 1 - Error: INSTALLATION FAILED: execution error at (kplane-components/templates/required.yaml:17:5): consul_fallback_token is required from values.yaml
...
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Edge Cloud- LTS2-Patch2 \[v-5.6.7-2624593].

## Cause

This is a known issue. Jira AIR-1218 has been filed to track and resolve it.

Platform9 Engineering team is actively working to fix this issue.

## Workaround

As a workaround, please follow the steps mentioned below:

1. Ensure your existing DU has no issues by running the following command and verifying that task state is ready

{% tabs %}
{% tab title="Javascript" %}

```javascript
airctl status
```

{% endtab %}
{% endtabs %}

```
  2.  Download LTS2-Patch4 [v-5.6.7-2658688] artifacts, following same steps as for LTS2-Patch#2
```

{% tabs %}
{% tab title="Bash" %}

```bash
bash ./install.sh v-5.6.7-2658688
```

{% endtab %}
{% endtabs %}

3. Run the upgrade operation following the upgrade guide. (Upgrade from LTS2-patch#2 to LTS2-patch#4)

{% tabs %}
{% tab title="None" %}

```none
airctl upgrade
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

The upgrade operation is expected to fail due a known issue which can be ignored. The upgrade, however it fails, fixes the state files which are essential for the restoration of LTS2 on SMCP. But the upgrade from LTS2-patch#2 to LTS2-patch#4 is affected due to removal of internal component known as decco and some related codebase changes.

The expected error message is shown below:
{% endhint %}

{% tabs %}
{% tab title="None" %}

```none
ERROR   upgrading management plane...
2023-10-11T10:08:24.585Z        error   fatal error:failed to upgrade: failed to upgrade kplane components: failed to upgrade helm chart /sbin/helm upgrade kplane-usermgr /opt/pf9/airctl/conf/helm_charts/kplane-components-0.3.4.tgz -f /opt/pf9/airctl/conf/kplane_values.yaml -f /opt/pf9/airctl/conf/secrets.yaml: exit status 1 - Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Namespace "smcp-mgmt-kplane" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "kplane-usermgr"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
```

{% endtab %}
{% endtabs %}

4. After this, please follow [restore process of smcp](https://platform9.com/docs/v5.9/PEC/backup--restore-and-upgrade-for-smcp) with following change:

In step#7, while updating the **nodelet-bootstrap.yaml** file add the **kubedu-imgs tar** file from LTS2-Patch#2 to the **userImages** section as well. A snippet of the yaml file shown below for reference:

{% tabs %}
{% tab title="YAML" %}

```yaml
isAirgapped: true
systemImages:
- /opt/pf9/airctl/imgs/kubedu-imgs-v-5.9.0-2847602.tar.gz
- /opt/pf9/airctl/imgs/nodelet-imgs-v-5.9.0-2847602.tar.gz
userImages:
- /home/centos/patch2/kubedu-imgs-v-5.6.7-2624593.tar.gz
- /home/centos/patch4/kubedu-imgs-v-5.6.7-2658688.tar.gz
```

{% endtab %}
{% endtabs %}

## Additional Information

In some cases, especially on systems with limited resources, the container runtime can perform a garbage collection of some of the kubedu images which have not been used yet. This can cause some of the operations like airctl upgrade/upgrade-hosts to fail due to ImagePullBackOff errors.

We can determine whether the images need to be reloaded by running and making sure the images that we need for du-upgrade or host-upgrade have not been cleaned up.

{% tabs %}
{% tab title="None" %}

```none
sudo /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io images
```

{% endtab %}
{% endtabs %}

For reference, some of the images we should look for are quay.io/platform9/k8s-helm-runner and quay.io/platform9/kplane-host-upg.

If we find that images are missing we can run the following command before the upgrade/upgrade-hosts operations

{% tabs %}
{% tab title="None" %}

```none
sudo /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io load -i /var/opt/pf9/images/kubedu-imgs-v-5.9.0-2847602.tar.gz
```

{% endtab %}
{% endtabs %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://platform9.com/kb/smcp/solution/while-upgrade-from-lts2-patch2-to-scmp--management-plane-cluster.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `build a script that syncs our docs to a CMS` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.