# How to Upgrade PMK Cluster in Batches

## Problem

How to Upgrade PMK Cluster in Batches?

## Environment

* Platform9 Managed Kubernetes - v5.4 and Higher
* Qbert v4 API

## Procedure

1. Fetch Keystone authentication token in order to access the PMK REST API by following [Keystone Identity API](https://platform9.com/docs/kubernetes/keystone-identity-api).
2. Check if the PMK Cluster is available for an upgrade.

{% tabs %}
{% tab title="GET Cluster API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/clusters/<CLUSTER_UUID>
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example GET Cluster API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": true,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.20.11-pmk.2064",
  "canMinorUpgrade": 1,
  "minorUpgradeRoleVersion": "1.21.3-pmk.72",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": ""
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

Cluster is currently on version *1.20.11-pmk.2064* and there is a minor upgrade available to version *1.21.3-pmk.72*.
{% endhint %}

3. Cluster Node Information

* Master Node

{% tabs %}
{% tab title="GET Master Node API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='1')'
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example GET Master Node API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
```

{% endtab %}
{% endtabs %}

* Worker Node

{% tabs %}
{% tab title="GET Worker Node API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='0')'
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example GET Worker Node API" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
```

{% endtab %}
{% endtabs %}

4. Performing Batch Cluster Upgrade

**Important**

As part of the Upgrade API with Batch Option, *only Worker Nodes* can/should be specified in the `batchUpgradeNodes` parameter.

The *Master Nodes* will always be upgraded first in a sequential manner followed by the worker nodes specified in the *batchUpgradeNodes* argument.

{% tabs %}
{% tab title="Upgrade API with Batch Option" %}

```bash
curl --request POST \
 --url https://platform9.io/qbert/v4/{project_uuid}/clusters/{uuid}/upgrade \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --header "content-type: application/json" \
 --data '{
 "batchUpgradeNodes": [
  "{array[string]...}"
 ],
}'
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example Upgrade API with Batch Option for Minor Type" %}

```bash
curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["522b7a45-b98a-4990-9482-826beab2dfbf"] }'

OK
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Note**

If Patch type cluster upgrade is available and that is desired, the same needs to be specified in the API call as `upgrade/\?type\="patch"`

`--header "content-type: application/json"` should be specified as shown above, otherwise the API call will upgrade all nodes in a sequential manner.
{% endhint %}

* The above example API upgraded the master nodes within the cluster first followed by the provided single worker node in batch.

{% tabs %}
{% tab title="Example Master Node Status" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72"
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example Worker Node Status" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",
```

{% endtab %}
{% endtabs %}

* Upgrade the remaining worker nodes as part of the next batch.

{% tabs %}
{% tab title="Example Upgrade API with Batch Option for Minor Type on Remaining Worker Nodes" %}

```bash
curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["64e08702-041a-43fb-a708-00cb02112f19", "f3b0cbb0-efb9-407e-939a-21809421c376"] }'

OK
```

{% endtab %}
{% endtabs %}

* Post Upgrade Status

{% tabs %}
{% tab title="Example Cluster Status Post Upgrade" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": false,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.21.3-pmk.72",
  "canMinorUpgrade": 0,
  "minorUpgradeRoleVersion": "",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": "",
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Example Node Status Post Upgrade" %}

```bash
curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90")' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.21.3-pmk.72",
```

{% endtab %}
{% endtabs %}