How to Upgrade PMK Cluster in Batches

Problem

How to Upgrade PMK Cluster in Batches?

Environment

• Platform9 Managed Kubernetes - v5.4 and Higher

• Qbert v4 API

Procedure

1. Fetch Keystone authentication token in order to access the PMK REST API by following Keystone Identity API.

2. Check if the PMK Cluster is available for an upgrade.

GET Cluster API

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/clusters/<CLUSTER_UUID>

Example GET Cluster API

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": true,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.20.11-pmk.2064",
  "canMinorUpgrade": 1,
  "minorUpgradeRoleVersion": "1.21.3-pmk.72",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": ""

Info

Cluster is currently on version 1.20.11-pmk.2064 and there is a minor upgrade available to version 1.21.3-pmk.72.

1. Cluster Node Information

• Master Node

GET Master Node API

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='1')'

Example GET Master Node API

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",

• Worker Node

GET Worker Node API

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='0')'

Example GET Worker Node API

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",

1. Performing Batch Cluster Upgrade

Important

As part of the Upgrade API with Batch Option, only Worker Nodes can/should be specified in the batchUpgradeNodes parameter.

The Master Nodes will always be upgraded first in a sequential manner followed by the worker nodes specified in the batchUpgradeNodes argument.

Upgrade API with Batch Option

curl --request POST \
 --url https://platform9.io/qbert/v4/{project_uuid}/clusters/{uuid}/upgrade \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --header "content-type: application/json" \
 --data '{
 "batchUpgradeNodes": [
  "{array[string]...}"
 ],
}'

Example Upgrade API with Batch Option for Minor Type

curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["522b7a45-b98a-4990-9482-826beab2dfbf"] }'

OK

Note

If Patch type cluster upgrade is available and that is desired, the same needs to be specified in the API call as upgrade/\?type\="patch"

--header "content-type: application/json" should be specified as shown above, otherwise the API call will upgrade all nodes in a sequential manner.

• The above example API upgraded the master nodes within the cluster first followed by the provided single worker node in batch.

Example Master Node Status

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72"

Example Worker Node Status

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",

• Upgrade the remaining worker nodes as part of the next batch.

Example Upgrade API with Batch Option for Minor Type on Remaining Worker Nodes

curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["64e08702-041a-43fb-a708-00cb02112f19", "f3b0cbb0-efb9-407e-939a-21809421c376"] }'

OK

• Post Upgrade Status

Example Cluster Status Post Upgrade

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": false,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.21.3-pmk.72",
  "canMinorUpgrade": 0,
  "minorUpgradeRoleVersion": "",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": "",

Example Node Status Post Upgrade

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90")' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.21.3-pmk.72",