How to Upgrade PMK Cluster in Batches

Problem

How to Upgrade PMK Cluster in Batches?

Environment

Platform9 Managed Kubernetes - v5.4 and Higher
Qbert v4 API

Procedure

Fetch Keystone authentication token in order to access the PMK REST API by following Keystone Identity API.
Check if the PMK Cluster is available for an upgrade.

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/clusters/<CLUSTER_UUID>

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": true,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.20.11-pmk.2064",
  "canMinorUpgrade": 1,
  "minorUpgradeRoleVersion": "1.21.3-pmk.72",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": ""

Info

Cluster is currently on version 1.20.11-pmk.2064 and there is a minor upgrade available to version 1.21.3-pmk.72.

Cluster Node Information

Master Node

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='1')'

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",

Worker Node

curl --header "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/nodes | jq '.[] | select(.clusterUuid=="<CLUSTER_UUID>" and .isMaster=='0')'

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",

Performing Batch Cluster Upgrade

Important

As part of the Upgrade API with Batch Option, only Worker Nodes can/should be specified in the batchUpgradeNodes parameter.

The Master Nodes will always be upgraded first in a sequential manner followed by the worker nodes specified in the batchUpgradeNodes argument.

curl --request POST \
 --url https://platform9.io/qbert/v4/{project_uuid}/clusters/{uuid}/upgrade \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --header "content-type: application/json" \
 --data '{
 "batchUpgradeNodes": [
  "{array[string]...}"
 ],
}'

curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["522b7a45-b98a-4990-9482-826beab2dfbf"] }'

OK

Note

If Patch type cluster upgrade is available and that is desired, the same needs to be specified in the API call as upgrade/\?type\="patch"

--header "content-type: application/json" should be specified as shown above, otherwise the API call will upgrade all nodes in a sequential manner.

The above example API upgraded the master nodes within the cluster first followed by the provided single worker node in batch.

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='1')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72"

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90" and .isMaster=='0')' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.20.11-pmk.2064",

Upgrade the remaining worker nodes as part of the next batch.

curl --header "X-Auth-Token: $TOKEN" --header "content-type: application/json" --request POST https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90/upgrade/\?type\="minor" --data '{ "batchUpgradeNodes": ["64e08702-041a-43fb-a708-00cb02112f19", "f3b0cbb0-efb9-407e-939a-21809421c376"] }'

OK

Post Upgrade Status

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/clusters/41596ba7-b6e5-4008-85e4-42080f43eb90 | jq | grep -e name -e uuid -e projectId -e kubeRoleVersion -e canUpgrade -e canMinorUpgrade -e minorUpgradeRoleVersion -e canPatchUpgrade -e patchUpgradeRoleVersion

  "name": "batch-upgrade-test",
  "uuid": "41596ba7-b6e5-4008-85e4-42080f43eb90",
  "canUpgrade": false,
  "projectId": "86f43d7fc7ce4287a6be1d861a9ac8b4",
  "kubeRoleVersion": "1.21.3-pmk.72",
  "canMinorUpgrade": 0,
  "minorUpgradeRoleVersion": "",
  "canPatchUpgrade": 0,
  "patchUpgradeRoleVersion": "",

curl --header "X-Auth-Token: $TOKEN" https://sanchit.platform9.horse/qbert/v4/86f43d7fc7ce4287a6be1d861a9ac8b4/nodes | jq '.[] | select(.clusterUuid=="41596ba7-b6e5-4008-85e4-42080f43eb90")' | grep -e uuid -e actualKubeRoleVersion

  "uuid": "522b7a45-b98a-4990-9482-826beab2dfbf",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "ba53cb28-14ab-4406-b352-9dd6a1314320",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "64e08702-041a-43fb-a708-00cb02112f19",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

  "uuid": "f3b0cbb0-efb9-407e-939a-21809421c376",
  "actualKubeRoleVersion": "1.21.3-pmk.72",

PreviousHow To Forcefully Remove a Terminating Namespace Stuck Due to Finalizers.

NextHow-To Implement CPU and Memory System Reservation for Kubernetes Nodes

Last updated 9 months ago