PMK Release 5.9 Release Notes

List of Kubernetes versions that are supported on PMK 5.9.x series:

PMK 5.9.0 Release Summary (Released 2023-09-12)

The Platform9 Managed Kubernetes (PMK) version 5.9 release is now generally available with support for Kubernetes v1.25. This release brings new features, enhancements and critical bug fixes to improve overall user experience and stability.

Release Highlights

New Features

• Added support to upgrade Platform9 DU directly from 5.7.2 to 5.9, instead of having to go through 5.8 as well.
• Added support for RHEL 8.7, Ubuntu 22.04 & Rocky in PMK for BareOS clusters. Read more on Supported OS here: Managed Kubernetes Support Matrix
• Added support to choose a custom network interface during cluster creation for BareOS clusters.
• Added support for GP3 volumes in AWS clusters. It will be the default storage type going forward for AWS clusters.
• Added Kubernetes Version support status information on the UI. Cluster versions are categorized in End of Life, Deprecated and Supported versions. Read More here: Managed Kubernetes Support Matrix
• Added changes in response traffic redirection from the older k8s.gcr.io to registry.k8s.io. https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
• Improved clusters view with consolidated clusters list & new row actions.

• Added Advance Networking configuration options in the UI.

• Added new options to left navigations in UI. Users can now manage Configurations such as ConfigMaps and Secrets & Advanced networking kubernetes CRDs from UI.

Deprecations and Feature Removal and EOL information

• Managed Argo CD and Arlon are removed from the PMK 5.9. Users can still deploy Argo CD in PMK clusters similar to any other application.

Platform9 CLI

The 1.23 pf9ctl release is now available and can be installed by running the following command.

bash <(curl -sL https://pmkft-assets.s3-us-west-1.amazonaws.com/pf9ctl_setup)

• Added an option in PF9Cli to move an existing node behind proxy.

Bug Fixes

Fixed Fixed an issue due to which certificate generation fails if CA validity is less than the amount of TTL with which the certificate is attempted to be generated in Vault

Fixed Fixed a bug which prevented Sunpike CA rotation if the CA had the expiry of more than a year.

Fixed Fixed an issue that cause some pods to be stuck in a Terminating state on RHEL nodes, due to missing sysctl parameter /proc/sys/fs/may_detach_ mounts set to 1

Fixed Fixed issue with Prometheus when cluster is created using Node Hostname option.

Fixed Removed the hostpath-csi-driver from our appcatalog.

Fixed Fixed the issue which caused Calico BIRD CPU usage spike.

Fixed Fixed a bug which prevented the persistence of the etcd logs on the master nodes in case of etcd container failure. Not applicable if the runtime is docker. Docker runtime is no longer supported.

Fixed Fixed the issue of seeing stale clusteraddon entries for removed/deleted clusters.

Fixed Fixed an issue where CAPI controller pods are crashing as they were getting invalid tokens issued.

Fixed Fixed the format of etcd logfile name.

Fixed Fixed the issue of allowing hosts to be downgraded, which is not a supported action.

Fixed Fixed an issue with cluster upgrade getting stuck if qbert pod restarts during the upgrade.

Fixed Fixed the issue of Nodelet failing to update Dynamic Kubelet Configuration from Sunpike in 1.24 version.

Fixed Fixed the issue where prometheus alert is not fired for PersistsentVolumeClaim full event.

Fixed Fixed a bug on pf9ctl which caused the check to fail when the package python3-policycoreutils pre-existed on the nodes.

Fixed Fixed issue with executing Kubectl logs/exec when the cluster is configured with httpproxy

Fixed Updated the Kubernetes Dashboard version to support 2.4 version for v1.23 and v1.24 k8s clusters.

Fixed Fixed issue where alerts configured for pod eviction events are not getting triggered.

Fixed Fixed a bug in pf9ctl which caused an unintended node to be decommissioned due to wrong choice of node uuid based on IP obtained from hostname -I

Fixed Fixed the issue with decommission node not draining the node. We explicitly warn the users now to detach the node before decommisioning.

Fixed Fixed an issue preventing the reuse of PVC attached for prometheus after addon upgrade.

Fixed Fixed the KubeStateMetrics addon version,1.24 clusters use 2.6.0 and 1.25 clusters use 2.7.0 here onwards.

Fixed Fixed an issue which caused the calico Typha pods on a customer’s cluster were getting OOM killed due to resource constraints. Updates made to current limits and Typha auto scaler ladder.

Fixed Updated the resource limits for CoreDNS to support higher scale.

Fixed Fixed an issue with prometheus pod crashing when a PVC backend is configured

Fixed Fixed issue of EtcdInsufficientMembers alert getting triggered frequently.

Fixed Fixed a bug in UI where role assignment data was not being shown correctly when editing a user.

Fixed Fixed a bug in UI which prevented the Self-service users with access to a particular namespace from being able to view resources in that namespace.

Fixed Fixed a bug in UI which prevented Add Topology Manager support option from being honoured, for BareOS Cluster Creation

Fixed Fixed a bug in UI which prevented the kube-apiserver flags to be added to apiserver configuration during cluster creation from UI.

Package Updates

The following packed components have been upgraded in latest v1.25.11 Kubernetes version:

Please refer to the Managed Kubernetes Support Matrix for v5.9 to view all currently deployed or supported upstream component versions.

Known Issues

Known Issue AWS New generations(CAPI) cluster creation might fail in some instance where the DU is upgraded from an earlier version to 5.9.

Known Issue AWS clusters using flannel CNI need to be updated to use port 2379 instead of 4001 from1.22 version onwards. Workaround is to go to the “Edit cluster” option on the UI and clicked on “Update cluster” without making any changes. This adds the 2379 ingress rule to the master ELB.

Known Issue When a detach operation is performed on a master node in a multi master cluster, it takes approximately 30 minutes to complete all the detach operations and perform cleanup on the node. Therefore, if you want to reattach this node to any other cluster, you need to wait for the nodelet to stop all the phases and perform cleanup before attempting to reattach the node.

Known Issue In some scenarios, after a node is removed from the qbert clusters, nodelet fails to cleanup the data. Workaround is to check and remove the /var/opt/pf9/kube directory if present, even after the node is deauthorized.

Known Issue AWS New generations(CAPI) cluster creation might fail in some instance where the DU is upgraded from an earlier version to 5.9.

Known Issue On k8s 1.24 cluster, Generate certs phase fails when more than 1 default route is configured on the node. To resolve this duplicate default routes need to be deleted so that ip commands correctly picks up just one default route for interface discovery. This is resolved in k8s 1.25.

Known Issue External Clusters with name more than 63 characters will not be discoverable for Import operation. This is due to the limitation from K8s specs.

Known Issue Calico IPAM is only supported when using Calico CNI.

Known Issue EKS, AKS, or GKE Cluster Import “401 Unauthorized” Notification and Empty Dashboards.

If an AWS Cloud Provider is configured to import clusters without the correct identity being added to the target cluster, Platform9 will be unable to access the cluster.

It's important to note that if you have used a Cloud Provider to register an EKS, AKS, or GKE cluster that was created with IAM user credentials, which no longer have access to the EKS, AKS, or GKE K8s clusters, Platform9 will fail with an 401 unauthorized error until that IAM user is given access to the K8s cluster.

View the EKS documentation here to ensure the correct access has been provisioned at for each imported cluster. https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/

Known Issue Platform9 monitoring won't work on ARM-based nodes on EKS, AKS, or GKE.

Known Issue Cluster upgrade attempt is blocked on UI post a cluster upgrade failure due to nodes being in a converging/not converged state.

Known Issue Kubelet authorization mode is marked set to AlwaysAllow instead of Webhook.

Known Issue UI throws error when using SSO with Azure AD and passwordless logins.

Known Issue PMK Cloud provider created directly in Sunpike cannot be used to create qbert clusters. Qbert cloud providers will work to create both qbert and sunpike clusters. But cloud providers created directly in sunpike CANNOT be used to create qbert clusters. Please use the appropriate one based on your needs.

Known Issue Certificate generation fails if CA validity is less than the amount of TTL with which certificate is attempted to be generated in Vault

CAPI AWS & EKS clusters

Known Issue For CAPI based clusters, labels with special characters like - cannot be associated to the cluster. A known bug leads to only one part of the label being used when applying it to the cluster.

Known Issue For CAPI based clusters, downloading multiple kubeconfigs for a given user will invalidate the previous kubeconfigs of that user. Only the latest downloaded kubeconfig will be valid for that user account to use. User are recommended to use caution when sharing user accounts for such clusters.

Known Issue Users should not make changes to EKS clusters created using PMK, from AWS Console. Platform9 manages the lifecycle of the EKS control-plane and EKS nodes. Making any changes from AWS Console might result in undesired effects or render cluster in non-functional state.

Known Issue The Kubernetes dashboard is not accessible for CAPI based clusters by uploading the kubeconfig, because of an upstream issue where dashboard does not support OIDC-based kubeconfigs.

As a workaround, authenticate with the ID Token.

1. Download and open the kubeconfig of a cluster.
2. Copy the value of the id_token field.
3. In the dashboard, select "token" authentication and paste the value in the form.

Note: refreshing is not supported by the dashboard, this means you lose access after the token expires (10-20 min)

To refresh the ID token, simply run a kubectl command with it. kubectl will replace the ID token in the kubeconfig with a valid one if it has expired. Then afterwards follow the steps above again.

Known Issue EKS cluster created with with MachinePool / MachineDeployment type of worker node groups with desired node count can sometimes get stuck in “provisioned” state, for more than an hour after control plane is ready.

Current workarounds:

• Machine Deployment type Node Group: Scale down the affected node group to 0 and then scale it back up to desired count.
• Machine Pool type Node Group: Delete the affected node group and add new node group to the cluster.
• Create an EKS cluster with MachinePool / MachineDeployment with replica count as 0. Once the cluster is healthy change the replica count to desired value.

Known Issue In some instances a CAPI cluster when deleted during provisioning phase can get stuck into deleting phase. Please contact Platform9 support for possible options of resolution.

Known Issue CAPI Cluster creation in EKS with Private API Server Endpoint access is not supported. Will be added in a futures release.

Platform9 Managed KubeVirt

The Platform9 Managed KubeVirt version 5.9 release is now generally available with support for Kubernetes v1.25 and Kubevirt v0.59.2 bringing new features, enhancements and bug fixes. Please refer Platform9 Managed KubeVirt 5.9 Release Notes for more information.

PMK 5.9.1 Patch Update (Released 2023-11-27)

Added Kubernetes 1.26 promoted to active state from early access.

Added Added security patches for components in Kubernetes 1.26 and Kubernetes 1.25.

Added Added a feature to configure containerd to pull images from a custom registry mirror. Documentation link.

Added Kube versions for 1.25 and 1.26 have been patched to 1.25.15 & 1.26.10 respectively.

Added Added two new alerts to catapult monitoring: "KubePodOOMKilled" to alert if a pod crashes due to OOMKill events, and "KubePodCrashLoopingThreshold" to alert if any specific pod restarts for more than the threshold value, in a span of 15 minutes.

Fixed Fixed a bug where nodelet ignores a disk of size larger than 100 GB for mounting runtime images.

Fixed Increased limits for kube-state-metrics pod.

Fixed Calico version has been updated to incorporate upstream bug fixes.

Fixed Fixed a bug in the add-on operator where it detects deletion of managed resources, but does not recreate them.

Fixed Fixed a bug in the add-on operator where it scaled down the pod replicas which were scaled up by the Coredns autoscaler.

Fixed Fixed a bug where nodelet does not handle partitioned devices for mounting runtime images path.

Fixed Fixed a bug where the add-on operator frequently deletes and recreates metrics server deployment, whenever it detects a resource version change.

Fixed Upgraded kube-prometheus-stack version to v52.1.0.

Known Issue PMK nodelet picks devices larger than 100GB for containerd mount. It should support partitions and lvm volumes.

Known Issue Kubectl logs command may fail for certain pods when used on the clusters created using the feature that allows users to choose a specific network-interface for cluster creation

Known Issue The metrics server addon may show in error state on the UI when upgrading kube version from 1.24 to 1.25.

PMK 5.9.2 Patch Update (Released 2024-02-20)

Added Added support for Kubernetes v1.27 as active version with build 1.27.8-pmk.47. Note: We have identified an issue where 1.27 go-client or kubectl CLI are not able to process the API response, when using the Platform9 proxy context. While we are working to address this issue, the workaround is to use 1.25 or lower version of the client if not using 'default' context.

Added Added Operating System support for RHEL 8.8.

Added Kube versions for 1.25 and 1.26 have been patched to 1.25.15 & 1.26.10 respectively.

Fixed Fixed an issue that caused MetalLB add-on to go in error, due to configuration moving from ConfigMap to Custom Resources in new MetalLB version(v0.13.11) during Kubernetes 1.24 to 1.25 upgrade.

Fixed Fixed an issue with clusters using custom interface defaulting the interface detection method to FirstFound upon upgrade to Kubernetes 1.25 or higher.

Fixed Fixed an issue with support bundle generation due to change in permissions of the /var/lib/containerd directory.

Fixed Fixed an issue which caused Master node to be stuck in NotReady state after reboot on a partially upgraded cluster with forced cert rotation. Added API validations to ensure that only worker nodes can be specified in batchUpgradeNodes parameter during cluster ugprade.

Fixed Added log rotation for kube-proxy and bouncer log files and moved them to /var/log/pf9/kube directory.

Fixed Fixed the issue causing kubectl logs to fail for certain pods when using the custom interface to create cluster.

Fixed Fixed the issue causing the metrics server add-on to show in error state on the UI when upgrading kube version from 1.24 to 1.25.

Fixed Fixed the issue with incorrect file encoding for sunpike causing the host agent to be disconnected from management plane in some cases.

Fixed Fixed an issue with nodeport assignment at scale by increasing nofile ulimit for kube-proxy containers.

Fixed Fixed a race condition in nginx causing host package download failure intermittently.

Known Issue For Management Planes configured with SSO login, PMK upgrade from 5.9.0/5.9.1 to 5.9.2 fails. Workaround is available to upgrade to 5.9.2, please reach out to Platform9 support before upgrading to PMK 5.9.2.

Known Issue Specifying a custom interface using Interface Detection Method option (not the one for Calico CNI) at a cluster level fails new cluster creation due to etcd certificates using primary interface IP.

Known Issue Grafana 'Kubernetes' Dashboard fails to load due to changed metric names with kube-state-metrics v2.0

Known Issue False positive alerts might get generated by cluster monitoring AlertManager for alert KubePersistentVolumeErrors.

Known Issue Force remove action on the does not detach from clusters. Users need to detach nodes from the clusters before using force remove.

Known Issue Accessing PMK clusters using pf9 context in kubeconfig might fail for pmk kuberole 1.27.8-pmk.47.

Known Issue Grafana UI fails to load Kubernetes dashboard metrics for clusters with more than 50 nodes.

Known Issue For nodes in clusters using Ubuntu18 as OS, post docker to containerd (k8s 1.24) upgrade, PMK does not utilize docker mountpoint for containerd.

Known Issue Whereabouts IP reconciler schedule is not customizable in v0.6 used in PMK and requires upgrade to higher version.

Known Issue Setting streamingConnectionIdleTimeout via dynamic kubelet config is not respected by containerd.

Known Issue Disabling the pf9-monitoring add-on deletes CRDs of Prometheus. This may impact any other prometheus deployment running along with platform9 monitoring add-on.

Known Issue Wrong Interface gets assigned to kube-apiserver for master nodes with multiple-interfaces defined in the V4 interface file. The interface defined in the file gets used correctly for ETCD and all other components except the Kube API server.

Known Issue IPv6 enabled Calico-node PODs "/proc/sys/net/ipv6/conf/*/accept_ra: read-only filesystem" warnings floods logs.

PMK 5.9.2 Latest Kubernetes Components List

PMK 5.9.3 Patch Update (Released 2024-03-28)

Added Added support for Kubernetes v1.28 as active version with build 1.28.6-pmk.26.

Added Kube versions for 1.25, 1.26 and 1.27 have been patched to 1.25.16, 1.26.14 and 1.27.10 respectively.

Fixed Fixed the issue where the Whereabouts IP reconciler schedule was not customizable in v0.6 used in PMK.

Fixed Fixed an issue where Nodelet Phases failed to retrieve Node IP with VLAN Interface on bonded NIC.

Fixed Fixed an issue which caused Grafana 'Kubernetes' Dashboard to fail to load due to changed metric names with kube-state-metrics v2.0.

Fixed Fixed an issue that caused Grafana UI to fail to load Kubernetes dashboard metrics for clusters with more than 50 nodes.

Known Issue Host upgrade may fail if the nginx server on the DU becomes temporarily unavailable during the upgrade.

Known Issue Removing an address pool from Metallb addon configuration does not delete the IP address pool from the cluster. Workaround is to update the address pools in the IPAddressPool Custom Resource present on the cluster.

Known Issue Proxy configuration file ( /etc/systemd/system/containerd.service.d/00-pf9-proxy.conf ) gets left over even after node decommission, not allowing the node to be re-used in another cluster. Workaround is to manually delete the file on the node.

Known Issue An intermittent issue might be seen in the case when more than 50 nodes are onboarded at the same time.

Known Issue For Management Planes configured with SSO login, PMK upgrade from 5.9.0/5.9.1 to 5.9.3 fails. Workaround is available to upgrade to 5.9.3, please reach out to Platform9 support before upgrading to PMK 5.9.3.

Known Issue Specifying a custom interface using Interface Detection Method option (not the one for Calico CNI) at a cluster level fails new cluster creation due to etcd certificates using primary interface IP.

Known Issue False positive alerts might get generated by cluster monitoring AlertManager for alert KubePersistentVolumeErrors.

Known Issue Force remove action on the does not detach from clusters. Users need to detach nodes from the clusters before using force remove.

Known Issue For nodes in clusters using Ubuntu18 as OS, post docker to containerd (k8s 1.24) upgrade, PMK does not utilize docker mountpoint for containerd.

Known Issue Setting streamingConnectionIdleTimeout via dynamic kubelet config is not respected by containerd.

Known Issue Disabling the pf9-monitoring add-on deletes CRDs of Prometheus. This may impact any other prometheus deployment running along with platform9 monitoring add-on.

Known Issue Wrong Interface gets assigned to kube-apiserver for master nodes with multiple-interfaces defined in the V4 interface file. The interface defined in the file gets used correctly for ETCD and all other components except the Kube API server.

Known Issue IPv6 enabled Calico-node PODs "/proc/sys/net/ipv6/conf/*/accept_ra: read-only filesystem" warnings floods logs.

PMK 5.9.3 Latest Kubernetes Components List

PMK 5.9.4 Patch Update (Released 2024-09-26)

Added The MySQL8 database used in PMK management plane deployment is upgraded behind the scenes to use the new MySQL8 database engine, since MySQL 5.7 reached EoL

Process Change Support Bundle generation and upload process is updated now with following changes:

• Starting PMK 5.9.4 and above versions, the generated support bundle will be gpg encrypted. The encryption key from https://gpg.platform9.com/publickey.txt is sourced and placed at /etc/pf9/public_key.asc on hosts.
• Users are required to upload only the encrypted support bundles to Platform9 support.
• Generated support bundles are now redacted of any sensitive data such as certificates, token, passwords etc.
• The encrypted support bundle file name contains fingerprint of the public key used to encrypt the bundle using the following pattern: /tmp/pf9-support.tgz.<GPG key fingerprint>.gpg
• pf9ctl does not support support bundle generation and auto-uploads with PMK 5.9.4+ and 5.10.2+
• Procedure to generate a support bundle using the datagatherer script can be found here.
• Please reach out to Platform9 support to get on-boarded to the new the upload process.

Known Issue All existing and new AWS clusters in PMK must be configured with an is_update flag and restricted security group rules. Without this cluster updates(such as AMI updated) and upgrades may fail. Please reach out to Platform9 support for this configuration.

PMK 5.9.4 Latest Kubernetes Components List