PEC LTS2 Release

Platform9 Edge Cloud LTS 2 Patch #4

2023-05-07

Features/Updates

Fixed (1453437) Fixed an issue where the DU deployment was not working with user specified FQDNs.

Platform9 Edge Cloud LTS 2 Patch #3

2023-05-01

Features/Updates

Added (1453437) RHEL 8.6 Support

Enhanced Simplfied DU deployment architecture. The nginx-ingress image has been upgraded to 1.3.1 as part of this.

Platform9 Edge Cloud LTS 2 Patch #2

2023-04-07

Features/Updates

Added (1451770) Ability to use custom FQDNs when deploying the management plane.

Added (1451770) Support for user provided certificate authorities and certificates. Please see here for more details.

Added Various management plane and PMK cluster components have been patched with the latest security fixes.

Added (1397440) Added support for insecure registries in the containerd config.

Added Display host upgrade status with airctl.

Bug Fixes

Fixed (1451920) Fixed an issue where duplicate entries were added to the bootstrap cluster YAML every time airctl start is executed.

Fixed (1451925, 1452063) Fixed an issue where airctl status failed to run after a management plane node reboot.

Fixed Fixed an issue where subsequent runs of airctl start after an airctl unconfigure-du was failing.

Fixed (1452270) Increased the keystone pod limits to 4CPU/4GB to handle resource crunches in keystone.

Fixed Improved validation for image uploads to containerd in the management plane.

Fixed (1451770) Fixed an issue where the management plane logs were growing uncontrollably.

Fixed (1452055) Fixed an issue where the hagrid-init pod was trying to talk to the internet to fetch packages.

Fixed (1452055,1452533) Bumped up readiness and liveness probe timeouts for percona.

Fixed (1452055,1452533) Fixed an issue where warnings thrown by helm were interfering with the output parsing.

Known Issues

Known Issue The Platform9 Profile Agent is only supported on Kubernetes versions 1.22+.

2022-22-12 Platform9 Edge Cloud LTS2 Patch #1

Platform9 Kube Version:

K8s 1.21 version: 1.21.3-pmk.408

K8s 1.22 version: 1.22.9-pmk.244

K8s 1.23 version: 1.23.8-pmk.218

Airctl/Release Build Version: v-5.6.0-2481858

Features/Updates

Added Support for dual stack (IPv4 &IPv6) for the management clusters.

Added ArgoCD support in edge/air-gapped deployments.

Enhanced Removed root requirement for installation.

Enhanced Improved security by updating upstream images to the latest versions which had vulnerability fixes implemented.

Enhanced Secured airctl__ __**get-creds** with a custom password. Users now need run airctl start --password__ ____. This same password would be used for get-creds command.

Enhanced Upgraded the PF9 add-on operator to 7.0

Bug Fixes

Fixed (AIR-670)Resolved and issue where the upgrade to 1.22 does not work for calico on single node PMK cluster

Fixed (AIR-678) Migration from LTS1 to LTS2 fails if it cannot talk to the MSSQL DB.

Known Issues

Known Issue PVCs fails to get created when node hostname is larger than 42 characters, because of a limitation in hostpath provisioner and Kubernetes labels character limit of 63. This is only an issue on IPv6 and dual stack management. clusters

Known Issue If the consul write fails error "etcdserver: leader changed" then run airctl unconfigure-du --force and then run airctl start__ ____.

Known Issue If the decco-consul pod is stuck in init restart the pod to resolve the issue.

Known Issue When upgrading Luigi from 0.3 to 0.4 manual cleanup of pods is required for whereabouts and hostplumber. Run the following command to force delete.

kubectl delete pod whereabouts-XXXXX--grace-period=0 --force

2022-11-18 Platform9 Edge Cloud LTS2 Release

The Platform 9 Edge Cloud (PEC) LTS2 release is now available with support for Kubernetes 1.22 and 1.23 versions. The LTS2 release is the most significant release for Edge Cloud since becoming generally available, containing multiple architectural changes, new features, and product improvements with the intent of creating a more highly available, intuitive, scalable, and secure edge cloud.

Platform9 Kube Version:

K8s 1.21 version: 1.21.3-pmk.374

K8s 1.22 version: 1.22.9-pmk.183

K8s 1.23 version: 1.23.8-pmk.140

Airctl/Release Build Version: v-5.6.0-2338142

LTS2 Release Highlights

Containerized Management Plane

The LTS2 release brings about a significant architectural change for the on-premises control plane by migrating away from running on VM-based DUs to a containerized management plane in KDUs. PMK customers have long been able to benefit from their management plane running on KDUs, now with the release LTS2 Edge airgap deployments can now benefit from leveraging k8s constructs in their management plane.

High Availability for the Management Cluster

HA is a big requirement for Telco/air-gapped environments and is a must-have tier-1 critical applications such as management of the K8s environment. By shifting away from the legacy VM-based DU to the containerized KDUs we have also added the high availability in order to maintain access to the management plane and operational functionality. Simply meaning that the management plane can tolerate an outage operational tasks such cluster upgrades or adding a user can still occur.

Offline Pf9 Kube Support/Disconnected Mode

Enable customers to run Platform9 in remote disconnected locations for private network use cases. This ensures that the environment continues to run without issue when the connection to the management lane is broken or purposely disconnected.

Migration to ContainerD

Switching to containerd as the container runtime eliminates the middleman. All the same, containers can be run by container runtimes like containerd as before. But now, since containers schedule directly with the container runtime, they are not visible to Docker. So we cannot get container information using docker ps or docker inspectcommands. As we cannot list containers, we cannot get logs, stop containers, or execute something inside the container using docker exec. Switching to containerd as the container runtime eliminates the middleman. All the same, containers can be run by container runtimes like containerd as before. But now, since containers schedule directly with the container runtime, they are not visible to Docker. So we cannot get container information using docker ps or docker inspectcommands. As we cannot list containers, we cannot get logs, stop containers, or execute something inside the container using docker exec

In phase script we have dependencies on docker-cli, so we have to replace that with containerd cli’s. containerd already has its own CLI called ctr. However, ctr was made only for testing very low-level functionality of contained. While nerdctl has the same UI/UX as Docker

Security & Reliability

Security is not an after thought but built into Platform9's development process. When we find security vulnerabilities within Platform9 or the upstream components we proactively resolved the issues in the LTS 2 release. In addition we built the framework in this release to apply patches for components like Luigi, Calico, etc., non-disruptively.

Enhancements & Updates

Added support for containerized management plane by transitioning the Platform9 control plane from a VM to run as a Kubernetes deployment.

Added Support HA support for the management cluster.

Added Migrated to containerd from Docker.

Added Enable disconnected mode by adding support for disconnected state between cluster and management plane for large periods of time, after initial deployment.

Added The ability to schedule the IP-reconciler job on a specific node using node selector using Luigi.

Added The ability to generate a support bundle for a custom list of nodes.

Added Support for K1.22 & K1.23

Enhanced Scaled KDU (containerized management plane) to support 2000+ nodes.

Enhanced Improved security by updating upstream images to the latest versions which had vulnerability fixes implemented.

Enhanced Updated API server health checks to be more reliable by replacing healthz (deprecated) with livez for Kube API server health checks.

Enhanced Hostagent logs are now rotated out to save the last 10 files, each 1G so that the Hostagent Daemon Log is kept from consuming all free disk space.

Enhanced airctl status now lists the health of DU services.

Enhanced SSO configurations are now self-service and can be done from the UI. https://platform9.com/docs/kubernetes/enable-sso-SAML-groups#sso

Bug Fixes

Fixed (#1403076 & #1450133) Resolved an issue where keepalived had multiple masters in a cluster holding the VIP at the same time. Resolved by updating to version of keepalived v2.1.3

Fixed (AIR-412) Resolved an issue where the admin is unable to create a cluster with nodes that had multus previously Installed.

Fixed (#1398567) Resolved an issue where keepalived does not assign the VIP to any other master after electing the new leader. Resolved by updating to version of keepalived v2.1.3

Fixed (#1393131) Resolved a split-brain scenario caused by keepalived assigning VIP to all masters. Resolved by updating to version of keepalived v2.1.3

Fixed (#1392999,#1397806,#1398353,#1402635,#1403100,#1404524) Resolved an issue where there is an exec probe timeout resulting in calico pods to fail liveness/readiness probes as default timeout is 1 second. Calico upgraded to 3.23

Fixed (#1353110) Resolved an issue where the UI queries external resources from the internet.

Fixed (# 1397440) Resolved an issue where docker daemon.json needed to be configurable when docker is managed by PMK. This was resolved by moving to containerd

Fixed (#1403245) Resolved an issue when the NetworkPlugin resource removal followed by Stack Restart/Reboot causes pods to fail to start as the Multus Daemonset reads current primary CNI config and then installs itself as primary CNI by writing to file /etc/cni/net.d/00-multus.conf via init container as well as the kubeconfigs.

Known Issues

Known Issue PVCs fails to get created when node hostname is larger than 42 characters, because of a limitation in hostpath provisioner and Kubernetes labels character limit of 63.

Known Issue There is an issue where after a reboot the cluster nodes remain disconnected from DU if the FQDN is missing in /etc/hosts.

Known Issue Ensure the the SSO API uses the XML file and not the upstream file path for metadata.

Known Issue airctl start failed with error "failed to deploy region: request to initiate deployment failed: status=503". If this occurs, run airctl unconfgure-du --force then retry the start operation.

Known Issue Migration from LTS1 to LTS2 fails if it cannot talk to the MSSQL DB.

Known Issue The KDU does not run in dual stack mode in LTS2.

Known Issue airctl get-creds is still unauthenticated.

Known Issue Calico runs in non-privileged mode but requires the privileged flag the initial setup.

Known Issue ArgoCD is not running in the LTS2 release.

Known Issue The DU can only be deployed with a root user.

Known Issue The metrics server is running 0.5.0. This will be upgraded in the next patch to 0.5.2

Known Issue Intermittent issue where Pods do not come up healthy after the node reboots (on the DU).

Known Issue Upgrade to 1.22 does not work for a single node PMK cluster.

Package Updates

The following packed components have been upgraded in latest v1.23.8 Kubernetes version:

Please refer to the Managed Kubernetes Support Matrix for v5.6 to view all currently deployed or supported upstream component versions.