Clusters

PCD uses Kubernetes Cluster API to create and manage the lifecycle of the Kubernetes Clusters.

What is CAPI?

Cluster API (CAPI) is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

Started by the Kubernetes Special Interest Group (SIG) Cluster Lifecycle, the Cluster API project uses Kubernetes-style APIs and patterns to automate cluster lifecycle management for platform operators. The supporting infrastructure, like virtual machines, networks, load balancers, and VPCs, as well as the Kubernetes cluster configuration are all defined in the same way that application developers operate deploying and managing their workloads. This enables consistent and repeatable cluster deployments across a wide variety of infrastructure environments.

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

For more information about Cluster API, read The Cluster API Book.

PCD CAPI Integration

PCD Cluster API Integration offers you the portability and extensibility of the open-source Cluster API(CAPI) project, with the streamlined and fully-managed experience of Platform9.

PCD integrates Cluster API into Platform9's SaaS Management plane so that you don't have to manage CAPI lifecycle; which means cluster provisioning & lifecycle-management is easy and out of the box for you.

PCD Cluster API Concepts

PCD SaaS Management Plane Cluster

Cluster API requires a Kubernetes cluster known as a Management Plane Cluster that manages the lifecycle of Workload Clusters. It is where one or more Infrastructure Providers run, and resources such as Machines are stored.

Platform9 has integrated the Management Cluster capabilities in the Platform9 SaaS Management Plane so that you don't have to run and manage the life cycle of the management cluster. It also means that all the Cluster API components are always up to date and managed for you.

Workload cluster

A workload cluster is a Kubernetes cluster whose lifecycle is managed by a CAPI Management Cluster.

Infrastructure provider

Infrastructure providers are sources of computational resources, such as compute and networking. For example, Infrastructure Providers include OpenStack Provider which provisions resources on OpenStack. They run as controllers along with other CAPI services and serve as interfaces between the management cluster and the IaaS Platform.

Bootstrap provider

The Bootstrap Provider is responsible for:

• Generating the cluster certificates, if not otherwise specified.
• Initialising the control plane, and gating the creation of other nodes until it is complete.
• Joining control plane and worker nodes to the cluster.

PCD uses upstream Kubernetes Kubeadm bootstrap provider.

Control plane

Platform9 uses its own provider HostedControlPlane to bootstrap the control plane, which provides a managed Kubernetes controlplane experience, and is hosted in PCD management plane.

Custom Resource Definitions

A CustomResourceDefinition (CRD) is a built-in resource that lets you extend the Kubernetes API. Each CustomResourceDefinition represents a customization of a Kubernetes installation.

CRDs are CAPI specific, Provider Specific(Infrastructure, Bootstrap, Control Plane etc) or PCD specific.

A Group of Custom Resources(CRs - Instances of CRDs) define and construct a CAPI cluster.

Organization Namespace

By default PCD creates namespaces for tenant organizations in the management plane which is also a Kubernetes cluster. You also start with a namespace created for your organization/tenants to isolate your Kubernetes resources from other tenants on the same management plane.

HCP class

HCP class is the a core PCD API that lets you define the core specifications for your Hosted Control Planes. You can define multiple HCPClass resources to provide different capabilities for control planes for different users.

Hosted Control Plane

Hosted Control Plane is a Kubernetes Control Plane that manages multiple Kubernetes clusters using control planes as pods on a hosted on the management plane cluster without the need for dedicated virtual or physical machines for each control plane.

Benefits of PCD Cluster API Integtration

• Multi-tenancy: PCD enables multi-tenancy which provides enterprises to the ability to host and manage multiple, isolated tenants (or users) within the same platform, where each tenant has its own distinct resources and operational environment. This architecture allows different organizations, departments, or teams to operate on the same underlying infrastructure but remain logically/physically isolated from one another.
• Hosted Control Plane: Hosted Control Plane is a Kubernetes Control Plane that manages multiple Kubernetes clusters using control planes as pods on a hosted on the management cluster without the need for dedicated virtual or physical machines for each control plane.
• IAM & RBAC management: PCD provides mechanisms to ensure that the right users have access to the right resources without compromising security or isolation between tenants. Its also helps manage integration with external identity providers.
• Add-on management: In PCD, Kubernetes add-on management is an essential feature that enhances cluster functionality by enabling the deployment and management of key services and integrations on top of Kubernetes. Add-ons provide additional capabilities such as monitoring, logging, networking, and security without the need to modify the core Kubernetes platform.
• Declarative resource management: PCD enables declarative resource management as all the APIs are kubernetes native, which means that user can user cloud native solutions such as GitOps using ArgoCD/Flux to manage and scale clusters.

Example of CRD based PCD Cluster API based Kubernetes cluster

apiVersion: v1

items:

- apiVersion: controlplane.platform9.io/v1alpha1

  kind: HCPClass

  metadata:

    creationTimestamp: "2024-10-16T11:50:16Z"

    generation: 2

    labels:

      app.kubernetes.io/managed-by: kustomize

      app.kubernetes.io/name: hostedcontrolplaneprovider

      hcpclass.controlplane.platform9.io/default: "true"

    name: hcpclass-sample

    resourceVersion: "72975"

    uid: 91ebd9a7-3db7-401a-8a29-0312e869b298

  spec:

    dataStore: default

    deployment:

      additionalMetadata: {}

      extraArgs: {}

      extraVolumes:

      - hostPath:

          path: /etc/ssl/certs

          type: Directory

        name: etc-ssl-certs-from-host

      podAdditionalMetadata: {}

      strategy: {}

    kine:

      resources:

        limits:

          cpu: 100m

    network:

      serviceAnnotations:

        metallb.universe.tf/address-pool: hcp-pool

        service.beta.kubernetes.io/openstack-internal-load-balancer: "false"

      serviceType: LoadBalancer

    registry: registry.k8s.io

    replicas: 1

---

apiVersion: cluster.x-k8s.io/v1beta1

kind: Cluster

metadata:

  name: k1

  namespace: org-platform9-product-team

  labels:

    core-addons: enabled

    proxy-addons: enabled

spec:

  controlPlaneEndpoint:

    host: 172.29.22.200

    port: 443

  clusterNetwork:

    apiServerPort: 443

    pods:

      cidrBlocks:

        - 10.244.0.0/16

    services:

      cidrBlocks:

        - 10.96.0.0/16

  controlPlaneRef:

    apiVersion: controlplane.platform9.io/v1alpha1

    kind: HostedControlPlane

    name: k1cp

  infrastructureRef:

    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1

    kind: OpenStackCluster

    name: k1

---

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1

kind: OpenStackCluster

metadata:

  name: k1

  namespace: org-platform9-product-team

spec:

  apiServerLoadBalancer:

    enabled: false

  controlPlaneAvailabilityZones:

  - nova

  disableAPIServerFloatingIP: true

  disableExternalNetwork: true

  apiServerFixedIP: ""

  network:

    id: 2ed5a3d2-e1fe-4b1d-a2ce-6f33d2c1c2ab

  subnets:

  - id: 2b53fc07-4f94-471c-96c2-d829c261ddea

  identityRef:

    name: k1

    cloudName: openstack

  managedSecurityGroups:

---

apiVersion: controlplane.platform9.io/v1alpha1

kind: HostedControlPlane

metadata:

  name: k1cp

  namespace: org-platform9-product-team

spec:

  apiServer:

    extraArgs:

      - --cloud-provider=external

      - --advertise-address=10.96.0.40

  controllerManager:

    extraArgs:

      - --cloud-provider=external

  addons:

    coreDNS: {}

    konnectivity:

      server:

        port: 8132

  kubelet:

    preferredAddressTypes:

      - InternalIP

      - ExternalIP

      - Hostname

    cgroupfs: systemd

  version: 1.29.6

  hcpClass: hcpclass-sample

---

apiVersion: cluster.x-k8s.io/v1beta1

kind: MachineDeployment

metadata:

  name: k1-md-0

  namespace: org-platform9-product-team

spec:

  clusterName: k1

  replicas: 3

  template:

    spec:

      bootstrap:

        configRef:

          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1

          kind: KubeadmConfigTemplate

          name: k1-md-0

      clusterName: k1

      failureDomain: compute-az

      infrastructureRef:

        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1

        kind: OpenStackMachineTemplate

        name: k1-md-0

      version: v1.29.6

---

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1

kind: OpenStackMachineTemplate

metadata:

  name: k1-md-0

  namespace: org-platform9-product-team

spec:

  template:

    spec:

      flavor: m1.small

      image:

        filter:

          name: anmol

      sshKeyName: anmol

      identityRef:

        name: k1

        cloudName: openstack

      ports:

      - network:

          id: 2ed5a3d2-e1fe-4b1d-a2ce-6f33d2c1c2ab

---

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1

kind: KubeadmConfigTemplate

metadata:

  name: k1-md-0

  namespace: org-platform9-product-team

spec:

  template:

    spec:

      preKubeadmCommands:

…………..