How to Deploy a Custom File to Every PCD Kubernetes Node via ClusterClass

Problem

How do you get an arbitrary file, such as a containerd registry config, a sysctl drop-in, or a custom CA certificate, written to every node of a PCD Kubernetes cluster, without hand-editing files on individual VMs (which won't survive a node replacement)?

Environment

• Platform9 Private Cloud Director (PCD) with PCD Kubernetes clusters (Cluster API / ClusterClass-based)
• API access to the cluster.x-k8s.io Cluster API resources for your cluster (see Prerequisites)

How it works

PCD Kubernetes clusters are provisioned by Cluster API from a ClusterClass. The Cluster resource's spec.topology.variables carries a kubeadmConfigFiles variable, a list of {content, path, owner, permissions} objects, that the ClusterClass patches into the rendered KubeadmConfigTemplate. kubeadm writes every one of those files to disk on every node at boot, via cloud-init, before kubelet starts.

Your cluster's ClusterClass typically ships a few such entries by default, including a base containerd configuration that already enables per-registry overrides under /etc/containerd/certs.d. In most cases you don't need to replace the whole containerd config, just add a new file under certs.d.

Editing kubeadmConfigFiles changes the rendered KubeadmConfigTemplate's content hash, which Cluster API detects as a spec change and uses to trigger a rolling replacement of the affected Machines (workers, and/or control plane if it isn't a managed hosted control plane). Set up Load Balancers using MetalLB describes the same rollout behavior for another topology-variable change.

Procedure

Prerequisites

• The base URL, namespace, and a bearer token for the Kubernetes Cluster API. See the Kubernetes Service API Overview for how to construct these.
• Sufficient permissions to read and update Cluster resources in that namespace.
• Your cluster's exact name.

Step 1: Fetch the current Cluster resource

curl -sk -H "Authorization: Bearer <token>" \
  "<base-url>/apis/cluster.x-k8s.io/v1beta1/namespaces/<namespace>/clusters/<cluster-name>" -o cluster.json

Step 2: Extend kubeadmConfigFiles

Find the kubeadmConfigFiles variable under spec.topology.variables and append your new file object. Do not replace the whole array with just your entry; the existing entries are load-bearing for the cluster. Example, trusting a private, HTTP-only container registry reachable at 10.10.8.7:80:

{
  "content": "server = \"http://10.10.8.7:80\"\n\n[host.\"http://10.10.8.7:80\"]\n  capabilities = [\"pull\", \"resolve\", \"push\"]\n",
  "owner": "root:root",
  "path": "/etc/containerd/certs.d/10.10.8.7:80/hosts.toml",
  "permissions": "0644"
}

Step 3: Write the whole object back

spec.topology.variables is replaced wholesale on write, so always GET, modify in place, then PUT the complete object back:

curl -sk -X PUT \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @cluster.json \
  "<base-url>/apis/cluster.x-k8s.io/v1beta1/namespaces/<namespace>/clusters/<cluster-name>"

A 200 response means the update was accepted.

Step 4: Wait for the rollout

kubectl get nodes -w

Watch until the old node names are gone and the new ones are all Ready.

Step 5: Verify

Connect to a (new) node and confirm the file landed:

ssh <user>@<node-ip> "cat /etc/containerd/certs.d/10.10.8.7:80/hosts.toml"

Additional Information

• The same GET, modify, PUT pattern applies to any variable in spec.topology.variables, not just kubeadmConfigFiles.
• Related: How to Expose a Kubernetes MetalLB Load Balancer in a Virtual Network documents another topology-variable change (allowedAddressPairs via Expose LB) that causes the same kind of rollout.