Nodelet Failing at Configure Container Runtime Phase due to Unexpected Disk Mounts

Problem

• Nodes have unused disks (~100Gi) that are being taken as mount point for containerd storage by nodelet.

• There is no defined way of stating containerd storage mount point in the PMK nodes.

• Nodelet is failing while configuring container runtime in PMK nodes.

Nodelet Phases

INDEX NUMBER  FILE                                         NAME     PHASE STATUS
  1             Generate certs / Send signing request to CA  running
  2             Prepare configuration                        running
  3             Configure Container Runtime                  running
  4             Start Container Runtime                      failed

• Nodelet is unable to detect the filesystem to be used as container runtime storage.

Nodelet Logs

[2024-05-06 16:31:09] mount: /var/lib/containerd: unknown filesystem type 'ceph_bluestore'.

Environment

• Platform9 Managed Kubernetes

• All Kubernetes versions

Cause

• In cases where /var/lib/containerd is not mounted, then Nodelet looks for local disks of size >= 100 GB on the node. A disk which is not in use (mounted anywhere) would be considered as the containerd mountpoint. In the below error log, the nodelet detected an available filesystem satisfying the requirement for containerd storage, however due to unexpected ceph filesystem, it didn't go through.

[2024-05-06 16:31:09] mount: /var/lib/containerd: unknown filesystem type 'ceph_bluestore'.

• In cases where nodelet is unable to find the device for /var/lib/containerd, it'll have an ext4 filesystem ready to be used as containerd storage.

Resolution

• Platform9 recommends configuring additional storage for /var/lib/containerd directory, sizing to be around 100Gi. This is the place where all the container images and other containerd metadata is stored.

• To uniformly manage the containerd storage through the PMK nodes, following flags can be used to specify the disk to be used:

  • **DISABLE_CONTAINERD_MOUNT**: Disable discovery mechanism for free disks during containerd configuration.

    • Example: export DISABLE_CONTAINERD_MOUNT=true

  • **CONTAINERD_DEVICE**: Specify the device that nodelet should use to mount /var/lib/containerd.

    • Example: export CONTAINERD_DEVICE=/dev/sdc1

The above flags need to be specified in /etc/pf9/kube_override.env file on every node of the cluster.

NOTE

• Ensure to have persistent mount point configurations to handle unexpected reboots; for example via /etc/fstab

Additional Information

Please refer to Platform9 Official Documentation to configure containerd storage:

https://platform9.com/docs/kubernetes/configuring-containerd-storage