How to Allow Unsafe Sysctls on a PMK Cluster

Problem

How to Allow Unsafe Sysctls on a PMK Cluster?

Environment

• Platform9 Managed Kubernetes - All Versions
• Kubelet

Procedure

• Depending on whether the change is needed to be done on a single node or all nodes part of the cluster, refer to

How To Change Configuration for Kubelet Service on a Single Worker or Master Node in a PMK Cluster

OR

How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster

• Stop the pf9-hostagent & pf9-nodeletd services on ALL the worker/master nodes depending on which configmap is being edited.

Bash

sudo systemctl stop pf9-{hostagent,nodeletd}

• Edit the ConfigMap as needed and save it. Here as an example, we are editing it master-default-kubelet-config ConfigMap for the single master node cluster.

Example

# kubectl get configmap master-default-kubelet-config  -n kube-system -o yaml
...
    allowedUnsafeSysctls:
    - "kernel.msg*"
    - "net.core.somaxconn"

• Start the pf9-hostagent service ALL the worker/master nodes where it was initially stopped.

Bash

sudo systemctl start pf9-hostagent

On Master node post-Kubelet restart (it initially always logs the default values, then loads the new config)

# less /var/log/pf9/kubelet/kubelet.INFO | grep -i unsafe
I0209 01:40:12.118508   10415 flags.go:59] FLAG: --allowed-unsafe-sysctls="[]"
...
I0209 01:40:12.161606   10415 controller.go:267] kubelet config controller: ensuring filesystem is set up correctly
I0209 01:40:12.161624   10415 fsstore.go:59] kubelet config controller: initializing config checkpoints directory "/var/opt/pf9/kube/kubelet-config/dynamic-config/store"
I0209 01:40:12.163696   10415 fsstore.go:116] kubelet config controller: loading Kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/master-default-kubelet-config, UID: ae560d51-931a-4a1c-ae90-0ea9fee5d5b7, ResourceVersion: 1187451

• Deployed a pod with security context.

Pod Spec

# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  securityContext:
    sysctls:
    - name: kernel.shm_rmid_forced
      value: "0"
    - name: net.core.somaxconn
      value: "1024"
    - name: kernel.msgmax
      value: "65536"
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

• Pod has been created and is in running state.

Example

# kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          67s

• If we deploy the same pod on a different cluster which does not have the Kubelet setting for allowedUnsafeSysctls added.

Example

# kubectl get pods
NAME    READY   STATUS            RESTARTS   AGE
nginx   0/1     SysctlForbidden   0          13s

# kubectl describe pod nginx
Events:
  Type     Reason           Age        From                     Message
  ----     ------           ----       ----                     -------
  Normal   Scheduled        <unknown>                           Successfully assigned default/nginx to 10.128.146.164
  Warning  SysctlForbidden  27s        kubelet, 10.128.146.164  forbidden sysctl: "net.core.somaxconn" not whitelisted

Additional Information

Using sysctls in a Kubernetes Cluster