Pod Exec Fails With Permission Issue

Problem

After you upgrade runtime from Docker to Containerd, nodes upgrades successfully, but if you try to exec some pods it fails with the below error logs:

Pod logs

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/etc/ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/etc/ssh/ssh_host_ed25519_key' are too open.

Environment

• Platform9 Managed Kubernetes - v5.6.8

• Kubernetes version - 1.22.9

Cause

The directory listed below does not have correct permission inside directory /var/lib/containerd after runtime upgrade to containerd.

Permission

drwxr-xr-x 3 root root 4096 Aug 14 10:47 io.containerd.snapshotter.v1.aufs
drwxr-xr-x 3 root root 4096 Aug 14 10:47 io.containerd.snapshotter.v1.native
drwxr-xr-x 3 root root 4096 Aug 14 10:47 io.containerd.snapshotter.v1.overlayfs

Resolution

You need to manually change the permission to 0700 for above directory using chmod command.

Change permission

# chmod -R 0700  io.containerd.snapshotter.v1.aufs
# chmod -R 0700  io.containerd.snapshotter.v1.native
# chmod -R 0700  io.containerd.snapshotter.v1.overlayfs

Directory permissions inside /var/lib/containerd should look like below:

/var/lib/containerd

# ls -ll /var/lib/containerd
total 40
drwxr-xr-x 4 root root 4096 Aug  2 07:56 io.containerd.content.v1.content
drwxr-xr-x 4 root root 4096 Aug  2 07:57 io.containerd.grpc.v1.cri
drwx--x--x 2 root root 4096 Aug  2 07:56 io.containerd.metadata.v1.bolt
drwx--x--x 2 root root 4096 Aug  2 07:56 io.containerd.runtime.v1.linux
drwx--x--x 3 root root 4096 Aug  2 07:57 io.containerd.runtime.v2.task
drwx------ 3 root root 4096 Aug  2 07:56 io.containerd.snapshotter.v1.aufs
drwx------ 2 root root 4096 Aug  2 07:56 io.containerd.snapshotter.v1.btrfs
drwx------ 3 root root 4096 Aug  2 07:56 io.containerd.snapshotter.v1.native
drwx------ 3 root root 4096 Aug  2 07:56 io.containerd.snapshotter.v1.overlayfs
drwx------ 2 root root 4096 Sep 21 02:00 tmpmounts