Pods Failing with "cannot allocate memory" Error

Problem

Pods failing with "cannot allocate memory" error.

Environment

• Platform9 Managed Kubernetes - All Versions
• CentOS Linux 7

Answer

• Error Observed

Mar  3 17:26:11 kubernetes-worker02 kubelet: E0303 17:26:11.092988   23949 pod_workers.go:191] Error syncing pod 00c8233c-8a81-487b-b5be-3f9befc13771 ("test(00c8233c-8a81-487b-b5be-3f9befc13771)"), skipping: failed to ensure that the pod: 00c8233c-8a81-487b-b5be-3f9befc13771 cgroups exist and are correctly applied: failed to create container for [kubepods burstable pod00c8233c-8a81-487b-b5be-3f9befc13771] : mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod00c8233c-8a81-487b-b5be-3f9befc13771: cannot allocate memory

• Following is the upstream issue link regarding this behavior Docker fails to start containers with cgroup memory allocation error.
• Indicates that the fix for the issue [LLNL 7.5 Bug] slab leak causing a crash when using kmem control group should be in Kernel Version 3.10.0-1062.4.1 or later.
• However the discussion also talks about similar issues being present/reported on Kernel version above this.

$ uname -r

3.10.0-1160.36.2.el7.x86_64

• One of the temporary workaround mentioned is to add the parameter cgroup.memory=nokmem to the GRUB_CMDLINE_LINUX line in the configuration file /etc/default/grub. After that,

$ grub2-mkconfig -o /boot/grub2/grub.cfg

$ reboot

Long term solution can be to upgrade the Kernel version to the latest version and see if it resolves the issue OR contact the OS vendor for further assistance.