Unable to Create Pods on Node With Error "cannot allocate memory"

Problem

Unable to create pods on a node due to insufficient memory.

Sample output

$ kubectl describe pod <pod-name>
...
Events:
  Type     Reason                    Age                       From               Message
  ----     ------                    ----                      ----               -------
  Normal   Scheduled                 2m30s                     default-scheduler  Successfully assigned <namespace>/<podname> to <node name>

  Warning  FailedCreatePodContainer  <invalid> (x12 over 17s)  kubelet            unable to ensure pod container exists: failed to create container for [kubepods <pod-id>] : mkdir /sys/fs/cgroup/memory/kubepods/<pod-id>: cannot allocate memory
...

Environment

• Platform9 Managed Kubernetes - All Versions

Cause

The system is running out of memory as the cache is not getting reclaimed on the node. This results in the system unable to provide the requested amount of memory for creation of pods.

Resolution

1. Verify if the system cache is not getting reclaimed by kernel. Below sample output shows that the memory in cache is very high and not reclaimed as the "MemFree" confirms the actual free memory on the system is very less.

Sample Output

# cat /proc/meminfo 
MemTotal: 263845016 kB  <<<<<<<< Total Memory
MemFree: 2865212 kB     <<<<<<<< Currently Free memory 
MemAvailable: 250746552 kB  <<<<<<<<< Memory Available including cache.
...
Cached: 243472860 kB  <<<< Memory in cache.
...

1. Also you can capture of output of below commands and share with Platform9 Support Team

commands

$ cat /sys/fs/cgroup/memory/memory.usage_in_bytes
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
$ free -m

1. Consult with your System Administration Team on Further troubleshooting steps.