Unable to Create Pods on Node With Error "cannot allocate memory"
Problem
Unable to create pods on a node due to insufficient memory.
x
$ kubectl describe pod <pod-name>...Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m30s default-scheduler Successfully assigned <namespace>/<podname> to <node name> Warning FailedCreatePodContainer <invalid> (x12 over 17s) kubelet unable to ensure pod container exists: failed to create container for [kubepods <pod-id>] : mkdir /sys/fs/cgroup/memory/kubepods/<pod-id>: cannot allocate memory...Environment
- Platform9 Managed Kubernetes - All Versions
Cause
The system is running out of memory as the cache is not getting reclaimed on the node. This results in the system unable to provide the requested amount of memory for creation of pods.
Resolution
- Verify if the system cache is not getting reclaimed by kernel. Below sample output shows that the memory in cache is very high and not reclaimed as the "MemFree" confirms the actual free memory on the system is very less.
# cat /proc/meminfo MemTotal: 263845016 kB <<<<<<<< Total MemoryMemFree: 2865212 kB <<<<<<<< Currently Free memory MemAvailable: 250746552 kB <<<<<<<<< Memory Available including cache....Cached: 243472860 kB <<<< Memory in cache....- Also you can capture of output of below commands and share with Platform9 Support Team
$ cat /sys/fs/cgroup/memory/memory.usage_in_bytes$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes$ free -m- Consult with your System Administration Team on Further troubleshooting steps.
Was this page helpful?