Nodes Are Experiencing Disk Pressure

Problem

• Nodes are experiencing disk pressure and the following events are seen in the kubelet events.

• failed to garbage collect messages can be seen in the node description trying to free up space.

Events

$ kubectl describe node <Nodename>
......
4m22s Warning EvictionThresholdMet node/<NODE_NAME> Attempting to reclaim ephemeral-storage 9m18s Warning FreeDiskSpaceFailed node/nodename (combined from similar events): failed to garbage collect required amount of images. Wanted to free 15477086617 bytes, but freed 0 bytes.
.....

Environment

• Platform9 Managed Kubernetes - All versions.

Cause

• When /var or the container root directory where the containers and their dependents resides, reaches a threshold value(80% of the total in general), the kubelet records this as a disk pressure event.

• Filling up of this directory could be due to :

  • Pods running on the respective node consuming unlimited storage(possibly not having ephemeral limits assigned) by writing huge log file or the data associated with it.

  • Overloading container images that are not in use and thereby consuming space.

  • Other system services generating logs and filling up /var

Resolution

Addressing the possible cases as mentioned in the Cause section using the most appropriate approach will avoid filling up of the respective directory. Thus preventing disk pressure.