Service 'pf9-kubelet' Keeps Restarting

Problem

The pf9-kubelet service is continuously restarting.

The following errors are present in the /var/log/pf9/kube/kube.log on the node.

Copy

Environment

  • Platform9 Managed Kubernetes - v4.0 and Higher
  • kubelet

Cause

The kubelet service fails to start due to inotify resource issues. The error itself means that system is getting low on inotify watches, which enable programs to monitor file or directory changes.

Resolution

  1. Identify the current setting.
Copy
  1. The output resembles the following code
Copy
  1. Increase value.
Copy
  1. To make the changes persistent across reboot.
Copy

Additional Information

  • This is a known Bug #10421 in Kubernetes.
  • Use the attached script inotify_watcher_count.sh to find out which application is using the inotify resource and the exact count of inotify watches on a system.
    • Note: It will need the root access to parse the details from /proc filesystem
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard