# Service 'pf9-kubelet' Keeps Restarting

## Problem

The **pf9-kubelet** service is continuously restarting.

The following errors are present in the ***/var/log/pf9/kube/kube.log*** on the node.

{% tabs %}
{% tab title="None" %}

```none
watcher.go:146] Failed to watch directory "/sys/fs/cgroup/memory/system.slice/run-[UUID].scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-[UUID].scope: no space left on devicekubelet.go:1365] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-[UUID].scope: no space left on device
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Managed Kubernetes - v4.0 and Higher
* kubelet

## Cause

The **kubelet** service fails to start due to **inotify** resource issues. The error itself means that system is getting low on **inotify** watches, which enable programs to monitor file or directory changes.

## Resolution

1. Identify the current setting.

{% tabs %}
{% tab title="None" %}

```none
# cat /proc/sys/fs/inotify/max_user_watches
```

{% endtab %}
{% endtabs %}

2. The output resembles the following code

{% tabs %}
{% tab title="None" %}

```none
8192
```

{% endtab %}
{% endtabs %}

3. Increase value.

{% tabs %}
{% tab title="None" %}

```none
# echo 1048576 > /proc/sys/fs/inotify/max_user_watches
```

{% endtab %}
{% endtabs %}

4. To make the changes persistent across reboot.

{% tabs %}
{% tab title="None" %}

```none
# echo "fs.inotify.max_user_watches = 1048576" >> /etc/sysctl.conf
```

{% endtab %}
{% endtabs %}

## Additional Information

* This is a known [Bug #10421](https://github.com/kubernetes/kubernetes/issues/10421) in Kubernetes.
* Use the attached script **inotify\_watcher\_count.sh** to find out which application is using the **inotify** resource and the exact count of **inotify** watches on a system.
  * **Note:** It will need the root access to parse the details from */proc* filesystem


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/service-pf9-kubelet-keeps-restarting.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.