> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmk/solutions/exec-probe-timeout-calico-readiness-liveness-probe-failure.md).

# Exec Probe Timeout Fixed from K8s v1.20 Resulting in Calico Pods to Fail Liveness/Readiness Probes a

## Problem

Exec probe timeout fixed from K8s v1.20 resulting in calico pods to fail liveness/readiness probes as default timeout is 1 second.

## Environment

* Platform9 Managed Kubernetes - K8s v1.20 and above
* Calico CNI v3.18

## Answer

* Both the `calico-kube-controller` pod and all the `calico-node` pods, the probe failures are taking place as starting from K8s v1.20, the exec probe timeouts were fixed to actually respect the timeout values. The default 1 second timeout value can prove to be too low for loaded clusters.

{% hint style="info" %}
**Info**

From Calico v3.20 they increased probe timeouts from 1s to 10s. Part of the reason is that in k8s v1.20, exec probe timeouts were [fixed](https://kubernetes.io/blog/2020/12/08/kubernetes-1-20-release-announcement/#exec-probe-timeout-handling) to actually respect the timeout values.
{% endhint %}

{% tabs %}
{% tab title="Describe Example" %}

```bash
# kubectl describe pod calico-node-25d6p -n kube-system
Name:                 calico-node-25d6p
Containers:
  calico-node:
   Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
 Events:
  Type     Reason     Age                  From     Message
  ----     ------     ----                 ----     -------
  Warning  Unhealthy  6m5s (x124 over 9d)  kubelet  Readiness probe failed:
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Kubelet Logs" %}

```bash
E0113 05:24:17.195056   23705 remote_runtime.go:392] ExecSync 4b779e0770fe8633778d71732591687263be8502b7444c95560d44bc61505d72 '/bin/calico-node -felix-live -bird-live' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0113 05:24:17.195205   23705 prober.go:117] Liveness probe for "calico-node-25d6p_kube-system(3f482be0-6e69-4dfd-9553-d6166f23ab75):calico-node" failed (failure):
```

{% endtab %}
{% endtabs %}

#### Workaround Option 1

* On ALL master nodes part of the cluster, add parameter `timeoutSeconds: 10`in files `/opt/pf9/pf9-kube/conf/networkapps/calico-v1.20.11.yaml` & `/opt/pf9/pf9-kube/conf/networkapps/calico-v1.20.11-configured.yaml` respectively at 2 sections i.e in Probes for calico-kube-controllers Deployment & calico-node DaemonSet Spec.
* Perform complete PMK stack restart on each Master Node ONE NODE AT A TIME.

```bash
sudo systemctl stop pf9-hostagent pf9-nodeletd
sudo /opt/pf9/nodelet/nodeletd phases stop
sudo systemctl start pf9-hostagent
```

* Post stack restart, the `calico-kube-controller` pod and all the `calico-node pods` will be recreated with spec consisting of `timeoutSeconds=10`.

{% tabs %}
{% tab title="Post Restart: New Applied Spec \[Note the addition of timeoutSeconds: 10 in respective sections]" %}

```bash
# kubectl get deployment calico-kube-controllers -n kube-system -o yaml
 ...
         readinessProbe:
          exec:
            command:
            - /usr/bin/check-status
            - -r
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10

# kubectl get daemonset calico-node -n kube-system -o yaml
...
    spec:
      containers:
...    
        livenessProbe:
          exec:
            command:
            - /bin/calico-node
            - -felix-live
            - -bird-live
          failureThreshold: 6
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        name: calico-node
        readinessProbe:
          exec:
            command:
            - /bin/calico-node
            - -felix-ready
            - -bird-ready
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Note**

The above procedure will not persist post cluster upgrade.
{% endhint %}

#### Workaround Option 2

Set `ExecProbeTimeout: false` as a feature gate in the Dynamic Kubelet Config of the nodes. This reverts to the previous behaviour where the timeouts are ignored in K8s v1.19 and below.

{% hint style="info" %}
**Info**

If changes are made to the configmap, all nodes running the *pf9-kubelet* service that use the configuration will detect the changes in the dynamic kubelet configuration and then integrate those changes into the ConfigMap settings and then restart the *pf9-kubelet* service. Once restarted, the *pf9-kubelet* service will use the new configuration in the ConfigMap.
{% endhint %}

**Reference:**

* [Configure Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes)
* [How To Change Configuration for Kubelet Service on All Worker or Master Nodes in a PMK Cluster](https://platform9.com/docs/kubernetes/dynamic-kubelet-configuration)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/exec-probe-timeout-calico-readiness-liveness-probe-failure.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.