Kubectl Exec Command is Failing Intermittently With i/o Timeout Error

Problem

  • The kubectl exec command is failing intermittently with i/o timeout Error.
Kubectl Command
Copy
  • Corresponding API Server logs on the Master Node.
API Server Logs
Copy

Environment

  • Platform9 Managed Kubenetes - v5.9 and Higher
  • Platform9 Self Managed Cloud Platform - v-5.9.2-3199093 and Higher.

Cause

  • Not all nodes in the Kubernetes cluster have the net.ipv4.ip_local_reserved_ports setting configured to reserve the NodePort range. This lead to ephemeral port conflicts and service binding failures.

The net.ipv4.ip_local_reserved_ports setting in Linux is a sysctl parameter that allows you to reserve specific local port numbers so they won’t be used for automatic port assignments during outbound connections.

Resolution

  • Ensure consistent reservation of the NodePort range across all cluster nodes (both workers and master nodes) using the net.ipv4.ip_local_reserved_ports sysctl parameter.

Implementation Steps:

  • Determine the NodePort range used in your cluster. Default is 30000–32767, unless customized in the Kubernetes API server configuration.
  • Apply the reservation on all nodes.
sysctl command
Copy
  • Confirm the change is active.
sysctl command
Copy
  • Use automation (e.g., Ansible, scripts, or DaemonSets) to enforce this setting across all current and future nodes.

Additional Information

Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard