Kubectl Command Gives a Delayed Response

Problem

• The kubectl commands takes time to show the outputs.
• The below errors could be seen when run using --v=8 option.

Bash

$ time kubectl get nodes --v=8
----------Snip----------
I0811 16:58:18.680294  143430 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0811 16:58:18.682150  143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32   

<========================= NOTICE THE TIME GAP =========================>

I0811 16:58:23.689536  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5007 milliseconds. 
I0811 16:58:23.689584  143430 round_trippers.go:451] Response Headers:
I0811 16:58:23.689597  143430 round_trippers.go:454]     Date: Thu, 11 Aug 2022 16:58:23 GMT
I0811 16:58:23.689610  143430 round_trippers.go:454]     Audit-Id: 302ec193-57a5-47e0-8fca-9dc1a391222a
.......
I0811 16:58:23.689681  143430 round_trippers.go:454]     Content-Length: 57
I0811 16:58:23.691529  143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0811 16:58:23.692455  143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32s
I0811 16:58:23.692478  143430 round_trippers.go:429] Request Headers:

<========================= NOTICE THE TIME GAP =========================>

I0811 16:58:28.695759  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.  
I0811 16:58:28.695799  143430 round_trippers.go:451] Response Headers:
I0811 16:58:28.695809  143430 round_trippers.go:454]     Audit-Id: 56774c23-e5c7-4437-912e-c95ed640df06
I0811 16:58:28.695819  143430 round_trippers.go:454]     Cache-Control: no-cache, private
I0811 16:58:28.695827  143430 round_trippers.go:454]     Content-Type: text/plain; charset=utf-8
----------Snip----------
NAME                 STATUS   ROLES    AGE   VERSION
my-cluster-master0   Ready    master   25h   v1.20.15
my-cluster-master1   Ready    master   25h   v1.20.15
my-cluster-master2   Ready    master   25h   v1.20.15
my-cluster-worker0   Ready    worker   25h   v1.20.15

real    0m15.160s
user    0m0.159s
sys     0m0.077s

Environment

• Platform9 Managed Kubernetes - All versions

Cause

• While querying for the requested resource using kubectl commands, it also queries the metrics services to validate few things before giving the actual outputs.
• The query to metrics service holds a timeout of 5003 milliseconds(5seconds) and retries 2 times during the kubectl call as seen above.

Bash

I0811 16:58:23.691529  143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
....
I0811 16:58:28.695759  143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.

• If the metrics service is unavailable due to some reason/issues, the total timeout adds up to 10 seconds, causing the actual output to return after ~15 seconds.

Resolution

• Check the status of the resources associated with metrics apiservice.
• The issue could be with any of the metrics resource. Below commands could be used to troubleshoot in respective direction.

Bash

# kubectl top nodes
# kubectl describe po <metric-server-pod> -n kube-system
# kubectl logs <metric-server-pod> -n kube-system -c metrics-server-nanny
# kubectl get --raw /apis/metrics.k8s.io/v1beta1 
# kubectl get deployment <metrics-server>-n kube-system -oyaml
# kubectl api-resources