Kubectl Command Gives a Delayed Response
Problem
- The
kubectlcommands takes time to show the outputs. - The below errors could be seen when run using
--v=8option.
x
$ time kubectl get nodes --v=8----------Snip----------I0811 16:58:18.680294 143430 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the requestI0811 16:58:18.682150 143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32 <========================= NOTICE THE TIME GAP =========================>I0811 16:58:23.689536 143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5007 milliseconds. I0811 16:58:23.689584 143430 round_trippers.go:451] Response Headers:I0811 16:58:23.689597 143430 round_trippers.go:454] Date: Thu, 11 Aug 2022 16:58:23 GMTI0811 16:58:23.689610 143430 round_trippers.go:454] Audit-Id: 302ec193-57a5-47e0-8fca-9dc1a391222a.......I0811 16:58:23.689681 143430 round_trippers.go:454] Content-Length: 57I0811 16:58:23.691529 143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the requestI0811 16:58:23.692455 143430 round_trippers.go:422] GET https://localhost:443/apis/metrics.k8s.io/v1beta1?timeout=32sI0811 16:58:23.692478 143430 round_trippers.go:429] Request Headers:<========================= NOTICE THE TIME GAP =========================>I0811 16:58:28.695759 143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds. I0811 16:58:28.695799 143430 round_trippers.go:451] Response Headers:I0811 16:58:28.695809 143430 round_trippers.go:454] Audit-Id: 56774c23-e5c7-4437-912e-c95ed640df06I0811 16:58:28.695819 143430 round_trippers.go:454] Cache-Control: no-cache, privateI0811 16:58:28.695827 143430 round_trippers.go:454] Content-Type: text/plain; charset=utf-8----------Snip----------NAME STATUS ROLES AGE VERSIONmy-cluster-master0 Ready master 25h v1.20.15my-cluster-master1 Ready master 25h v1.20.15my-cluster-master2 Ready master 25h v1.20.15my-cluster-worker0 Ready worker 25h v1.20.15real 0m15.160suser 0m0.159ssys 0m0.077sEnvironment
- Platform9 Managed Kubernetes - All versions
Cause
- While querying for the requested resource using
kubectlcommands, it also queries the metrics services to validate few things before giving the actual outputs. - The query to metrics service holds a timeout of 5003 milliseconds(5seconds) and retries 2 times during the
kubectlcall as seen above.
I0811 16:58:23.691529 143430 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request....I0811 16:58:28.695759 143430 round_trippers.go:448] Response Status: 503 Service Unavailable in 5003 milliseconds.- If the metrics service is unavailable due to some reason/issues, the total timeout adds up to 10 seconds, causing the actual output to return after ~15 seconds.
Resolution
- Check the status of the resources associated with metrics apiservice.
- The issue could be with any of the metrics resource. Below commands could be used to troubleshoot in respective direction.
# kubectl top nodes# kubectl describe po <metric-server-pod> -n kube-system# kubectl logs <metric-server-pod> -n kube-system -c metrics-server-nanny# kubectl get --raw /apis/metrics.k8s.io/v1beta1 # kubectl get deployment <metrics-server>-n kube-system -oyaml# kubectl api-resourcesWas this page helpful?