kubectl is really slow and sometimes times out with i/o timeout

admin

Everything with kubectl takes forever. That was not the case before. What could be the problem?

Getting pods with logging as you can see it takes more than 20 seconds:

kubectl get nodes –v=8

I0626 11:05:30.744121   86247 loader.go:374] Config loaded from file:  /Users/[REMOVED]/.kube/config
I0626 11:05:30.747674   86247 round_trippers.go:463] GET https://[REMOVED].gr7.eu-central-1.eks.amazonaws.com/api/v1/namespaces/default/pods?limit=500
I0626 11:05:30.747690   86247 round_trippers.go:469] Request Headers:
I0626 11:05:30.747698   86247 round_trippers.go:473]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I0626 11:05:30.747702   86247 round_trippers.go:473]     User-Agent: kubectl/v1.25.4 (darwin/arm64) kubernetes/872a965
I0626 11:05:51.346710   86247 round_trippers.go:574] Response Status: 200 OK in 20599 milliseconds
I0626 11:05:51.346891   86247 round_trippers.go:577] Response Headers:
I0626 11:05:51.346917   86247 round_trippers.go:580]     Audit-Id: 6cad6d86-1cbc-4df7-8979-ac95dccbb6c3
I0626 11:05:51.346941   86247 round_trippers.go:580]     Cache-Control: no-cache, private
I0626 11:05:51.346959   86247 round_trippers.go:580]     Content-Type: application/json
I0626 11:05:51.346976   86247 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: f65d7b50-0090-499b-8be0-6fcf34055c25
I0626 11:05:51.346995   86247 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: ddeeb77c-23bd-4fb0-bb3e-d67e223ba442
I0626 11:05:51.347013   86247 round_trippers.go:580]     Date: Wed, 26 Jun 2024 09:05:51 GMT
I0626 11:05:51.359686   86247 request.go:1154] Response Body: [REMOVED]

Checking the memory and cpu pressure, but it looks fine:

kubectl describe nodes:

Node 1:
  Resource           Requests       Limits
  --------           --------       ------
  cpu                880m (22%)     0 (0%)
  memory             10290Mi (33%)  10598Mi (34%)
  ephemeral-storage  0 (0%)         0 (0%)
  hugepages-1Gi      0 (0%)         0 (0%)
  hugepages-2Mi      0 (0%)         0 (0%)
Node 2:
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1080m (55%)   0 (0%)
  memory             2029Mi (28%)  4897Mi (69%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)

Checking Checking with kubectl top for each node:

kubectl top node []

Node 1:
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
[REMOVED]   144m         7%     3876Mi          54%   
Node 2:
NAME                                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
[REMOVED]   64m          1%     5950Mi          19%

Read more here: Source link