"kubectl get " cli response time too high on k8s cluster - kubectl

Question

I had two small k8s cluster (one with flannel and one with calico).
Exact steps were used to install the two k8s cluster only difference being selection of Pod network at install time (one uses flannel one uses calico).
Issue was that "kubectl get all" command had different response time on both cluster. It takes roughly a min to respond on k8s with calico while k8s with flannel gave instant response.
was sure the issue is not due to Pod network selection choice as had no issues in spinning pods etc on both cluster both are working as expected.
Time on flannel based k8s 0m0.167s
$ time kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-6db489d4b7-h2mvv 1/1 Running 0 17m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 17m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6db489d4b7 1 1 1 17m
real 0m0.167s
user 0m0.100s
sys 0m0.028s
Time on calico based k8s cluster hangs and responds only after nearly a minute .. 0m59.294s
$ time kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-6db489d4b7-b8c2g 1/1 Running 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6db489d4b7 1 1 1 11m
real 0m59.294s
user 0m0.316s
sys 0m0.072s
At cluster install time it was ensured to ran below commands as regular user
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ ll .kube/config
-rw------- 1 ubuntu ubuntu 5455 Jan 23 10:17 .kube/config
I tried setting the kubeconfig env variable as
export KUBECONFIG=$HOME/.kube/config
Still this did not fix the response time for kubectl command.

Answer 1

After spending a lot of time in wrong direction it was identified that the cache and http-cache folder under $HOME/.kube were not having the correct chown permission.
Once fixed the permissions on those two folders (cache and http-cache) as well under $HOME/.kube the kubectl get response time was back to normal.
ubuntu#k8s-calico-master-1:~/.kube$ ll
total 24
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 23 10:18 ./
drwxr-xr-x 5 ubuntu ubuntu 4096 Jan 23 10:17 ../
drwxr-x--- 3 ubuntu ubuntu 4096 Jan 23 10:18 cache/
-rw------- 1 ubuntu ubuntu 5455 Jan 23 10:17 config
drwxr-x--- 3 ubuntu ubuntu 4096 Jan 23 10:18 http-cache/
ubuntu#k8s-calico-master-1:~/.kube$ cd cache/
ubuntu#k8s-calico-master-1:~/.kube/cache$ ll
total 12
drwxr-x--- 3 ubuntu ubuntu 4096 Jan 23 10:18 ./
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 23 10:18 ../
drwxr-x--- 3 ubuntu ubuntu 4096 Jan 23 10:18 discovery/
ubuntu#k8s-calico-master-1:~/.kube$ cd http-cache/
ubuntu#k8s-calico-master-1:~/.kube/http-cache$ ll
total 164
drwxr-x--- 3 ubuntu ubuntu 4096 Jan 23 10:18 ./
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 23 10:18 ../
drwxr-x--- 2 ubuntu ubuntu 4096 Jan 23 10:18 .diskv-temp/
-rw-rw---- 1 ubuntu ubuntu 813 Jan 23 10:18 f436dd33b3ceee24aa367363c323688e
ubuntu#k8s-calico-master-1:~/.kube/http-cache$ time kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-6db489d4b7-b8c2g 1/1 Running 0 52m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 61m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 52m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6db489d4b7 1 1 1 52m
real 0m0.104s
user 0m0.104s
sys 0m0.024s