kubectl can not connect to docker desktop kubernetes cluster

I have 2 machines. A is m1 macbook pro with ip 192.168.2.152. B is a ubuntu vm running on a windows machine with ip 192.168.2.186.

I enable the kubernetes cluster in Docker Desktop on A and I can successfully use kubectl to connect to the cluster.

❯ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   27h   v1.29.2

However, when I copy the kubeconfig file in ~/.kube to the B, and try to use kubectl on B to access the cluster. It says connection refused.

rio@rio-virtual-machine ~/P/g/jenkins> kubectl get nodes
E0713 18:55:56.372257  247766 memcache.go:265] couldn't get current server API group list: Get "https://192.168.2.152:6443/api?timeout=32s": dial tcp 192.168.2.152:6443: connect: connection refused
E0713 18:55:56.378367  247766 memcache.go:265] couldn't get current server API group list: Get "https://192.168.2.152:6443/api?timeout=32s": dial tcp 192.168.2.152:6443: connect: connection refused
E0713 18:55:56.386471  247766 memcache.go:265] couldn't get current server API group list: Get "https://192.168.2.152:6443/api?timeout=32s": dial tcp 192.168.2.152:6443: connect: connection refused
E0713 18:55:56.392208  247766 memcache.go:265] couldn't get current server API group list: Get "https://192.168.2.152:6443/api?timeout=32s": dial tcp 192.168.2.152:6443: connect: connection refused
E0713 18:55:56.398323  247766 memcache.go:265] couldn't get current server API group list: Get "https://192.168.2.152:6443/api?timeout=32s": dial tcp 192.168.2.152:6443: connect: connection refused
The connection to the server 192.168.2.152:6443 was refused - did you specify the right host or port?

B can ping A successfully, so I make sure is not an internet problem. Does it have something to do with I simply copy the kubeconfig file to B? Or I need to regenerate the config file on B.