kubernetes – kubectl failed to ensure lease exists error

admin

I’m trying to deploy an at home k8s node to learn on and deploy some actual services I rely on to. I have taken the following steps:

  1. Install Fedora Server 39
  2. Follow the directions at https://docs.fedoraproject.org/en-US/quick-docs/using-kubernetes/ to deploy the k8s cluster, with the exception of using containerd instead of CRI-O, and using Canal instead of Flannel using the deployment yaml at https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/canal.yaml

I also ran all command printed by kubeadm after the init command completes to set up my profile for kubectl. I immediately started to get an issue where sometimes kubectl commands would run, and sometimes I’d get a Service Unreachable on 6443 error. Once I started digging I found the following errors in the kubelet logs:

I0129 10:39:19.409618    1170 kubelet_node_status.go:70] "Attempting to register node" node="localhost"
E0129 10:39:19.411296    1170 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes \"localhost\" is forbidden: node \"localhost.localdomain\" is not allowed to modify node \"localhost\"" node="localhost"
E0129 10:39:21.998836    1170 eviction_manager.go:262] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"localhost\" not found"
E0129 10:39:26.204801    1170 controller.go:146] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"localhost\" is forbidden: User \"system:node:localhost.localdomain\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\": can only access node lease with the same name as the requesting node" interval="7s"

I’ve tried the following:

  1. Re-disabling firewalld
  2. Re-disabling swap with swapoff --all
  3. Change SELinux Policy from ENFORCE to PERMISSIVE
  4. Removing all localdomain references in /etc/kubernetes/kubelet.conf
  5. Changing hostname with hostnamectl hostname localhost.localdomain
  6. Restarting the server

The issue persists. All solutions I find online seem to boil down to either option (1) or option (2) from above, to no avail. Any ideas on what I could try next?

Extra info:

Output from kubectl get nodes:

NAME                    STATUS     ROLES           AGE   VERSION
localhost.localdomain   NotReady   control-plane   12h   v1.27.10

Output from kubectl get pods --all-namespaces:

NAMESPACE     NAME                                            READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-867bf4f5b5-kz2r9        0/1     Pending   0             121m
kube-system   canal-pw6j2                                     0/2     Pending   0             121m
kube-system   coredns-5d78c9869d-rvmzw                        0/1     Pending   0             12h
kube-system   coredns-5d78c9869d-zn9lx                        0/1     Pending   0             12h
kube-system   etcd-localhost.localdomain                      1/1     Running   5 (12h ago)   12h
kube-system   kube-apiserver-localhost.localdomain            0/1     Running   8 (12h ago)   12h
kube-system   kube-controller-manager-localhost.localdomain   1/1     Running   7 (12h ago)   12h
kube-system   kube-proxy-mrsvp                                0/1     Unknown   6             12h
kube-system   kube-scheduler-localhost.localdomain            1/1     Running   6 (12h ago)   12h

Read more here: Source link