How to fix leader election failed in Kubernetes

KubernetesADVANCEDHIGH

Leader election failures prevent controllers from achieving high availability. When multiple replicas of a controller (operator, webhook, scheduler) cannot elect a leader, none may process events, disrupting cluster operations.

What this error means

Many Kubernetes controllers use leader election to ensure only one replica is active: 1. Multiple replicas attempt to acquire a lease 2. The winner becomes the leader and processes events 3. If leader fails, another replica takes over When leader election fails: - No lease is acquired - No replica becomes leader - Events are not processed - Cluster operations stall Common causes: missing RBAC permissions, etcd unavailable, network partition, or misconfigured lease mechanism.

How to fix "leader election failed"

1Verify leader election mechanism is configured

Check if controller has leader election enabled:

bash

kubectl get pods -n <namespace> -l app=<controller>
kubectl logs <controller-pod-name> -n <namespace> | grep -i "leader\|lease"

For external-dns, cert-manager, etc.:

bash

kubectl get lease -A

Should show lease resources like:

bash

NAMESPACE     NAME                          HOLDER                         AGE
default       external-dns                  external-dns-7f4c6f8d8       5m
default       cert-manager                  cert-manager-5d6c9f9d9       3m

If no leases appear, leader election hasn't succeeded.

2Check RBAC permissions for lease management

Verify the controller's ServiceAccount has lease permissions:

bash

kubectl get rolebinding -A | grep <controller>
kubectl get clusterrolebinding | grep <controller>

Check the role definition:

bash

kubectl get role <role-name> -n <namespace> -o yaml

Should include:

yaml

rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
  - patch  # For renewals

If missing, add permissions:

bash

kubectl create role <controller>-leader --verb=get,create,update,patch --resource=leases -n <namespace>
kubectl create rolebinding <controller>-leader --role=<controller>-leader --serviceaccount=<namespace>:<controller> -n <namespace>

For cluster-wide leadership:

bash

kubectl create clusterrole <controller>-leader --verb=get,create,update,patch --resource=leases
kubectl create clusterrolebinding <controller>-leader --clusterrole=<controller>-leader --serviceaccount=<namespace>:<controller>

3Verify API server and etcd are healthy

Leader election requires a working etcd backend:

bash

# Check API server
kubectl cluster-info
kubectl get componentstatus  # Deprecated but useful

# Check etcd
kubectl get pods -n kube-system -l component=etcd
kubectl logs -n kube-system etcd-<node> --tail=50

# Test API server connectivity
curl -k https://<api-server>:6443/api/v1

If API server is down, restart it:

bash

# For kubeadm clusters
sudo systemctl restart kubelet

# For managed services
# Contact provider

If etcd is corrupted:

bash

# Backup and restore
sudo systemctl stop kubelet
sudo etcdctl snapshot save backup.db --endpoints=https://127.0.0.1:2379
# Restore requires downtime
sudo systemctl start kubelet

4Check lease resource for conflicts

Inspect the lease that controller is trying to acquire:

bash

kubectl get lease -A
kubectl describe lease <lease-name> -n <namespace>

Should show current holder:

yaml

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-controller
  namespace: default
spec:
  holderIdentity: my-controller-pod-abc123
  leaseDurationSeconds: 60
  acquireTime: "2024-01-01T12:00:00Z"
  renewTime: "2024-01-01T12:01:00Z"

If holderIdentity is empty or stale:

bash

# Delete the stuck lease to force re-election
kubectl delete lease <lease-name> -n <namespace>

Controller pods will immediately re-attempt election:

bash

kubectl logs <controller-pod> -n <namespace> -f | grep -i leader

Watch for log entries like "became leader".

5Review controller pod logs for errors

Get detailed error logs:

bash

kubectl logs <controller-pod> -n <namespace> --tail=100

Look for:
- "failed to acquire lease"
- "permission denied"
- "connection refused"
- "deadline exceeded"
- "not found"

Enable debug logging:

bash

kubectl set env deployment/<controller> -c <container> -n <namespace> \
  LEADER_ELECTION_NAMESPACE=<namespace> \
  V=4  # Verbose logging

kubectl rollout status deployment/<controller> -n <namespace>
kubectl logs deployment/<controller> -n <namespace> -f

Common patterns:
- "permission denied" → RBAC issue
- "connection refused" → API server unreachable
- "deadline exceeded" → etcd slow or unavailable

6Verify controller replica count and network

Check if multiple replicas exist:

bash

kubectl get pods -l app=<controller> -n <namespace>

Should show multiple replicas (usually 2-3):

bash

NAME                    READY   STATUS    RESTARTS
my-controller-abc123    1/1     Running   0
my-controller-def456    1/1     Running   0

If only one replica exists, scale up:

bash

kubectl scale deployment <controller> --replicas=2 -n <namespace>

Test inter-pod network connectivity:

bash

kubectl exec <controller-pod-1> -n <namespace> -- \
  curl -v https://<api-server>:6443/api/v1/leases

kubectl exec <controller-pod-2> -n <namespace> -- \
  ping <controller-pod-1>.default.svc.cluster.local

If network is broken, check:
- CNI plugin status
- Network policies blocking communication
- Service DNS resolution

7Investigate etcd lock contention

If multiple controllers are competing for the same lease:

bash

kubectl get lease <lease-name> -n <namespace> -o yaml

If different pods keep acquiring the lease (frequent holder changes):
- Lease duration too short (increase leaseDurationSeconds)
- Rapid pod restarts
- Network latency causing renewal failures

Check etcd metrics:

bash

kubectl logs -n kube-system etcd-<node> | grep -i "slot\|contention"

For performance issues:
- Monitor etcd write latency
- Check if other heavy workloads are using etcd
- Consider splitting leadership into multiple leases (one per component)

8Restart controller to force re-election

If a stale leader is blocking new elections:

bash

# Delete all controller pods to force fresh start
kubectl delete pods -l app=<controller> -n <namespace>

# Monitor new pod startup and leader election
kubectl get pods -l app=<controller> -n <namespace> -w
kubectl logs -l app=<controller> -n <namespace> -f --all-containers=true

Watch logs for leader election:

bash

[INFO] controller: became leader
[INFO] controller: started processing events

Alternatively, delete just the stuck lease:

bash

kubectl delete lease <lease-name> -n <namespace>
# Controller will immediately try to re-acquire

Then verify new leader is elected:

bash

kubectl get lease <lease-name> -n <namespace>

How to fix leader election failed in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "leader election failed"

Advanced notes

Related errors

Official resources & further reading