How to fix leader election failed in Kubernetes

KubernetesADVANCEDHIGHUpdated Dec 6, 2025· 161 views

Leader election failures prevent controllers from achieving high availability. When multiple replicas of a controller (operator, webhook, scheduler) cannot elect a leader, none may process events, disrupting cluster operations.

Quick fix

Fixes leader election failed

kubectl get pods -n <namespace> -l app=<controller>
kubectl logs <controller-pod-name> -n <namespace> | grep -i "leader\|lease"
  1. Verify leader election mechanism is configured
  2. Check RBAC permissions for lease management
  3. Verify API server and etcd are healthy
  4. Check lease resource for conflicts
kubernetes — error
kubectl get pods -n <namespace> -l app=<controller>
kubectl logs <controller-pod-name> -n <namespace> | grep -i "leader\|lease"
leader election failed

What this error means

Many Kubernetes controllers use leader election to ensure only one replica is active: 1. Multiple replicas attempt to acquire a lease 2. The winner becomes the leader and processes events 3. If leader fails, another replica takes over When leader election fails: - No lease is acquired - No replica becomes leader - Events are not processed - Cluster operations stall Common causes: missing RBAC permissions, etcd unavailable, network partition, or misconfigured lease mechanism.

Typical symptoms

  • Controller pod logs show "failed to acquire lease" or "leader election failed"
  • Multiple controller replicas all show as non-leaders
  • Events are not processed or delayed significantly
  • Status of CRDs/resources not updating
  • No pod logs showing leader being elected
  • Lease resource shows "no holder" in all replicas
  • Controller crash loop due to failed election

Common causes

  1. RBAC role missing "lease" or "coordinationv1" permissions
  2. etcd or API server unavailable
  3. Network partition preventing replicas from communicating
  4. Lease resource corrupted or stuck
  5. Kubelet cannot resolve API server DNS
  6. TLS certificate expired or invalid
  7. Controller using wrong lease namespace or name

How to fix "leader election failed"

1Verify leader election mechanism is configured

Check if controller has leader election enabled:

bash
kubectl get pods -n <namespace> -l app=<controller>
kubectl logs <controller-pod-name> -n <namespace> | grep -i "leader\|lease"

For external-dns, cert-manager, etc.:

bash
kubectl get lease -A

Should show lease resources like:

bash
NAMESPACE     NAME                          HOLDER                         AGE
default       external-dns                  external-dns-7f4c6f8d8       5m
default       cert-manager                  cert-manager-5d6c9f9d9       3m

If no leases appear, leader election hasn't succeeded.

2Check RBAC permissions for lease management

Verify the controller's ServiceAccount has lease permissions:

bash
kubectl get rolebinding -A | grep <controller>
kubectl get clusterrolebinding | grep <controller>

Check the role definition:

bash
kubectl get role <role-name> -n <namespace> -o yaml

Should include:

yaml
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
  - patch  # For renewals

If missing, add permissions:

bash
kubectl create role <controller>-leader --verb=get,create,update,patch --resource=leases -n <namespace>
kubectl create rolebinding <controller>-leader --role=<controller>-leader --serviceaccount=<namespace>:<controller> -n <namespace>

For cluster-wide leadership:

bash
kubectl create clusterrole <controller>-leader --verb=get,create,update,patch --resource=leases
kubectl create clusterrolebinding <controller>-leader --clusterrole=<controller>-leader --serviceaccount=<namespace>:<controller>
3Verify API server and etcd are healthy

Leader election requires a working etcd backend:

bash
# Check API server
kubectl cluster-info
kubectl get componentstatus  # Deprecated but useful

# Check etcd
kubectl get pods -n kube-system -l component=etcd
kubectl logs -n kube-system etcd-<node> --tail=50

# Test API server connectivity
curl -k https://<api-server>:6443/api/v1

If API server is down, restart it:

bash
# For kubeadm clusters
sudo systemctl restart kubelet

# For managed services
# Contact provider

If etcd is corrupted:

bash
# Backup and restore
sudo systemctl stop kubelet
sudo etcdctl snapshot save backup.db --endpoints=https://127.0.0.1:2379
# Restore requires downtime
sudo systemctl start kubelet
4Check lease resource for conflicts

Inspect the lease that controller is trying to acquire:

bash
kubectl get lease -A
kubectl describe lease <lease-name> -n <namespace>

Should show current holder:

yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-controller
  namespace: default
spec:
  holderIdentity: my-controller-pod-abc123
  leaseDurationSeconds: 60
  acquireTime: "2024-01-01T12:00:00Z"
  renewTime: "2024-01-01T12:01:00Z"

If holderIdentity is empty or stale:

bash
# Delete the stuck lease to force re-election
kubectl delete lease <lease-name> -n <namespace>

Controller pods will immediately re-attempt election:

bash
kubectl logs <controller-pod> -n <namespace> -f | grep -i leader

Watch for log entries like "became leader".

5Review controller pod logs for errors

Get detailed error logs:

bash
kubectl logs <controller-pod> -n <namespace> --tail=100

Look for:
- "failed to acquire lease"
- "permission denied"
- "connection refused"
- "deadline exceeded"
- "not found"

Enable debug logging:

bash
kubectl set env deployment/<controller> -c <container> -n <namespace> \
  LEADER_ELECTION_NAMESPACE=<namespace> \
  V=4  # Verbose logging

kubectl rollout status deployment/<controller> -n <namespace>
kubectl logs deployment/<controller> -n <namespace> -f

Common patterns:
- "permission denied" → RBAC issue
- "connection refused" → API server unreachable
- "deadline exceeded" → etcd slow or unavailable

6Verify controller replica count and network

Check if multiple replicas exist:

bash
kubectl get pods -l app=<controller> -n <namespace>

Should show multiple replicas (usually 2-3):

bash
NAME                    READY   STATUS    RESTARTS
my-controller-abc123    1/1     Running   0
my-controller-def456    1/1     Running   0

If only one replica exists, scale up:

bash
kubectl scale deployment <controller> --replicas=2 -n <namespace>

Test inter-pod network connectivity:

bash
kubectl exec <controller-pod-1> -n <namespace> -- \
  curl -v https://<api-server>:6443/api/v1/leases

kubectl exec <controller-pod-2> -n <namespace> -- \
  ping <controller-pod-1>.default.svc.cluster.local

If network is broken, check:
- CNI plugin status
- Network policies blocking communication
- Service DNS resolution

7Investigate etcd lock contention

If multiple controllers are competing for the same lease:

bash
kubectl get lease <lease-name> -n <namespace> -o yaml

If different pods keep acquiring the lease (frequent holder changes):
- Lease duration too short (increase leaseDurationSeconds)
- Rapid pod restarts
- Network latency causing renewal failures

Check etcd metrics:

bash
kubectl logs -n kube-system etcd-<node> | grep -i "slot\|contention"

For performance issues:
- Monitor etcd write latency
- Check if other heavy workloads are using etcd
- Consider splitting leadership into multiple leases (one per component)

8Restart controller to force re-election

If a stale leader is blocking new elections:

bash
# Delete all controller pods to force fresh start
kubectl delete pods -l app=<controller> -n <namespace>

# Monitor new pod startup and leader election
kubectl get pods -l app=<controller> -n <namespace> -w
kubectl logs -l app=<controller> -n <namespace> -f --all-containers=true

Watch logs for leader election:

bash
[INFO] controller: became leader
[INFO] controller: started processing events

Alternatively, delete just the stuck lease:

bash
kubectl delete lease <lease-name> -n <namespace>
# Controller will immediately try to re-acquire

Then verify new leader is elected:

bash
kubectl get lease <lease-name> -n <namespace>

Advanced notes

Leader election is critical for HA controllers. Multiple replicas without leader election cause conflicting operations (two cert-manager instances issuing certs, two external-dns instances managing DNS, etc.). The election mechanism uses etcd Leases for atomicity. Lease duration (default 15s) balances between quick failover and election churn. For wide-area networks, increase lease duration. Operator frameworks (Kubebuilder, Operator SDK) provide built-in leader election. Monitor lease churn with Prometheus: kubernetes_build_info{job="controller"} tracks replica count. For mission-critical controllers, implement additional healthchecks beyond leader election. Some controllers use ConfigMap-based elections instead of Leases (older Kubernetes)—these are less reliable. Multi-cluster scenarios may need distributed leader election (Consul, etcd cluster). For testing, disable leader election locally to simplify debugging.

Frequently asked questions

What does "leader election failed" mean?
Many Kubernetes controllers use leader election to ensure only one replica is active: 1. Multiple replicas attempt to acquire a lease 2. The winner becomes the leader and processes events 3.
How do I fix "leader election failed"?
Leader election failures prevent controllers from achieving high availability. When multiple replicas of a controller (operator, webhook, scheduler) cannot elect a leader, none may process events, disrupting cluster operations.
What causes "leader election failed"?
RBAC role missing "lease" or "coordinationv1" permissions etcd or API server unavailable Network partition preventing replicas from communicating
Is the "leader election failed" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading