How to fix etcdserver: leader changed in Kubernetes

KubernetesADVANCEDMEDIUM

The etcd leader changed error indicates the etcd cluster detected a leadership change during operation. While occasional changes are normal, frequent changes signal network issues, slow disk performance, or resource constraints that prevent timely heartbeats.

What this error means

Etcd uses a distributed consensus algorithm with one elected leader. When the leader fails to send heartbeats within the election timeout, followers initiate a new election and elect a different leader. The "leader changed" error indicates this just happened. Occasional changes are normal in HA clusters, but frequent changes indicate infrastructure problems preventing stable leadership.

Typical symptoms

How to fix "etcdserver: leader changed"

1Check cluster infrastructure health

Diagnose network and disk performance:

bash

# Network latency between control nodes:
ping -c 100 <other-control-node-ip> | tail -1

# Disk I/O performance:
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread \
  --bs=4k --direct=1 --size=1G --numjobs=1 \
  --runtime=60 --group_reporting --directory=/var/lib/etcd

Look for:
- Network latency > 100ms (indicates potential issue)
- Disk read latency > 10ms

2Monitor etcd metrics

Check for excessive leader elections:

bash

# Prometheus query (if Prometheus installed):
etcd_server_has_leader  # Should be 1 (has leader)
etcd_server_leader_changes_seen_total  # Should be low

# Or check logs:
kubectl logs -n kube-system <etcd-pod> | grep "elected leader"
kubectl logs -n kube-system <etcd-pod> | grep "leader changed"

Frequent elections (> 1 per minute) indicate problems.

3Verify etcd cluster health

Check all etcd members are responding:

bash

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 \
  endpoint health

# Check member list:
ETCDCTL_API=3 etcdctl member list

All members should show healthy. Remove stale members if any show down.

4Clean up accumulated objects in etcd

Many objects in etcd slow down lookups:

bash

# Check pod count (rough etcd size indicator):
kubectl get pods --all-namespaces | wc -l

# Remove evicted or error pods:
kubectl delete pods --all-namespaces --field-selector=status.phase=Failed
kubectl delete pods --all-namespaces --field-selector=status.reason=Evicted

Large numbers of accumulated objects slow etcd responses.

5Monitor disk space for etcd

Ensure etcd has enough space:

bash

df -h /var/lib/etcd
ls -lh /var/lib/etcd/member/snap/db

Etcd has a default 2GB quota. Monitor size growth:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status

If approaching quota, increase it or compact the database.

6Defragment etcd if fragmented

After deleting many objects, defragment:

bash

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag

Defragmentation may trigger a leader election temporarily.

7Verify cluster topology is optimal

Check number of etcd members:

bash

ETCDCTL_API=3 etcdctl member list

Recommended:
- 3 members (tolerate 1 failure)
- 5 members (tolerate 2 failures)
- Avoid even numbers (4, 6) as quorum becomes fragile

If using even number, add/remove to make odd.

8Increase cluster resource allocation

Dedicate more resources to etcd:

bash

# For static pods, edit manifest:
sudo vi /etc/kubernetes/manifests/etcd.yaml

# Increase CPU/memory requests:
resources:
  requests:
    cpu: 500m  # Increase from 100m
    memory: 2Gi  # Increase from 1Gi

Then restart etcd:

bash

sudo systemctl restart kubelet  # Picks up manifest change

9Monitor for platform-specific snapshot operations

Some platforms (Azure AKS) take automated etcd snapshots:

bash

# Check Azure AKS backup settings:
az aks show --name <cluster> --resource-group <group>

Snapshots can trigger temporary leader changes. These are expected and not concerning if infrequent.

For multi-control-plane clusters, these operations should not be frequent.

How to fix etcdserver: leader changed in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "etcdserver: leader changed"

Advanced notes

Related errors

Official resources & further reading