How to fix etcdserver: leader changed in Kubernetes

KubernetesADVANCEDMEDIUMUpdated Dec 6, 2025· 148 views

The etcd leader changed error indicates the etcd cluster detected a leadership change during operation. While occasional changes are normal, frequent changes signal network issues, slow disk performance, or resource constraints that prevent timely heartbeats.

Quick fix

Fixes etcdserver: leader changed

# Network latency between control nodes:
ping -c 100 <other-control-node-ip> | tail -1

# Disk I/O performance:
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread \
  --bs=4k --direct=1 --size=1G --numjobs=1 \
  --runtime=60 --group_reporting --directory=/var/lib/etcd
  1. Check cluster infrastructure health
  2. Monitor etcd metrics
  3. Verify etcd cluster health
  4. Clean up accumulated objects in etcd
kubernetes — error
# Network latency between control nodes:
ping -c 100 <other-control-node-ip> | tail -1
# Disk I/O performance:
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread \
  --bs=4k --direct=1 --size=1G --numjobs=1 \
  --runtime=60 --group_reporting --directory=/var/lib/etcd
etcdserver: leader changed

What this error means

Etcd uses a distributed consensus algorithm with one elected leader. When the leader fails to send heartbeats within the election timeout, followers initiate a new election and elect a different leader. The "leader changed" error indicates this just happened. Occasional changes are normal in HA clusters, but frequent changes indicate infrastructure problems preventing stable leadership.

Typical symptoms

  • kubectl commands intermittently fail with "etcdserver: leader changed"
  • API server responsiveness becomes unreliable or slow
  • Helm installations or Kubernetes deployments timeout unexpectedly
  • Kubernetes Dashboard fails to communicate
  • Existing pods continue running but cluster is temporarily read-only
  • etcd metrics show excessive leader election frequency

Common causes

  1. Network latency or intermittent connectivity between control plane nodes
  2. Slow disk I/O causing etcd write-ahead log (WAL) synchronization delays
  3. High CPU or memory usage starving etcd of resources
  4. Maintenance operations like etcd snapshots triggering temporary leader changes
  5. Insufficient quorum causing failed elections
  6. Poor cluster topology (even number of nodes)

How to fix "etcdserver: leader changed"

1Check cluster infrastructure health

Diagnose network and disk performance:

bash
# Network latency between control nodes:
ping -c 100 <other-control-node-ip> | tail -1

# Disk I/O performance:
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread \
  --bs=4k --direct=1 --size=1G --numjobs=1 \
  --runtime=60 --group_reporting --directory=/var/lib/etcd

Look for:
- Network latency > 100ms (indicates potential issue)
- Disk read latency > 10ms

2Monitor etcd metrics

Check for excessive leader elections:

bash
# Prometheus query (if Prometheus installed):
etcd_server_has_leader  # Should be 1 (has leader)
etcd_server_leader_changes_seen_total  # Should be low

# Or check logs:
kubectl logs -n kube-system <etcd-pod> | grep "elected leader"
kubectl logs -n kube-system <etcd-pod> | grep "leader changed"

Frequent elections (> 1 per minute) indicate problems.

3Verify etcd cluster health

Check all etcd members are responding:

bash
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 \
  endpoint health

# Check member list:
ETCDCTL_API=3 etcdctl member list

All members should show healthy. Remove stale members if any show down.

4Clean up accumulated objects in etcd

Many objects in etcd slow down lookups:

bash
# Check pod count (rough etcd size indicator):
kubectl get pods --all-namespaces | wc -l

# Remove evicted or error pods:
kubectl delete pods --all-namespaces --field-selector=status.phase=Failed
kubectl delete pods --all-namespaces --field-selector=status.reason=Evicted

Large numbers of accumulated objects slow etcd responses.

5Monitor disk space for etcd

Ensure etcd has enough space:

bash
df -h /var/lib/etcd
ls -lh /var/lib/etcd/member/snap/db

Etcd has a default 2GB quota. Monitor size growth:

bash
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status

If approaching quota, increase it or compact the database.

6Defragment etcd if fragmented

After deleting many objects, defragment:

bash
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag

Defragmentation may trigger a leader election temporarily.

7Verify cluster topology is optimal

Check number of etcd members:

bash
ETCDCTL_API=3 etcdctl member list

Recommended:
- 3 members (tolerate 1 failure)
- 5 members (tolerate 2 failures)
- Avoid even numbers (4, 6) as quorum becomes fragile

If using even number, add/remove to make odd.

8Increase cluster resource allocation

Dedicate more resources to etcd:

bash
# For static pods, edit manifest:
sudo vi /etc/kubernetes/manifests/etcd.yaml

# Increase CPU/memory requests:
resources:
  requests:
    cpu: 500m  # Increase from 100m
    memory: 2Gi  # Increase from 1Gi

Then restart etcd:

bash
sudo systemctl restart kubelet  # Picks up manifest change
9Monitor for platform-specific snapshot operations

Some platforms (Azure AKS) take automated etcd snapshots:

bash
# Check Azure AKS backup settings:
az aks show --name <cluster> --resource-group <group>

Snapshots can trigger temporary leader changes. These are expected and not concerning if infrequent.

For multi-control-plane clusters, these operations should not be frequent.

Advanced notes

Leader changes are Kubernetes' self-healing mechanism—the cluster is working correctly even during elections. What matters is frequency and stability afterward. A single leader change is not a problem; multiple changes per minute indicate infrastructure issues. Network latency is the most common cause in cloud environments; use dedicated high-performance networking for control plane nodes. For very large clusters (1000+ nodes), consider splitting etcd into a dedicated external cluster. Kubernetes 1.20+ has better etcd performance optimizations; upgrading may help. Monitor etcd_disk_backend_commit_duration_seconds metric to identify slow disk operations.

Frequently asked questions

What does "etcdserver: leader changed" mean?
Etcd uses a distributed consensus algorithm with one elected leader. When the leader fails to send heartbeats within the election timeout, followers initiate a new election and elect a different leader. The "leader changed" error indicates this just happened.
How do I fix "etcdserver: leader changed"?
The etcd leader changed error indicates the etcd cluster detected a leadership change during operation. While occasional changes are normal, frequent changes signal network issues, slow disk performance, or resource constraints that prevent timely heartbeats.
What causes "etcdserver: leader changed"?
Network latency or intermittent connectivity between control plane nodes Slow disk I/O causing etcd write-ahead log (WAL) synchronization delays High CPU or memory usage starving etcd of resources
Is the "etcdserver: leader changed" error serious?
It is a medium-severity error — more of a blocker than a risk. The steps above resolve it.

Official resources & further reading