How to fix Error: context deadline exceeded (etcd) in Kubernetes

KubernetesADVANCEDCRITICAL

The etcd context deadline exceeded error indicates API requests are timing out when communicating with the etcd cluster. This is critical because etcd stores all Kubernetes cluster data and its unavailability affects the entire cluster.

What this error means

Etcd client requests (from kube-apiserver or etcdctl) are failing to reach or receive responses from the etcd cluster within the default timeout window (typically 5 seconds). Since etcd is the backing database for all Kubernetes state, this error prevents API server operations like creating pods, updating deployments, or querying cluster state.

Typical symptoms

How to fix "Error: context deadline exceeded (etcd)"

1Verify etcd cluster health

Check if etcd members are responding:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

All members should show "healthy: true". If any show "unhealthy" or timeout, that's the problem.

2Verify network connectivity between nodes

Test reachability between etcd members:

bash

ping <etcd-member-ip>
telnet <etcd-member-ip> 2379  # Client port
telnet <etcd-member-ip> 2380  # Peer port

Check firewall rules:

bash

sudo iptables -L -n | grep 2379
sudo iptables -L -n | grep 2380

Open ports if blocked:

bash

sudo iptables -A INPUT -p tcp --dport 2379 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 2380 -j ACCEPT

3Check etcd certificate configuration

Verify TLS certs are correct:

bash

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 \
  member list

If cert errors occur, certs may be expired or misconfigured. Regenerate if needed:

bash

kubeadm certs renew etcd-server
kubeadm certs renew etcd-peer

Then restart etcd.

4Increase timeout for diagnosis

Give etcd more time to respond:

bash

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --command-timeout=120s \
  --endpoints=https://127.0.0.1:2379 \
  endpoint health

If this succeeds, etcd is slow. Check for performance issues (disk I/O, CPU).

5Check etcd cluster quorum

Ensure majority of members are healthy:

bash

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 \
  member list

Count healthy members. For quorum:
- 3-member cluster needs 2 healthy
- 5-member cluster needs 3 healthy

If quorum is lost, remove unhealthy members:

bash

ETCDCTL_API=3 etcdctl member remove <member-id>

6Monitor etcd performance

Check if etcd is slow:

bash

# Monitor metrics:
kubectl logs -n kube-system -l component=etcd | grep slowRequestDuration

# Check CPU/memory:
kubectl top pod -n kube-system -l component=etcd

# Check database size:
ls -lh /var/lib/etcd/member/snap/db

If slow, defragment the database:

bash

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 \
  defrag

7Check API server logs for specifics

Get more details on the timeout:

bash

kubectl logs -n kube-system -l component=kube-apiserver | grep deadline
kubectl logs -n kube-system -l component=kube-apiserver | grep etcd

Look for:
- "context deadline exceeded" - request timed out
- "connection refused" - etcd not reachable
- "x509" - certificate issue

8Restart etcd as last resort

Rolling restart of etcd members (one at a time, wait for recovery):

bash

sudo systemctl restart etcd
# or for containerized:
sudo docker restart <etcd-container>

Monitor health between restarts:

bash

watch "ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://127.0.0.1:2379 endpoint health"

Wait 10-30 seconds between restarts for leader election.

How to fix Error: context deadline exceeded (etcd) in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "Error: context deadline exceeded (etcd)"

Advanced notes

Related errors

Official resources & further reading