How to fix etcdserver: member not found in Kubernetes

KubernetesADVANCEDCRITICALUpdated Dec 5, 2025· 152 views

This error occurs when an operation references a non-existent etcd cluster member, typically after a node removal, incomplete cluster scaling, or misconfigured member IDs. It requires careful member list reconciliation to resolve.

Quick fix

Fixes etcdserver: member not found

kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member list
  1. Verify current etcd cluster membership status
  2. Check and update the kubeadm ConfigMap
  3. Remove stale or phantom members from the cluster
  4. Add the node back as a new member
kubernetes — error
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://127.0.0.1:2379 \
member list
etcdserver: member not found

What this error means

The 'etcdserver: member not found' error indicates that an operation (such as promoting, removing, or updating a member) is being performed on a member ID that doesn't exist in the etcd cluster's membership registry. This typically happens when a member was removed from the cluster but references to it still exist in configuration. The error reflects a mismatch between the intended cluster topology and the actual cluster state—the etcd cluster maintains a list of active members, and any operation referencing a non-existent member ID will fail. This is a critical issue because unresolved member mismatches can lead to partial quorum loss, cluster instability, and an inability to schedule new workloads on Kubernetes.

Typical symptoms

  • API server cannot create, update, or delete resources and returns "member not found" errors
  • kubectl commands hang or time out when attempting state changes
  • etcdctl member list returns different results depending on which cluster member is queried
  • Master/control plane nodes fail to join the cluster after scaling operations
  • etcd pods are in CrashLoopBackOff state with member-related error logs
  • Cluster shows healthy RAFT consensus but operations still fail due to member inconsistency

Common causes

  1. A control plane node was removed without properly running "etcdctl member remove" before shutdown
  2. Node was decommissioned but kubeadm ConfigMap still references it as an API endpoint
  3. During cluster scaling, members were added with incorrect peer URLs or names
  4. etcd was upgraded to version 3.5.0 which has a known bug causing "zombie members" to reappear
  5. Attempting to promote, remove, or update a member using an ID that was never properly added
  6. Configuration mismatch between initial-cluster and actual member list after network splits
  7. New member added as a learner but configuration was incomplete

How to fix "etcdserver: member not found"

1Verify current etcd cluster membership status

Connect to a healthy etcd member and list all current members:

bash
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member list

Record the member IDs and their peer URLs. Look for members with status 'unstarted' or invalid names.

2Check and update the kubeadm ConfigMap

The kubeadm ConfigMap stores etcd cluster endpoints. Export and inspect it:

bash
kubectl get configmap kubeadm-config -n kube-system -o yaml > kubeadm-config.yaml

Edit the file and locate the etcd section. Remove any references to deleted or invalid nodes in the initialCluster entries.

Apply the updated configuration:

bash
kubectl apply -f kubeadm-config.yaml
3Remove stale or phantom members from the cluster

Use the member IDs from step 1 to remove members that are 'unstarted' or reference deleted nodes:

bash
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member remove <member-id>

Only remove members if you have a quorum (majority) of healthy members remaining.

4Add the node back as a new member

If the node needs to rejoin the cluster, use member add with correct peer URLs:

bash
kubectl exec etcd-<healthy-master-node> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member add <new-member-name> \
  --peer-urls=https://<NODE_IP>:2380

Save the output values (ETCD_NAME, ETCD_INITIAL_CLUSTER, ETCD_INITIAL_CLUSTER_STATE=existing) for the next step.

5Stop and clean up etcd data on the rejoining node

Before restarting etcd on the node being re-added, clear its existing state:

bash
# On the node being re-added, stop etcd first
sudo systemctl stop etcd

# Or if running as a static pod:
kubectl delete pod etcd-<node-name> -n kube-system

# Remove the stale data directory (BE CAREFUL - this is destructive)
sudo rm -rf /var/lib/etcd/member

# Verify it's gone
ls -la /var/lib/etcd/
6Restart etcd with the correct initial cluster configuration

Use the values from step 4 to restart etcd on the rejoining node:

bash
# For static pod (most common in Kubernetes):
# Edit /etc/kubernetes/manifests/etcd.yaml and ensure:
# - ETCD_INITIAL_CLUSTER contains the full member list
# - ETCD_INITIAL_CLUSTER_STATE=existing (not new)
# - The node name matches what was provided in 'member add'

# Start kubelet to restart the static pods
sudo systemctl start kubelet

# Monitor startup
sudo journalctl -u etcd -f

The node should sync its state from the cluster leader within seconds.

Advanced notes

etcd uses a membership registry to track cluster members. Members have unique IDs (UUIDs) and names. When a node is removed, its member entry must be explicitly deleted via 'etcdctl member remove' to prevent 'member not found' errors.

Known Bug in etcd 3.5.0: This version introduced a bug that resurrects 'zombie members' from ancient cluster states. Upgrade to etcd 3.5.2 or later to resolve this.

Learner Members: When adding a member, it can be added as a learner (read-only) until it catches up with the cluster snapshot. Use 'etcdctl member promote <member-id>' to convert it to a voting member.

Quorum Loss Prevention: Only remove members when you have a confirmed quorum (majority) of healthy members. Removing a member when quorum is already lost will cause permanent cluster failure.

Disaster Recovery: If more than half the cluster is failed (quorum loss), individual member recovery won't work. You must restore from an etcd snapshot backup using 'etcdctl snapshot restore' and rebuild with '--force-new-cluster'.

Frequently asked questions

What does "etcdserver: member not found" mean?
The 'etcdserver: member not found' error indicates that an operation (such as promoting, removing, or updating a member) is being performed on a member ID that doesn't exist in the etcd cluster's membership registry. This typically happens when a member was removed from the cluster but references to it still exist in configuration. The error reflects a mismatch between the intended cluster topology and the actual cluster state—the etcd cluster maintains a list of active members, and any operation referencing a non-existent member ID will fail.
How do I fix "etcdserver: member not found"?
This error occurs when an operation references a non-existent etcd cluster member, typically after a node removal, incomplete cluster scaling, or misconfigured member IDs. It requires careful member list reconciliation to resolve.
What causes "etcdserver: member not found"?
A control plane node was removed without properly running "etcdctl member remove" before shutdown Node was decommissioned but kubeadm ConfigMap still references it as an API endpoint During cluster scaling, members were added with incorrect peer URLs or names
Is the "etcdserver: member not found" error serious?
Yes. This is a critical-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading