How to fix etcdserver: member not found in Kubernetes
This error occurs when an operation references a non-existent etcd cluster member, typically after a node removal, incomplete cluster scaling, or misconfigured member IDs. It requires careful member list reconciliation to resolve.
Quick fix
Fixes etcdserver: member not found
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://127.0.0.1:2379 \
member list- Verify current etcd cluster membership status
- Check and update the kubeadm ConfigMap
- Remove stale or phantom members from the cluster
- Add the node back as a new member
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \--cacert /etc/kubernetes/pki/etcd/ca.crt \--cert /etc/kubernetes/pki/etcd/peer.crt \--key /etc/kubernetes/pki/etcd/peer.key \--endpoints https://127.0.0.1:2379 \member listetcdserver: member not found
What this error means
The 'etcdserver: member not found' error indicates that an operation (such as promoting, removing, or updating a member) is being performed on a member ID that doesn't exist in the etcd cluster's membership registry. This typically happens when a member was removed from the cluster but references to it still exist in configuration. The error reflects a mismatch between the intended cluster topology and the actual cluster state—the etcd cluster maintains a list of active members, and any operation referencing a non-existent member ID will fail. This is a critical issue because unresolved member mismatches can lead to partial quorum loss, cluster instability, and an inability to schedule new workloads on Kubernetes.
Typical symptoms
- API server cannot create, update, or delete resources and returns "member not found" errors
- kubectl commands hang or time out when attempting state changes
- etcdctl member list returns different results depending on which cluster member is queried
- Master/control plane nodes fail to join the cluster after scaling operations
- etcd pods are in CrashLoopBackOff state with member-related error logs
- Cluster shows healthy RAFT consensus but operations still fail due to member inconsistency
Common causes
- A control plane node was removed without properly running "etcdctl member remove" before shutdown
- Node was decommissioned but kubeadm ConfigMap still references it as an API endpoint
- During cluster scaling, members were added with incorrect peer URLs or names
- etcd was upgraded to version 3.5.0 which has a known bug causing "zombie members" to reappear
- Attempting to promote, remove, or update a member using an ID that was never properly added
- Configuration mismatch between initial-cluster and actual member list after network splits
- New member added as a learner but configuration was incomplete
How to fix "etcdserver: member not found"
Connect to a healthy etcd member and list all current members:
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://127.0.0.1:2379 \
member listRecord the member IDs and their peer URLs. Look for members with status 'unstarted' or invalid names.
The kubeadm ConfigMap stores etcd cluster endpoints. Export and inspect it:
kubectl get configmap kubeadm-config -n kube-system -o yaml > kubeadm-config.yamlEdit the file and locate the etcd section. Remove any references to deleted or invalid nodes in the initialCluster entries.
Apply the updated configuration:
kubectl apply -f kubeadm-config.yamlUse the member IDs from step 1 to remove members that are 'unstarted' or reference deleted nodes:
kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://127.0.0.1:2379 \
member remove <member-id>Only remove members if you have a quorum (majority) of healthy members remaining.
If the node needs to rejoin the cluster, use member add with correct peer URLs:
kubectl exec etcd-<healthy-master-node> -n kube-system -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://127.0.0.1:2379 \
member add <new-member-name> \
--peer-urls=https://<NODE_IP>:2380Save the output values (ETCD_NAME, ETCD_INITIAL_CLUSTER, ETCD_INITIAL_CLUSTER_STATE=existing) for the next step.
Before restarting etcd on the node being re-added, clear its existing state:
# On the node being re-added, stop etcd first
sudo systemctl stop etcd
# Or if running as a static pod:
kubectl delete pod etcd-<node-name> -n kube-system
# Remove the stale data directory (BE CAREFUL - this is destructive)
sudo rm -rf /var/lib/etcd/member
# Verify it's gone
ls -la /var/lib/etcd/Use the values from step 4 to restart etcd on the rejoining node:
# For static pod (most common in Kubernetes):
# Edit /etc/kubernetes/manifests/etcd.yaml and ensure:
# - ETCD_INITIAL_CLUSTER contains the full member list
# - ETCD_INITIAL_CLUSTER_STATE=existing (not new)
# - The node name matches what was provided in 'member add'
# Start kubelet to restart the static pods
sudo systemctl start kubelet
# Monitor startup
sudo journalctl -u etcd -fThe node should sync its state from the cluster leader within seconds.
Advanced notes
etcd uses a membership registry to track cluster members. Members have unique IDs (UUIDs) and names. When a node is removed, its member entry must be explicitly deleted via 'etcdctl member remove' to prevent 'member not found' errors.
Known Bug in etcd 3.5.0: This version introduced a bug that resurrects 'zombie members' from ancient cluster states. Upgrade to etcd 3.5.2 or later to resolve this.
Learner Members: When adding a member, it can be added as a learner (read-only) until it catches up with the cluster snapshot. Use 'etcdctl member promote <member-id>' to convert it to a voting member.
Quorum Loss Prevention: Only remove members when you have a confirmed quorum (majority) of healthy members. Removing a member when quorum is already lost will cause permanent cluster failure.
Disaster Recovery: If more than half the cluster is failed (quorum loss), individual member recovery won't work. You must restore from an etcd snapshot backup using 'etcdctl snapshot restore' and rebuild with '--force-new-cluster'.
Frequently asked questions
- What does "etcdserver: member not found" mean?
- The 'etcdserver: member not found' error indicates that an operation (such as promoting, removing, or updating a member) is being performed on a member ID that doesn't exist in the etcd cluster's membership registry. This typically happens when a member was removed from the cluster but references to it still exist in configuration. The error reflects a mismatch between the intended cluster topology and the actual cluster state—the etcd cluster maintains a list of active members, and any operation referencing a non-existent member ID will fail.
- How do I fix "etcdserver: member not found"?
- This error occurs when an operation references a non-existent etcd cluster member, typically after a node removal, incomplete cluster scaling, or misconfigured member IDs. It requires careful member list reconciliation to resolve.
- What causes "etcdserver: member not found"?
- A control plane node was removed without properly running "etcdctl member remove" before shutdown Node was decommissioned but kubeadm ConfigMap still references it as an API endpoint During cluster scaling, members were added with incorrect peer URLs or names
- Is the "etcdserver: member not found" error serious?
- Yes. This is a critical-severity error that usually blocks work until it is fixed, but the steps above resolve it.
Official resources & further reading
- 📚Kubernetes Official Documentation
- 📖Docs: etcd administration(Official Docs)
- 💬SO: etcd member issues(Stack Overflow)
- 🐙GitHub: etcd repository(GitHub)