How to fix etcdserver: member not found in Kubernetes

KubernetesADVANCEDCRITICAL

This error occurs when an operation references a non-existent etcd cluster member, typically after a node removal, incomplete cluster scaling, or misconfigured member IDs. It requires careful member list reconciliation to resolve.

What this error means

The 'etcdserver: member not found' error indicates that an operation (such as promoting, removing, or updating a member) is being performed on a member ID that doesn't exist in the etcd cluster's membership registry. This typically happens when a member was removed from the cluster but references to it still exist in configuration. The error reflects a mismatch between the intended cluster topology and the actual cluster state—the etcd cluster maintains a list of active members, and any operation referencing a non-existent member ID will fail. This is a critical issue because unresolved member mismatches can lead to partial quorum loss, cluster instability, and an inability to schedule new workloads on Kubernetes.

How to fix "etcdserver: member not found"

1Verify current etcd cluster membership status

Connect to a healthy etcd member and list all current members:

bash

kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member list

Record the member IDs and their peer URLs. Look for members with status 'unstarted' or invalid names.

2Check and update the kubeadm ConfigMap

The kubeadm ConfigMap stores etcd cluster endpoints. Export and inspect it:

bash

kubectl get configmap kubeadm-config -n kube-system -o yaml > kubeadm-config.yaml

Edit the file and locate the etcd section. Remove any references to deleted or invalid nodes in the initialCluster entries.

Apply the updated configuration:

bash

kubectl apply -f kubeadm-config.yaml

3Remove stale or phantom members from the cluster

Use the member IDs from step 1 to remove members that are 'unstarted' or reference deleted nodes:

bash

kubectl exec etcd-<master-node-name> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member remove <member-id>

Only remove members if you have a quorum (majority) of healthy members remaining.

4Add the node back as a new member

If the node needs to rejoin the cluster, use member add with correct peer URLs:

bash

kubectl exec etcd-<healthy-master-node> -n kube-system -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --endpoints https://127.0.0.1:2379 \
  member add <new-member-name> \
  --peer-urls=https://<NODE_IP>:2380

Save the output values (ETCD_NAME, ETCD_INITIAL_CLUSTER, ETCD_INITIAL_CLUSTER_STATE=existing) for the next step.

5Stop and clean up etcd data on the rejoining node

Before restarting etcd on the node being re-added, clear its existing state:

bash

# On the node being re-added, stop etcd first
sudo systemctl stop etcd

# Or if running as a static pod:
kubectl delete pod etcd-<node-name> -n kube-system

# Remove the stale data directory (BE CAREFUL - this is destructive)
sudo rm -rf /var/lib/etcd/member

# Verify it's gone
ls -la /var/lib/etcd/

6Restart etcd with the correct initial cluster configuration

Use the values from step 4 to restart etcd on the rejoining node:

bash

# For static pod (most common in Kubernetes):
# Edit /etc/kubernetes/manifests/etcd.yaml and ensure:
# - ETCD_INITIAL_CLUSTER contains the full member list
# - ETCD_INITIAL_CLUSTER_STATE=existing (not new)
# - The node name matches what was provided in 'member add'

# Start kubelet to restart the static pods
sudo systemctl start kubelet

# Monitor startup
sudo journalctl -u etcd -f

The node should sync its state from the cluster leader within seconds.

How to fix etcdserver: member not found in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "etcdserver: member not found"

Advanced notes

Related errors

Official resources & further reading