How to fix node.kubernetes.io/unreachable in Kubernetes

Run:

kubectl get nodes
kubectl describe node <node-name>

Look for taint: "node.kubernetes.io/unreachable:NoExecute". Check the "Ready" condition—it should show "False" or "Unknown". Examine recent events for clues about what caused unreachability.

Try to SSH to the node and check if kubelet is running:

ssh <node-ip>
sudo systemctl status kubelet
sudo journalctl -u kubelet -n 50  # Last 50 kubelet logs

If kubelet is down, restart it:

sudo systemctl restart kubelet

Monitor status:

kubectl get nodes -w  # Watch for status changes

From control plane, verify you can reach the node:

ping <node-ip>
kubectl get --raw /api/v1/nodes/<node-name>/proxy/metrics

Check kubelet port (10250):

netstat -tlnp | grep 10250  # On the node
curl -k https://<node-ip>:10250/metrics  # From control plane

If firewall is blocking, open required ports: 10250 (kubelet), 10255 (metrics).

IPv4 forwarding is required for pod networking. Check:

cat /proc/sys/net/ipv4/ip_forward

If it returns 0, enable it:

sudo sysctl -w net.ipv4.ip_forward=1

Make permanent by editing /etc/sysctl.conf:

net.ipv4.ip_forward = 1

Then apply: sudo sysctl -p

Check installed CNI plugin:

kubectl get ds -n kube-system  # DaemonSets including CNI
kubectl logs -n kube-system <cni-pod>  # Check logs for errors

Common CNI plugins: flannel, calico, weave. Verify:
- Pods are running: kubectl get pods -n kube-system | grep <cni>
- Node CIDR assignment: kubectl get node <node-name> -o jsonpath="{.spec.podCIDR}"

If misconfigured, redeploy CNI plugin from official manifests.

Review firewall rules:

sudo iptables -L -n  # On the node
sudo firewall-cmd --list-all  # If using firewalld

Ensure these ports are open:
- 10250/TCP (kubelet)
- 10255/TCP (kubelet metrics)
- 8285-8472/UDP (Flannel VxLAN, if using Flannel)
- 179/TCP (Calico BGP, if using Calico)

For testing, temporarily disable firewall:

sudo systemctl stop firewalld

If this fixes it, add proper firewall rules instead.

If temporary unreachability is expected, increase toleration:

spec:
  tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 6000  # 100 minutes instead of 300s
  # ... rest of pod spec

For critical pods that must stay running, increase tolerationSeconds even more or remove the tolerance entirely (not recommended for most workloads).

If the node cannot be recovered, gracefully remove it:

kubectl drain <node-name> --ignore-daemonsets
kubectl delete node <node-name>

For cloud providers (AWS, GCP, Azure), the node may auto-rejoin if the instance is still running. Stop the instance if you want permanent removal:

# AWS EC2
aws ec2 stop-instances --instance-ids <instance-id>

Watch the node status after recovery attempts:

kubectl get nodes -w
kubectl describe node <node-name>

Once Ready condition returns to True, the taint is automatically removed and pods can be scheduled again. Check pod eviction status:

kubectl get pods --all-namespaces --field-selector=status.reason=Evicted

Evicted pods need to be manually recreated (via deployments, etc.).