How to fix endpoint not ready in Kubernetes
Cilium endpoint not ready errors occur when pods fail to establish network connectivity. Fix by checking node networking, verifying pod IP allocation, and investigating eBPF program issues.
Quick fix
Fixes endpoint not ready
kubectl exec -it -n kube-system ds/cilium -- \
cilium-dbg endpoint list
kubectl exec -it -n kube-system ds/cilium -- \
cilium-dbg endpoint get <endpoint-id>- Check Cilium endpoint status
- Verify pod received IP address
- Monitor network traffic with Cilium monitor
- Verify IP forwarding is enabled
kubectl exec -it -n kube-system ds/cilium -- \cilium-dbg endpoint listkubectl exec -it -n kube-system ds/cilium -- \cilium-dbg endpoint get <endpoint-id>endpoint not ready
What this error means
In Cilium, an endpoint represents a pod's network interface state. When endpoints are not ready, pods cannot communicate with other pods, services, or external networks. This typically indicates issues with IP allocation, routing configuration, or eBPF program enforcement on the node.
Typical symptoms
- Pods unable to communicate across nodes
- Cilium endpoint list shows unhealthy endpoints
- Network connectivity working on some nodes but not others
- Pods on same node communicate, but cross-node fails
- cilium-dbg endpoint list shows endpoints in Failed state
- Packet drops visible in cilium monitor
- Pods cannot reach services or external networks
- DNS queries fail or timeout
Common causes
- IP address not properly allocated to pod
- eBPF programs not loaded or enforcing incorrectly
- Network routing not configured between nodes
- Node-to-node connectivity blocked by firewall
- Pod network namespace not properly initialized
- Cilium node selector not matching all nodes
- IP forwarding disabled on host
- Conflicting network policies or firewall rules
How to fix "endpoint not ready"
List and examine endpoint states:
kubectl exec -it -n kube-system ds/cilium -- \
cilium-dbg endpoint list
kubectl exec -it -n kube-system ds/cilium -- \
cilium-dbg endpoint get <endpoint-id>Check if pod has valid IP assigned:
kubectl get pod <pod-name> -o wide
kubectl exec -it <pod-name> -- ip addr
kubectl exec -it <pod-name> -- ip routeIf no IP, the IPAM system failed allocation.
Observe packet flow and drops:
kubectl exec -it -n kube-system ds/cilium -- \
cilium monitor -t drop
kubectl exec -it -n kube-system ds/cilium -- \
cilium monitor --type=traceLook for packet drops and their reason codes.
Check host kernel forwarding configuration:
sysctl net.ipv4.ip_forward
sysctl net.ipv6.conf.all.forwarding
# Enable if needed:
sudo sysctl -w net.ipv4.ip_forward=1Verify direct connectivity between nodes:
# From one node
ping <other-node-ip>
# Test from pod
kubectl exec -it <pod-name> -- ping <pod-on-other-node>Check if eBPF programs are properly loaded:
kubectl exec -it -n kube-system ds/cilium -- \
cilium-dbg bpf listIf programs are missing, agent may not have started properly.
Identify if network policies are causing drops:
kubectl get networkpolicy --all-namespaces
kubectl describe networkpolicy <policy-name>Verify policies allow traffic between pods.
Force reinit of Cilium on node:
kubectl delete pod -n kube-system -l app=cilium \
--field-selector spec.nodeName=<node-name>
kubectl wait --for=condition=Ready pod \
-n kube-system -l app=cilium --timeout=300sAdvanced notes
For production, use Cilium Hubble for continuous network monitoring. Test connectivity between every pair of nodes during cluster setup. Enable debug logging on Cilium agent for endpoint state machine tracking. For large clusters, use node selectors to limit Cilium deployment. Monitor endpoint creation/deletion metrics for high churn indicating instability.
Frequently asked questions
- What does "endpoint not ready" mean?
- In Cilium, an endpoint represents a pod's network interface state. When endpoints are not ready, pods cannot communicate with other pods, services, or external networks. This typically indicates issues with IP allocation, routing configuration, or eBPF program enforcement on the node.
- How do I fix "endpoint not ready"?
- Cilium endpoint not ready errors occur when pods fail to establish network connectivity. Fix by checking node networking, verifying pod IP allocation, and investigating eBPF program issues.
- What causes "endpoint not ready"?
- IP address not properly allocated to pod eBPF programs not loaded or enforcing incorrectly Network routing not configured between nodes
- Is the "endpoint not ready" error serious?
- Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.
Official resources & further reading
- ๐Kubernetes Official Documentation
- ๐Cilium Troubleshooting(Official Docs)
- ๐Unreachable Endpoints on Specific Nodes(Official Docs)
- ๐