IP pool exhaustion occurs when all available IP addresses in the Calico IP pool are allocated. Resolve by expanding the pool CIDR, reclaiming unused IPs, or monitoring IP usage.
Calico manages IP address allocation for pods using IP pools. When an IP pool is exhausted, there are no more available IP addresses to assign to new pods. This prevents pod creation and stops scaling operations. The issue occurs when the CIDR range configured for the pool is too small for the number of pods needed.
Get IP pool configuration and check available addresses:
calicoctl get ippools -o yaml
calicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"Determine how many IPs you need:
- Max pods in cluster = nodes × pods per node
- Add buffer for growth (usually 1.5-2x expected peak)
- Account for system pods, daemonsets, statefulsets
Example: 10 nodes × 50 pods/node × 2 = 1000 IPs needed
Add a new IP pool with larger CIDR range:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: large-pool
spec:
cidr: 10.48.0.0/16
blockSize: 25
EOFThen disable the old pool and migrate pods.
Reduce per-node allocation to conserve IPs:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
blockSize: 26 # Allocate /26 to each node (64 IPs) instead of /25 (128)
EOFSmaller blocks = fewer wasted IPs but requires more fragmentation management.
Gradually migrate from old to new pool:
# Disable old pool to prevent new allocation
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: old-pool
spec:
disabled: true
EOF
# Delete pods to reassign from new pool
kubectl rollout restart deployment -n your-namespaceClean up IPs not properly released:
# Check IP allocation status
calicoctl ipam show
# Release stuck allocations
calicoctl ipam release --ip <orphaned-ip>Set up monitoring to prevent future exhaustion:
# Get current usage
calicoctl get ippool default-ipv4-ippool -o yaml | grep -E "cidr|available"
# Create Prometheus alerts
alerting_rule:
- alert: CalicoIPPoolExhaustionWarning
expr: calico_ippool_available_ips < 100For large clusters, implement automation:
1. Monitor available IPs via Prometheus
2. Create alerts at 20%, 10%, 5% available
3. Use GitOps to automatically add new pools
4. Size pools to support expected growth (2-3 years)
For production clusters, plan IP pool sizing during initial deployment - retroactive expansion requires careful pod migration. Use multiple smaller IP pools rather than one large pool to limit blast radius of exhaustion. Consider IPv6 pools in addition to IPv4 for organizations with IPv6 infrastructure. Monitor IPAM churn rate to detect abnormal IP allocation patterns. For dynamic clusters with frequent scaling, leave 40-50% buffer in IP pools.
Failed to connect to server: connection refused (HTTP/2)
How to fix "HTTP/2 connection refused" error in Kubernetes
missing request for cpu in container
How to fix "missing request for cpu in container" in Kubernetes HPA
error: invalid configuration
How to fix "error: invalid configuration" in Kubernetes
etcdserver: cluster ID mismatch
How to fix "etcdserver: cluster ID mismatch" in Kubernetes
running with swap on is not supported
How to fix "running with swap on is not supported" in kubeadm