How to fix IP pool is exhausted in Kubernetes
IP pool exhaustion occurs when all available IP addresses in the Calico IP pool are allocated. Resolve by expanding the pool CIDR, reclaiming unused IPs, or monitoring IP usage.
Quick fix
Fixes IP pool is exhausted
calicoctl get ippools -o yaml
calicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"- Check current IP pool status and usage
- Calculate IP requirements
- Create a larger IP pool
- Adjust block size if using multiple pools
calicoctl get ippools -o yamlcalicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"IP pool is exhausted
What this error means
Calico manages IP address allocation for pods using IP pools. When an IP pool is exhausted, there are no more available IP addresses to assign to new pods. This prevents pod creation and stops scaling operations. The issue occurs when the CIDR range configured for the pool is too small for the number of pods needed.
Typical symptoms
- New pods fail to create with "Failed to allocate IP address"
- Error message "IP pool is exhausted" or "CIDRNotAvailable"
- Scaling deployments fails after reaching certain replica count
- calicoctl reports IP pool with 0 available addresses
- Pods stuck in Pending state unable to get IP
- Node cannot accept new pods due to lack of addresses
- IPAM errors in Calico logs indicating address exhaustion
- Cluster cannot autoscale due to IP shortage
Common causes
- IP pool CIDR range too small for number of pods needed
- Block size (per-node allocation) consuming too many addresses
- Orphaned IPs from deleted pods not being released
- Pods being deleted but IPs not reclaimed quickly
- High pod churn causing rapid IP allocation/deallocation
- Multiple IP pools overlapping or duplicated
- Node configuration reserving too many IP addresses
- External integrations consuming IPs unexpectedly
How to fix "IP pool is exhausted"
Get IP pool configuration and check available addresses:
calicoctl get ippools -o yaml
calicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"Determine how many IPs you need:
- Max pods in cluster = nodes × pods per node
- Add buffer for growth (usually 1.5-2x expected peak)
- Account for system pods, daemonsets, statefulsets
Example: 10 nodes × 50 pods/node × 2 = 1000 IPs needed
Add a new IP pool with larger CIDR range:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: large-pool
spec:
cidr: 10.48.0.0/16
blockSize: 25
EOFThen disable the old pool and migrate pods.
Reduce per-node allocation to conserve IPs:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
blockSize: 26 # Allocate /26 to each node (64 IPs) instead of /25 (128)
EOFSmaller blocks = fewer wasted IPs but requires more fragmentation management.
Gradually migrate from old to new pool:
# Disable old pool to prevent new allocation
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: old-pool
spec:
disabled: true
EOF
# Delete pods to reassign from new pool
kubectl rollout restart deployment -n your-namespaceClean up IPs not properly released:
# Check IP allocation status
calicoctl ipam show
# Release stuck allocations
calicoctl ipam release --ip <orphaned-ip>Set up monitoring to prevent future exhaustion:
# Get current usage
calicoctl get ippool default-ipv4-ippool -o yaml | grep -E "cidr|available"
# Create Prometheus alerts
alerting_rule:
- alert: CalicoIPPoolExhaustionWarning
expr: calico_ippool_available_ips < 100For large clusters, implement automation:
1. Monitor available IPs via Prometheus
2. Create alerts at 20%, 10%, 5% available
3. Use GitOps to automatically add new pools
4. Size pools to support expected growth (2-3 years)
Advanced notes
For production clusters, plan IP pool sizing during initial deployment - retroactive expansion requires careful pod migration. Use multiple smaller IP pools rather than one large pool to limit blast radius of exhaustion. Consider IPv6 pools in addition to IPv4 for organizations with IPv6 infrastructure. Monitor IPAM churn rate to detect abnormal IP allocation patterns. For dynamic clusters with frequent scaling, leave 40-50% buffer in IP pools.
Frequently asked questions
- What does "IP pool is exhausted" mean?
- Calico manages IP address allocation for pods using IP pools. When an IP pool is exhausted, there are no more available IP addresses to assign to new pods. This prevents pod creation and stops scaling operations.
- How do I fix "IP pool is exhausted"?
- IP pool exhaustion occurs when all available IP addresses in the Calico IP pool are allocated. Resolve by expanding the pool CIDR, reclaiming unused IPs, or monitoring IP usage.
- What causes "IP pool is exhausted"?
- IP pool CIDR range too small for number of pods needed Block size (per-node allocation) consuming too many addresses Orphaned IPs from deleted pods not being released
- Is the "IP pool is exhausted" error serious?
- Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.
Official resources & further reading
- 📚Kubernetes Official Documentation
- 📖IP Pool Exhaustion(Official Docs)
- 📖Calico IP Address Exhaustion(Official Docs)
- 🐙IPAM Pod IPs Not Released(GitHub)