IP pool exhaustion occurs when all available IP addresses in the Calico IP pool are allocated. Resolve by expanding the pool CIDR, reclaiming unused IPs, or monitoring IP usage.
Fixes IP pool is exhausted
calicoctl get ippools -o yaml
calicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"calicoctl get ippools -o yamlcalicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"IP pool is exhausted
Calico manages IP address allocation for pods using IP pools. When an IP pool is exhausted, there are no more available IP addresses to assign to new pods. This prevents pod creation and stops scaling operations. The issue occurs when the CIDR range configured for the pool is too small for the number of pods needed.
Get IP pool configuration and check available addresses:
calicoctl get ippools -o yaml
calicoctl get ippool default-ipv4-ippool -o yaml | grep -A 5 "cidr|blockSize"Determine how many IPs you need:
- Max pods in cluster = nodes × pods per node
- Add buffer for growth (usually 1.5-2x expected peak)
- Account for system pods, daemonsets, statefulsets
Example: 10 nodes × 50 pods/node × 2 = 1000 IPs needed
Add a new IP pool with larger CIDR range:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: large-pool
spec:
cidr: 10.48.0.0/16
blockSize: 25
EOFThen disable the old pool and migrate pods.
Reduce per-node allocation to conserve IPs:
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
blockSize: 26 # Allocate /26 to each node (64 IPs) instead of /25 (128)
EOFSmaller blocks = fewer wasted IPs but requires more fragmentation management.
Gradually migrate from old to new pool:
# Disable old pool to prevent new allocation
calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: old-pool
spec:
disabled: true
EOF
# Delete pods to reassign from new pool
kubectl rollout restart deployment -n your-namespaceClean up IPs not properly released:
# Check IP allocation status
calicoctl ipam show
# Release stuck allocations
calicoctl ipam release --ip <orphaned-ip>Set up monitoring to prevent future exhaustion:
# Get current usage
calicoctl get ippool default-ipv4-ippool -o yaml | grep -E "cidr|available"
# Create Prometheus alerts
alerting_rule:
- alert: CalicoIPPoolExhaustionWarning
expr: calico_ippool_available_ips < 100For large clusters, implement automation:
1. Monitor available IPs via Prometheus
2. Create alerts at 20%, 10%, 5% available
3. Use GitOps to automatically add new pools
4. Size pools to support expected growth (2-3 years)
For production clusters, plan IP pool sizing during initial deployment - retroactive expansion requires careful pod migration. Use multiple smaller IP pools rather than one large pool to limit blast radius of exhaustion. Consider IPv6 pools in addition to IPv4 for organizations with IPv6 infrastructure. Monitor IPAM churn rate to detect abnormal IP allocation patterns. For dynamic clusters with frequent scaling, leave 40-50% buffer in IP pools.