How to fix HTTP 429: Too Many Requests in Kubernetes

KubernetesADVANCEDHIGHUpdated Dec 6, 2025· 183 views

The Kubernetes API server is overwhelmed by excessive requests, returning 429 (Too Many Requests) errors. Causes include clients making unoptimized LIST queries, etcd performance degradation, or insufficient inflight request capacity. Fix by identifying offending clients, tuning etcd, and enabling API Priority and Fairness (APF).

Quick fix

Fixes HTTP 429: Too Many Requests

# Check if enabled (default: true in 1.20+)
kubectl api-resources | grep flowschema

# Create prioritized flows for critical workloads
kubectl apply -f - <<EOF
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10000
EOF
  1. Enable API Priority and Fairness (APF)
  2. Check etcd database size
  3. Identify offending clients
  4. Tune inflight request limits
kubernetes — error
# Check if enabled (default: true in 1.20+)
kubectl api-resources | grep flowschema
# Create prioritized flows for critical workloads
kubectl apply -f - <<EOF
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10000
EOF
HTTP 429: Too Many Requests

What this error means

API server rate limiting rejects requests when concurrent inflight requests exceed configured limits (default: 400). This protects the cluster from cascading failures but blocks legitimate operations. Root cause is usually etcd degradation (slow backing store) or client behavior (excessive LIST operations).

Typical symptoms

  • kubectl commands timeout or hang frequently
  • HTTP 429 responses from API server
  • Prometheus scrape failures (metrics collection times out)
  • Pod health checks failing (liveness/readiness probes timeout)
  • Controller manager unable to reconcile resources
  • CI/CD pipeline deployments stall
  • Increased latency in all kubectl operations

Common causes

  1. Clients making excessive LIST/GET operations (most resource-intensive)
  2. etcd database too large (>6GB, compaction needed)
  3. etcd slow disk operations (>25ms fsync latency)
  4. Insufficient --max-requests-inflight configuration (default: 400)
  5. Monitoring systems making high-frequency API calls
  6. Insufficient API server pod resources (CPU, memory)
  7. Traffic bursts exceeding cluster capacity

How to fix "HTTP 429: Too Many Requests"

1Enable API Priority and Fairness (APF)

Classifies and prioritizes requests to prevent cascade failures:

bash
# Check if enabled (default: true in 1.20+)
kubectl api-resources | grep flowschema

# Create prioritized flows for critical workloads
kubectl apply -f - <<EOF
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10000
EOF
2Check etcd database size

Large etcd causes API server slowdown:

bash
kubectl exec -n kube-system etcd-<node> -- etcdctl endpoint status
# Check DB size (alert if >6GB)
kubectl exec -n kube-system etcd-<node> -- etcdctl alarm list

If DB too large, compact and defragment:

bash
kubectl exec -n kube-system etcd-<node> -- etcdctl compact <revision>
kubectl exec -n kube-system etcd-<node> -- etcdctl defrag
3Identify offending clients

Find clients making excessive requests:

bash
# Query Prometheus for request rates by client
kubectl port-forward -n prometheus svc/prometheus 9090:9090
# In browser: http://localhost:9090
# Query: sum(rate(apiserver_request_total[5m])) by (user, client)

Optimize clients to batch requests, use watches instead of polls, and add field selectors to LIST queries.

4Tune inflight request limits

Increase limits if appropriate:

bash
kubectl edit -n kube-system deployment kube-apiserver
# Add or modify:
spec:
  containers:
  - name: kube-apiserver
    args:
    - --max-requests-inflight=800  # Increase from 400
    - --max-mutating-requests-inflight=400  # Increase from 200

Note: Monitor CPU/memory impact carefully.

5Monitor API server metrics

Track latency and throttling:

bash
# API request latency (P99)
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Throttled requests
sum(rate(apiserver_request_total{code="429"}[5m]))

# etcd size
etcd_mvcc_db_total_size_in_bytes / 1024 / 1024 / 1024

Alert if P99 latency > 2s or 429 rate > 0.

6Scale control plane horizontally

Add more API server replicas:

bash
# For managed clusters (GKE, AKS, EKS)
# Typically automatic or via cluster settings

# For self-managed clusters
kubectl scale deployment -n kube-system kube-apiserver --replicas=5
7Optimize client-side settings

Configure controller rate limits:

bash
kubectl set env deployment/<name> \
  KUBE_API_QPS=50 \
  KUBE_API_BURST=100

Defaults: qps=5, burst=10. Increase for high-throughput controllers.

Advanced notes

API overload is almost always a symptom of etcd degradation or client behavior problems, not API server limits. Fix root cause (etcd, clients) rather than just raising limits. Monitor with Prometheus; alert on 429 rate or latency trends. Use API Priority and Fairness (APF) for graceful degradation under load.

Frequently asked questions

What does "HTTP 429: Too Many Requests" mean?
API server rate limiting rejects requests when concurrent inflight requests exceed configured limits (default: 400). This protects the cluster from cascading failures but blocks legitimate operations. Root cause is usually etcd degradation (slow backing store) or client behavior (excessive LIST operations).
How do I fix "HTTP 429: Too Many Requests"?
The Kubernetes API server is overwhelmed by excessive requests, returning 429 (Too Many Requests) errors. Causes include clients making unoptimized LIST queries, etcd performance degradation, or insufficient inflight request capacity. Fix by identifying offending clients, tuning etcd, and enabling API Priority and Fairness (APF).
What causes "HTTP 429: Too Many Requests"?
Clients making excessive LIST/GET operations (most resource-intensive) etcd database too large (>6GB, compaction needed) etcd slow disk operations (>25ms fsync latency)
Is the "HTTP 429: Too Many Requests" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading