How to fix HTTP 429: Too Many Requests in Kubernetes

KubernetesADVANCEDHIGH

The Kubernetes API server is overwhelmed by excessive requests, returning 429 (Too Many Requests) errors. Causes include clients making unoptimized LIST queries, etcd performance degradation, or insufficient inflight request capacity. Fix by identifying offending clients, tuning etcd, and enabling API Priority and Fairness (APF).

What this error means

API server rate limiting rejects requests when concurrent inflight requests exceed configured limits (default: 400). This protects the cluster from cascading failures but blocks legitimate operations. Root cause is usually etcd degradation (slow backing store) or client behavior (excessive LIST operations).

How to fix "HTTP 429: Too Many Requests"

1Enable API Priority and Fairness (APF)

Classifies and prioritizes requests to prevent cascade failures:

bash

# Check if enabled (default: true in 1.20+)
kubectl api-resources | grep flowschema

# Create prioritized flows for critical workloads
kubectl apply -f - <<EOF
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10000
EOF

2Check etcd database size

Large etcd causes API server slowdown:

bash

kubectl exec -n kube-system etcd-<node> -- etcdctl endpoint status
# Check DB size (alert if >6GB)
kubectl exec -n kube-system etcd-<node> -- etcdctl alarm list

If DB too large, compact and defragment:

bash

kubectl exec -n kube-system etcd-<node> -- etcdctl compact <revision>
kubectl exec -n kube-system etcd-<node> -- etcdctl defrag

3Identify offending clients

Find clients making excessive requests:

bash

# Query Prometheus for request rates by client
kubectl port-forward -n prometheus svc/prometheus 9090:9090
# In browser: http://localhost:9090
# Query: sum(rate(apiserver_request_total[5m])) by (user, client)

Optimize clients to batch requests, use watches instead of polls, and add field selectors to LIST queries.

4Tune inflight request limits

Increase limits if appropriate:

bash

kubectl edit -n kube-system deployment kube-apiserver
# Add or modify:
spec:
  containers:
  - name: kube-apiserver
    args:
    - --max-requests-inflight=800  # Increase from 400
    - --max-mutating-requests-inflight=400  # Increase from 200

Note: Monitor CPU/memory impact carefully.

5Monitor API server metrics

Track latency and throttling:

bash

# API request latency (P99)
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Throttled requests
sum(rate(apiserver_request_total{code="429"}[5m]))

# etcd size
etcd_mvcc_db_total_size_in_bytes / 1024 / 1024 / 1024

Alert if P99 latency > 2s or 429 rate > 0.

6Scale control plane horizontally

Add more API server replicas:

bash

# For managed clusters (GKE, AKS, EKS)
# Typically automatic or via cluster settings

# For self-managed clusters
kubectl scale deployment -n kube-system kube-apiserver --replicas=5

7Optimize client-side settings

Configure controller rate limits:

bash

kubectl set env deployment/<name> \
  KUBE_API_QPS=50 \
  KUBE_API_BURST=100

Defaults: qps=5, burst=10. Increase for high-throughput controllers.

How to fix HTTP 429: Too Many Requests in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "HTTP 429: Too Many Requests"

Advanced notes

Related errors

Official resources & further reading