How to fix BackoffLimitExceeded in Kubernetes

KubernetesINTERMEDIATEHIGHUpdated Dec 5, 2025· 218 views

BackoffLimitExceeded occurs when a Kubernetes Job has reached its maximum retry attempts (default 6) and all pods have failed. The Job is marked as Failed and no further retries occur.

Quick fix

Fixes BackoffLimitExceeded

kubectl describe job <job-name> -n <namespace>
  1. Check Job and Pod status with kubectl describe
  2. Examine container logs to identify the root cause
  3. Review resource limits and allocations
  4. Verify container image and command configuration
kubernetes — error
kubectl describe job <job-name> -n <namespace>
BackoffLimitExceeded

What this error means

BackoffLimitExceeded occurs when a Kubernetes Job has reached its maximum number of retry attempts and all pods have failed. Kubernetes Jobs have a built-in retry mechanism controlled by the backoffLimit parameter (default: 6 attempts) that automatically recreates failed pods with exponential backoff delays (10s, 20s, 40s... capped at 6 minutes). When this limit is reached, the Job status is marked as Failed and no further retries occur. This is a permanent failure state that requires manual investigation and intervention to resolve, as the Job controller will not automatically restart the Job. The exponential backoff mechanism helps prevent overwhelming the cluster with rapidly failing pods, but once exceeded, you must fix the underlying issue and create a new Job or adjust the backoffLimit.

Typical symptoms

  • Job status shows "Failed" with condition type "Failed" and reason "BackoffLimitExceeded"
  • All pods associated with the Job have exited with non-zero exit codes
  • No new pods are being created despite the Job definition remaining unchanged
  • kubectl describe job shows "backoff limit has been exceeded" in the events section
  • Application workload does not execute even though the Job is scheduled
  • Pod count stabilizes because retry attempts have exhausted

Common causes

  1. Container application crashes with unhandled exceptions or logic errors (exit code 1, 139, etc.)
  2. Out-of-Memory (OOM) errors when container exceeds allocated memory limits (exit code 137)
  3. Missing dependencies, libraries, or configuration files required for application startup
  4. Invalid or non-existent command specified in container entrypoint or args
  5. Init container failures preventing main container from starting
  6. Pod evicted from node due to resource constraints or node failure
  7. Liveness probe failures repeatedly killing the container

How to fix "BackoffLimitExceeded"

1Check Job and Pod status with kubectl describe

Get detailed information about the failed Job:

bash
kubectl describe job <job-name> -n <namespace>

This shows the Job's status, failure reason, and event history. Then check the associated pods:

bash
kubectl get pods -n <namespace> -l job-name=<job-name>
kubectl describe pod <pod-name> -n <namespace>

Look for the container status, exit codes, and termination reason. The 'Last State' section reveals the exit code and reason (e.g., OOMKilled, Error).

2Examine container logs to identify the root cause

Retrieve logs from the failed pod:

bash
kubectl logs <pod-name> -n <namespace>

For pods that crashed during startup, check previous logs:

bash
kubectl logs <pod-name> -n <namespace> --previous

For init container failures:

bash
kubectl logs <pod-name> -n <namespace> -c <init-container-name>

Look for error messages, stack traces, or exceptions. Common indicators: 'FATAL', 'ERROR', 'panic', 'out of memory', 'command not found'.

3Review resource limits and allocations

Check if containers are being killed due to resource constraints:

bash
kubectl describe node <node-name>
kubectl top nodes
kubectl top pods -n <namespace>

If the exit code is 137 (OOMKilled), increase the memory limit:

yaml
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
4Verify container image and command configuration

Ensure the container image exists and can be pulled:

bash
kubectl describe pod <pod-name> -n <namespace> | grep -i 'image\|pull'

Check the command and args in your Job spec. Exit code 127 means the command is not found. Exit code 126 means insufficient permissions.

Test the command locally:

bash
docker run --rm myimage:v1 ls -la /path/to/command
docker run --rm myimage:v1 /bin/sh -c "./run.sh"
5Increase backoffLimit if failures are transient

If the failures are transient (temporary network issues, brief resource spikes), increase the backoffLimit:

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  backoffLimit: 10  # Increased from default of 6
  activeDeadlineSeconds: 3600  # Job will fail if not done in 1 hour
  template:
    spec:
      containers:
      - name: app
        image: myimage:v1
      restartPolicy: Never

Do NOT blindly increase backoffLimit for bugs in application code—this wastes cluster resources.

6Fix the application and redeploy

Once you've identified the root cause, implement the fix:

For application errors, fix the code, rebuild the image, and update the Job:

bash
docker build -t myimage:v2 .
docker push myimage:v2

Delete the old Job and reapply:

bash
kubectl delete job <old-job-name> -n <namespace>
kubectl apply -f job.yaml

For configuration issues, fix init containers or environment variables before recreating.

Advanced notes

BackoffLimit behavior varies with Job configuration: (1) With restartPolicy: Never (default), the Job creates new pods on failure up to backoffLimit. (2) With restartPolicy: OnFailure, the same pod is restarted within the container—backoffLimit still applies to pod creation attempts.

For Indexed Jobs (Kubernetes 1.27+), you can use backoffLimitPerIndex to set per-index retry limits instead of a global limit, providing better control over large parallel jobs.

Pod failure policy (Kubernetes 1.25+) allows you to mark certain exit codes as non-retriable (e.g., application bugs with exit code 1) to fail the Job faster without wasting retries on unrecoverable errors.

When combined with activeDeadlineSeconds, a Job will fail immediately if the total duration exceeds this limit, regardless of backoffLimit remaining.

Frequently asked questions

What does "BackoffLimitExceeded" mean?
BackoffLimitExceeded occurs when a Kubernetes Job has reached its maximum number of retry attempts and all pods have failed. Kubernetes Jobs have a built-in retry mechanism controlled by the backoffLimit parameter (default: 6 attempts) that automatically recreates failed pods with exponential backoff delays (10s, 20s, 40s... capped at 6 minutes).
How do I fix "BackoffLimitExceeded"?
BackoffLimitExceeded occurs when a Kubernetes Job has reached its maximum retry attempts (default 6) and all pods have failed. The Job is marked as Failed and no further retries occur.
What causes "BackoffLimitExceeded"?
Container application crashes with unhandled exceptions or logic errors (exit code 1, 139, etc.) Out-of-Memory (OOM) errors when container exceeds allocated memory limits (exit code 137) Missing dependencies, libraries, or configuration files required for application startup
Is the "BackoffLimitExceeded" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading