How to fix BackoffLimitExceeded in Kubernetes

Get detailed information about the failed Job:

kubectl describe job <job-name> -n <namespace>

This shows the Job's status, failure reason, and event history. Then check the associated pods:

kubectl get pods -n <namespace> -l job-name=<job-name>
kubectl describe pod <pod-name> -n <namespace>

Look for the container status, exit codes, and termination reason. The 'Last State' section reveals the exit code and reason (e.g., OOMKilled, Error).

Retrieve logs from the failed pod:

kubectl logs <pod-name> -n <namespace>

For pods that crashed during startup, check previous logs:

kubectl logs <pod-name> -n <namespace> --previous

For init container failures:

kubectl logs <pod-name> -n <namespace> -c <init-container-name>

Look for error messages, stack traces, or exceptions. Common indicators: 'FATAL', 'ERROR', 'panic', 'out of memory', 'command not found'.

Check if containers are being killed due to resource constraints:

kubectl describe node <node-name>
kubectl top nodes
kubectl top pods -n <namespace>

If the exit code is 137 (OOMKilled), increase the memory limit:

spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Ensure the container image exists and can be pulled:

kubectl describe pod <pod-name> -n <namespace> | grep -i 'image\|pull'

Check the command and args in your Job spec. Exit code 127 means the command is not found. Exit code 126 means insufficient permissions.

Test the command locally:

docker run --rm myimage:v1 ls -la /path/to/command
docker run --rm myimage:v1 /bin/sh -c "./run.sh"

If the failures are transient (temporary network issues, brief resource spikes), increase the backoffLimit:

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  backoffLimit: 10  # Increased from default of 6
  activeDeadlineSeconds: 3600  # Job will fail if not done in 1 hour
  template:
    spec:
      containers:
      - name: app
        image: myimage:v1
      restartPolicy: Never

Do NOT blindly increase backoffLimit for bugs in application code—this wastes cluster resources.

Once you've identified the root cause, implement the fix:

For application errors, fix the code, rebuild the image, and update the Job:

docker build -t myimage:v2 .
docker push myimage:v2

Delete the old Job and reapply:

kubectl delete job <old-job-name> -n <namespace>
kubectl apply -f job.yaml

For configuration issues, fix init containers or environment variables before recreating.