How to fix Preempted in Kubernetes

KubernetesINTERMEDIATEMEDIUM

Preempted pods were evicted to make room for higher-priority pods. Adjust priority classes, add cluster capacity, or review resource requests.

What this error means

Pod preemption occurs when a higher-priority pod cannot be scheduled due to insufficient resources, so the scheduler evicts lower-priority pods to make room. Unlike node-pressure eviction (kubelet protecting node health), preemption is a scheduling decision to prioritize important workloads. Preemption is controlled by PriorityClass resources. Pods with higher priority values can preempt pods with lower values. The scheduler terminates victim pods gracefully and schedules the pending high-priority pod on the freed resources.

Typical symptoms

How to fix "Preempted"

1Understand the preemption event

Check why the pod was preempted:

bash

kubectl describe pod <preempted-pod>

Look at Events for:
- "Preempted by <namespace>/<pod-name>" - shows what preempted it
- Priority class of victim and preemptor

Check priority classes in use:

bash

kubectl get priorityclasses
kubectl get pod <pod-name> -o yaml | grep priorityClassName

2Review and fix priority class assignments

Audit priority usage across the cluster:

bash

# See all pods with their priorities
kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PRIORITY:.spec.priorityClassName

Common priority hierarchy:

yaml

# System critical (built-in)
system-cluster-critical: 2000000000
system-node-critical: 2000001000

# Production workloads
production: 1000000

# Default/batch workloads
default: 0
batch: -100

Ensure non-critical workloads don't have elevated priorities.

3Add cluster capacity

If legitimate high-priority pods are preempting needed workloads:

bash

# Check node utilization
kubectl top nodes

# Check pending pods
kubectl get pods --field-selector=status.phase=Pending

Solutions:
- Add more nodes to the cluster
- Use larger node instance types
- Enable cluster autoscaler for dynamic scaling

bash

# Example: Scale node pool (cloud-specific)
# GKE
gcloud container clusters resize CLUSTER --num-nodes=5

# EKS (via node group)
aws eks update-nodegroup-config --cluster-name CLUSTER --nodegroup-name NODEGROUP --scaling-config minSize=3,maxSize=10,desiredSize=5

4Use non-preempting priority classes

Create priority classes that queue without preempting:

yaml

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority-non-preempting
value: 1000000
preemptionPolicy: Never  # Won't evict other pods
globalDefault: false
description: "High priority but won't preempt others"

Pods with this class wait in queue rather than evicting existing pods.

5Protect critical workloads with PodDisruptionBudget

PDBs limit voluntary disruptions but don't prevent preemption directly. They're more useful for planned operations:

yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2  # or maxUnavailable: 1
  selector:
    matchLabels:
      app: critical-app

For preemption protection, use appropriate priority classes instead.

6Optimize resource requests

Right-size resource requests to reduce resource contention:

bash

# Check actual vs requested resources
kubectl top pods
kubectl get pods -o custom-columns=NAME:.metadata.name,REQ_CPU:.spec.containers[*].resources.requests.cpu,REQ_MEM:.spec.containers[*].resources.requests.memory

Reduce over-provisioned requests:

yaml

resources:
  requests:
    cpu: "100m"      # Actual need, not worst-case
    memory: "256Mi"
  limits:
    cpu: "500m"      # Allow bursting
    memory: "512Mi"

Lower requests mean more pods fit, reducing preemption pressure.

How to fix Preempted in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "Preempted"

Advanced notes

Related errors

Official resources & further reading