How to fix node(s) had taints that the pod didn't tolerate in Kubernetes

KubernetesINTERMEDIATEMEDIUM

Taint/toleration mismatch means nodes are tainted but your pod lacks matching tolerations. Add tolerations to your pod or remove taints from nodes.

What this error means

This error occurs when all available nodes have taints that your pod doesn't tolerate. Taints are a way to repel pods from nodes—only pods with matching tolerations can be scheduled on tainted nodes. Taints have three effects: - NoSchedule: Pods won't be scheduled (but existing pods stay) - PreferNoSchedule: Scheduler avoids the node but may use it if necessary - NoExecute: Pods are evicted if they don't tolerate the taint

Typical symptoms

How to fix "node(s) had taints that the pod didn't tolerate"

1Check node taints

View taints on all nodes:

bash

kubectl describe nodes | grep -A 2 Taints

# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'

Common taints:
- node-role.kubernetes.io/control-plane:NoSchedule - Control plane
- node.kubernetes.io/unschedulable:NoSchedule - Cordoned nodes
- nvidia.com/gpu:NoSchedule - GPU nodes

2Add tolerations to your pod

Match the taint with a toleration:

yaml

spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "app-team"
    effect: "NoSchedule"

For control plane nodes:

yaml

tolerations:
- key: "node-role.kubernetes.io/control-plane"
  operator: "Exists"
  effect: "NoSchedule"

The Exists operator tolerates any value for the key.

3Remove taints from nodes

If you want all pods to schedule on the node:

bash

# Remove specific taint (note the minus at the end)
kubectl taint nodes <node-name> dedicated:NoSchedule-

# Remove control plane taint (for single-node clusters)
kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-

Warning: Removing control plane taints in production can affect cluster stability.

4Understand toleration operators

Two operators for matching taints:

Equal - Matches specific key and value:

yaml

tolerations:
- key: "environment"
  operator: "Equal"
  value: "production"
  effect: "NoSchedule"

Exists - Matches any value for the key:

yaml

tolerations:
- key: "environment"
  operator: "Exists"
  effect: "NoSchedule"

To tolerate ALL taints (use carefully):

yaml

tolerations:
- operator: "Exists"

5Handle NoExecute taints

NoExecute taints evict running pods. Add tolerationSeconds to delay eviction:

yaml

tolerations:
- key: "node.kubernetes.io/not-ready"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 300  # Wait 5 minutes before eviction

Kubernetes automatically adds tolerations for node problems:
- node.kubernetes.io/not-ready
- node.kubernetes.io/unreachable

Default tolerationSeconds is 300.

6Combine taints with node affinity for exclusive nodes

Taints alone don't guarantee pods go to specific nodes. Combine with affinity:

yaml

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values:
            - gpu-workloads
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu-workloads"
    effect: "NoSchedule"

This ensures pods both tolerate the taint AND are attracted to the node.

How to fix node(s) had taints that the pod didn't tolerate in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "node(s) had taints that the pod didn't tolerate"

Advanced notes

Related errors

Official resources & further reading