How to fix node(s) had taints that the pod didn't tolerate in Kubernetes
Taint/toleration mismatch means nodes are tainted but your pod lacks matching tolerations. Add tolerations to your pod or remove taints from nodes.
Quick fix
Fixes node(s) had taints that the pod didn't tolerate
kubectl describe nodes | grep -A 2 Taints
# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'- Check node taints
- Add tolerations to your pod
- Remove taints from nodes
- Understand toleration operators
kubectl describe nodes | grep -A 2 Taints# Or per nodekubectl get node <node-name> -o jsonpath='{.spec.taints}'node(s) had taints that the pod didn't tolerate
What this error means
This error occurs when all available nodes have taints that your pod doesn't tolerate. Taints are a way to repel pods from nodes—only pods with matching tolerations can be scheduled on tainted nodes. Taints have three effects: - NoSchedule: Pods won't be scheduled (but existing pods stay) - PreferNoSchedule: Scheduler avoids the node but may use it if necessary - NoExecute: Pods are evicted if they don't tolerate the taint
Typical symptoms
- Pod stuck in Pending state
- Events show N node(s) had taints that the pod didn't tolerate
- Pods cannot schedule on control plane nodes
- Dedicated node pools reject workloads
- All nodes tainted leaving no schedulable targets
- Pods evicted after taint applied with NoExecute
Common causes
- Control plane nodes tainted by default (NoSchedule)
- Custom taints applied for node dedication
- GPU nodes tainted to restrict workloads
- Node maintenance taints not removed
- Spot/preemptible instance taints
- Missing tolerations in pod spec
- Toleration key/value/effect mismatch
How to fix "node(s) had taints that the pod didn't tolerate"
View taints on all nodes:
kubectl describe nodes | grep -A 2 Taints
# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'Common taints:
- node-role.kubernetes.io/control-plane:NoSchedule - Control plane
- node.kubernetes.io/unschedulable:NoSchedule - Cordoned nodes
- nvidia.com/gpu:NoSchedule - GPU nodes
Match the taint with a toleration:
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "app-team"
effect: "NoSchedule"For control plane nodes:
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"The Exists operator tolerates any value for the key.
If you want all pods to schedule on the node:
# Remove specific taint (note the minus at the end)
kubectl taint nodes <node-name> dedicated:NoSchedule-
# Remove control plane taint (for single-node clusters)
kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-Warning: Removing control plane taints in production can affect cluster stability.
Two operators for matching taints:
Equal - Matches specific key and value:
tolerations:
- key: "environment"
operator: "Equal"
value: "production"
effect: "NoSchedule"Exists - Matches any value for the key:
tolerations:
- key: "environment"
operator: "Exists"
effect: "NoSchedule"To tolerate ALL taints (use carefully):
tolerations:
- operator: "Exists"NoExecute taints evict running pods. Add tolerationSeconds to delay eviction:
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300 # Wait 5 minutes before evictionKubernetes automatically adds tolerations for node problems:
- node.kubernetes.io/not-ready
- node.kubernetes.io/unreachable
Default tolerationSeconds is 300.
Taints alone don't guarantee pods go to specific nodes. Combine with affinity:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- gpu-workloads
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu-workloads"
effect: "NoSchedule"This ensures pods both tolerate the taint AND are attracted to the node.
Advanced notes
Kubernetes 1.24+ changed control-plane taint from node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane. Update tolerations when upgrading.
Taint effects explained:
- NoSchedule: New pods won't schedule, existing pods unaffected
- PreferNoSchedule: Soft version, scheduler avoids but doesn't guarantee
- NoExecute: Evicts existing pods without tolerations
System pods (kube-proxy, CNI, etc.) have built-in tolerations for critical taints. Don't remove these tolerations from system DaemonSets.
For dedicated node pools, use both taints (to repel unwanted pods) and labels/affinity (to attract wanted pods). This prevents both unwanted workloads on dedicated nodes and wanted workloads escaping to other nodes.
Debug scheduling decisions:
kubectl get events --field-selector reason=FailedSchedulingFrequently asked questions
- What does "node(s) had taints that the pod didn't tolerate" mean?
- This error occurs when all available nodes have taints that your pod doesn't tolerate. Taints are a way to repel pods from nodes—only pods with matching tolerations can be scheduled on tainted nodes.
- How do I fix "node(s) had taints that the pod didn't tolerate"?
- Taint/toleration mismatch means nodes are tainted but your pod lacks matching tolerations. Add tolerations to your pod or remove taints from nodes.
- What causes "node(s) had taints that the pod didn't tolerate"?
- Control plane nodes tainted by default (NoSchedule) Custom taints applied for node dedication GPU nodes tainted to restrict workloads
- Is the "node(s) had taints that the pod didn't tolerate" error serious?
- It is a medium-severity error — more of a blocker than a risk. The steps above resolve it.
Official resources & further reading
- 📚Kubernetes Official Documentation
- 📖Kubernetes Docs: taint and toleration(Official Docs)
- 💬
- 📝