Taint/toleration mismatch means nodes are tainted but your pod lacks matching tolerations. Add tolerations to your pod or remove taints from nodes.
Fixes node(s) had taints that the pod didn't tolerate
kubectl describe nodes | grep -A 2 Taints
# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'kubectl describe nodes | grep -A 2 Taints# Or per nodekubectl get node <node-name> -o jsonpath='{.spec.taints}'node(s) had taints that the pod didn't tolerate
This error occurs when all available nodes have taints that your pod doesn't tolerate. Taints are a way to repel pods from nodes—only pods with matching tolerations can be scheduled on tainted nodes. Taints have three effects: - NoSchedule: Pods won't be scheduled (but existing pods stay) - PreferNoSchedule: Scheduler avoids the node but may use it if necessary - NoExecute: Pods are evicted if they don't tolerate the taint
View taints on all nodes:
kubectl describe nodes | grep -A 2 Taints
# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'Common taints:
- node-role.kubernetes.io/control-plane:NoSchedule - Control plane
- node.kubernetes.io/unschedulable:NoSchedule - Cordoned nodes
- nvidia.com/gpu:NoSchedule - GPU nodes
Match the taint with a toleration:
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "app-team"
effect: "NoSchedule"For control plane nodes:
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"The Exists operator tolerates any value for the key.
If you want all pods to schedule on the node:
# Remove specific taint (note the minus at the end)
kubectl taint nodes <node-name> dedicated:NoSchedule-
# Remove control plane taint (for single-node clusters)
kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-Warning: Removing control plane taints in production can affect cluster stability.
Two operators for matching taints:
Equal - Matches specific key and value:
tolerations:
- key: "environment"
operator: "Equal"
value: "production"
effect: "NoSchedule"Exists - Matches any value for the key:
tolerations:
- key: "environment"
operator: "Exists"
effect: "NoSchedule"To tolerate ALL taints (use carefully):
tolerations:
- operator: "Exists"NoExecute taints evict running pods. Add tolerationSeconds to delay eviction:
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300 # Wait 5 minutes before evictionKubernetes automatically adds tolerations for node problems:
- node.kubernetes.io/not-ready
- node.kubernetes.io/unreachable
Default tolerationSeconds is 300.
Taints alone don't guarantee pods go to specific nodes. Combine with affinity:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- gpu-workloads
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu-workloads"
effect: "NoSchedule"This ensures pods both tolerate the taint AND are attracted to the node.
Kubernetes 1.24+ changed control-plane taint from node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane. Update tolerations when upgrading.
Taint effects explained:
- NoSchedule: New pods won't schedule, existing pods unaffected
- PreferNoSchedule: Soft version, scheduler avoids but doesn't guarantee
- NoExecute: Evicts existing pods without tolerations
System pods (kube-proxy, CNI, etc.) have built-in tolerations for critical taints. Don't remove these tolerations from system DaemonSets.
For dedicated node pools, use both taints (to repel unwanted pods) and labels/affinity (to attract wanted pods). This prevents both unwanted workloads on dedicated nodes and wanted workloads escaping to other nodes.
Debug scheduling decisions:
kubectl get events --field-selector reason=FailedScheduling