Taint/toleration mismatch means nodes are tainted but your pod lacks matching tolerations. Add tolerations to your pod or remove taints from nodes.
This error occurs when all available nodes have taints that your pod doesn't tolerate. Taints are a way to repel pods from nodes—only pods with matching tolerations can be scheduled on tainted nodes. Taints have three effects: - NoSchedule: Pods won't be scheduled (but existing pods stay) - PreferNoSchedule: Scheduler avoids the node but may use it if necessary - NoExecute: Pods are evicted if they don't tolerate the taint
View taints on all nodes:
kubectl describe nodes | grep -A 2 Taints
# Or per node
kubectl get node <node-name> -o jsonpath='{.spec.taints}'Common taints:
- node-role.kubernetes.io/control-plane:NoSchedule - Control plane
- node.kubernetes.io/unschedulable:NoSchedule - Cordoned nodes
- nvidia.com/gpu:NoSchedule - GPU nodes
Match the taint with a toleration:
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "app-team"
effect: "NoSchedule"For control plane nodes:
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"The Exists operator tolerates any value for the key.
If you want all pods to schedule on the node:
# Remove specific taint (note the minus at the end)
kubectl taint nodes <node-name> dedicated:NoSchedule-
# Remove control plane taint (for single-node clusters)
kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-Warning: Removing control plane taints in production can affect cluster stability.
Two operators for matching taints:
Equal - Matches specific key and value:
tolerations:
- key: "environment"
operator: "Equal"
value: "production"
effect: "NoSchedule"Exists - Matches any value for the key:
tolerations:
- key: "environment"
operator: "Exists"
effect: "NoSchedule"To tolerate ALL taints (use carefully):
tolerations:
- operator: "Exists"NoExecute taints evict running pods. Add tolerationSeconds to delay eviction:
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300 # Wait 5 minutes before evictionKubernetes automatically adds tolerations for node problems:
- node.kubernetes.io/not-ready
- node.kubernetes.io/unreachable
Default tolerationSeconds is 300.
Taints alone don't guarantee pods go to specific nodes. Combine with affinity:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- gpu-workloads
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu-workloads"
effect: "NoSchedule"This ensures pods both tolerate the taint AND are attracted to the node.
Kubernetes 1.24+ changed control-plane taint from node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane. Update tolerations when upgrading.
Taint effects explained:
- NoSchedule: New pods won't schedule, existing pods unaffected
- PreferNoSchedule: Soft version, scheduler avoids but doesn't guarantee
- NoExecute: Evicts existing pods without tolerations
System pods (kube-proxy, CNI, etc.) have built-in tolerations for critical taints. Don't remove these tolerations from system DaemonSets.
For dedicated node pools, use both taints (to repel unwanted pods) and labels/affinity (to attract wanted pods). This prevents both unwanted workloads on dedicated nodes and wanted workloads escaping to other nodes.
Debug scheduling decisions:
kubectl get events --field-selector reason=FailedSchedulingNo subnets found for EKS cluster
How to fix "eks subnet not found" in Kubernetes
unable to compute replica count
How to fix "unable to compute replica count" in Kubernetes HPA
error: context not found
How to fix "error: context not found" in Kubernetes
default backend - 404
How to fix "default backend - 404" in Kubernetes Ingress
serviceaccount cannot list resource
How to fix "serviceaccount cannot list resource" in Kubernetes