A Kubernetes node is running out of available process IDs (PIDs). This prevents new Pods and processes from starting, causing resource exhaustion.
Each Linux system has a maximum number of processes (PIDs) it can run. When the number of running processes approaches this limit, Kubernetes marks the node with PIDPressure condition. This is a sign of either too many Pods on the node or processes creating excessive child processes without cleanup.
SSH into the node and check the max PID limit:
cat /proc/sys/kernel/pid_maxDefault is usually 32768 or 4194304. A value below 10000 is likely too low.
Safely drain Pods from the affected node:
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-dataThis allows the cluster to reschedule Pods on other nodes.
On the node, increase the PID limit (temporary):
echo 2097152 | sudo tee /proc/sys/kernel/pid_maxFor persistence, add to /etc/sysctl.conf:
kernel.pid_max = 2097152Then apply:
sudo sysctl -pCheck which Pods are consuming the most PIDs:
kubectl top pods -A --sort-by=cpuLook for Pods with unusual process counts. Delete or update them:
kubectl delete pod POD_NAME -n NAMESPACEAfter fixing the issue, uncordon the node:
kubectl uncordon NODE_NAMEMonitor the PIDPressure condition:
kubectl get nodes -w
kubectl describe node NODE_NAME | grep -A5 "Conditions:"The PIDPressure should return to False.
In Kubernetes 1.24+, you can set per-pod process limits using the pod security policy or resource limits. Consider using process-level monitoring to catch runaway processes early. Use cgroups v2 (cgroup2) for better process isolation. Implement proper Pod eviction policies to prevent over-packing on nodes.
Failed to connect to server: connection refused (HTTP/2)
How to fix "HTTP/2 connection refused" error in Kubernetes
No subnets found for EKS cluster
How to fix "eks subnet not found" in Kubernetes
missing request for cpu in container
How to fix "missing request for cpu in container" in Kubernetes HPA
unable to compute replica count
How to fix "unable to compute replica count" in Kubernetes HPA
error: context not found
How to fix "error: context not found" in Kubernetes