A Kubernetes node is running out of available process IDs (PIDs). This prevents new Pods and processes from starting, causing resource exhaustion.
Fixes PIDPressure
cat /proc/sys/kernel/pid_maxcat /proc/sys/kernel/pid_maxPIDPressure
Each Linux system has a maximum number of processes (PIDs) it can run. When the number of running processes approaches this limit, Kubernetes marks the node with PIDPressure condition. This is a sign of either too many Pods on the node or processes creating excessive child processes without cleanup.
SSH into the node and check the max PID limit:
cat /proc/sys/kernel/pid_maxDefault is usually 32768 or 4194304. A value below 10000 is likely too low.
Safely drain Pods from the affected node:
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-dataThis allows the cluster to reschedule Pods on other nodes.
On the node, increase the PID limit (temporary):
echo 2097152 | sudo tee /proc/sys/kernel/pid_maxFor persistence, add to /etc/sysctl.conf:
kernel.pid_max = 2097152Then apply:
sudo sysctl -pCheck which Pods are consuming the most PIDs:
kubectl top pods -A --sort-by=cpuLook for Pods with unusual process counts. Delete or update them:
kubectl delete pod POD_NAME -n NAMESPACEAfter fixing the issue, uncordon the node:
kubectl uncordon NODE_NAMEMonitor the PIDPressure condition:
kubectl get nodes -w
kubectl describe node NODE_NAME | grep -A5 "Conditions:"The PIDPressure should return to False.
In Kubernetes 1.24+, you can set per-pod process limits using the pod security policy or resource limits. Consider using process-level monitoring to catch runaway processes early. Use cgroups v2 (cgroup2) for better process isolation. Implement proper Pod eviction policies to prevent over-packing on nodes.