How to fix DiskPressure in Kubernetes

Run kubectl get nodes to list all nodes, then check which ones have DiskPressure:

kubectl describe nodes | grep -A 5 DiskPressure

For a specific node:

kubectl describe node <node-name>

Look for DiskPressure=True in the Conditions section. Also check the kubelet logs:

ssh <node-ip>
sudo journalctl -u kubelet -n 100

SSH into the affected node and use crictl to remove unused images:

ssh <node-ip>
# List all images
sudo crictl images

# Remove unused images (prune)
sudo crictl rmi --prune

For Docker-based nodes:

sudo docker image prune -a --force

This will delete all dangling images and images not used by any running container. Check the freed space with:

df -h /var/lib/kubelet

Adjust the kubelet arguments to prevent disk pressure from occurring so quickly. Edit the kubelet service configuration:

ssh <node-ip>
sudo nano /etc/default/kubelet

Add or modify these arguments:

KUBELET_EXTRA_ARGS="--image-gc-high-threshold=70 --image-gc-low-threshold=60 --eviction-hard=nodefs.available<5Gi --eviction-soft=nodefs.available<10Gi --eviction-soft-grace-period=nodefs.available=2m"

Restart the kubelet:

sudo systemctl restart kubelet

Container logs in /var/lib/kubelet/pods can grow very large. Configure log rotation. For Docker, edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}

Restart Docker:

sudo systemctl restart docker

Also apply log rotation at the OS level using logrotate:

sudo cat > /etc/logrotate.d/kubernetes << EOF
/var/log/pods/*/*.log {
  rotate 5
  daily
  compress
  delaycompress
  missingok
  notifempty
}
EOF

Update your pod manifests to define ephemeral storage limits:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: my-app:latest
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "5Gi"

The ephemeral storage limit should account for container logs. If log rotation is set to max-size=100m with max-file=10, set the ephemeral limit to at least 1Gi.

If recurring disk pressure indicates insufficient total disk capacity:

For cloud-managed nodes (EKS, GKE, AKS), scale up the node group to use larger instances with more disk.

For on-premises/self-managed clusters, add storage to nodes or provision new nodes with larger disks.

# Check current disk usage
df -h /var/lib/kubelet

# Drain the node before making changes
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

Monitor the migration to ensure pods are successfully rescheduled.