How to fix DiskPressure in Kubernetes

KubernetesINTERMEDIATEHIGHUpdated Dec 5, 2025· 246 views

DiskPressure indicates a node is running critically low on available disk space. The kubelet stops scheduling new pods and begins evicting existing pods to reclaim disk space.

Quick fix

Fixes DiskPressure

kubectl describe nodes | grep -A 5 DiskPressure
  1. Check current disk usage and identify the node under pressure
  2. Manually trigger image garbage collection
  3. Configure kubelet image garbage collection thresholds
  4. Configure log rotation for container logs
kubernetes — error
kubectl describe nodes | grep -A 5 DiskPressure
DiskPressure

What this error means

DiskPressure is a Kubernetes node condition that indicates a node is running critically low on available disk space. The kubelet continuously tracks disk usage on two filesystems: the node filesystem (nodefs) which stores container data and logs, and the image filesystem (imagefs) which stores container images. When disk usage crosses the configured threshold (typically 85% by default), the kubelet marks the node with DiskPressure=True and stops scheduling new pods. Once DiskPressure is triggered, the kubelet initiates node-pressure eviction. It attempts to free disk space in a specific order: first by garbage collecting dead pods and containers, then by deleting unused container images. If disk pressure persists, the scheduler will evict running pods, starting with those that consume the most disk space. This condition is particularly critical because it can impact essential cluster components like the API server, etcd, and the kubelet itself, potentially causing cascading failures across your entire Kubernetes cluster.

Typical symptoms

  • Pods are evicted from the node with "Evicted" status and reason "DiskPressure"
  • kubectl describe node shows DiskPressure condition as True
  • New pods cannot be scheduled to the node (they remain in Pending state)
  • Kubelet logs show messages like "failed to garbage collect required amount of images"
  • Container runtime alerts about insufficient disk space or ImageGCFailed events
  • Applications become unavailable when their pods are evicted and cannot find available nodes

Common causes

  1. Container logs growing unbounded without rotation - applications logging extensively can quickly consume gigabytes
  2. Accumulation of unused and dangling container images that are pulled but never cleaned up
  3. Too many pods scheduled on a single node creating excessive ephemeral storage demand
  4. Persistent volumes with large datasets not being properly cleaned up after pod termination
  5. Missing or misconfigured ephemeral storage limits allowing pods to consume entire node disk
  6. Default image garbage collection threshold (85%) being too aggressive for available disk capacity
  7. Dead containers and pod artifacts accumulating when container garbage collection is misconfigured

How to fix "DiskPressure"

1Check current disk usage and identify the node under pressure

Run kubectl get nodes to list all nodes, then check which ones have DiskPressure:

bash
kubectl describe nodes | grep -A 5 DiskPressure

For a specific node:

bash
kubectl describe node <node-name>

Look for DiskPressure=True in the Conditions section. Also check the kubelet logs:

bash
ssh <node-ip>
sudo journalctl -u kubelet -n 100
2Manually trigger image garbage collection

SSH into the affected node and use crictl to remove unused images:

bash
ssh <node-ip>
# List all images
sudo crictl images

# Remove unused images (prune)
sudo crictl rmi --prune

For Docker-based nodes:

bash
sudo docker image prune -a --force

This will delete all dangling images and images not used by any running container. Check the freed space with:

bash
df -h /var/lib/kubelet
3Configure kubelet image garbage collection thresholds

Adjust the kubelet arguments to prevent disk pressure from occurring so quickly. Edit the kubelet service configuration:

bash
ssh <node-ip>
sudo nano /etc/default/kubelet

Add or modify these arguments:

bash
KUBELET_EXTRA_ARGS="--image-gc-high-threshold=70 --image-gc-low-threshold=60 --eviction-hard=nodefs.available<5Gi --eviction-soft=nodefs.available<10Gi --eviction-soft-grace-period=nodefs.available=2m"

Restart the kubelet:

bash
sudo systemctl restart kubelet
4Configure log rotation for container logs

Container logs in /var/lib/kubelet/pods can grow very large. Configure log rotation. For Docker, edit /etc/docker/daemon.json:

json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}

Restart Docker:

bash
sudo systemctl restart docker

Also apply log rotation at the OS level using logrotate:

bash
sudo cat > /etc/logrotate.d/kubernetes << EOF
/var/log/pods/*/*.log {
  rotate 5
  daily
  compress
  delaycompress
  missingok
  notifempty
}
EOF
5Set ephemeral storage requests and limits on pods

Update your pod manifests to define ephemeral storage limits:

yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: my-app:latest
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "5Gi"

The ephemeral storage limit should account for container logs. If log rotation is set to max-size=100m with max-file=10, set the ephemeral limit to at least 1Gi.

6Increase node disk capacity or scale out the cluster

If recurring disk pressure indicates insufficient total disk capacity:

For cloud-managed nodes (EKS, GKE, AKS), scale up the node group to use larger instances with more disk.

For on-premises/self-managed clusters, add storage to nodes or provision new nodes with larger disks.

bash
# Check current disk usage
df -h /var/lib/kubelet

# Drain the node before making changes
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

Monitor the migration to ensure pods are successfully rescheduled.

Advanced notes

The kubelet supports both soft and hard eviction thresholds. Soft thresholds allow a grace period (default 5 minutes) for pods to gracefully terminate before force termination, while hard thresholds trigger immediate eviction. Configure soft eviction: --eviction-soft=nodefs.available<10Gi --eviction-soft-grace-period=nodefs.available=2m.

When image GC runs, it first marks images as unused if no container is using them, then deletes them starting with the oldest. However, if the node is already critically low on space, the kubelet may fail to GC enough images. To prevent this, set image-gc-high-threshold 10-15% lower than your eviction threshold.

Container logs and temporary files consume ephemeral storage (local node disk), while data stored in PersistentVolumes does not count against node-level limits. If your pods need to write large temporary datasets, explicitly request ephemeral storage.

Set up Prometheus alerts for disk usage trending upward. Query kubelet metrics like kubelet_volume_stats_used_bytes and node_filesystem_avail_bytes to catch pressure before it becomes critical. Consider setting a Kubernetes resource quota on ephemeral-storage at the namespace level.

Frequently asked questions

What does "DiskPressure" mean?
DiskPressure is a Kubernetes node condition that indicates a node is running critically low on available disk space. The kubelet continuously tracks disk usage on two filesystems: the node filesystem (nodefs) which stores container data and logs, and the image filesystem (imagefs) which stores container images. When disk usage crosses the configured threshold (typically 85% by default), the kubelet marks the node with DiskPressure=True and stops scheduling new pods.
How do I fix "DiskPressure"?
DiskPressure indicates a node is running critically low on available disk space. The kubelet stops scheduling new pods and begins evicting existing pods to reclaim disk space.
What causes "DiskPressure"?
Container logs growing unbounded without rotation - applications logging extensively can quickly consume gigabytes Accumulation of unused and dangling container images that are pulled but never cleaned up Too many pods scheduled on a single node creating excessive ephemeral storage demand
Is the "DiskPressure" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading