How to fix PLEG is not healthy in Kubernetes

KubernetesADVANCEDHIGHUpdated Dec 6, 2025· 231 views

PLEG (Pod Lifecycle Event Generator) is a critical kubelet component that monitors container runtime state. When PLEG becomes unhealthy, the node transitions to NotReady and prevents pod scheduling. This is usually caused by container runtime latency, high pod density, or resource exhaustion.

Quick fix

Fixes PLEG is not healthy

kubectl top nodes
kubectl top pods -n <namespace>
  1. Check node resource usage immediately
  2. Count pods and containers on the node
  3. Check container runtime logs
  4. Reduce pod density by draining and redistributing
kubernetes — error
kubectl top nodes
kubectl top pods -n <namespace>
PLEG is not healthy

What this error means

PLEG is a Kubernetes kubelet component that continuously polls the container runtime (Docker, containerd, CRI-O) to detect container state changes and synchronize them with the pod cache. The "PLEG is not healthy" error means the kubelet's relist process—which checks container status—took longer than 3 minutes to complete. When PLEG times out, the kubelet transitions the node to NotReady to prevent pod scheduling. This is a safety mechanism because if the kubelet can't track container state, it can't reliably manage pods. The underlying cause is usually the container runtime being too slow to respond to status queries.

Typical symptoms

  • Node transitions to NotReady state with condition "PLEG is not healthy"
  • Error messages show: "PLEG was last seen active 3m ago, threshold is 3m0s"
  • Pods stuck in "ContainerCreating" or "Terminating" states
  • Node repeatedly flaps between Ready/NotReady (recurring relist timeouts)
  • Kubelet and Docker daemon showing high CPU usage or lag
  • kubectl top nodes/pods returns metrics unavailable
  • Combined error: "container runtime is down, PLEG is not healthy"

Common causes

  1. Container runtime (Docker, containerd, CRI-O) slow or unresponsive to status queries
  2. High pod/container density exceeding ~400 containers per node
  3. Resource exhaustion (CPU bottleneck, out-of-memory, disk pressure on host)
  4. Slow or faulty storage (EBS volumes, local disk with bad sectors)
  5. CNI/networking issues causing network status retrieval delays
  6. Container state corruption preventing reliable listing/inspection
  7. Docker/containerd hangs during inspect operations (known bug moby#44300)
  8. Kubernetes versions < 1.14 with PLEG deadlock bug

How to fix "PLEG is not healthy"

1Check node resource usage immediately

Get real-time metrics:

bash
kubectl top nodes
kubectl top pods -n <namespace>

Check if CPU or memory is near 100%. From the node itself:

bash
top  # Interactive: shows CPU/memory usage
free -h  # Memory usage
df -h  # Disk usage
ps aux | grep kubelet  # Check kubelet CPU %
ps aux | grep docker  # Check Docker CPU %

If CPU is high, the relist can't complete in time.

2Count pods and containers on the node

Check current pod density:

bash
kubectl get pods --all-namespaces --field-selector=spec.nodeName=<node-name> | wc -l
kubectl describe node <node-name> | grep "Allocated resources"

Also check from the node:

bash
docker ps | wc -l  # Number of running containers
docker ps -a | wc -l  # Total containers (including exited)

If exceeding ~400 containers, that's likely the bottleneck.

3Check container runtime logs

View Docker/containerd logs for hanging operations:

bash
# Docker:
sudo journalctl -u docker -n 100
sudo docker info  # Check for warnings

# containerd:
sudo journalctl -u containerd -n 100

# CRI-O:
sudo journalctl -u crio -n 100

Look for:
- "inspect" operations timing out
- Hung goroutines or deadlocks
- Resource allocation warnings

4Reduce pod density by draining and redistributing

Move pods to other nodes to reduce density below ~400 containers:

bash
# Prevent new pods:
kubectl cordon <node-name>

# Gracefully move existing pods:
kubectl drain <node-name> --ignore-daemonsets

# Wait for PLEG to recover (2-5 minutes):
kubectl describe node <node-name>

# Once healthy, allow scheduling again:
kubectl uncordon <node-name>

Monitor recovery:

bash
kubectl get nodes -w
5Restart container runtime daemon

Restarting Docker/containerd clears hung connections and resets state:

bash
# Docker:
sudo systemctl restart docker

# containerd:
sudo systemctl restart containerd

Kubelet will automatically reconnect. Monitor:

bash
kubectl describe node <node-name>

Wait 30-60 seconds for PLEG to recover.

6Clean up exited and dangling containers

Remove containers consuming memory/disk:

bash
# Docker:
sudo docker container prune  # Removes all exited containers
sudo docker image prune      # Removes unused images

# containerd:
sudo crictl rm <container-id>  # Remove specific container
sudo ctr container rm <container-id>

This reduces relist scan overhead by decreasing the number of containers to inspect.

7Upgrade Kubernetes if using old versions

PLEG deadlock was fixed in Kubernetes 1.14+. Check current version:

bash
kubectl version --short

If using 1.13 or earlier, upgrade the cluster. This fixes inherited PLEG issues. For cloud providers:

bash
# GKE:
gcloud container clusters upgrade <cluster-name> --master-zone <zone>

# EKS:
aws eks update-cluster-version --name <cluster-name> --kubernetes-version 1.27

# AKS:
az aks upgrade --resource-group <group> --name <cluster-name> --kubernetes-version 1.27
8Enable Event-based PLEG (Kubernetes 1.27+)

Modern Kubernetes supports event-based container status instead of polling. This dramatically reduces overhead:

bash
# Check if supported:
kubectl version --short

If using 1.27+, enable in kubelet config:

bash
sudo vi /etc/kubernetes/kubelet.conf
# Add or update:
eventRecordQPS: 10
featureGates:
  EventedPLEG: true

Restart kubelet:

bash
sudo systemctl restart kubelet

This prevents PLEG polling altogether.

9Scale infrastructure to add capacity

If resource exhaustion is persistent, increase node capacity:

Add more nodes:

bash
# Scale auto-scaling group (AWS):
aws autoscaling set-desired-capacity --auto-scaling-group-name <asg> --desired-capacity 5

# Scale GKE node pool:
gcloud container node-pools update <pool> --num-nodes 5

Upgrade node specs:
- Increase CPU/RAM on existing nodes
- Use larger instance types
- Add SSDs for faster disk I/O

Redistribute pods using node affinity or manually drain/reschedule.

Advanced notes

PLEG is the critical health check for kubelet's container state tracking. When unhealthy, the kubelet intentionally marks the node NotReady to prevent cascading failures. This is a safety feature, not a bug. Docker has known inspect() hangs (moby#44300)—upgrading to 20.10.19+ improves stability. For high-density clusters, event-based PLEG (Kubernetes 1.27+) is transformational, reducing polling overhead by 90%+. In resource-constrained environments (edge, IoT), reduce pod density or allocate reserved CPU/memory for kubelet/containerd. For stateful workloads, high pod churn can cause repeated PLEG timeouts during rapid scaling—use pod disruption budgets to slow down evictions. WSL2-based Kubernetes may experience PLEG issues during heavy container churn—ensure adequate memory allocation to WSL VM.

Frequently asked questions

What does "PLEG is not healthy" mean?
PLEG is a Kubernetes kubelet component that continuously polls the container runtime (Docker, containerd, CRI-O) to detect container state changes and synchronize them with the pod cache. The "PLEG is not healthy" error means the kubelet's relist process—which checks container status—took longer than 3 minutes to complete. When PLEG times out, the kubelet transitions the node to NotReady to prevent pod scheduling.
How do I fix "PLEG is not healthy"?
PLEG (Pod Lifecycle Event Generator) is a critical kubelet component that monitors container runtime state. When PLEG becomes unhealthy, the node transitions to NotReady and prevents pod scheduling. This is usually caused by container runtime latency, high pod density, or resource exhaustion.
What causes "PLEG is not healthy"?
Container runtime (Docker, containerd, CRI-O) slow or unresponsive to status queries High pod/container density exceeding ~400 containers per node Resource exhaustion (CPU bottleneck, out-of-memory, disk pressure on host)
Is the "PLEG is not healthy" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading