This critical error occurs when Docker's devicemapper thin pool is completely exhausted with zero free blocks remaining. The Docker daemon becomes unresponsive and cannot create containers. Immediate action is required to recover storage space or extend the thin pool.
The "Thin Pool has 0 free data blocks" error is the most severe form of thin pool exhaustion in Docker's devicemapper storage driver. Unlike warnings about "less than minimum required" blocks, having exactly 0 free blocks means the thin pool is completely full and Docker operations are entirely blocked. When the thin pool reaches zero free data blocks, the Docker daemon enters a critical state. It cannot allocate any new storage for containers, images, or volumes. Existing containers may continue running but cannot write new data. The daemon itself may become unresponsive, making normal cleanup commands like `docker system prune` fail or hang indefinitely. This error typically occurs on systems using the legacy devicemapper storage driver, which was common on RHEL/CentOS 6 and 7, older Amazon Linux AMIs, and some cloud deployments. The devicemapper driver was deprecated in Docker 25.0 (January 2024) in favor of overlay2, but many legacy systems still encounter this critical failure mode.
When the thin pool has 0 free blocks, the Docker daemon may be unresponsive. First, try to stop containers to free any reclaimable space:
# Try stopping all containers (may hang if daemon is unresponsive)
docker stop $(docker ps -q) 2>/dev/null || true
# Check if Docker is responding at all
timeout 10 docker infoIf Docker is completely unresponsive, you may need to kill the daemon:
# Kill the Docker daemon (last resort)
sudo systemctl kill -s SIGKILL dockerWarning: Killing the daemon can leave containers in an inconsistent state.
Since Docker may be unresponsive, use LVM tools directly to assess the situation:
# View all logical volumes
sudo lvs -a
# Get detailed thin pool status
sudo lvs -a -o +devices,lv_health_status
# Check volume group free space
sudo vgsLook for the thin pool (usually named docker-pool, docker/thinpool, or similar) and note:
- Data%: Should show 100% if completely full
- VG Free: Available space in the volume group for extension
Example output when full:
LV VG Attr LSize Pool Origin Data% Meta%
docker-pool docker twi-a-t--- 100.00g 100.00 50.00If any containers are still running, try to reclaim unused blocks:
# Run fstrim on all running containers
sudo sh -c "docker ps -q | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/" 2>/dev/nullThis command:
1. Gets PIDs of running containers
2. Runs fstrim on each container's root filesystem
3. Discards unused blocks back to the thin pool
Note: This only works if Docker is still somewhat responsive and containers are running.
If sudo vgs shows free space in the volume group, extend the thin pool immediately:
# Find exact thin pool name
sudo lvs -a | grep -i pool
# Extend thin pool data volume (adjust path to match your setup)
sudo lvextend -l+100%FREE docker/thinpool
# Or extend by a specific size
sudo lvextend -L+50G docker/thinpoolAfter extending:
# Restart Docker to recognize new space
sudo systemctl restart docker
# Verify space is available
docker info | grep "Data Space"Common thin pool paths:
- /dev/docker/thinpool
- /dev/mapper/docker-pool
- /dev/mapper/docker-thinpool
If the volume group itself has no free space, add a new disk or partition:
# Create physical volume on new disk (DESTROYS ALL DATA on disk)
sudo pvcreate /dev/sdX
# Add to the docker volume group
sudo vgextend docker /dev/sdX
# Now extend the thin pool
sudo lvextend -l+100%FREE docker/thinpool
# Restart Docker
sudo systemctl restart dockerFor AWS/cloud instances, you may need to:
1. Attach a new EBS volume
2. Create partition: sudo fdisk /dev/xvdf
3. Create PV: sudo pvcreate /dev/xvdf1
4. Extend VG: sudo vgextend docker /dev/xvdf1
If extension is not possible and Docker is completely stuck, you may need to delete Docker's data directory:
Warning: This removes ALL images, containers, and non-persistent volumes. Back up any important data first.
# Stop Docker (force if necessary)
sudo systemctl stop docker || sudo systemctl kill docker
# Back up important volumes if accessible
sudo cp -r /var/lib/docker/volumes/important_volume /backup/
# Remove Docker data directory
sudo rm -rf /var/lib/docker
# Restart Docker (will recreate directory structure)
sudo systemctl start docker
# Verify clean state
docker infoYou will need to re-pull all images and recreate containers after this operation.
Once Docker is responsive again, aggressively clean up to prevent recurrence:
# Remove all stopped containers
docker container prune -f
# Remove all unused images (including tagged ones)
docker image prune -a -f
# Remove all unused volumes (careful - may delete data!)
docker volume prune -f
# Remove all unused networks
docker network prune -f
# Nuclear option: remove everything unused
docker system prune -a --volumes -fCheck space recovered:
docker system df
docker info | grep "Data Space"Set up automatic thin pool extension to avoid reaching 0 blocks again:
# Create or edit LVM profile
sudo vi /etc/lvm/profile/docker-thinpool.profileAdd the following configuration:
activation {
thin_pool_autoextend_threshold=70
thin_pool_autoextend_percent=20
}Apply the profile:
# Find your thin pool logical volume path
sudo lvs -a | grep pool
# Apply profile (adjust path as needed)
sudo lvchange --metadataprofile docker-thinpool docker/thinpoolThis automatically extends the pool by 20% when usage reaches 70%.
The devicemapper driver is deprecated. Migrate to overlay2 to eliminate thin pool issues entirely:
Warning: This removes all existing images and containers.
# Stop Docker
sudo systemctl stop docker
# Back up Docker directory (optional but recommended)
sudo cp -au /var/lib/docker /var/lib/docker.backup
# Remove Docker data
sudo rm -rf /var/lib/docker
# Configure overlay2
sudo vi /etc/docker/daemon.jsonSet contents to:
{
"storage-driver": "overlay2"
}Start Docker:
sudo systemctl start docker
docker info | grep "Storage Driver"Should show: Storage Driver: overlay2
Re-pull required images after migration.
Why 0 free blocks is worse than "minimum required" warnings: The devicemapper driver has a safety threshold (configurable via dm.min_free_space) that triggers warnings before exhaustion. Having 0 blocks means you've bypassed or ignored these warnings, or the pool filled suddenly (e.g., during a large build). At 0 blocks, the kernel's dm-thinp module cannot allocate any new blocks, causing all write operations to fail.
Docker daemon unresponsiveness: When the thin pool is completely exhausted, the Docker daemon itself may hang on internal operations. This is because the daemon needs to write state changes to disk. Commands that should be read-only (like docker ps) may still hang because they trigger internal state updates.
Recovery order of operations: The most reliable recovery sequence is:
1. Kill the Docker daemon if unresponsive
2. Use LVM tools to extend thin pool (if possible)
3. Restart Docker
4. Clean up resources
5. Configure prevention measures
Loopback vs direct-lvm considerations: Loopback mode uses sparse files with a maximum size limit. Even if your disk has space, the loopback file cannot grow beyond its configured limit. Direct-lvm mode can use entire block devices and is easier to extend, but requires more initial setup.
AWS ECS/Elastic Beanstalk specific recovery: AWS environments using the default Amazon Linux AMI with devicemapper may need:
1. Terminate and relaunch instances with larger storage volumes
2. Or attach additional EBS volumes and extend the thin pool
The default 12GB xvdcz volume fills quickly with multiple container deployments.
Monitoring recommendations: After recovery, implement monitoring:
# Add to monitoring/alerting system
USAGE=$(sudo lvs --noheadings -o data_percent docker/thinpool | tr -d ' ')
if (( $(echo "$USAGE > 80" | bc -l) )); then
echo "ALERT: Docker thin pool at $USAGE%"
fiKernel-level recovery: In extreme cases where LVM commands also hang, you may need to:
1. Reboot the system
2. Boot into rescue mode
3. Manually deactivate the thin pool: lvchange -an docker/thinpool
4. Delete Docker data: rm -rf /var/lib/docker
5. Reactivate volumes: vgchange -ay
6. Reboot normally
unable to configure the Docker daemon with file /etc/docker/daemon.json
How to fix 'unable to configure the Docker daemon with file daemon.json' in Docker
image operating system "linux" cannot be used on this platform
How to fix 'image operating system linux cannot be used on this platform' in Docker
dockerfile parse error line 5: unknown instruction: RRUN
How to fix 'unknown instruction' Dockerfile parse error in Docker
manifest unknown: manifest unknown
How to fix 'manifest unknown' in Docker
cannot open '/etc/passwd': Permission denied
How to fix 'cannot open: Permission denied' in Docker