What causes "etcdserver: mvcc: database space exceeded"?

Accumulated Event objects - Kubernetes generates thousands of events during normal operation that consume space rapidly Excessive Pod and ReplicaSet revisions - Each update creates new versions in etcd that are never cleaned without compaction Unevicted Pods in Pending/Failed state - Pods remaining in Failed or Pending status occupy etcd space

Is the "etcdserver: mvcc: database space exceeded" error serious?

Yes. This is a critical-severity error that usually blocks work until it is fixed, but the steps above resolve it.

How to fix etcdserver: mvcc: database space exceeded in Kubernetes

KubernetesADVANCEDCRITICALUpdated Dec 5, 2025· 203 views

This critical error indicates your Kubernetes cluster's etcd database has exceeded its storage quota (default 2GB). etcd enters read-only mode, preventing any cluster modifications until space is reclaimed through compaction and defragmentation.

Quick fix

Fixes etcdserver: mvcc: database space exceeded

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

Check the current etcd alarm status
Compact the etcd keyspace to remove historical revisions
Defragment the etcd database to reclaim file system space
Disarm the NOSPACE alarm and verify recovery

kubernetes — error

$ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list
etcdserver: mvcc: database space exceeded

What this error means

This error indicates that your Kubernetes cluster's etcd database has exceeded its configured storage quota limit, typically the default 2GB. etcd uses MVCC (Multi-Version Concurrent Control) to maintain a complete history of all key-value changes, never deleting old revisions automatically. When the database size reaches the quota, etcd enters a maintenance mode that only accepts read and delete operations, preventing any new writes to the cluster. The error occurs because etcd enforces a space quota to prevent the database from consuming unlimited disk space and degrading cluster performance. Without this quota, etcd would continue growing with every update to objects like Pods, Services, ConfigMaps, and Events, eventually exhausting your storage. Once the NOSPACE alarm is triggered, you cannot create or modify any Kubernetes objects—no deployments, no pods, no configuration changes—until the database space issue is resolved through compaction, defragmentation, and/or cleanup of accumulated objects.

Typical symptoms

API server returns "etcdserver: mvcc: database space exceeded" errors on all write operations
Kubectl commands fail with connection errors or "provide credentials" messages
Cluster becomes read-only; you can list resources but cannot create, scale, or modify any objects
Pod evictions fail because the API server cannot perform deletion operations
etcd health check shows NOSPACE alarm when running "etcdctl alarm list"
Kubernetes controller manager and scheduler log errors about inability to update status

Common causes

Accumulated Event objects - Kubernetes generates thousands of events during normal operation that consume space rapidly
Excessive Pod and ReplicaSet revisions - Each update creates new versions in etcd that are never cleaned without compaction
Unevicted Pods in Pending/Failed state - Pods remaining in Failed or Pending status occupy etcd space
Unfinished Jobs and CronJobs - Completed or failed jobs are not automatically cleaned up
Missing or infrequent automatic compaction - Space is consumed faster than it can be reclaimed
Stale ConfigMaps, Secrets, and custom resources - Applications may create and abandon configuration objects

How to fix "etcdserver: mvcc: database space exceeded"

1Check the current etcd alarm status

Verify that the NOSPACE alarm is active. SSH into a control plane node and run:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

Also check the etcd database size:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=json | jq '.[] | {endpoint, dbSize}'

2Compact the etcd keyspace to remove historical revisions

Compaction removes old versions of keys, freeing up logical space. First, get the current revision:

bash

rev=$(ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out="json" | jq '.[] | .revision')

echo "Current revision: $rev"

Then compact to that revision:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  compact $rev

3Defragment the etcd database to reclaim file system space

After compaction, defragment to return free space to the file system. For etcd v3.5.x, use etcdutl for offline defragmentation:

bash

# Stop etcd pod first
kubectl -n kube-system delete pod etcd-<node-name> --ignore-not-found=true

# Perform offline defragmentation
ETCDCTL_API=3 etcdutl defrag --data-dir=/var/lib/etcd

For etcd v3.3 or v3.4, use online defragmentation:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag --cluster

4Disarm the NOSPACE alarm and verify recovery

After defragmentation, clear the NOSPACE alarm:

bash

# Wait for etcd to be ready
sleep 30

# Disarm the alarm
ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm disarm

Verify alarms are cleared:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

The output should be empty. Your cluster should now accept write operations.

5Clean up accumulated Kubernetes objects to prevent recurrence

Remove unnecessary objects that consume etcd space:

bash

# Delete completed jobs
kubectl delete jobs --field-selector status.successful=1 -A

# Delete failed jobs
kubectl delete jobs --field-selector status.failed=1 -A

# Clean up evicted pods
kubectl get pods -A --field-selector=status.phase=Failed -o json | kubectl delete -f -

# Delete all events (use with caution)
kubectl delete event -A --all

Configure event TTL in the API server by editing /etc/kubernetes/manifests/kube-apiserver.yaml:

yaml

- --event-ttl=12h

6Configure auto-compaction and increase quota for long-term stability

Edit /etc/kubernetes/manifests/etcd.yaml on each control plane node:

yaml

spec:
  containers:
  - name: etcd
    command:
    - etcd
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention=1h
    - --quota-backend-bytes=4294967296  # 4GB

Restart etcd by deleting the static pod:

bash

kubectl -n kube-system delete pod etcd-<node-name> --wait=false

The --auto-compaction-retention=1h tells etcd to keep only 1 hour of history. Do not exceed 8GB for the quota—etcd performance degrades significantly above this threshold.

Advanced notes

etcd's Multi-Version Concurrent Control mechanism stores every historical version of every key. This enables strong consistency guarantees but requires explicit cleanup through compaction. Deleting a key does not immediately free space; it only marks the key as deleted. Only compaction removes old revisions.

Recent etcd versions (v3.5.x) have data inconsistency bugs if defragmentation is interrupted. Use etcdutl for offline defragmentation instead of online defragmentation via etcdctl.

Monitor these Prometheus metrics to catch issues early:
- etcd_mvcc_db_total_size_in_use_in_bytes: Actual database size after compaction
- etcd_debugging_mvcc_db_total_size_in_bytes: Total size including fragmentation
- etcd_server_quota_backend_bytes: The configured quota

Set up alerts when the ratio of used space exceeds 70% of the quota.

The default 2GB quota is suitable for clusters with <5000 nodes. For larger clusters or high-churn environments, 4GB is recommended. Never exceed 8GB. If you regularly hit quota limits, investigate what's consuming space rather than just raising quotas.

Frequently asked questions

What does "etcdserver: mvcc: database space exceeded" mean?: This error indicates that your Kubernetes cluster's etcd database has exceeded its configured storage quota limit, typically the default 2GB. etcd uses MVCC (Multi-Version Concurrent Control) to maintain a complete history of all key-value changes, never deleting old revisions automatically. When the database size reaches the quota, etcd enters a maintenance mode that only accepts read and delete operations, preventing any new writes to the cluster.
How do I fix "etcdserver: mvcc: database space exceeded"?: This critical error indicates your Kubernetes cluster's etcd database has exceeded its storage quota (default 2GB). etcd enters read-only mode, preventing any cluster modifications until space is reclaimed through compaction and defragmentation.
What causes "etcdserver: mvcc: database space exceeded"?: Accumulated Event objects - Kubernetes generates thousands of events during normal operation that consume space rapidly Excessive Pod and ReplicaSet revisions - Each update creates new versions in etcd that are never cleaned without compaction Unevicted Pods in Pending/Failed state - Pods remaining in Failed or Pending status occupy etcd space
Is the "etcdserver: mvcc: database space exceeded" error serious?: Yes. This is a critical-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading

📚Kubernetes Official Documentation
📖
Docs: Manage etcd disk space(Official Docs)
💬
SO: etcd space issues(Stack Overflow)
🐙
GitHub: etcd repository(GitHub)

How to fix etcdserver: mvcc: database space exceeded in Kubernetes

Quick fix

What this error means

Typical symptoms

Common causes

How to fix "etcdserver: mvcc: database space exceeded"

Advanced notes

Frequently asked questions

Related errors

Official resources & further reading