How to fix etcdserver: mvcc: database space exceeded in Kubernetes

KubernetesADVANCEDCRITICAL

This critical error indicates your Kubernetes cluster's etcd database has exceeded its storage quota (default 2GB). etcd enters read-only mode, preventing any cluster modifications until space is reclaimed through compaction and defragmentation.

What this error means

This error indicates that your Kubernetes cluster's etcd database has exceeded its configured storage quota limit, typically the default 2GB. etcd uses MVCC (Multi-Version Concurrent Control) to maintain a complete history of all key-value changes, never deleting old revisions automatically. When the database size reaches the quota, etcd enters a maintenance mode that only accepts read and delete operations, preventing any new writes to the cluster. The error occurs because etcd enforces a space quota to prevent the database from consuming unlimited disk space and degrading cluster performance. Without this quota, etcd would continue growing with every update to objects like Pods, Services, ConfigMaps, and Events, eventually exhausting your storage. Once the NOSPACE alarm is triggered, you cannot create or modify any Kubernetes objects—no deployments, no pods, no configuration changes—until the database space issue is resolved through compaction, defragmentation, and/or cleanup of accumulated objects.

How to fix "etcdserver: mvcc: database space exceeded"

1Check the current etcd alarm status

Verify that the NOSPACE alarm is active. SSH into a control plane node and run:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

Also check the etcd database size:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=json | jq '.[] | {endpoint, dbSize}'

2Compact the etcd keyspace to remove historical revisions

Compaction removes old versions of keys, freeing up logical space. First, get the current revision:

bash

rev=$(ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out="json" | jq '.[] | .revision')

echo "Current revision: $rev"

Then compact to that revision:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  compact $rev

3Defragment the etcd database to reclaim file system space

After compaction, defragment to return free space to the file system. For etcd v3.5.x, use etcdutl for offline defragmentation:

bash

# Stop etcd pod first
kubectl -n kube-system delete pod etcd-<node-name> --ignore-not-found=true

# Perform offline defragmentation
ETCDCTL_API=3 etcdutl defrag --data-dir=/var/lib/etcd

For etcd v3.3 or v3.4, use online defragmentation:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag --cluster

4Disarm the NOSPACE alarm and verify recovery

After defragmentation, clear the NOSPACE alarm:

bash

# Wait for etcd to be ready
sleep 30

# Disarm the alarm
ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm disarm

Verify alarms are cleared:

bash

ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

The output should be empty. Your cluster should now accept write operations.

5Clean up accumulated Kubernetes objects to prevent recurrence

Remove unnecessary objects that consume etcd space:

bash

# Delete completed jobs
kubectl delete jobs --field-selector status.successful=1 -A

# Delete failed jobs
kubectl delete jobs --field-selector status.failed=1 -A

# Clean up evicted pods
kubectl get pods -A --field-selector=status.phase=Failed -o json | kubectl delete -f -

# Delete all events (use with caution)
kubectl delete event -A --all

Configure event TTL in the API server by editing /etc/kubernetes/manifests/kube-apiserver.yaml:

yaml

- --event-ttl=12h

6Configure auto-compaction and increase quota for long-term stability

Edit /etc/kubernetes/manifests/etcd.yaml on each control plane node:

yaml

spec:
  containers:
  - name: etcd
    command:
    - etcd
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention=1h
    - --quota-backend-bytes=4294967296  # 4GB

Restart etcd by deleting the static pod:

bash

kubectl -n kube-system delete pod etcd-<node-name> --wait=false

The --auto-compaction-retention=1h tells etcd to keep only 1 hour of history. Do not exceed 8GB for the quota—etcd performance degrades significantly above this threshold.

How to fix etcdserver: mvcc: database space exceeded in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "etcdserver: mvcc: database space exceeded"

Advanced notes

Related errors

Official resources & further reading