How to fix volume mount failed in Kubernetes

KubernetesINTERMEDIATEHIGHUpdated Dec 6, 2025· 241 views

Volume mount failures prevent pods from accessing storage. This occurs when persistent volumes are unavailable, mount permissions are wrong, or storage backends fail. Pods remain Pending until the issue is resolved.

Quick fix

Fixes volume mount failed

kubectl get pvc -A
  1. Check if PVC is bound to a PersistentVolume
  2. Verify StorageClass is configured correctly
  3. Check storage provisioner pod status
  4. Verify storage backend is accessible
kubernetes — error
kubectl get pvc -A
volume mount failed

What this error means

When a pod requests a PersistentVolumeClaim (PVC), the kubelet must: 1. Attach the storage volume to the node 2. Format the volume (if new) 3. Mount the volume to the node filesystem 4. Bind-mount the volume into the container Failure at any step results in "volume mount failed". Common causes: - PVC not provisioned (PVC stuck in Pending) - StorageClass misconfigured - Storage backend unavailable - Node cannot access storage (NFS server down, iSCSI unavailable) - Container filesystem permissions prevent mounting

Typical symptoms

  • Pod stuck in Pending with "volume not attached" or "unable to mount"
  • Pod shows "unmounted" or "timeout" for volumes in events
  • kubelet logs show "failed to mount volume" or "attachment timeout"
  • PVC remains Pending (not bound to PV)
  • Storage provisioner pod in error state
  • Pod cannot access mounted volume (permission denied)
  • Volume appears in pod spec but not in mounted volumes

Common causes

  1. PersistentVolumeClaim (PVC) not yet provisioned
  2. StorageClass does not exist or is misconfigured
  3. Storage provisioner pod crashed or unable to provision volumes
  4. Insufficient storage capacity in the storage backend
  5. Node unable to attach storage (iSCSI, NFS, cloud volume)
  6. Mount permission denied (NFS uid mismatch, cloud provider auth)
  7. Storage backend offline or unreachable

How to fix "volume mount failed"

1Check if PVC is bound to a PersistentVolume

List all PVCs:

bash
kubectl get pvc -A

For the problematic pod's namespace:

bash
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

Look for:
- Status should be "Bound", not "Pending"
- Volume field should list a PV name
- Events section for provisioning errors

If status is Pending:

bash
kubectl describe pvc <pvc-name> -n <namespace> | grep -A5 Events:

Common events:
- "no persistent volumes available"
- "provision failed"
- "timeout"

Check if StorageClass exists:

bash
kubectl get storageclass
kubectl describe storageclass <class-name>
2Verify StorageClass is configured correctly

Inspect the StorageClass:

bash
kubectl get sc <storage-class> -o yaml

Should include:

yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs  # or other provisioner
parameters:
  type: gp2
  iops: "100"
reclaimPolicy: Delete
allowVolumeExpansion: true

Common issues:
- Provisioner does not exist or is misspelled
- Parameters invalid for the provisioner
- No default StorageClass (if pod doesn't specify one)

Create a StorageClass if missing:

yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

Apply:

bash
kubectl apply -f storageclass.yaml
3Check storage provisioner pod status

Storage provisioners (local-path, NFS, AWS EBS, etc.) run as DaemonSets or Deployments:

bash
kubectl get pods -n kube-system | grep -i storage
kubectl get pods -n kube-system | grep -i csi  # CSI drivers

For each provisioner pod:

bash
kubectl describe pod <provisioner-pod> -n kube-system
kubectl logs <provisioner-pod> -n kube-system | tail -50

Common provisioners:
- csi-*-plugin (cloud or third-party CSI drivers)
- local-path-provisioner (local storage)
- nfs-provisioner (NFS)

If provisioner pod is CrashLoopBackOff:

bash
kubectl logs <provisioner-pod> -n kube-system --previous

Look for:
- Image pull failures (see kubelet-temporary-failure article)
- Configuration errors
- RBAC permission denied

4Verify storage backend is accessible

Test backend connectivity based on storage type:

For NFS:

bash
# From a pod
kubectl run test-nfs --image=ubuntu -it --rm -- bash
# Inside pod:
apt-get update && apt-get install -y nfs-common
mountpoint /mnt/test || mount -t nfs <nfs-server>:/export /mnt/test
ls -la /mnt/test

For cloud storage (EBS, GCE Persistent Disk, Azure Disk):

bash
# AWS EBS
aws ec2 describe-volumes --volume-ids <volume-id>
aws ec2 describe-volume-attachments --volume-ids <volume-id>

# GCP
gcloud compute disks list
gcloud compute disks describe <disk-name>

# Azure
az disk list
az disk show -g <group> -n <disk-name>

For iSCSI:

bash
# From node
sudo iscsiadm -m session
sudo iscsiadm -m discoverydb -t st -p <target-ip>
5Check volume attachment and mount status on node

SSH into the node where pod is scheduled:

bash
kubectl describe pod <pod-name> -n <namespace> | grep Node:
kubectl get pod <pod-name> -n <namespace> -o wide  # See assigned node

On the node:

bash
ls -la /var/lib/kubelet/pods/*/volumes/  # List mounted volumes
mount | grep -i kubernetes  # Show mounted volumes
df -h | grep /var/lib/kubelet  # Disk usage

For specific volume:

bash
kubectl get volumeattachment  # Check if volume is attached
kubectl describe volumeattachment <attachment>

If volume not attached:

bash
kubectl get pvc <pvc-name> -o yaml | grep volumeName
# Get the PV name, then check:
kubectl describe pv <pv-name>

For mount permission issues:

bash
sudo mount | grep <pvc-name>
# Check mount options and uid/gid
ls -la /var/lib/kubelet/pods/<pod-id>/volumes/<volume-type>/
6Check pod volume configuration

Inspect the pod spec for volume mounts:

bash
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A20 volumes:

Should include both volumes section (declare PVC) and containers with volumeMounts:

yaml
spec:
  containers:
  - name: app
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

Common issues:
- claimName references non-existent PVC
- mountPath conflicts with application paths
- Volume mode mismatch (Filesystem vs Block)

Verify the PVC exists:

bash
kubectl get pvc <claimName> -n <namespace>

If volume mount is read-only:

yaml
volumeMounts:
- name: data
  mountPath: /data
  readOnly: true  # May cause permission errors
7Fix mount permission issues

If volume is mounted but inaccessible (permission denied):

bash
# Check mount options
mount | grep <volume-path>

# For NFS, check ownership
ls -la /var/lib/kubelet/pods/<pod-id>/volumes/kubernetes.io~nfs/<volume>/

# Change ownership if needed
sudo chown <uid>:<gid> <mount-path>
sudo chmod 755 <mount-path>

For container mounts, use securityContext:

yaml
spec:
  securityContext:
    fsGroup: 1000  # Set group on mounted volume
  containers:
  - name: app
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
    volumeMounts:
    - name: data
      mountPath: /data

For NFS mount options (e.g., anonuid):

yaml
volumes:
- name: data
  nfs:
    server: nfs.example.com
    path: /export
    mountOptions:
    - vers=4.1
    - anonuid=65534  # Map remote root to nobody
8Recreate PVC and volume if stuck

If PVC remains Pending despite correct configuration:

bash
# Check if volume provisioning is truly stuck
kubectl describe pvc <pvc-name> -n <namespace>

If provisioner has given up or errors are permanent:

1. Delete the PVC (and its bound PV if needed):

bash
kubectl delete pvc <pvc-name> -n <namespace>
kubectl delete pv <pv-name>  # If not auto-deleted

2. Optionally, pre-create the PV:

yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  nfs:
    server: nfs.example.com
    path: /export

3. Recreate the PVC:

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

4. Redeploy the pod:

bash
kubectl delete pod <pod-name>
kubectl get pods -w  # Watch for new pod to start

Advanced notes

Volume mount failures are among the most common Kubernetes issues in production. The root cause often lies outside Kubernetes (storage backend unavailable, NFS server down, cloud API rate-limited). Implement monitoring on storage provisioners and backends. For stateful workloads, use StatefulSets with volumeClaimTemplates to auto-create PVCs per replica. Consider storage quotas (ResourceQuota) to prevent runaway provisioning. Backup strategies should account for mounted volumes; use snapshots or persistent backup containers. For disaster recovery, implement volume replication at the storage layer. CSI drivers provide better error handling and logging than in-tree provisioners. Test storage failover scenarios: what happens if an NFS server is down for 5 minutes? Do pods recover automatically? Multi-zone clusters need storage replication or highly available backends. For local storage (local-path provisioner), ensure node failures don't lose data. Monitoring volume mount latency helps identify slow storage backends. Implement PVC garbage collection to clean up orphaned volumes.

Frequently asked questions

What does "volume mount failed" mean?
When a pod requests a PersistentVolumeClaim (PVC), the kubelet must: 1. Attach the storage volume to the node 2. Format the volume (if new) 3.
How do I fix "volume mount failed"?
Volume mount failures prevent pods from accessing storage. This occurs when persistent volumes are unavailable, mount permissions are wrong, or storage backends fail. Pods remain Pending until the issue is resolved.
What causes "volume mount failed"?
PersistentVolumeClaim (PVC) not yet provisioned StorageClass does not exist or is misconfigured Storage provisioner pod crashed or unable to provision volumes
Is the "volume mount failed" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading