How to fix volume mount failed in Kubernetes

KubernetesINTERMEDIATEHIGH

Volume mount failures prevent pods from accessing storage. This occurs when persistent volumes are unavailable, mount permissions are wrong, or storage backends fail. Pods remain Pending until the issue is resolved.

What this error means

When a pod requests a PersistentVolumeClaim (PVC), the kubelet must: 1. Attach the storage volume to the node 2. Format the volume (if new) 3. Mount the volume to the node filesystem 4. Bind-mount the volume into the container Failure at any step results in "volume mount failed". Common causes: - PVC not provisioned (PVC stuck in Pending) - StorageClass misconfigured - Storage backend unavailable - Node cannot access storage (NFS server down, iSCSI unavailable) - Container filesystem permissions prevent mounting

How to fix "volume mount failed"

1Check if PVC is bound to a PersistentVolume

List all PVCs:

bash

kubectl get pvc -A

For the problematic pod's namespace:

bash

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

Look for:
- Status should be "Bound", not "Pending"
- Volume field should list a PV name
- Events section for provisioning errors

If status is Pending:

bash

kubectl describe pvc <pvc-name> -n <namespace> | grep -A5 Events:

Common events:
- "no persistent volumes available"
- "provision failed"
- "timeout"

Check if StorageClass exists:

bash

kubectl get storageclass
kubectl describe storageclass <class-name>

2Verify StorageClass is configured correctly

Inspect the StorageClass:

bash

kubectl get sc <storage-class> -o yaml

Should include:

yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs  # or other provisioner
parameters:
  type: gp2
  iops: "100"
reclaimPolicy: Delete
allowVolumeExpansion: true

Common issues:
- Provisioner does not exist or is misspelled
- Parameters invalid for the provisioner
- No default StorageClass (if pod doesn't specify one)

Create a StorageClass if missing:

yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

Apply:

bash

kubectl apply -f storageclass.yaml

3Check storage provisioner pod status

Storage provisioners (local-path, NFS, AWS EBS, etc.) run as DaemonSets or Deployments:

bash

kubectl get pods -n kube-system | grep -i storage
kubectl get pods -n kube-system | grep -i csi  # CSI drivers

For each provisioner pod:

bash

kubectl describe pod <provisioner-pod> -n kube-system
kubectl logs <provisioner-pod> -n kube-system | tail -50

Common provisioners:
- csi-*-plugin (cloud or third-party CSI drivers)
- local-path-provisioner (local storage)
- nfs-provisioner (NFS)

If provisioner pod is CrashLoopBackOff:

bash

kubectl logs <provisioner-pod> -n kube-system --previous

Look for:
- Image pull failures (see kubelet-temporary-failure article)
- Configuration errors
- RBAC permission denied

4Verify storage backend is accessible

Test backend connectivity based on storage type:

For NFS:

bash

# From a pod
kubectl run test-nfs --image=ubuntu -it --rm -- bash
# Inside pod:
apt-get update && apt-get install -y nfs-common
mountpoint /mnt/test || mount -t nfs <nfs-server>:/export /mnt/test
ls -la /mnt/test

For cloud storage (EBS, GCE Persistent Disk, Azure Disk):

bash

# AWS EBS
aws ec2 describe-volumes --volume-ids <volume-id>
aws ec2 describe-volume-attachments --volume-ids <volume-id>

# GCP
gcloud compute disks list
gcloud compute disks describe <disk-name>

# Azure
az disk list
az disk show -g <group> -n <disk-name>

For iSCSI:

bash

# From node
sudo iscsiadm -m session
sudo iscsiadm -m discoverydb -t st -p <target-ip>

5Check volume attachment and mount status on node

SSH into the node where pod is scheduled:

bash

kubectl describe pod <pod-name> -n <namespace> | grep Node:
kubectl get pod <pod-name> -n <namespace> -o wide  # See assigned node

On the node:

bash

ls -la /var/lib/kubelet/pods/*/volumes/  # List mounted volumes
mount | grep -i kubernetes  # Show mounted volumes
df -h | grep /var/lib/kubelet  # Disk usage

For specific volume:

bash

kubectl get volumeattachment  # Check if volume is attached
kubectl describe volumeattachment <attachment>

If volume not attached:

bash

kubectl get pvc <pvc-name> -o yaml | grep volumeName
# Get the PV name, then check:
kubectl describe pv <pv-name>

For mount permission issues:

bash

sudo mount | grep <pvc-name>
# Check mount options and uid/gid
ls -la /var/lib/kubelet/pods/<pod-id>/volumes/<volume-type>/

6Check pod volume configuration

Inspect the pod spec for volume mounts:

bash

kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A20 volumes:

Should include both volumes section (declare PVC) and containers with volumeMounts:

yaml

spec:
  containers:
  - name: app
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

Common issues:
- claimName references non-existent PVC
- mountPath conflicts with application paths
- Volume mode mismatch (Filesystem vs Block)

Verify the PVC exists:

bash

kubectl get pvc <claimName> -n <namespace>

If volume mount is read-only:

yaml

volumeMounts:
- name: data
  mountPath: /data
  readOnly: true  # May cause permission errors

7Fix mount permission issues

If volume is mounted but inaccessible (permission denied):

bash

# Check mount options
mount | grep <volume-path>

# For NFS, check ownership
ls -la /var/lib/kubelet/pods/<pod-id>/volumes/kubernetes.io~nfs/<volume>/

# Change ownership if needed
sudo chown <uid>:<gid> <mount-path>
sudo chmod 755 <mount-path>

For container mounts, use securityContext:

yaml

spec:
  securityContext:
    fsGroup: 1000  # Set group on mounted volume
  containers:
  - name: app
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
    volumeMounts:
    - name: data
      mountPath: /data

For NFS mount options (e.g., anonuid):

yaml

volumes:
- name: data
  nfs:
    server: nfs.example.com
    path: /export
    mountOptions:
    - vers=4.1
    - anonuid=65534  # Map remote root to nobody

8Recreate PVC and volume if stuck

If PVC remains Pending despite correct configuration:

bash

# Check if volume provisioning is truly stuck
kubectl describe pvc <pvc-name> -n <namespace>

If provisioner has given up or errors are permanent:

1. Delete the PVC (and its bound PV if needed):

bash

kubectl delete pvc <pvc-name> -n <namespace>
kubectl delete pv <pv-name>  # If not auto-deleted

2. Optionally, pre-create the PV:

yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  nfs:
    server: nfs.example.com
    path: /export

3. Recreate the PVC:

yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

4. Redeploy the pod:

bash

kubectl delete pod <pod-name>
kubectl get pods -w  # Watch for new pod to start

How to fix volume mount failed in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "volume mount failed"

Advanced notes

Related errors

Official resources & further reading