title: Unable to attach or mount volumes in Kubernetes
body: <p>Since I updated my Digital Ocean cluster to Kubernetes 1.21, I have multiple errors mounting nfs volumes on deployments and statefulsets like this one :</p>
  <pre><code>Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[storage kube-api-access-p2rxg config]: timed out waiting for the condition
  </code></pre>
  <p>Or this one :</p>
  <pre><code>MountVolume.SetUp failed for volume &quot;pvc-45b42793-b47d-4fab-afd7-c05a38c59d94&quot; : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs -o vers=3 10.245.244.118:/export/pvc-45b42793-b47d-4fab-afd7-c05a38c59d94 /var/lib/kubelet/pods/f5f46e8f-4078-4a54-8c40-f2f2f50e3cfa/volumes/kubernetes.io~nfs/pvc-45b42793-b47d-4fab-afd7-c05a38c59d94 Output: mount.nfs: access denied by server while mounting 10.245.244.118:/export/pvc-45b42793-b47d-4fab-afd7-c05a38c59d94
  </code></pre>
  <p>Output: mount.nfs: Operation not permitted</p>
  <p>The PV and PVC are good, bound and shows no errors, I find no way to resolve it...</p>
  <p>Redis configuration for example :</p>
  <pre><code>Name:           redis-0
  Namespace:      default
  Priority:       0
  Node:           pool-8gb-4vcpu-8s3aw/10.135.114.203
  Start Time:     Thu, 22 Jul 2021 14:08:27 +0200
  Labels:         app=redis
                  chart=redis-10.7.11
                  controller-revision-hash=redis-ccb4d5475
                  release=example
                  role=master
                  statefulset.kubernetes.io/pod-name=redis-0
  Annotations:    checksum/configmap: 0737f5e9fab2f644768e1df5f0b7c23b6f98ebfb2dacf2ed33bec6fceac68dae
                  checksum/health: 9ecbeadea17dee912ca49a508bd60b76ae75ef4e0b1577f98838e4bd939adc46
                  checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  prometheus.io/port: 9121
                  prometheus.io/scrape: true
  Status:         Pending
  IP:             
  IPs:            &lt;none&gt;
  Controlled By:  StatefulSet/redis
  Containers:
    redis:
      Container ID:  
      Image:         docker.io/bitnami/redis:6.0.5-debian-10-r32
      Image ID:      
      Port:          6379/TCP
      Host Port:     0/TCP
      Command:
        /bin/bash
        -c
        if [[ -n $REDIS_PASSWORD_FILE ]]; then
          password_aux=`cat ${REDIS_PASSWORD_FILE}`
          export REDIS_PASSWORD=$password_aux
        fi
        if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
          cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
        fi
        if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
          cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
        fi
        ARGS=(&quot;--port&quot; &quot;${REDIS_PORT}&quot;)
        ARGS+=(&quot;--protected-mode&quot; &quot;no&quot;)
        ARGS+=(&quot;--include&quot; &quot;/opt/bitnami/redis/etc/redis.conf&quot;)
        ARGS+=(&quot;--include&quot; &quot;/opt/bitnami/redis/etc/master.conf&quot;)
        /run.sh ${ARGS[@]}
        
      State:          Waiting
        Reason:       ContainerCreating
      Ready:          False
      Restart Count:  0
      Liveness:       exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
      Readiness:      exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
      Environment:
        REDIS_REPLICATION_MODE:  master
        ALLOW_EMPTY_PASSWORD:    yes
        REDIS_TLS_ENABLED:       no
        REDIS_PORT:              6379
      Mounts:
        /data from redis-data (rw)
        /health from health (rw)
        /opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
        /opt/bitnami/redis/mounted-etc from config (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kdlx4 (ro)
    metrics:
      Container ID:  
      Image:         docker.io/bitnami/redis-exporter:1.9.0-debian-10-r2
      Image ID:      
      Port:          9121/TCP
      Host Port:     0/TCP
      Command:
        /bin/bash
        -c
        if [[ -f '/secrets/redis-password' ]]; then
        export REDIS_PASSWORD=$(cat /secrets/redis-password)
        fi
        redis_exporter
        
      State:          Waiting
        Reason:       ContainerCreating
      Ready:          False
      Restart Count:  0
      Environment:
        REDIS_ALIAS:  redis
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kdlx4 (ro)
  Conditions:
    Type              Status
    Initialized       True 
    Ready             False 
    ContainersReady   False 
    PodScheduled      True 
  Volumes:
    redis-data:
      Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
      ClaimName:  redis-data-redis-0
      ReadOnly:   false
    health:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      redis-health
      Optional:  false
    config:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      redis
      Optional:  false
    redis-tmp-conf:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  &lt;unset&gt;
    kube-api-access-kdlx4:
      Type:                    Projected (a volume that contains injected data from multiple sources)
      TokenExpirationSeconds:  3607
      ConfigMapName:           kube-root-ca.crt
      ConfigMapOptional:       &lt;nil&gt;
      DownwardAPI:             true
  QoS Class:                   BestEffort
  Node-Selectors:              &lt;none&gt;
  Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                               node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason       Age    From               Message
    ----     ------       ----   ----               -------
    Normal   Scheduled    2m58s  default-scheduler  Successfully assigned default/toolib-dev-redis-0 to pool-8gb-4vcpu-8s3aw
    Warning  FailedMount  55s    kubelet            Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[health redis-data config redis-tmp-conf kube-api-access-kdlx4]: timed out waiting for the condition
    Warning  FailedMount  47s    kubelet            MountVolume.SetUp failed for volume &quot;pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081&quot; : mount failed: exit status 32
  Mounting command: mount
  Mounting arguments: -t nfs -o vers=3 10.245.165.123:/export/pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081 /var/lib/kubelet/pods/384b9414-5522-4731-96e6-28c7d5514c29/volumes/kubernetes.io~nfs/pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081
  Output: mount.nfs: Connection timed out
  </code></pre>
  <p>The PVC :</p>
  <pre><code>Name:          redis-data-redis-0
  Namespace:     default
  StorageClass:  nfs
  Status:        Bound
  Volume:        pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081
  Labels:        app=redis
                 component=master
                 heritage=Helm
                 release=example
                 role=master
  Annotations:   pv.kubernetes.io/bind-completed: yes
                 pv.kubernetes.io/bound-by-controller: yes
                 volume.beta.kubernetes.io/storage-provisioner: cluster.local/toolib-dev-nfs-server-provisioner
  Finalizers:    [kubernetes.io/pvc-protection]
  Capacity:      1Gi
  Access Modes:  RWO
  VolumeMode:    Filesystem
  Used By:       redis-0
  Events:        &lt;none&gt;
  </code></pre>
  <p>The PV :</p>
  <pre><code>Name:            pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081
  Labels:          &lt;none&gt;
  Annotations:     EXPORT_block:
                     
                     EXPORT
                     {
                       Export_Id = 3;
                       Path = /export/pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081;
                       Pseudo = /export/pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081;
                       Access_Type = RW;
                       Squash = no_root_squash;
                       SecType = sys;
                       Filesystem_id = 3.3;
                       FSAL {
                         Name = VFS;
                       }
                     }
                   Export_Id: 3
                   Project_Id: 0
                   Project_block: 
                   Provisioner_Id: 395bd01e-ad2b-11e9-8ba9-da1e9ed97b98
                   kubernetes.io/createdby: nfs-dynamic-provisioner
                   pv.kubernetes.io/provisioned-by: cluster.local/toolib-dev-nfs-server-provisioner
  Finalizers:      [kubernetes.io/pv-protection]
  StorageClass:    nfs
  Status:          Bound
  Claim:           default/redis-data-redis-0
  Reclaim Policy:  Delete
  Access Modes:    RWO
  VolumeMode:      Filesystem
  Capacity:        1Gi
  Node Affinity:   &lt;none&gt;
  Message:         
  Source:
      Type:      NFS (an NFS mount that lasts the lifetime of a pod)
      Server:    10.245.165.123
      Path:      /export/pvc-0f1b9dca-e53a-47e5-aaa9-a0c33af58081
      ReadOnly:  false
  Events:        &lt;none&gt;
  </code></pre>
  <p>The kube-api-access seems new in this kubernetes version, I don't know how to deal with it</p>
  <p>What I tried :</p>
  <ul>
  <li><p>update eache nodes and install libnfs-utils and other useful stuffs for nfs volumes</p>
  </li>
  <li><p>update the charts that I am using</p>
  </li>
  <li><p>delete volume and recreate it</p>
  </li>
  <li><p>update my nfs-server-provisioner</p>
  </li>
  <li><p>try some stack overflow solutions I found (like this one <a href="https://stackoverflow.com/questions/47423453/mountvolume-setup-failed-for-volume-nfs-mount-failed-exit-status-32">MountVolume.SetUp failed for volume &quot;nfs&quot; : mount failed: exit status 32</a>)</p>
  </li>
  </ul>
  <p>First question on stack, hope I will find help, thanks !</p>