title: EKS aws-ebs-csi-driver failed to mount volume
body: <p>We are using EKS varsion v1.17.17-eks-087e67</p>
  <p>With installed aws-ebs-csi-driver components versions:</p>
  <ul>
  <li><p>aws-ebs-csi-driver:v1.1.3</p>
  </li>
  <li><p>csi-provisioner:v2.1.1</p>
  </li>
  <li><p>csi-attacher:v3.1.0</p>
  </li>
  <li><p>csi-snapshotter:v3.0.3</p>
  </li>
  <li><p>csi-resizer:v1.0.0</p>
  </li>
  </ul>
  <p>When we create PVC driver could not mount volume. As I can see, AWS volume continuously creating and deleting (from cloud trail):</p>
  <pre><code>{
      &quot;eventVersion&quot;: &quot;1.08&quot;,
      &quot;userIdentity&quot;: {
          &quot;type&quot;: &quot;AssumedRole&quot;,
          &quot;principalId&quot;: &quot;AROAV5QH66QYOM4FMMPFI:1631165222580844502&quot;,
          &quot;arn&quot;: &quot;arn:aws:sts::XXXXXXXXXX:assumed-role/EKSEBSCSIServiceRole-cluster01-eks-external-sandbox/XXXXXXXXXXXXXXXXXXXXXXXX&quot;,
          &quot;accountId&quot;: &quot;XXXXXXXXXX&quot;,
          &quot;accessKeyId&quot;: &quot;ASIAV5QH66QYFCKRZG43&quot;,
          &quot;sessionContext&quot;: {
              &quot;sessionIssuer&quot;: {
                  &quot;type&quot;: &quot;Role&quot;,
                  &quot;principalId&quot;: &quot;AROAV5QH66QYOM4FMMPFI&quot;,
                  &quot;arn&quot;: &quot;arn:aws:iam::XXXXXXXXXX:role/eks/EKSEBSCSIServiceRole-cluster01-eks-external-sandbox&quot;,
                  &quot;accountId&quot;: &quot;XXXXXXXXXX&quot;,
                  &quot;userName&quot;: &quot;EKSEBSCSIServiceRole-cluster01-eks-external-sandbox&quot;
              },
              &quot;webIdFederationData&quot;: {
                  &quot;federatedProvider&quot;: &quot;arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXX&quot;,
                  &quot;attributes&quot;: {}
              },
              &quot;attributes&quot;: {
                  &quot;creationDate&quot;: &quot;2021-09-09T05:27:03Z&quot;,
                  &quot;mfaAuthenticated&quot;: &quot;false&quot;
              }
          }
      },
      &quot;eventTime&quot;: &quot;2021-09-09T06:11:12Z&quot;,
      &quot;eventSource&quot;: &quot;ec2.amazonaws.com&quot;,
      &quot;eventName&quot;: &quot;CreateVolume&quot;,
      &quot;awsRegion&quot;: &quot;eu-central-1&quot;,
      &quot;sourceIPAddress&quot;: &quot;18.157.68.62&quot;,
      &quot;userAgent&quot;: &quot;aws-sdk-go/1.35.37 (go1.15.6; linux; amd64) exec-env/aws-ebs-csi-driver-v1.1.3&quot;,
      &quot;requestParameters&quot;: {
          &quot;size&quot;: &quot;8&quot;,
          &quot;zone&quot;: &quot;eu-central-1a&quot;,
          &quot;volumeType&quot;: &quot;gp2&quot;,
          &quot;encrypted&quot;: true,
          &quot;tagSpecificationSet&quot;: {
              &quot;items&quot;: [
                  {
                      &quot;resourceType&quot;: &quot;volume&quot;,
                      &quot;tags&quot;: [
                          {
                              &quot;key&quot;: &quot;ebs.csi.aws.com/cluster&quot;,
                              &quot;value&quot;: &quot;true&quot;
                          },
                          {
                              &quot;key&quot;: &quot;CSIVolumeName&quot;,
                              &quot;value&quot;: &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;
                          },
                          {
                              &quot;key&quot;: &quot;kubernetes.io/created-for/pv/name&quot;,
                              &quot;value&quot;: &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;
                          },
                          {
                              &quot;key&quot;: &quot;kubernetes.io/created-for/pvc/name&quot;,
                              &quot;value&quot;: &quot;data-postgres-postgresql-0&quot;
                          },
                          {
                              &quot;key&quot;: &quot;kubernetes.io/created-for/pvc/namespace&quot;,
                              &quot;value&quot;: &quot;default&quot;
                          }
                      ]
                  }
              ]
          }
      },
      &quot;responseElements&quot;: {
          &quot;requestId&quot;: &quot;5404a63c-a8d6-4bfa-b18f-ce1fba1060ee&quot;,
          &quot;volumeId&quot;: &quot;vol-032b5c6671123cc35&quot;,
          &quot;size&quot;: &quot;8&quot;,
          &quot;zone&quot;: &quot;eu-central-1a&quot;,
          &quot;status&quot;: &quot;creating&quot;,
          &quot;createTime&quot;: 1631167872000,
          &quot;volumeType&quot;: &quot;gp2&quot;,
          &quot;iops&quot;: 100,
          &quot;encrypted&quot;: true,
          &quot;masterEncryptionKeyId&quot;: &quot;arn:aws:kms:eu-central-1:XXXXXXXXXX:key/ef3b2237-00c3-4fd0-b556-91cda7f7db95&quot;,
          &quot;tagSet&quot;: {
              &quot;items&quot;: [
                  {
                      &quot;key&quot;: &quot;ebs.csi.aws.com/cluster&quot;,
                      &quot;value&quot;: &quot;true&quot;
                  },
                  {
                      &quot;key&quot;: &quot;CSIVolumeName&quot;,
                      &quot;value&quot;: &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;
                  },
                  {
                      &quot;key&quot;: &quot;kubernetes.io/created-for/pv/name&quot;,
                      &quot;value&quot;: &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;
                  },
                  {
                      &quot;key&quot;: &quot;kubernetes.io/created-for/pvc/name&quot;,
                      &quot;value&quot;: &quot;data-postgres-postgresql-0&quot;
                  },
                  {
                      &quot;key&quot;: &quot;kubernetes.io/created-for/pvc/namespace&quot;,
                      &quot;value&quot;: &quot;default&quot;
                  }
              ]
          },
          &quot;multiAttachEnabled&quot;: false
      },
      &quot;requestID&quot;: &quot;5404a63c-a8d6-4bfa-b18f-ce1fba1060ee&quot;,
      &quot;eventID&quot;: &quot;0941702c-119c-45fb-8c9e-6ef8918db6da&quot;,
      &quot;readOnly&quot;: false,
      &quot;eventType&quot;: &quot;AwsApiCall&quot;,
      &quot;managementEvent&quot;: true,
      &quot;recipientAccountId&quot;: &quot;XXXXXXXXXX&quot;,
      &quot;eventCategory&quot;: &quot;Management&quot;
  }
  
  </code></pre>
  <pre><code>  &quot;eventTime&quot;: &quot;2021-09-09T06:11:15Z&quot;,
      &quot;eventSource&quot;: &quot;ec2.amazonaws.com&quot;,
      &quot;eventName&quot;: &quot;DeleteVolume&quot;,
      &quot;awsRegion&quot;: &quot;eu-central-1&quot;,
      &quot;sourceIPAddress&quot;: &quot;x.x.x.x&quot;,
      &quot;userAgent&quot;: &quot;aws-sdk-go/1.35.37 (go1.15.6; linux; amd64) exec-env/aws-ebs-csi-driver-v1.1.3&quot;,
      &quot;errorCode&quot;: &quot;Client.InvalidVolume.NotFound&quot;,
      &quot;errorMessage&quot;: &quot;The volume 'vol-032b5c6671123cc35' does not exist.&quot;,
      &quot;requestParameters&quot;: {
          &quot;volumeId&quot;: &quot;vol-032b5c6671123cc35&quot;
      },
      &quot;responseElements&quot;: null,
      &quot;requestID&quot;: &quot;3cf2ce00-5845-436b-8470-3e1918dd24af&quot;,
      &quot;eventID&quot;: &quot;e5fbd13c-fc72-4cc1-9468-2a928d52a186&quot;,
      &quot;readOnly&quot;: false,
      &quot;eventType&quot;: &quot;AwsApiCall&quot;,
      &quot;managementEvent&quot;: true,
      &quot;recipientAccountId&quot;: &quot;XXXXXXXXXX&quot;,
      &quot;eventCategory&quot;: &quot;Management&quot;
  }
  </code></pre>
  <p>But eventually provisioner could not find this volume</p>
  <pre><code>0909 06:11:12.088851       1 controller.go:1332] provision &quot;default/data-postgres-postgresql-0&quot; class &quot;ebs-default&quot;: started
  I0909 06:11:12.089028       1 event.go:282] Event(v1.ObjectReference{Kind:&quot;PersistentVolumeClaim&quot;, Namespace:&quot;default&quot;, Name:&quot;data-postgres-postgresql-0&quot;, UID:&quot;27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;145344106&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim &quot;default/data-postgres-postgresql-0&quot;
  I0909 06:11:15.565942       1 controller.go:1099] Final error received, removing PVC 27fa1e04-c99d-48d2-9efa-0633ee3669d2 from claims in progress
  W0909 06:11:15.565962       1 controller.go:958] Retrying syncing claim &quot;27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;, failure 18
  E0909 06:11:15.565981       1 controller.go:981] error syncing claim &quot;27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;: failed to provision volume with StorageClass &quot;ebs-default&quot;: rpc error: code = Internal desc = Could not create volume &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;: failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-032b5c6671123cc35' does not exist.
          status code: 400, request id: a396c26c-71c6-4c88-8f2f-ebb3aa492447
  I0909 06:11:15.566164       1 event.go:282] Event(v1.ObjectReference{Kind:&quot;PersistentVolumeClaim&quot;, Namespace:&quot;default&quot;, Name:&quot;data-postgres-postgresql-0&quot;, UID:&quot;27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;145344106&quot;, FieldPath:&quot;&quot;}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass &quot;ebs-default&quot;: rpc error: code = Internal desc = Could not create volume &quot;pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2&quot;: failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-032b5c6671123cc35' does not exist.
          status code: 400, request id: a396c26c-71c6-4c88-8f2f-ebb3aa492447
  </code></pre>
  <p>Here is the policy from AWS Role for annotated CA:</p>
  <pre><code>{
      &quot;Version&quot;: &quot;2012-10-17&quot;,
      &quot;Statement&quot;: [
          {
              &quot;Effect&quot;: &quot;Allow&quot;,
              &quot;Action&quot;: [
                  &quot;ec2:AttachVolume&quot;,
                  &quot;ec2:CreateSnapshot&quot;,
                  &quot;ec2:CreateTags&quot;,
                  &quot;ec2:CreateVolume&quot;,
                  &quot;ec2:DeleteSnapshot&quot;,
                  &quot;ec2:DeleteTags&quot;,
                  &quot;ec2:DeleteVolume&quot;,
                  &quot;ec2:DescribeAvailabilityZones&quot;,
                  &quot;ec2:DescribeInstances&quot;,
                  &quot;ec2:DescribeSnapshots&quot;,
                  &quot;ec2:DescribeTags&quot;,
                  &quot;ec2:DescribeVolumes&quot;,
                  &quot;ec2:DescribeVolumesModifications&quot;,
                  &quot;ec2:DetachVolume&quot;,
                  &quot;ec2:ModifyVolume&quot;
              ],
              &quot;Resource&quot;: &quot;*&quot;
          }
      ]
  }
  </code></pre>
  <p>Here is StorageClass:</p>
  <pre><code>allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: &quot;true&quot;
    name: ebs-default
  parameters:
    csi.storage.k8s.io/fstype: ext4
    encrypted: &quot;true&quot;
    type: gp2
  provisioner: ebs.csi.aws.com
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer
  ~                                         
  </code></pre>
  <p>We are running workers in eu-central-1 region in 3 AZ</p>