<p dir="auto">I'm using Kube 1.4.5, and AWS storage.</p>
<p dir="auto">When I try to attach multiple volumes using PVC's, one of the volumes consistently gets stuck in the attaching state whilst the other is successful.</p>
<p dir="auto">Below are the definitions that I used.</p>
<p dir="auto"><em>sonar-persistence.yml</em></p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: sonarqube-data
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: aws://eu-west-1a/vol-XXXXXXX
    fsType: ext4

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: sonarqube-extensions
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: aws://eu-west-1a/vol-XXXXXX
    fsType: ext4"><pre class="notranslate">---
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">PersistentVolume</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-data</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">capacity</span>:
    <span class="pl-ent">storage</span>: <span class="pl-s">5Gi</span>
  <span class="pl-ent">accessModes</span>:
    - <span class="pl-s">ReadWriteOnce</span>
  <span class="pl-ent">awsElasticBlockStore</span>:
    <span class="pl-ent">volumeID</span>: <span class="pl-s">aws://eu-west-1a/vol-XXXXXXX</span>
    <span class="pl-ent">fsType</span>: <span class="pl-s">ext4</span>

---
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">PersistentVolume</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-extensions</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">capacity</span>:
    <span class="pl-ent">storage</span>: <span class="pl-s">5Gi</span>
  <span class="pl-ent">accessModes</span>:
    - <span class="pl-s">ReadWriteOnce</span>
  <span class="pl-ent">awsElasticBlockStore</span>:
    <span class="pl-ent">volumeID</span>: <span class="pl-s">aws://eu-west-1a/vol-XXXXXX</span>
    <span class="pl-ent">fsType</span>: <span class="pl-s">ext4</span></pre></div>
<p dir="auto"><em>sonar-claim.yml</em></p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: sonarqube-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: sonarqube-extensions
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi"><pre class="notranslate">---
<span class="pl-ent">kind</span>: <span class="pl-s">PersistentVolumeClaim</span>
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-data</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">accessModes</span>:
    - <span class="pl-s">ReadWriteOnce</span>
  <span class="pl-ent">resources</span>:
    <span class="pl-ent">requests</span>:
      <span class="pl-ent">storage</span>: <span class="pl-s">5Gi</span>

---
<span class="pl-ent">kind</span>: <span class="pl-s">PersistentVolumeClaim</span>
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-extensions</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">accessModes</span>:
    - <span class="pl-s">ReadWriteOnce</span>
  <span class="pl-ent">resources</span>:
    <span class="pl-ent">requests</span>:
      <span class="pl-ent">storage</span>: <span class="pl-s">5Gi</span></pre></div>
<p dir="auto"><em>sonar-deployment.yml</em></p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sonar
spec:
  replicas: 1
  template:
    metadata:
      name: sonar
      labels:
        name: sonar
    spec:
      containers:
        - image: sonarqube:lts
          args:
            - -Dsonar.web.context=/sonar
          name: sonar
          env:
            - name: SONARQUBE_JDBC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-pwd
                  key: password
            - name: SONARQUBE_JDBC_URL
              value: jdbc:postgresql://sonar:5432/sonar
          ports:
            - containerPort: 9000
              name: sonar
          volumeMounts:
          - name: sonarqube-data
            mountPath: /opt/sonarqube/data
          - name: sonarqube-extensions
            mountPath: /opt/sonarqube/extensions
      volumes:
        - name: sonarqube-data
          persistentVolumeClaim:
            claimName: sonarqube-data
        - name: sonarqube-extensions
          persistentVolumeClaim:
            claimName: sonarqube-extensions"><pre class="notranslate">---
<span class="pl-ent">apiVersion</span>: <span class="pl-s">extensions/v1beta1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">Deployment</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">sonar</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">replicas</span>: <span class="pl-c1">1</span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">name</span>: <span class="pl-s">sonar</span>
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">name</span>: <span class="pl-s">sonar</span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
        - <span class="pl-ent">image</span>: <span class="pl-s">sonarqube:lts</span>
          <span class="pl-ent">args</span>:
            - <span class="pl-s">-Dsonar.web.context=/sonar</span>
          <span class="pl-ent">name</span>: <span class="pl-s">sonar</span>
          <span class="pl-ent">env</span>:
            - <span class="pl-ent">name</span>: <span class="pl-s">SONARQUBE_JDBC_PASSWORD</span>
              <span class="pl-ent">valueFrom</span>:
                <span class="pl-ent">secretKeyRef</span>:
                  <span class="pl-ent">name</span>: <span class="pl-s">postgres-pwd</span>
                  <span class="pl-ent">key</span>: <span class="pl-s">password</span>
            - <span class="pl-ent">name</span>: <span class="pl-s">SONARQUBE_JDBC_URL</span>
              <span class="pl-ent">value</span>: <span class="pl-s">jdbc:postgresql://sonar:5432/sonar</span>
          <span class="pl-ent">ports</span>:
            - <span class="pl-ent">containerPort</span>: <span class="pl-c1">9000</span>
              <span class="pl-ent">name</span>: <span class="pl-s">sonar</span>
          <span class="pl-ent">volumeMounts</span>:
          - <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-data</span>
            <span class="pl-ent">mountPath</span>: <span class="pl-s">/opt/sonarqube/data</span>
          - <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-extensions</span>
            <span class="pl-ent">mountPath</span>: <span class="pl-s">/opt/sonarqube/extensions</span>
      <span class="pl-ent">volumes</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-data</span>
          <span class="pl-ent">persistentVolumeClaim</span>:
            <span class="pl-ent">claimName</span>: <span class="pl-s">sonarqube-data</span>
        - <span class="pl-ent">name</span>: <span class="pl-s">sonarqube-extensions</span>
          <span class="pl-ent">persistentVolumeClaim</span>:
            <span class="pl-ent">claimName</span>: <span class="pl-s">sonarqube-extensions</span></pre></div>
<p dir="auto">The data volume always appears to be successful, and maybe coincidentally is first in the list. I have tried this multiple times but the result is always the same.</p>
<p dir="auto">The error message is as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Unable to mount volumes for pod &quot;sonar-3504269494-tnzwo_default(2cc5292c-a5d4-11e6-bd99-0a82a8a86ebf)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;sonar-3504269494-tnzwo&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;sonar-3504269494-tnzwo&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]"><pre class="notranslate"><code class="notranslate">Unable to mount volumes for pod "sonar-3504269494-tnzwo_default(2cc5292c-a5d4-11e6-bd99-0a82a8a86ebf)": timeout expired waiting for volumes to attach/mount for pod "sonar-3504269494-tnzwo"/"default". list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "sonar-3504269494-tnzwo"/"default". list of unattached/unmounted volumes=[sonarqube-data sonarqube-extensions]
</code></pre></div>
<p dir="auto"><strong>Kubernetes version</strong>:</p>
<p dir="auto"><code class="notranslate">Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+coreos.0", GitCommit:"f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd", GitTreeState:"clean", BuildDate:"2016-08-29T17:01:01Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</code></p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong>: CoreOS</li>
<li><strong>Kernel</strong>: 4.6.3-coreos</li>
<li><strong>Install tools</strong>: kube-aws</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Some EBS volumes remain in the attachment state "attaching". These EBS volumes are attached on a device name that was used for a previous attachment.<br>
Looking on the worker node, the volume is not visible in the device list (eg by running <code class="notranslate">lsblk</code>). After a reboot of the node the volume is attached successfully. The attachment state then is "attached".</p>
<p dir="auto"><strong>Expected behavior</strong>:</p>
<p dir="auto">The EBS volumes are attached correctly.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<ol dir="auto">
<li>Spin up a worker node</li>
<li>Create 20 EBS volumes</li>
<li>Create 19 pods with volumes attached on this node</li>
<li>Remove the first pod</li>
<li>Create a 20th pod with the 20th volume attached</li>
</ol>
<p dir="auto">Now the 20th volume remains in the attachment state 'attaching'.</p>
<p dir="auto">20 is not necessary the lower bound but the issue did not appear with 3 volumes.</p>
<p dir="auto"><strong>Additional information</strong>:</p>
<p dir="auto">It seems related to the reuse of device names. Only when the device name /dev/xvdXX was already used for a previous attachment, the volume remains in the attachment state "attaching".<br>
When using a lower number of volumes, it is still possible to detach-attach. Only when attaching a considerable amount of volumes (about 20), the issue appears.</p>
<p dir="auto">According to <a href="https://aws.amazon.com/premiumsupport/knowledge-center/ebs-stuck-attaching/" rel="nofollow">this AWS support article</a> it is possible that the device name was not released by the block device driver after detaching.</p>