<p dir="auto">Forked from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159700661" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27194" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27194/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27194">#27194</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ichekrygin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ichekrygin">@ichekrygin</a> reported a weird cpu/usage_rate issue from heapster</p>
<p dir="auto">I am running on v1.3.5 and this still seems to be an issue</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/production/pods/rain-377063289-tz0tq/metrics/cpu/usage_rate
{
  &quot;metrics&quot;: [
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:04:00Z&quot;,
    &quot;value&quot;: 18446744073709551449
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:05:00Z&quot;,
    &quot;value&quot;: 705
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:06:00Z&quot;,
    &quot;value&quot;: 18446744073709551028
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:07:00Z&quot;,
    &quot;value&quot;: 696
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:08:00Z&quot;,
    &quot;value&quot;: 18446744073709550587
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:09:00Z&quot;,
    &quot;value&quot;: 0
   },"><pre class="notranslate"><code class="notranslate">curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/production/pods/rain-377063289-tz0tq/metrics/cpu/usage_rate
{
  "metrics": [
   {
    "timestamp": "2016-08-18T03:04:00Z",
    "value": 18446744073709551449
   },
   {
    "timestamp": "2016-08-18T03:05:00Z",
    "value": 705
   },
   {
    "timestamp": "2016-08-18T03:06:00Z",
    "value": 18446744073709551028
   },
   {
    "timestamp": "2016-08-18T03:07:00Z",
    "value": 696
   },
   {
    "timestamp": "2016-08-18T03:08:00Z",
    "value": 18446744073709550587
   },
   {
    "timestamp": "2016-08-18T03:09:00Z",
    "value": 0
   },
</code></pre></div>
<p dir="auto">Node version:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubelet --version
Kubernetes v1.3.5"><pre class="notranslate"><code class="notranslate">kubelet --version
Kubernetes v1.3.5
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/timstclair/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/timstclair">@timstclair</a> and I checked his node stats by collecting  <code class="notranslate">curl -s http://localhost:10255/stats/summary | jq '.node.cpu.usageCoreNanoSeconds'</code>. usageCoreNanoSeconds is the accumulative cpu usage, and it is only stats used by heapster and HPA. This stat is mapping to heapster's cpu usage, and it is used to calculate cpu/usage_rate together with some timestamps. We noticed that the usageCoreNanoSeconds are sane, but corresponding heapster's cpu/usage_rate is insane as the above. I believe heapster ran into some overflow issue cAdvisor encountered before. Filed this one as a record to make sure the problem is addressed at the proper layer.</p>
<p dir="auto">cc/ <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mwielgus/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mwielgus">@mwielgus</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fgrzadkowski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fgrzadkowski">@fgrzadkowski</a></p>
<p dir="auto"><strong>Kubernetes version</strong>:</p>
<p dir="auto"><code class="notranslate">Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+coreos.0", GitCommit:"f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd", GitTreeState:"clean", BuildDate:"2016-08-29T17:01:01Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</code></p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong>: CoreOS</li>
<li><strong>Kernel</strong>: 4.6.3-coreos</li>
<li><strong>Install tools</strong>: kube-aws</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Some EBS volumes remain in the attachment state "attaching". These EBS volumes are attached on a device name that was used for a previous attachment.<br>
Looking on the worker node, the volume is not visible in the device list (eg by running <code class="notranslate">lsblk</code>). After a reboot of the node the volume is attached successfully. The attachment state then is "attached".</p>
<p dir="auto"><strong>Expected behavior</strong>:</p>
<p dir="auto">The EBS volumes are attached correctly.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<ol dir="auto">
<li>Spin up a worker node</li>
<li>Create 20 EBS volumes</li>
<li>Create 19 pods with volumes attached on this node</li>
<li>Remove the first pod</li>
<li>Create a 20th pod with the 20th volume attached</li>
</ol>
<p dir="auto">Now the 20th volume remains in the attachment state 'attaching'.</p>
<p dir="auto">20 is not necessary the lower bound but the issue did not appear with 3 volumes.</p>
<p dir="auto"><strong>Additional information</strong>:</p>
<p dir="auto">It seems related to the reuse of device names. Only when the device name /dev/xvdXX was already used for a previous attachment, the volume remains in the attachment state "attaching".<br>
When using a lower number of volumes, it is still possible to detach-attach. Only when attaching a considerable amount of volumes (about 20), the issue appears.</p>
<p dir="auto">According to <a href="https://aws.amazon.com/premiumsupport/knowledge-center/ebs-stuck-attaching/" rel="nofollow">this AWS support article</a> it is possible that the device name was not released by the block device driver after detaching.</p>