<p dir="auto">This test is failing often on the gci-serial suite both on master and release-1.4.</p>
<p dir="auto"><a href="https://k8s-testgrid.appspot.com/google-gci#gci-serial&amp;sort-by-failures=" rel="nofollow">Test Grid link</a></p>
<p dir="auto">GCI is the default in v1.4 and so it is imperative to at-least triage the flakes soon.</p>
<p dir="auto">cc @kubernetes/autoscaling</p>
<p dir="auto">Forked from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159700661" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27194" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27194/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27194">#27194</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ichekrygin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ichekrygin">@ichekrygin</a> reported a weird cpu/usage_rate issue from heapster</p>
<p dir="auto">I am running on v1.3.5 and this still seems to be an issue</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/production/pods/rain-377063289-tz0tq/metrics/cpu/usage_rate
{
  &quot;metrics&quot;: [
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:04:00Z&quot;,
    &quot;value&quot;: 18446744073709551449
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:05:00Z&quot;,
    &quot;value&quot;: 705
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:06:00Z&quot;,
    &quot;value&quot;: 18446744073709551028
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:07:00Z&quot;,
    &quot;value&quot;: 696
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:08:00Z&quot;,
    &quot;value&quot;: 18446744073709550587
   },
   {
    &quot;timestamp&quot;: &quot;2016-08-18T03:09:00Z&quot;,
    &quot;value&quot;: 0
   },"><pre class="notranslate"><code class="notranslate">curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/production/pods/rain-377063289-tz0tq/metrics/cpu/usage_rate
{
  "metrics": [
   {
    "timestamp": "2016-08-18T03:04:00Z",
    "value": 18446744073709551449
   },
   {
    "timestamp": "2016-08-18T03:05:00Z",
    "value": 705
   },
   {
    "timestamp": "2016-08-18T03:06:00Z",
    "value": 18446744073709551028
   },
   {
    "timestamp": "2016-08-18T03:07:00Z",
    "value": 696
   },
   {
    "timestamp": "2016-08-18T03:08:00Z",
    "value": 18446744073709550587
   },
   {
    "timestamp": "2016-08-18T03:09:00Z",
    "value": 0
   },
</code></pre></div>
<p dir="auto">Node version:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubelet --version
Kubernetes v1.3.5"><pre class="notranslate"><code class="notranslate">kubelet --version
Kubernetes v1.3.5
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/timstclair/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/timstclair">@timstclair</a> and I checked his node stats by collecting  <code class="notranslate">curl -s http://localhost:10255/stats/summary | jq '.node.cpu.usageCoreNanoSeconds'</code>. usageCoreNanoSeconds is the accumulative cpu usage, and it is only stats used by heapster and HPA. This stat is mapping to heapster's cpu usage, and it is used to calculate cpu/usage_rate together with some timestamps. We noticed that the usageCoreNanoSeconds are sane, but corresponding heapster's cpu/usage_rate is insane as the above. I believe heapster ran into some overflow issue cAdvisor encountered before. Filed this one as a record to make sure the problem is addressed at the proper layer.</p>
<p dir="auto">cc/ <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mwielgus/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mwielgus">@mwielgus</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fgrzadkowski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fgrzadkowski">@fgrzadkowski</a></p>