<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">NO, it is not a request of help, this is a BUG REPORT</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">HPA not down-scaling deployment<br>
HPA not able to get load info</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Tested to repeat on 1.3.5 and on 1.3.6</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE/GKE managed</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-k8s-test1-default-pool-72fe9503-8tvm 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: gcloud</li>
<li><strong>Others</strong>: N/A</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">We have development cluster with one of node pools of --min-nodes=2 and --max-nodes=3.<br>
Normally cluster operates on 2 nodes in this pool.<br>
Deployment under question specifically has a nodeSelector set to allocate nodes only to this pool.<br>
Deployment under question has an HPA configured with maximum pod count of 20.</p>
<p dir="auto">We observed a spike in the load, where HPA decided to allocate all 20 pods.</p>
<p dir="auto">Due to CPU resource constraint, it triggered an additional node allocation: everything was as-expected at this time.</p>
<p dir="auto">After additional node allocation, it was STILL NOT ENOUGH CPU to start all pods, but now limit on number of nodes was reached and pods were stuck in "Pending" state: as-expected.</p>
<p dir="auto">What went wrong: after load was removed, HPA did not want to scale back to 1-2 pods we see normally running.</p>
<p dir="auto">We found that HPA can not get CPU consumption info for some pods:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  4m            4m              2       {horizontal-pod-autoscaler }                    Warning         FailedGetMetrics        failed to get CPU consumption and request: metrics obtained for 20/22 of pods
  4m            4m              2       {horizontal-pod-autoscaler }                    Warning         FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 20/22 of pods
"><pre class="notranslate"><code class="notranslate">  4m            4m              2       {horizontal-pod-autoscaler }                    Warning         FailedGetMetrics        failed to get CPU consumption and request: metrics obtained for 20/22 of pods
  4m            4m              2       {horizontal-pod-autoscaler }                    Warning         FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 20/22 of pods

</code></pre></div>
<p dir="auto">It was suspicious it reports more pods than we see with "kubectl get pod -l ...", so we found the missing ones to be "stuck" and waiting to be reaped:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get pod -a -l run=cm-worker
NAME                         READY     STATUS     RESTARTS   AGE
cm-worker-2850697223-1wtmo   1/1       Running    0          11m
cm-worker-2850697223-9evqk   1/1       Running    0          11m
cm-worker-2850697223-dfhiu   0/1       OutOfCPU   0          5h
cm-worker-2850697223-f6h9o   1/1       Running    0          11m
[...]
cm-worker-2850697223-wogui   1/1       Running    0          11m
cm-worker-2850697223-zdxjg   0/1       OutOfCPU   0          4h
"><pre class="notranslate"><code class="notranslate">kubectl get pod -a -l run=cm-worker
NAME                         READY     STATUS     RESTARTS   AGE
cm-worker-2850697223-1wtmo   1/1       Running    0          11m
cm-worker-2850697223-9evqk   1/1       Running    0          11m
cm-worker-2850697223-dfhiu   0/1       OutOfCPU   0          5h
cm-worker-2850697223-f6h9o   1/1       Running    0          11m
[...]
cm-worker-2850697223-wogui   1/1       Running    0          11m
cm-worker-2850697223-zdxjg   0/1       OutOfCPU   0          4h

</code></pre></div>
<p dir="auto">At this point, we upgraded master from 1.3.5 to 1.3.6, but it did not resolve the situation.</p>
<p dir="auto">Upon manual kill on these pods, cluster and HPA started to operate as appropriate: HPA scaled down and cluster scaled down node it did not need.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<ol dir="auto">
<li>It was expected that Kubernetes would properly discover and reap failed pods.</li>
<li>It was expected that HPA would not consider dead pods for query of current load.</li>
</ol>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">We are not sure we can reproduce, so we investigated to the root cause right in place.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">N/A</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Kubernetes version</strong>: 1.3.5</li>
<li><strong>Cloud provider or hardware configuration</strong>: Google Container Engine</li>
<li><strong>Kernel</strong>: Linux 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
We're using a HorizontalPodAutoscaler in our cluster to scale pods based on CPU load. While this is working for some deployments, one deployment seems to be in a broken state where the HPA tries to get cpu consumption from heapster for pods that do not exist.<br>
The HPA was working for said deployment but started to misbehave after a scaling event.</p>
<p dir="auto">Result from <code class="notranslate">kubectl get pods</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="…
gosu-main-api-v1-3735438065-5n8t7                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-j027w                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-rydr6                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-vf59h                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-w4hzm                    2/2       Running   1          2h
…"><pre class="notranslate"><code class="notranslate">…
gosu-main-api-v1-3735438065-5n8t7                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-j027w                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-rydr6                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-vf59h                    2/2       Running   1          2h
gosu-main-api-v1-3735438065-w4hzm                    2/2       Running   1          2h
…
</code></pre></div>
<p dir="auto">Result from <code class="notranslate">kubectl get hpa</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME                      REFERENCE                            TARGET    CURRENT     MINPODS   MAXPODS   AGE
…
gosu-main-api-v1          Deployment/gosu-main-api-v1          75%       &lt;waiting&gt;   3         20        28m
…"><pre class="notranslate"><code class="notranslate">NAME                      REFERENCE                            TARGET    CURRENT     MINPODS   MAXPODS   AGE
…
gosu-main-api-v1          Deployment/gosu-main-api-v1          75%       &lt;waiting&gt;   3         20        28m
…
</code></pre></div>
<p dir="auto">Result from <code class="notranslate">kubectl describe hpa gosu-main-api-v1</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Name:               gosu-main-api-v1
Namespace:          default
Labels:             &lt;none&gt;
Annotations:            &lt;none&gt;
CreationTimestamp:      Sat, 27 Aug 2016 17:17:31 -0700
Reference:          Deployment/gosu-main-api-v1
Target CPU utilization:     75%
Current CPU utilization:    &lt;unset&gt;
Min replicas:           3
Max replicas:           20
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  18m       16m     4   {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 0/12 of pods
  18m       16m     4   {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/12 of pods
  30m       30s     58  {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 5/12 of pods
  30m       30s     58  {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 5/12 of pods"><pre class="notranslate"><code class="notranslate">Name:               gosu-main-api-v1
Namespace:          default
Labels:             &lt;none&gt;
Annotations:            &lt;none&gt;
CreationTimestamp:      Sat, 27 Aug 2016 17:17:31 -0700
Reference:          Deployment/gosu-main-api-v1
Target CPU utilization:     75%
Current CPU utilization:    &lt;unset&gt;
Min replicas:           3
Max replicas:           20
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  18m       16m     4   {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 0/12 of pods
  18m       16m     4   {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/12 of pods
  30m       30s     58  {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 5/12 of pods
  30m       30s     58  {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 5/12 of pods
</code></pre></div>
<p dir="auto">Excerpt from heapster logs</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0828 00:47:34.525902       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526057       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526107       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526165       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526249       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526304       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526387       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526444       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526493       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526548       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526591       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526671       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526754       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:47:34.526822       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584791       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584816       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584837       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584866       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584893       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584961       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584986       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585005       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585048       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585066       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585097       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585186       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585201       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:48:04.585206       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9"><pre class="notranslate"><code class="notranslate">I0828 00:47:34.525902       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526057       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:47:34.526107       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526165       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:47:34.526249       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526304       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:47:34.526387       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526444       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:47:34.526493       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526548       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:47:34.526591       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526671       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:47:34.526754       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:47:34.526822       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584791       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584816       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-x19pc
I0828 00:48:04.584837       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584866       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-ipr93
I0828 00:48:04.584893       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584961       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-02y6n
I0828 00:48:04.584986       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585005       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-wkcbo
I0828 00:48:04.585048       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585066       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-1499485845-vylz6
I0828 00:48:04.585097       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585186       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-3b1q3
I0828 00:48:04.585201       1 handlers.go:242] No metrics for container gosu-main-api in pod default/gosu-main-api-v1-786519656-qscb9
I0828 00:48:04.585206       1 handlers.go:178] No metrics for pod default/gosu-main-api-v1-786519656-qscb9
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
The deployment is currently set to 5 replicas, so I would expect for the HPA to get the resource consumption for the 5 pods that are scheduled.</p>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
Cannot reliably bring other deployments into this broken state. Had the same issue with another deployment, but it started to work again without outer influence (maybe after we updated the deployment)</p>