<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): kinda</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): OutOfCPU</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.2&quot;, GitCommit:&quot;9bafa3400a77c14ee50782bb05f9efc5c91b3185&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-17T18:30:39Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.2", GitCommit:"9bafa3400a77c14ee50782bb05f9efc5c91b3185", GitTreeState:"clean", BuildDate:"2016-07-17T18:30:39Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
</code></pre></div>
<p dir="auto">(cluster is now torn down, but is also 1.3.2 with the same commit)</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS - 80 c4.4xlarge nodes</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1068.8.0 (stable)</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Scheduling roughly 10k Pods on 80 nodes with 16 cores each, lots of pods end up having issues with being scheduled and end up with an OutOfCPU error. In the case below, we were experimenting with increasing <code class="notranslate">limits</code> to beyond the node capacity, but we found the behaviour on the <code class="notranslate">Guaranteed</code> (limits and requests set to 14) and <code class="notranslate">Burstable</code> classes. To me, it appears like the scheduler is having problems keeping track of the resources in use and is trying to over-provision the node.</p>
<p dir="auto">Here's the <code class="notranslate">describe pods</code> Note that the extra 100m CPU reported as being used is from system containers (fluentd, etc.)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe pods worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Name:           worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Namespace:      default
Node:           &lt;REDACTED&gt;
Start Time:     Tue, 16 Aug 2016 13:02:38 +0800
Labels:         sample=&lt;REDACTED&gt;,submitter=&lt;REDACTED&gt;
Status:         Failed
Reason:         OutOfCPU
Message:        Pod Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000
IP:
Controllers:    &lt;none&gt;
Containers:
  pod-template:
    Image:      &lt;REDACTED&gt;
    Port:
    Command:
      &lt;REDACTED&gt;
    QoS Tier:
      cpu:      Burstable
      memory:   BestEffort
    Limits:
      cpu:      32
    Requests:
      cpu:      14
Volumes:
  workdir:
    Type:       HostPath (bare host directory volume)
    Path:       /mnt
  default-token-u3fnm:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-u3fnm
Events:
  FirstSeen     LastSeen        Count   From                                                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                                                    -------------   --------        ------          -------
  31m           31m             1       {kubelet &lt;REDACTED&gt;}                     Warning         OutOfCPU        Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000"><pre class="notranslate"><code class="notranslate">$ kubectl describe pods worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Name:           worker-7f1df241-5c3f-45cd-8df3-6caa2c972f68
Namespace:      default
Node:           &lt;REDACTED&gt;
Start Time:     Tue, 16 Aug 2016 13:02:38 +0800
Labels:         sample=&lt;REDACTED&gt;,submitter=&lt;REDACTED&gt;
Status:         Failed
Reason:         OutOfCPU
Message:        Pod Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000
IP:
Controllers:    &lt;none&gt;
Containers:
  pod-template:
    Image:      &lt;REDACTED&gt;
    Port:
    Command:
      &lt;REDACTED&gt;
    QoS Tier:
      cpu:      Burstable
      memory:   BestEffort
    Limits:
      cpu:      32
    Requests:
      cpu:      14
Volumes:
  workdir:
    Type:       HostPath (bare host directory volume)
    Path:       /mnt
  default-token-u3fnm:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-u3fnm
Events:
  FirstSeen     LastSeen        Count   From                                                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                                                    -------------   --------        ------          -------
  31m           31m             1       {kubelet &lt;REDACTED&gt;}                     Warning         OutOfCPU        Node didn't have enough resource: CPU, requested: 14000, used: 14100, capacity: 16000
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Pod to stay in a <code class="notranslate">Pending</code> state whilst resources are not available for scheduling.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Submit 10k jobs requiring 14 CPUs on a cluster containing 16 cores each. Note that in this case, we are submitting all 10k jobs within a 1 minute time window.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto"><strong>Is this a request for help?</strong>: No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> OutOfCpu<br>
Issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="168610759" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29846" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/29846/hovercard" href="https://github.com/kubernetes/kubernetes/issues/29846">#29846</a> may be related; however, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="168610759" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29846" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/29846/hovercard" href="https://github.com/kubernetes/kubernetes/issues/29846">#29846</a> is not related to cluster autoscaling and is limited to an OutOfCpu condition.</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.0</p>
<p dir="auto"><strong>Environment</strong>: Google GKE</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Google</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Google Container-VM Image V55</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-omega-cluster-default-pool-d07fc610-clps 4.4.14+ <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Tue Sep 20 10:32:07 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
Using cluster of n1-standard-1 nodes, 1 initial node and enabled cluster autoscaling, I started 34 pods requesting 100m CPU each. Since each node has only 1 CPU and 200m CPU is required for Kubernetes services on each node (300m CPU on the first node), each node can run 8 pods (7 for the first node). All 34 pods would, therefore, required 5 nodes. The sequence of events:</p>
<p dir="auto">Around 15:06:30: All pods are created. As expected, most pods remain in Pending state.<br>
15:06:42: Kubernetes correctly triggers scale-up; however, the scheduler only requests a single node.<br>
15:07:34: Two pods ("out-of-cpurf679" and "out-of-cpuwi5ge") are killed with "Deleted by rescheduler in order to schedule critical pod kube-system_heapster-v1.2.0-3455740371-j973x".<br>
15:09:33: Scale-up is triggered again, scaling from 2 to 3 nodes.<br>
15:11:48: Two other pods ("out-of-cpu3gsro" and "out-of-cpun30p1") fail with OutOfCpu when 10 pods are assigned to the newly added 3rd node.<br>
15:12:31: Next scale-up, going from 3 to 4 nodes.<br>
Eventually, 30 pods are running, 2 pods are in OutOfCpu state, and 2 pods have disappeared.<br>
15:15:16: Another scale-up is triggered from 4 to 5 nodes. The 5th node remains idle as there are no more pods to schedule.</p>
<p dir="auto">Please refer to this zip file for details:<br>
<a href="https://github.com/kubernetes/kubernetes/files/528186/out-of-cpu-bug.zip">out-of-cpu-bug.zip</a></p>
<p dir="auto">"out-of-cpu-bug.yaml" is the YAML that was used to run the experiment through the following commands:</p>
<p dir="auto">1&gt; gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8<br>
2&gt; bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'</p>
<p dir="auto">"out-of-cpu-bug-events.txt" is the capture of events from 'kubectl get events -w' up to the final scale-up. "out-of-cpu-bug-pods1.txt" shows the output of 'kubectl get nodes,pods -a' after the OutOfCpu condition of two pods. Note that only 32 pods show up, because 2 pods have been killed earlier and have "disappeared". "out-of-cpu-bug.pods2.txt" shows the nodes and pods after the final scale-up. The new node "gke-omega-cluster-default-pool-d07fc610-w79v" is unused.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<ol dir="auto">
<li>Scale-up should have been triggered to bring the cluster from 1 to 5 nodes in a single step.</li>
<li>Pods "out-of-cpurf679" and "out-of-cpuwi5ge" should not have been killed and disappeared.</li>
<li>Pods "out-of-cpu3gsro" and "out-of-cpun30p1" should not have been scheduled onto a full node and should not have received an OutOfCpu condition.</li>
<li>Scale-up should not have produced an idle node.</li>
</ol>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">See above. Create a cluster on Google with<br>
gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8</p>
<p dir="auto">Schedule pods with the supplied YAML:<br>
bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'</p>
<p dir="auto">The exact numbers don't matter.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">The bug(s) appear to include some race condition as the error does not always (although frequently) happen. The more pods that are being created, the more likely the situation is observed. The YAML has a commented-out section to request pod affinity. This seems to increase the likelyhood of OutOfCpu failures. Nonetheless, the above commands will show the failure with a 1 in 2 chance. Increase the number of pods to 60 or more and the problem occurs almost every time. I consider this problem a showstopper for any serious cluster autoscaling application.</p>