title: Cluster autoscaler not downscaling
body: <p>I have a regional cluster set up in <strong>google kubernetes engine (GKE)</strong>. The node group is a single <strong>vm in each region (3 total)</strong>. I have a deployment with <strong>3 replicas minimum</strong> controlled by a HPA.
  The <strong>nodegroup is configured to be autoscaling</strong> (cluster autoscaling aka CA).
  The problem scenario:</p>
  
  <p>Update deployment image. Kubernetes automatically creates new pods and the CA identifies that a new node is needed. I now have 4.
  The old pods get removed when all new pods have started, which means I have the exact same CPU request as the minute before. But the after the 10 min maximum downscale time I still have 4 nodes.</p>
  
  <p>The CPU requests for the nodes is now:</p>
  
  <pre><code>CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ------------  ----------  ---------------  -------------
    358m (38%)    138m (14%)  516896Ki (19%)   609056Ki (22%)
  --
    CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ------------  ----------  ---------------  -------------
    800m (85%)    0 (0%)      200Mi (7%)       300Mi (11%)
  --
    CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ------------  ----------  ---------------  -------------
    510m (54%)    100m (10%)  410Mi (15%)      770Mi (29%)
  --
    CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ------------  ----------  ---------------  -------------
    823m (87%)    158m (16%)  484Mi (18%)      894Mi (33%)
  </code></pre>
  
  <p>The 38% node is running:</p>
  
  <pre><code>Namespace                  Name                                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ---------                  ----                                                            ------------  ----------  ---------------  -------------
    kube-system                event-exporter-v0.1.9-5c8fb98cdb-8v48h                          0 (0%)        0 (0%)      0 (0%)           0 (0%)
    kube-system                fluentd-gcp-v2.0.17-q29t2                                       100m (10%)    0 (0%)      200Mi (7%)       300Mi (11%)
    kube-system                heapster-v1.5.2-585f569d7f-886xx                                138m (14%)    138m (14%)  301856Ki (11%)   301856Ki (11%)
    kube-system                kube-dns-autoscaler-69c5cbdcdd-rk7sd                            20m (2%)      0 (0%)      10Mi (0%)        0 (0%)
    kube-system                kube-proxy-gke-production-cluster-default-pool-0fd62aac-7kls    100m (10%)    0 (0%)      0 (0%)           0 (0%)
  </code></pre>
  
  <p>I suspect it wont downscale because heapster or kube-dns-autoscaler.
  But the 85% pod contains:</p>
  
  <pre><code>Namespace                  Name                                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits
    ---------                  ----                                                            ------------  ----------  ---------------  -------------
    kube-system                fluentd-gcp-v2.0.17-s25bk                                       100m (10%)    0 (0%)      200Mi (7%)       300Mi (11%)
    kube-system                kube-proxy-gke-production-cluster-default-pool-7ffeacff-mh6p    100m (10%)    0 (0%)      0 (0%)           0 (0%)
    my-deploy                  my-deploy-54fc6b67cf-7nklb                                      300m (31%)    0 (0%)      0 (0%)           0 (0%)
    my-deploy                  my-deploy-54fc6b67cf-zl7mr                                      300m (31%)    0 (0%)      0 (0%)           0 (0%)
  </code></pre>
  
  <p>The fluentd and kube-proxy pods are present on every node, so I assume they are not needed without the node. Which means that my deployment could be relocated to the other nodes since it only has a request of 300m (31% since only 94% of node CPU is allocatable).</p>
  
  <p>So I figured that Ill check the logs. But if I run <code>kubectl get pods --all-namespaces</code> there are no pod visible on GKE for the CA. And if I use the command <code>kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml</code> it only tells me if it is about to scale, not why or why not.
  Another option is to look at <code>/var/log/cluster-autoscaler.log</code> in the master node. I SSH:ed in the all 4 nodes and only found a <code>gcp-cluster-autoscaler.log.pos</code> file that says: <code>/var/log/cluster-autoscaler.log 0000000000000000        0000000000000000</code> meaning the file should be right there but is empty.
  Last option according to the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noreferrer">FAQ</a>, is to check the events for the pods, but as far as i can tell they are empty.</p>
  
  <p>Anyone know why it wont downscale or atleast where to find the logs?</p>