<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="13:41:01 Bringing down cluster
13:41:49 ERROR: (gcloud.compute.instance-groups.managed.delete) Some requests did not succeed:
13:41:49  - Internal Error
13:41:49 
13:41:51 ERROR: (gcloud.compute.instance-templates.delete) Some requests did not succeed:
13:41:51  - The instance_template resource 'e2e-slow-minion-template' is already being used by 'e2e-slow-minion-group'
13:41:51 
13:41:57 Updated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/zones/us-central1-f/instances/e2e-slow-master].
13:42:44 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/zones/us-central1-f/instances/e2e-slow-master].
13:43:09 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/firewalls/e2e-slow-master-https].
13:43:26 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/firewalls/e2e-slow-minion-all].
13:43:27 Deleting routes e2e-slow-5f46aa47-79b1-11e5-bafa-42010af00002 e2e-slow-96a9332a-79b7-11e5-bafa-42010af00002 e2e-slow-a8196f66-79b0-11e5-bafa-42010af00002
13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-5f46aa47-79b1-11e5-bafa-42010af00002].
13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-96a9332a-79b7-11e5-bafa-42010af00002].
13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-a8196f66-79b0-11e5-bafa-42010af00002].
13:43:57 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/regions/us-central1/addresses/e2e-slow-master-ip]."><pre class="notranslate"><span class="pl-c1">13:41:01 Bringing down cluster</span>
<span class="pl-c1">13:41:49 ERROR: (gcloud.compute.instance-groups.managed.delete) Some requests did not succeed:</span>
<span class="pl-c1">13:41:49  - Internal Error</span>
<span class="pl-c1">13:41:49 </span>
<span class="pl-c1">13:41:51 ERROR: (gcloud.compute.instance-templates.delete) Some requests did not succeed:</span>
<span class="pl-c1">13:41:51  - The instance_template resource 'e2e-slow-minion-template' is already being used by 'e2e-slow-minion-group'</span>
<span class="pl-c1">13:41:51 </span>
<span class="pl-c1">13:41:57 Updated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/zones/us-central1-f/instances/e2e-slow-master].</span>
<span class="pl-c1">13:42:44 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/zones/us-central1-f/instances/e2e-slow-master].</span>
<span class="pl-c1">13:43:09 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/firewalls/e2e-slow-master-https].</span>
<span class="pl-c1">13:43:26 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/firewalls/e2e-slow-minion-all].</span>
<span class="pl-c1">13:43:27 Deleting routes e2e-slow-5f46aa47-79b1-11e5-bafa-42010af00002 e2e-slow-96a9332a-79b7-11e5-bafa-42010af00002 e2e-slow-a8196f66-79b0-11e5-bafa-42010af00002</span>
<span class="pl-c1">13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-5f46aa47-79b1-11e5-bafa-42010af00002].</span>
<span class="pl-c1">13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-96a9332a-79b7-11e5-bafa-42010af00002].</span>
<span class="pl-c1">13:43:49 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/global/routes/e2e-slow-a8196f66-79b0-11e5-bafa-42010af00002].</span>
<span class="pl-c1">13:43:57 Deleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-slow/regions/us-central1/addresses/e2e-slow-master-ip].</span></pre></div>
<p dir="auto">Our <code class="notranslate">kubernetes-upgrade-gce</code> Jenkins job has been broken for quite a while now, and it looks like it's due to target pools and/or forwarding rules leaking and exceeding quota:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sep  2 00:48:45.094: INFO: event for mutability-service-test: {service-controller } creating loadbalancer: creating loadbalancer
Sep  2 00:48:45.094: INFO: event for mutability-service-test: {service-controller } creating loadbalancer failed: error creating load balancer (will retry): failed to create external load balancer for service e2e-tests-services-mrlvq/mutability-service-test: googleapi: Error 403: Quota 'TARGET_POOLS' exceeded.  Limit: 50.0"><pre class="notranslate"><code class="notranslate">Sep  2 00:48:45.094: INFO: event for mutability-service-test: {service-controller } creating loadbalancer: creating loadbalancer
Sep  2 00:48:45.094: INFO: event for mutability-service-test: {service-controller } creating loadbalancer failed: error creating load balancer (will retry): failed to create external load balancer for service e2e-tests-services-mrlvq/mutability-service-test: googleapi: Error 403: Quota 'TARGET_POOLS' exceeded.  Limit: 50.0
</code></pre></div>
<p dir="auto">I've cleared out the old rules, and I'll try to diagnose what's leaking them and why.</p>