<p dir="auto">I find it curious that this doesn't already exist. I assume there is a reason for that but I couldn't find any references for it. What is the reason?</p>
<p dir="auto">If there is hope of adding custom gke scopes, I'd be happy to submit a PR, but I'd like some feedback first. I noticed notes for <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mbforbes/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mbforbes">@mbforbes</a> like "Refactor setting scope flags." What refactor was in mind? Currently scopes are set in the MINION_SCOPES variable which is a bash array. Because it's an array it needs a join_csv function. But as far as I can tell, the variable is never "pushed" or "popped" so I see no benefit from using the array. Although if we eventually want to push e.g. monitoring, logging scopes onto the array, we should keep it. Alternatively it might be easy to just concat comma-separated strings.</p>
<p dir="auto">I don't mean to bike shed here, I'd just like to do the simplest PR that will be accepted and enable custom scopes in gke.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">unable to delete filesystem</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT.</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:08:43Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GKE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<p dir="auto">PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"<br>
NAME="Debian GNU/Linux"<br>
VERSION_ID="7"<br>
VERSION="7 (wheezy)"<br>
ID=debian<br>
ANSI_COLOR="1;31"<br>
HOME_URL="<a href="http://www.debian.org/" rel="nofollow">http://www.debian.org/</a>"<br>
SUPPORT_URL="<a href="http://www.debian.org/support/" rel="nofollow">http://www.debian.org/support/</a>"<br>
BUG_REPORT_URL="<a href="http://bugs.debian.org/" rel="nofollow">http://bugs.debian.org/</a>"</p>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-xxx-default-pool-xxx-xxx 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: None.</li>
<li><strong>Others</strong>: None.</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Garbage collector fails to remove old containers. Eventually nodes run out of disc space and docker daemon and/or other things start to crash. Instances get stuck with 100% CPU usage.</p>
<p dir="auto">I've since replaced the stuck instances (but this is not the first time). I'm still seeing following error in Google Logs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubelet: &quot;Failed to remove dead container &quot;/k8s_POD.xxxx&quot;: Error response from daemon: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy&quot;

docker: Handler for DELETE /containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e returned error: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy&quot;
"><pre class="notranslate"><code class="notranslate">kubelet: "Failed to remove dead container "/k8s_POD.xxxx": Error response from daemon: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy"

docker: Handler for DELETE /containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e returned error: Unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy"

</code></pre></div>
<p dir="auto">Before I saw more interesting errors:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="docker: &quot;collecting stats for 2356f6b7eb7f993d6ea1e9286e5edc0da11dda4e9d2e4d4beb05c35e71878067: invalid id: 

kubelet:  &quot;Failed to create summary reader for &quot;/docker/1a2010b84bf49f396e5acd8a4d0d96458043ff10eeb5dc4126ad3213ef213155&quot;: none of the resources are being tracked.&quot; "><pre class="notranslate"><code class="notranslate">docker: "collecting stats for 2356f6b7eb7f993d6ea1e9286e5edc0da11dda4e9d2e4d4beb05c35e71878067: invalid id: 

kubelet:  "Failed to create summary reader for "/docker/1a2010b84bf49f396e5acd8a4d0d96458043ff10eeb5dc4126ad3213ef213155": none of the resources are being tracked." 
</code></pre></div>
<p dir="auto">A bit later:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="22:06:11.000
eviction manager: unexpected err: failed GetContainerInfoV2: unknown container: &quot;/&quot;
22:06:11.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:12.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:14.000
Failed to create summary reader for &quot;/docker/9dd56e5a42bb17701e322476aeeb3767295c1705afc3bb37bb0c2c83dcdc9750&quot;: none of the resources are being tracked.
22:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
22:06:16.000
while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
22:06:16.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:17.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:17.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:18.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
Failed to create summary reader for &quot;/docker/82ddd29cbe84685adf1322ad77016b1c4721a5e228fe23381b320c480cbb904f&quot;: none of the resources are being tracked.
22:06:19.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:19.000
Failed to check if disk space is available for the runtime: failed to get fs info for &quot;runtime&quot;: unable to find data for container /
22:06:19.000
Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: unable to find data for container /"><pre class="notranslate"><code class="notranslate">22:06:11.000
eviction manager: unexpected err: failed GetContainerInfoV2: unknown container: "/"
22:06:11.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:12.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:13.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:14.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:14.000
Failed to create summary reader for "/docker/9dd56e5a42bb17701e322476aeeb3767295c1705afc3bb37bb0c2c83dcdc9750": none of the resources are being tracked.
22:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:16.000
while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
22:06:16.000
while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
22:06:16.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:17.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:17.000
Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead.
22:06:18.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:18.000
Failed to create summary reader for "/docker/82ddd29cbe84685adf1322ad77016b1c4721a5e228fe23381b320c480cbb904f": none of the resources are being tracked.
22:06:19.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
22:06:19.000
Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
22:06:19.000
Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
</code></pre></div>
<p dir="auto">and</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="15:22:03.000
Failed to delete container: operation timeout: context deadline exceeded; Skipping pod &quot;ec5aa9ac-7e6e-11e6-b32a-42010a800070&quot;
15:22:04.000
Failed to create pod infra container: RunContainerError; Skipping pod &quot;xxx(323984f3-7fd4-11e6-b32a-42010a800070)&quot;: runContainer: operation timeout: context deadline exceeded
15:22:19.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:22:29.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:22:33.000
GenericPLEG: Unable to retrieve pods: operation timeout: context deadline exceeded
15:22:34.000
Error listing containers: dockertools.operationTimeout{err:(*errors.errorString)(0xc82000f310)}
15:22:34.000
Failed cleaning pods: operation timeout: context deadline exceeded
15:22:39.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:22:40.000
Container garbage collection failed: operation timeout: context deadline exceeded
15:22:49.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:22:59.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:23:05.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq
15:23:09.000
Missing default interface &quot;eth0&quot; for pod:xxx-446723216-jbngq"><pre class="notranslate"><code class="notranslate">15:22:03.000
Failed to delete container: operation timeout: context deadline exceeded; Skipping pod "ec5aa9ac-7e6e-11e6-b32a-42010a800070"
15:22:04.000
Failed to create pod infra container: RunContainerError; Skipping pod "xxx(323984f3-7fd4-11e6-b32a-42010a800070)": runContainer: operation timeout: context deadline exceeded
15:22:19.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:29.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:33.000
GenericPLEG: Unable to retrieve pods: operation timeout: context deadline exceeded
15:22:34.000
Error listing containers: dockertools.operationTimeout{err:(*errors.errorString)(0xc82000f310)}
15:22:34.000
Failed cleaning pods: operation timeout: context deadline exceeded
15:22:39.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:40.000
Container garbage collection failed: operation timeout: context deadline exceeded
15:22:49.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:22:59.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:23:05.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
15:23:09.000
Missing default interface "eth0" for pod:xxx-446723216-jbngq
</code></pre></div>
<p dir="auto">Eventually leading to this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="03:06:14.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Failed to check if disk space is available for the runtime: failed to get fs info for &quot;runtime&quot;: unable to find data for container /
03:06:15.000
Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: unable to find data for container /
03:06:15.000
Failed to create summary reader for &quot;/docker/93f4b7062198072d70f8ec855639464926747ba8ddb902265c0155623c5f1c63&quot;: none of the resources are being tracked.
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
eviction manager: unexpected err: failed ImageStats: failed to list docker images - Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:17.000
"><pre class="notranslate"><code class="notranslate">03:06:14.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:14.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Error getting image list: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
03:06:15.000
Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
03:06:15.000
Failed to create summary reader for "/docker/93f4b7062198072d70f8ec855639464926747ba8ddb902265c0155623c5f1c63": none of the resources are being tracked.
03:06:15.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:15.000
eviction manager: unexpected err: failed ImageStats: failed to list docker images - Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
Container runtime sanity check failed: docker: failed to get docker version: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:16.000
GenericPLEG: Unable to retrieve pods: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
03:06:17.000

</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Garbage collector doesn't let nodes run out of disc space.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">I'm not exactly sure. We have multiple deployments in multiple namespaces. The only out of ordinary thing is that we use big container images ~1gb. This has happened couple times. We were running <code class="notranslate">weave-scope</code> before and it seemed that maybe it might be related because when I inspected failing nodes using <code class="notranslate">kubectl describe</code> they showed event "deleting all containers" and the only container left running was the scope. It was stuck in "terminating" state and no way to remove it (<code class="notranslate">kubectl delete pod</code>) didn't work.</p>
<p dir="auto">I have since removed <code class="notranslate">weave-scope</code>, but I still see the <code class="notranslate">Failed to remove dead container</code> errors.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">I've tracked down a docker issue which seems to be related - <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="145033091" data-permission-text="Title is private" data-url="https://github.com/moby/moby/issues/21704" data-hovercard-type="issue" data-hovercard-url="/moby/moby/issues/21704/hovercard" href="https://github.com/moby/moby/issues/21704">moby/moby#21704</a>.</p>
<p dir="auto">Also seems related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="156065049" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26000" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/26000/hovercard" href="https://github.com/kubernetes/kubernetes/issues/26000">#26000</a>, but we're using standard GKE image using AUFS.</p>
<p dir="auto">Docker issue mentions that upgrading kernel to <code class="notranslate">&gt;=3.19</code> might help. Any chances we could have newer kernel in GKE image?</p>