<p dir="auto">We've been encountering increasing number of issues where the <code class="notranslate">kubelet</code> seems to be in some sort of hung state while handling processing events (like creating or destroying a pod):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get po
NAME                                   READY     STATUS              
appname-2331821309-lgj2o      0/1       ContainerCreating"><pre class="notranslate"><code class="notranslate">$ kubectl get po
NAME                                   READY     STATUS              
appname-2331821309-lgj2o      0/1       ContainerCreating
</code></pre></div>
<p dir="auto">Getting on the <code class="notranslate">kubelet</code> where the container is hung, the logs seem to have a lot of transport errors (maybe our <code class="notranslate">kube-apiserver</code> crashed or disappeared):</p>
<p dir="auto"><code class="notranslate">kubelet.log</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubelet: E0613 11:01:53.943206   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6439574&amp;timeoutSeconds=420: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943276   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6439299&amp;timeoutSeconds=487: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943315   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6468007&amp;timeoutSeconds=505: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:55.640953   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6439574 (6468022)
kubelet: E0613 11:01:55.667667   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6439299 (6468022)
kubelet: E0613 11:01:55.843807   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6468007 (6468022)
kubelet: E0613 11:09:54.814446   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468023 (6468279)
kubelet: E0613 15:34:56.610783   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6496591&amp;timeoutSeconds=398: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615108   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6468845&amp;timeoutSeconds=374: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615166   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&amp;timeoutSeconds=532: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.613588   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6496591&amp;timeoutSeconds=405: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617199   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&amp;timeoutSeconds=429: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617255   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6468845&amp;timeoutSeconds=526: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:58.786094   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6496591 (6496612)
kubelet: E0613 15:34:58.969776   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6468023 (6496611)
kubelet: E0613 15:34:59.037563   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468845 (6496611)
kubelet: E0613 15:35:00.512631   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6496612 (6496617)"><pre class="notranslate"><code class="notranslate">kubelet: E0613 11:01:53.943206   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6439574&amp;timeoutSeconds=420: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943276   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6439299&amp;timeoutSeconds=487: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943315   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6468007&amp;timeoutSeconds=505: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:55.640953   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6439574 (6468022)
kubelet: E0613 11:01:55.667667   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6439299 (6468022)
kubelet: E0613 11:01:55.843807   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6468007 (6468022)
kubelet: E0613 11:09:54.814446   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468023 (6468279)
kubelet: E0613 15:34:56.610783   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6496591&amp;timeoutSeconds=398: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615108   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6468845&amp;timeoutSeconds=374: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615166   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&amp;timeoutSeconds=532: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.613588   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&amp;resourceVersion=6496591&amp;timeoutSeconds=405: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617199   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&amp;timeoutSeconds=429: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617255   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&amp;resourceVersion=6468845&amp;timeoutSeconds=526: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:58.786094   28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6496591 (6496612)
kubelet: E0613 15:34:58.969776   28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6468023 (6496611)
kubelet: E0613 15:34:59.037563   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468845 (6496611)
kubelet: E0613 15:35:00.512631   28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6496612 (6496617)
</code></pre></div>
<p dir="auto">Restarting the <code class="notranslate">kubelet</code> process seems to make everything process as expected.</p>
<p dir="auto">While it's definitely not ideal (the apiserver being unavailable) I would not expect what is a very possible intermittent network failure to result in a completely unresponsive kubelet. Instead I'd expect it to gracefully resolve itself and resume communication with the apiserver or to crash altogether. The latter would be a lot easier from an operators standpoint as at least we would have some idea there is an issue occurring.</p>
<p dir="auto">Sorry in advance for the sparse information in this issue. I'm happy to provide as much as possible but the issue has been pretty intermittent and the <code class="notranslate">kubelet</code> logs don't seem overly helpful.</p>
<p dir="auto">Following this doc <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/libvirt-coreos/README.md">https://github.com/kubernetes/kubernetes/blob/master/cluster/libvirt-coreos/README.md</a></p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="git clone https://github.com/kubernetes/kubernetes/
cd kubernetes
make release-skip-tests
export KUBERNETES_PROVIDER=libvirt-coreos
./cluster/kube-up.sh
ssh core@192.168.10.2
docker run -d --name nginx nginx
docker inspect  --format '{{ .NetworkSettings.IPAddress }}' nginx # i.e. returns 10.10.1.3"><pre class="notranslate">git clone https://github.com/kubernetes/kubernetes/
<span class="pl-c1">cd</span> kubernetes
make release-skip-tests
<span class="pl-k">export</span> KUBERNETES_PROVIDER=libvirt-coreos
./cluster/kube-up.sh
ssh core@192.168.10.2
docker run -d --name nginx nginx
docker inspect  --format <span class="pl-s"><span class="pl-pds">'</span>{{ .NetworkSettings.IPAddress }}<span class="pl-pds">'</span></span> nginx <span class="pl-c"><span class="pl-c">#</span> i.e. returns 10.10.1.3</span></pre></div>
<p dir="auto">then run curl on another k8s minion:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ssh core@192.168.10.3
curl 10.10.1.3"><pre class="notranslate">ssh core@192.168.10.3
curl 10.10.1.3</pre></div>
<p dir="auto">And you'll see result below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;

&lt;p&gt;For online documentation and support please refer to
&lt;a href=&quot;http://nginx.org/&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
Commercial support is available at
&lt;a href=&quot;http://nginx.com/&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer"><pre class="notranslate"><code class="notranslate">core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;

&lt;p&gt;For online documentation and support please refer to
&lt;a href="http://nginx.org/"&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
Commercial support is available at
&lt;a href="http://nginx.com/"&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
</code></pre></div>
<p dir="auto">So looks like there is something wrong with the bridge configuration in <code class="notranslate">libvirt-coreos</code> provider.</p>
<p dir="auto">Here is tcpdump:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="10:50:44.059742 IP 10.10.2.1.59138 &gt; 10.10.1.3.http: Flags [S], seq 3020067180, win 29200, options [mss 1460,sackOK,TS val 3556468 ecr 0,nop,wscale 6], length 0
10:50:44.059777 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.059793 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.060083 IP 10.10.2.1.59138 &gt; 10.10.1.3.http: Flags [.], ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 0
10:50:44.060090 IP 10.10.2.1.59138 &gt; 10.10.1.3.http: Flags [P.], seq 1:74, ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 73
10:50:44.060110 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060114 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060131 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060133 IP 10.10.1.3.http &gt; 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0"><pre class="notranslate">10:50:44.059742 IP 10.10.2.1.59138 <span class="pl-k">&gt;</span> 10.10.1.3.http: Flags [S], seq 3020067180, win 29200, options [mss 1460,sackOK,TS val 3556468 ecr 0,nop,wscale 6], length 0
10:50:44.059777 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.059793 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.060083 IP 10.10.2.1.59138 <span class="pl-k">&gt;</span> 10.10.1.3.http: Flags [.], ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 0
10:50:44.060090 IP 10.10.2.1.59138 <span class="pl-k">&gt;</span> 10.10.1.3.http: Flags [P.], seq 1:74, ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 73
10:50:44.060110 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060114 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060131 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060133 IP 10.10.1.3.http <span class="pl-k">&gt;</span> 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0</pre></div>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lhuard1A/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lhuard1A">@lhuard1A</a></p>