<p dir="auto">See log here: <a href="https://gist.github.com/sebv/914c9b73b48e4542bfeb">https://gist.github.com/sebv/914c9b73b48e4542bfeb</a> , the symlink to /mnt/ephemeral/docker/containers is not valid within container.</p>
<p dir="auto">This is a new issue on 1.1 (it's working on 1.0.7)</p>
<p dir="auto">On the node:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ubuntu@ip-172-20-0-155:~$ sudo ls -l /var/log/containers
total 80
lrwxrwxrwx 1 root root 171 Nov 23 21:24 elasticsearch-logging-v1-4yjm2_kube-system_elasticsearch-logging-b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041.log -&gt; /mnt/ephemeral/docker/containers/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041-json.log"><pre class="notranslate"><code class="notranslate">ubuntu@ip-172-20-0-155:~$ sudo ls -l /var/log/containers
total 80
lrwxrwxrwx 1 root root 171 Nov 23 21:24 elasticsearch-logging-v1-4yjm2_kube-system_elasticsearch-logging-b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041.log -&gt; /mnt/ephemeral/docker/containers/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041-json.log
</code></pre></div>
<p dir="auto">My version:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  ~  kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;0+&quot;, GitVersion:&quot;v1.0.8-beta.3+27719743e6b69e&quot;, GitCommit:&quot;27719743e6b69e9526d5133b6d6b71cc6f1024c2&quot;, GitTreeState:&quot;clean&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;1&quot;, GitVersion:&quot;v1.1.1&quot;, GitCommit:&quot;92635e23dfafb2ddc828c8ac6c03c7a7205a84d8&quot;, GitTreeState:&quot;clean&quot;}"><pre class="notranslate"><code class="notranslate">  ~  kubectl version
Client Version: version.Info{Major:"1", Minor:"0+", GitVersion:"v1.0.8-beta.3+27719743e6b69e", GitCommit:"27719743e6b69e9526d5133b6d6b71cc6f1024c2", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}
</code></pre></div>
<p dir="auto">It seems that there are potentially two separate issues here:</p>
<ul dir="auto">
<li>The inconsistent docker root due to a potential race between kubelet and docker.</li>
<li>The misconfiguration (on AWS) of fluentd to mount /var/lib/docker instead of /mnt/ephemeral/docker.</li>
</ul>
<p dir="auto">For the first issue, perhaps the kubelet should fail if it is not able to retrieve the docker root from docker info?</p>
<p dir="auto">An alternative solution would be to mount our volume for AWS to /var/lib/docker instead of /mnt/ephemeral/docker. This would solve both issues. But, it seems that the kubelet is providing the ability to dynamically locate the docker root for some reason.</p>
<h5 dir="auto">Details:</h5>
<p dir="auto">I'm having intermittent issues with fluentd pushing logs to elasticsearch. It seems that fluentd is unable to see the log files, even though it is able list the containers directory. This is happening on an AWS cluster.</p>
<p dir="auto">From inside the fluentd I see the following:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" root@fluentd-elasticsearch-ip-172-20-0-194:/# cat /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log
 cat: /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log: No such file or directory"><pre class="notranslate"> root@fluentd-elasticsearch-ip-172-20-0-194:/<span class="pl-c"><span class="pl-c">#</span> cat /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log</span>
 cat: /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log: No such file or directory</pre></div>
<p dir="auto">On AWS the log file symlinks look like this:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" root@fluentd-elasticsearch-ip-172-20-0-194:/# ls -al /varlog/containers/vapor-web-v1-x894w_*
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -&gt; /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -&gt; /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -&gt; /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log"><pre class="notranslate"> root@fluentd-elasticsearch-ip-172-20-0-194:/<span class="pl-c"><span class="pl-c">#</span> ls -al /varlog/containers/vapor-web-v1-x894w_*</span>
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
 lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log</pre></div>
<p dir="auto">Fluentd seems to expect that the symlinks will point to /var/lib/docker/containers but on AWS we use /mnt/ephemeral/docker/containers.</p>
<p dir="auto">From the node I see:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="root@ip-172-20-0-194:/home/ubuntu# ls -al /var/log/containers/vapor-web-v1-x894w_vapor-alpha_*
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -&gt; /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -&gt; /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -&gt; /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log"><pre class="notranslate">root@ip-172-20-0-194:/home/ubuntu# ls -al /var/log/containers/vapor-web-v1-x894w_vapor-alpha_<span class="pl-k">*</span>
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -<span class="pl-k">&gt;</span> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log</pre></div>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" root@ip-172-20-0-194:/home/ubuntu# cat /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log | tail -n 5
 {&quot;log&quot;:&quot;172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] \&quot;GET / HTTP/1.1\&quot; 302 116 \&quot;-\&quot; \&quot;Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)\&quot;\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2015-08-28T17:29:26.242591558Z&quot;}
 {&quot;log&quot;:&quot;172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] \&quot;GET /users/sign_in HTTP/1.1\&quot; 200 1954 \&quot;-\&quot; \&quot;Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)\&quot;\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2015-08-28T17:29:26.265429648Z&quot;}
 {&quot;log&quot;:&quot;2015/08/28 17:29:26 [info] 5#5: *58423 client 172.20.0.70 closed keepalive connection\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2015-08-28T17:29:26.271201715Z&quot;}
 {&quot;log&quot;:&quot;2015/08/28 17:29:38 [info] 5#5: *58426 client closed connection while waiting for request, client: 172.20.0.171, server: 0.0.0.0:80\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2015-08-28T17:29:38.549273837Z&quot;}
 {&quot;log&quot;:&quot;2015/08/28 17:29:46 [info] 5#5: *58427 client closed connection while waiting for request, client: 172.20.0.33, server: 0.0.0.0:80\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2015-08-28T17:29:46.549263618Z&quot;}"><pre class="notranslate"> root@ip-172-20-0-194:/home/ubuntu# cat /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log <span class="pl-k">|</span> tail -n 5
 {<span class="pl-s"><span class="pl-pds">"</span>log<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] <span class="pl-cce">\"</span>GET / HTTP/1.1<span class="pl-cce">\"</span> 302 116 <span class="pl-cce">\"</span>-<span class="pl-cce">\"</span> <span class="pl-cce">\"</span>Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)<span class="pl-cce">\"</span>\n<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>stream<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>stdout<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015-08-28T17:29:26.242591558Z<span class="pl-pds">"</span></span>}
 {<span class="pl-s"><span class="pl-pds">"</span>log<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] <span class="pl-cce">\"</span>GET /users/sign_in HTTP/1.1<span class="pl-cce">\"</span> 200 1954 <span class="pl-cce">\"</span>-<span class="pl-cce">\"</span> <span class="pl-cce">\"</span>Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)<span class="pl-cce">\"</span>\n<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>stream<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>stdout<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015-08-28T17:29:26.265429648Z<span class="pl-pds">"</span></span>}
 {<span class="pl-s"><span class="pl-pds">"</span>log<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015/08/28 17:29:26 [info] 5#5: *58423 client 172.20.0.70 closed keepalive connection\n<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>stream<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>stdout<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015-08-28T17:29:26.271201715Z<span class="pl-pds">"</span></span>}
 {<span class="pl-s"><span class="pl-pds">"</span>log<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015/08/28 17:29:38 [info] 5#5: *58426 client closed connection while waiting for request, client: 172.20.0.171, server: 0.0.0.0:80\n<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>stream<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>stdout<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015-08-28T17:29:38.549273837Z<span class="pl-pds">"</span></span>}
 {<span class="pl-s"><span class="pl-pds">"</span>log<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015/08/28 17:29:46 [info] 5#5: *58427 client closed connection while waiting for request, client: 172.20.0.33, server: 0.0.0.0:80\n<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>stream<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>stdout<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>2015-08-28T17:29:46.549263618Z<span class="pl-pds">"</span></span>}</pre></div>