<p dir="auto">This is blocking <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="72252383" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/7580" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/7580/hovercard" href="https://github.com/kubernetes/kubernetes/pull/7580">#7580</a> (test reboot).</p>
<p dir="auto">I originally ran into this trying to run the guestbook test after the nodes had all rebooted; here is the <a href="https://gist.github.com/mbforbes/ff74c87346f724b0fc8e">log for the failed guestbook test</a>.</p>
<p dir="auto">I added more logging to help understand what is happening. I added the following logging <a href="https://github.com/GoogleCloudPlatform/kubernetes/blob/40ec627cd4df486916572a495a7f139c3faedddb/test/e2e/kubectl.go#L153">after this line</a>:</p>
<div class="highlight highlight-source-go notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Logf(&quot;Call to makeRequestToGuestbook(client, cmd='%s', arg='%s', ns='%s') failed.&quot;, cmd, arg, ns)
Logf(&quot;\t response: expected '%s', got: '%s'&quot;, expectedResponse, res)
Logf(&quot;\t error:    expected '&lt;nil&gt;', got: '%v'&quot;, err)"><pre class="notranslate"><span class="pl-en">Logf</span>(<span class="pl-s">"Call to makeRequestToGuestbook(client, cmd='%s', arg='%s', ns='%s') failed."</span>, <span class="pl-s1">cmd</span>, <span class="pl-s1">arg</span>, <span class="pl-s1">ns</span>)
<span class="pl-en">Logf</span>(<span class="pl-s">"<span class="pl-cce">\t</span> response: expected '%s', got: '%s'"</span>, <span class="pl-s1">expectedResponse</span>, <span class="pl-s1">res</span>)
<span class="pl-en">Logf</span>(<span class="pl-s">"<span class="pl-cce">\t</span> error:    expected '&lt;nil&gt;', got: '%v'"</span>, <span class="pl-s1">err</span>)</pre></div>
<p dir="auto">The first time it's called I get an error that says <code class="notranslate">no endpoints available for "frontend"</code>, but then after that, I get the following repeated errors:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INFO: Call to makeRequestToGuestbook(client, cmd='get', arg='', ns='e2e-tests-kubectl-7d744f0d-775f-4fd0-93b2-88135adbc11e') failed.
INFO:    response: expected '{&quot;data&quot;: &quot;&quot;}', got: '&lt;br /&gt;
&lt;b&gt;Fatal error&lt;/b&gt;:  Uncaught exception 'Predis\Connection\ConnectionException' with message 'php_network_getaddresses: getaddrinfo failed: Name or service not known [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141
Stack trace:
#0 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(96): Predis\Connection\AbstractConnection-&amp;gt;onConnectionError('php_network_get...', 0)
#1 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(70): Predis\Connection\StreamConnection-&amp;gt;tcpStreamInitializer(Object(Predis\Connection\ConnectionParameters))
#2 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(96): Predis\Connection\StreamConnection-&amp;gt;createResource()
#3 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(144): Predis\Connection\AbstractConnection-&amp;gt;connect()
#4 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(181): Predis\Connection\StreamConnection-&amp;gt;connect()
#5 /vendor/predis/predis/lib/Predis/Connection/StreamConnectio in &lt;b&gt;/vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php&lt;/b&gt; on line &lt;b&gt;141&lt;/b&gt;&lt;br /&gt;
'
INFO:    error:    expected '&lt;nil&gt;', got: '&lt;nil&gt;'"><pre class="notranslate"><code class="notranslate">INFO: Call to makeRequestToGuestbook(client, cmd='get', arg='', ns='e2e-tests-kubectl-7d744f0d-775f-4fd0-93b2-88135adbc11e') failed.
INFO:    response: expected '{"data": ""}', got: '&lt;br /&gt;
&lt;b&gt;Fatal error&lt;/b&gt;:  Uncaught exception 'Predis\Connection\ConnectionException' with message 'php_network_getaddresses: getaddrinfo failed: Name or service not known [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141
Stack trace:
#0 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(96): Predis\Connection\AbstractConnection-&amp;gt;onConnectionError('php_network_get...', 0)
#1 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(70): Predis\Connection\StreamConnection-&amp;gt;tcpStreamInitializer(Object(Predis\Connection\ConnectionParameters))
#2 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(96): Predis\Connection\StreamConnection-&amp;gt;createResource()
#3 /vendor/predis/predis/lib/Predis/Connection/StreamConnection.php(144): Predis\Connection\AbstractConnection-&amp;gt;connect()
#4 /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php(181): Predis\Connection\StreamConnection-&amp;gt;connect()
#5 /vendor/predis/predis/lib/Predis/Connection/StreamConnectio in &lt;b&gt;/vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php&lt;/b&gt; on line &lt;b&gt;141&lt;/b&gt;&lt;br /&gt;
'
INFO:    error:    expected '&lt;nil&gt;', got: '&lt;nil&gt;'
</code></pre></div>
<p dir="auto">Just to make that more readable, the error before the stack trace is: <code class="notranslate">Uncaught exception 'Predis\Connection\ConnectionException' with message 'php_network_getaddresses: getaddrinfo failed: Name or service not known [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141</code></p>
<p dir="auto">Any tips or help greatly appreciated. I'm not sure where in the tcp / services / php / redis stack this is going wrong.</p>
<p dir="auto">Apologies in advance for the large cc:<br>
+cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/roberthbailey/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/roberthbailey">@roberthbailey</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zmerlynn/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zmerlynn">@zmerlynn</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/quinton-hoole/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/quinton-hoole">@quinton-hoole</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ixdy/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ixdy">@ixdy</a></p>
<p dir="auto">I believe I'm seeing an issue where secrets are not available after server reboot.</p>
<p dir="auto">The steps to reproduce are below.  Note that the following steps have been run on v.0.16.2:</p>
<p dir="auto">Create the following simple secret:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# From kubernetes/examples/secrets/secret.yaml
apiVersion: v1beta3
kind: Secret
metadata:
  name: test-secret
data:
  data-1: dmFsdWUtMQ0K
  data-2: dmFsdWUtMg0KDQo="><pre class="notranslate"><code class="notranslate"># From kubernetes/examples/secrets/secret.yaml
apiVersion: v1beta3
kind: Secret
metadata:
  name: test-secret
data:
  data-1: dmFsdWUtMQ0K
  data-2: dmFsdWUtMg0KDQo=
</code></pre></div>
<p dir="auto">Then create the following pod:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1beta3
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: test
    image: centos:centos7.0.1406
    command: [ &quot;bash&quot;, &quot;-c&quot;, &quot;while [ true ]; do cat /etc/test-secret/data-1; sleep 5; done&quot; ]
    volumeMounts:
    - mountPath: /etc/test-secret
      name: test-secret
      readOnly: true
  volumes:
  - name: test-secret
    secret:
      secretName: test-secret
"><pre class="notranslate"><code class="notranslate">apiVersion: v1beta3
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: test
    image: centos:centos7.0.1406
    command: [ "bash", "-c", "while [ true ]; do cat /etc/test-secret/data-1; sleep 5; done" ]
    volumeMounts:
    - mountPath: /etc/test-secret
      name: test-secret
      readOnly: true
  volumes:
  - name: test-secret
    secret:
      secretName: test-secret

</code></pre></div>
<p dir="auto">After starting the pod you can see that it is correctly slowing the data-1 value from the secret</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl log test
2015-05-08T16:06:37.433196624Z value-1"><pre class="notranslate"><code class="notranslate">kubectl log test
2015-05-08T16:06:37.433196624Z value-1
</code></pre></div>
<p dir="auto">Now reboot the server running the pod.  After reboot I see the following in the log output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl log test
2015-05-08T16:08:53.172253741Z cat: /etc/test-secret/data-1: No such file or directory"><pre class="notranslate"><code class="notranslate">kubectl log test
2015-05-08T16:08:53.172253741Z cat: /etc/test-secret/data-1: No such file or directory
</code></pre></div>
<p dir="auto">That seems to indicate to me that the pod is re-starting after the reboot but that it is unable to get the secret.  Note that after the reboot manually stopping ( kubectl stop pod test ) and re-starting the pod shows the expected results.</p>
<p dir="auto">My expectation of the behavior is that if the server a pod is running on reboots that the pod will re-start with the same secret it had when it started previously.  If the secret had been updated since the pod had been originally started I would expect the original secret to be available.</p>