<p dir="auto">I setup o 2-node kubernetes cluster using docker containers, as described here: <a href="https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/docker-multinode.md">https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/docker-multinode.md</a></p>
<p dir="auto">Now, I created a secret and a test pod:<br>
(using files from the branch 'release-1.1' and commit 'ac3f023ab28ed4f4c4f88a3fb518e90f4a8d4749')</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl create -f docs/user-guide/secrets/secret.yaml
kubectl create -f docs/user-guide/secrets/secret-pod.yaml"><pre class="notranslate"><code class="notranslate">kubectl create -f docs/user-guide/secrets/secret.yaml
kubectl create -f docs/user-guide/secrets/secret-pod.yaml
</code></pre></div>
<p dir="auto">Both secret and pod are created:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe secret test-secret
Name:           test-secret
Namespace:      default
Labels:         &lt;none&gt;
Annotations:    &lt;none&gt;

Type:   Opaque

Data
====
data-1: 9 bytes
data-2: 11 bytes"><pre class="notranslate"><code class="notranslate">$ kubectl describe secret test-secret
Name:           test-secret
Namespace:      default
Labels:         &lt;none&gt;
Annotations:    &lt;none&gt;

Type:   Opaque

Data
====
data-1: 9 bytes
data-2: 11 bytes
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe po/secret-test-pod
Name:                           secret-test-pod
Namespace:                      default
Image(s):                       kubernetes/mounttest:0.1
Node:                           127.0.0.1/127.0.0.1
Start Time:                     Thu, 10 Dec 2015 16:20:49 +0100
Labels:                         &lt;none&gt;
Status:                         Running
Reason: 
Message:
IP:                             10.1.61.8
Replication Controllers:        &lt;none&gt;
Containers:
  test-container:
    Container ID:       docker://82409e45ad6fcd73a8100a1b0767692d5e823cc3a14d8154644293d6b4c86854
    Image:              kubernetes/mounttest:0.1
    Image ID:           docker://b14127f747a0569e9ab96f0ee2de1b07630b7dbc16f8facf0b0adabd5848d41c
    State:              Running
      Started:          Thu, 10 Dec 2015 16:20:51 +0100
    Ready:              False
    Restart Count:      0
    Environment Variables:
Conditions:
  Type          Status
  Ready         False 
Volumes:
  secret-volume:
    Type:       Secret (a secret that should populate this volume)
    SecretName: test-secret
Events:
  FirstSeen     LastSeen        Count   From                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                    ─────────────                           ──────          ───────
  9s            9s              1       {scheduler }                                                    Scheduled       Successfully assigned secret-test-pod to 127.0.0.1
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Pulled          Container image &quot;gcr.io/google_containers/pause:0.8.0&quot; already present on machine
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Created         Created with docker id cb1290a496c8
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Started         Started with docker id cb1290a496c8
  8s            8s              1       {kubelet 127.0.0.1}                                             FailedSync      Error syncing pod, skipping: open /var/lib/docker/containers/cb1290a496c8d3721064573b27ddd0530c864462374ec0106b5a2cb25fdee916/resolv.conf: read-only file system
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Pulled          Container image &quot;kubernetes/mounttest:0.1&quot; already present on machine
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Created         Created with docker id 82409e45ad6f
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Started         Started with docker id 82409e45ad6f"><pre class="notranslate"><code class="notranslate">$ kubectl describe po/secret-test-pod
Name:                           secret-test-pod
Namespace:                      default
Image(s):                       kubernetes/mounttest:0.1
Node:                           127.0.0.1/127.0.0.1
Start Time:                     Thu, 10 Dec 2015 16:20:49 +0100
Labels:                         &lt;none&gt;
Status:                         Running
Reason: 
Message:
IP:                             10.1.61.8
Replication Controllers:        &lt;none&gt;
Containers:
  test-container:
    Container ID:       docker://82409e45ad6fcd73a8100a1b0767692d5e823cc3a14d8154644293d6b4c86854
    Image:              kubernetes/mounttest:0.1
    Image ID:           docker://b14127f747a0569e9ab96f0ee2de1b07630b7dbc16f8facf0b0adabd5848d41c
    State:              Running
      Started:          Thu, 10 Dec 2015 16:20:51 +0100
    Ready:              False
    Restart Count:      0
    Environment Variables:
Conditions:
  Type          Status
  Ready         False 
Volumes:
  secret-volume:
    Type:       Secret (a secret that should populate this volume)
    SecretName: test-secret
Events:
  FirstSeen     LastSeen        Count   From                    SubobjectPath                           Reason          Message
  ─────────     ────────        ─────   ────                    ─────────────                           ──────          ───────
  9s            9s              1       {scheduler }                                                    Scheduled       Successfully assigned secret-test-pod to 127.0.0.1
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Pulled          Container image "gcr.io/google_containers/pause:0.8.0" already present on machine
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Created         Created with docker id cb1290a496c8
  8s            8s              1       {kubelet 127.0.0.1}     implicitly required container POD       Started         Started with docker id cb1290a496c8
  8s            8s              1       {kubelet 127.0.0.1}                                             FailedSync      Error syncing pod, skipping: open /var/lib/docker/containers/cb1290a496c8d3721064573b27ddd0530c864462374ec0106b5a2cb25fdee916/resolv.conf: read-only file system
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Pulled          Container image "kubernetes/mounttest:0.1" already present on machine
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Created         Created with docker id 82409e45ad6f
  7s            7s              1       {kubelet 127.0.0.1}     spec.containers{test-container}         Started         Started with docker id 82409e45ad6f
</code></pre></div>
<p dir="auto">but the secret directory inside the container is empty:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl logs secret-test-pod
error reading file content for &quot;/etc/secret-volume/data-1&quot;: open /etc/secret-volume/data-1: no such file or directory"><pre class="notranslate"><code class="notranslate">$ kubectl logs secret-test-pod
error reading file content for "/etc/secret-volume/data-1": open /etc/secret-volume/data-1: no such file or directory
</code></pre></div>
<p dir="auto">This is tested on ubuntu 14.04.3 and docker "version 1.9.1, build a34a1d5" with AUFS on ext4.</p>
<p dir="auto">What does the "FailedSync Error" in the status of a pod mean? The kubelet's log is also full of lines like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="W1210 15:25:08.504876    8522 container_manager_linux.go:278] [ContainerManager] Failed to ensure state of &quot;/docker-daemon&quot;: failed to move PID 20479 (in &quot;/user/1000.user/14.session&quot;) to &quot;/docker-daemon&quot;
I1210 15:25:09.337270    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13 with output du: cannot access '/var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13': No such file or directory
 - exit status 1
I1210 15:25:10.939578    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16 with output du: cannot access '/var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16': No such file or directory
 - exit status 1
I1210 15:25:10.940009    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08 with output du: cannot access '/var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08': No such file or directory
 - exit status 1
I1210 15:25:11.010029    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38 with output du: cannot access '/var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38': No such file or directory
 - exit status 1
I1210 15:25:13.054248    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160 with output du: cannot access '/var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160': No such file or directory
 - exit status 1
I1210 15:25:14.771897    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b with output du: cannot access '/var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b': No such file or directory
 - exit status 1
I1210 15:25:16.292393    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476 with output du: cannot access '/var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476': No such file or directory
 - exit status 1"><pre class="notranslate"><code class="notranslate">W1210 15:25:08.504876    8522 container_manager_linux.go:278] [ContainerManager] Failed to ensure state of "/docker-daemon": failed to move PID 20479 (in "/user/1000.user/14.session") to "/docker-daemon"
I1210 15:25:09.337270    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13 with output du: cannot access '/var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13': No such file or directory
 - exit status 1
I1210 15:25:10.939578    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16 with output du: cannot access '/var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16': No such file or directory
 - exit status 1
I1210 15:25:10.940009    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08 with output du: cannot access '/var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08': No such file or directory
 - exit status 1
I1210 15:25:11.010029    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38 with output du: cannot access '/var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38': No such file or directory
 - exit status 1
I1210 15:25:13.054248    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160 with output du: cannot access '/var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160': No such file or directory
 - exit status 1
I1210 15:25:14.771897    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b with output du: cannot access '/var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b': No such file or directory
 - exit status 1
I1210 15:25:16.292393    8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476 with output du: cannot access '/var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476': No such file or directory
 - exit status 1
</code></pre></div>
<p dir="auto">but none of them is "error"</p>
<p dir="auto">When running a containerized kubernetes cluster I cannot spawn pods due to <code class="notranslate">file does not exist</code> errors when trying to mount their secret volumes:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl create -f - &lt;&lt;EOF
apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - &quot;1000000&quot;
EOF
pod &quot;busybox-sleep&quot; created"><pre class="notranslate"><code class="notranslate">$ kubectl create -f - &lt;&lt;EOF
apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
EOF
pod "busybox-sleep" created
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pods
NAME                   READY     STATUS              RESTARTS   AGE
busybox-sleep          0/1       ContainerCreating   0          15s
k8s-master-127.0.0.1   3/3       Running             0          8m"><pre class="notranslate"><code class="notranslate">$ kubectl get pods
NAME                   READY     STATUS              RESTARTS   AGE
busybox-sleep          0/1       ContainerCreating   0          15s
k8s-master-127.0.0.1   3/3       Running             0          8m
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe pod busybox-sleep
Name:               busybox-sleep
Namespace:          default
Image(s):           busybox
Node:               127.0.0.1/127.0.0.1
Start Time:         Fri, 04 Dec 2015 22:48:56 +0000
Labels:             &lt;none&gt;
Status:             Pending
Reason:             
Message:            
IP:             
Replication Controllers:    &lt;none&gt;
Containers:
  busybox:
    Container ID:   
    Image:      busybox
    Image ID:       
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  default-token-6dwl1:
    Type:   Secret (a secret that should populate this volume)
    SecretName: default-token-6dwl1
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Reason      Message
  ─────────   ────────    ───── ────            ───────────── ──────      ───────
  32s       32s     1   {scheduler }                Scheduled   Successfully assigned busybox-sleep to 127.0.0.1
  32s       10s     3   {kubelet 127.0.0.1}         FailedMount Unable to mount volumes for pod &quot;busybox-sleep_default&quot;: IsLikelyNotMountPoint(&quot;/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1&quot;): file does not exist
  32s       10s     3   {kubelet 127.0.0.1}         FailedSync  Error syncing pod, skipping: IsLikelyNotMountPoint(&quot;/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1&quot;): file does not exist"><pre class="notranslate"><code class="notranslate">$ kubectl describe pod busybox-sleep
Name:               busybox-sleep
Namespace:          default
Image(s):           busybox
Node:               127.0.0.1/127.0.0.1
Start Time:         Fri, 04 Dec 2015 22:48:56 +0000
Labels:             &lt;none&gt;
Status:             Pending
Reason:             
Message:            
IP:             
Replication Controllers:    &lt;none&gt;
Containers:
  busybox:
    Container ID:   
    Image:      busybox
    Image ID:       
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  default-token-6dwl1:
    Type:   Secret (a secret that should populate this volume)
    SecretName: default-token-6dwl1
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Reason      Message
  ─────────   ────────    ───── ────            ───────────── ──────      ───────
  32s       32s     1   {scheduler }                Scheduled   Successfully assigned busybox-sleep to 127.0.0.1
  32s       10s     3   {kubelet 127.0.0.1}         FailedMount Unable to mount volumes for pod "busybox-sleep_default": IsLikelyNotMountPoint("/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1"): file does not exist
  32s       10s     3   {kubelet 127.0.0.1}         FailedSync  Error syncing pod, skipping: IsLikelyNotMountPoint("/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1"): file does not exist
</code></pre></div>
<p dir="auto">The secrets are there as expected:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get secrets
NAME                  TYPE                                  DATA      AGE
default-token-6dwl1   kubernetes.io/service-account-token   2         8m"><pre class="notranslate"><code class="notranslate">$ kubectl get secrets
NAME                  TYPE                                  DATA      AGE
default-token-6dwl1   kubernetes.io/service-account-token   2         8m
</code></pre></div>
<p dir="auto">Here are the full kubelet logs: <a href="https://gist.github.com/2opremio/6fcbae55d83f20bc3c82">https://gist.github.com/2opremio/6fcbae55d83f20bc3c82</a></p>
<p dir="auto">The single-node containerized cluster is created using <a href="https://gist.github.com/2opremio/12eafc0c2485d8d13f2a">this script</a>, which is based on <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md">https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md</a> and spawns a cluster based on a recent version of master (<a href="https://github.com/kubernetes/kubernetes/tree/81b6ac47557359429e519d889d9ad48b5c8a2411">https://github.com/kubernetes/kubernetes/tree/81b6ac47557359429e519d889d9ad48b5c8a2411</a>).</p>
<p dir="auto">This is the docker version I am running:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker info
Containers: 7
Images: 62
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: 77RC:MDFB:UNPM:62Z6:FYLF:TZTG:43NZ:77DD:IL22:QI6S:LDSV:UN6W
Debug mode (server): true
File Descriptors: 54
Goroutines: 90
System Time: 2015-12-04T23:01:25.308787463Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox"><pre class="notranslate"><code class="notranslate">$ docker info
Containers: 7
Images: 62
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: 77RC:MDFB:UNPM:62Z6:FYLF:TZTG:43NZ:77DD:IL22:QI6S:LDSV:UN6W
Debug mode (server): true
File Descriptors: 54
Goroutines: 90
System Time: 2015-12-04T23:01:25.308787463Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64"><pre class="notranslate"><code class="notranslate">$ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64
</code></pre></div>
<p dir="auto">The client runs in an OSX laptop and the daemon runs inside a VM provisioned with <code class="notranslate">docker-machine create --driver virtualbox --engine-storage-driver overlay default</code></p>
<p dir="auto">Surprisingly, everything works when using a vagrant VM provisioned with <a href="https://github.com/weaveworks/weave/blob/vagrant-pin-docker/Vagrantfile">https://github.com/weaveworks/weave/blob/vagrant-pin-docker/Vagrantfile</a>, so the problem must somehow be environmental but I haven't managed to triage the difference causing the problem.</p>