title: Kubernetes kube-dns Pause container in crashloop with Error adding network: failed to Statfs \"/proc/54226/ns/net\":
body: <p>I have a Kubernetes onebox deployment with the following (containerized) components, all running as <code>--net=host</code>, with kubelet running as a privileged Docker container with the kubernetes flag <code>--allow-privileged</code> set to true.</p>
  
  <pre><code>gcr.io/google_containers/hyperkube-amd64:v1.7.9   "/bin/bash -c './hype"     kubelet
  gcr.io/google_containers/hyperkube-amd64:v1.7.9   "/bin/bash -c './hype"     kube-proxy
  gcr.io/google_containers/hyperkube-amd64:v1.7.9   "/bin/bash -c './hype"     kube-scheduler
  gcr.io/google_containers/hyperkube-amd64:v1.7.9   "/bin/bash -c './hype"     kube-controller-manager
  gcr.io/google_containers/hyperkube-amd64:v1.7.9   "/bin/bash -c './hype"     kube-apiserver
  quay.io/coreos/etcd:v3.1.0                        "/usr/local/bin/etcd "     etcd
  </code></pre>
  
  <p>On top of this, I enabled the addon manager with <code>kubectl create -f https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/resources/manifests/kube-addon-manager.yaml</code>, 
  with the default yaml manifests for calico 2.6.1 and kube-dns 1.14.5 mounted to <code>/etc/kubernetes/addons/</code>. The calico pod comes up with two nodes (install-cni and calico-node) as expected. </p>
  
  <p>However, kube-dns gets stuck in ContainerCreating or ContainerCannotRun, with the following error while trying to start the Kubernetes pause container: </p>
  
  <pre><code>{"log":"I1111 00:35:19.549318       1 manager.go:913] Added container: \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\" (aliases: [k8s_POD_kube-dns-v20-141138543-pmdww_kube-system_3173eef3-c678-11e7-ac4b-e41d2d59689e_0 1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54], namespace: \"docker\")\n","stream":"stderr","time":"2017-11-11T00:35:19.5526284Z"}
  {"log":"I1111 00:35:19.549433       1 cni.go:291] About to add CNI network cni-loopback (type=loopback)\n","stream":"stderr","time":"2017-11-11T00:35:19.5526748Z"}
  {"log":"I1111 00:35:19.549504       1 handler.go:325] Added event \u0026{/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54 2017-11-11 00:35:19.3931718 +0000 UTC containerCreation {\u003cnil\u003e}}\n","stream":"stderr","time":"2017-11-11T00:35:19.5527217Z"}
  {"log":"I1111 00:35:19.551134       1 container.go:407] Start housekeeping for container \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\"\n","stream":"stderr","time":"2017-11-11T00:35:19.5527441Z"}
  {"log":"E1111 00:35:19.555099       1 cni.go:294] Error adding network: failed to Statfs \"/proc/54226/ns/net\": no such file or directory\n","stream":"stderr","time":"2017-11-11T00:35:19.5553606Z"}
  {"log":"E1111 00:35:19.555122       1 cni.go:237] Error while adding to cni lo network: failed to Statfs \"/proc/54226/ns/net\": no such file or directory\n","stream":"stderr","time":"2017-11-11T00:35:19.5553887Z"}
  {"log":"I1111 00:35:19.600281       1 manager.go:970] Destroyed container: \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\" (aliases: [k8s_POD_kube-dns-v20-141138543-pmdww_kube-system_3173eef3-c678-11e7-ac4b-e41d2d59689e_0 1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54], namespace: \"docker\")\n","stream":"stderr","time":"2017-11-11T00:35:19.6005722Z"}
  </code></pre>
  
  <p>I see \pause containers keep coming up just to exit a second later, with an innocuous error message (this one is old, I stopped the cluster so it wouldn't keep spawning more containers):</p>
  
  <pre><code>ubuntu@r172-16-6-39:~$ docker ps -a | grep 216e39defa36
  216e39defa36        gcr.io/google_containers/pause-amd64:3.0          "/pause"                 About an hour ago   Exited (0) About an hour ago                         k8s_POD_kube-dns-v20-141138543-xvdmv_kube-system_0594732f-c688-11e7-9da5-e41d2d59689e_17
  ubuntu@r172-16-6-39:~$ docker logs 216e39defa36
  shutting down, got signal: Terminated
  </code></pre>
  
  <p>The dir <code>/proc/54226</code> doesn't exist on my host, which I assume is why CNI is complaining. But the pause containers for Calico are fine, running the same image, so something must be either failing to write only in the case of kube-dns, or not trying to write in the case of Calico. I found some references to a similar <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1414963" rel="nofollow noreferrer">SELinux-related error on Openshift</a>, but I'm running a bare Ubuntu 14.04 VM without SELinux even installed. </p>
  
  <pre><code>ubuntu@r172-16-6-39:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 14.04.4 LTS
  Release:        14.04
  Codename:       trusty
  ubuntu@r172-16-6-39:~$ setenforce
  The program 'setenforce' is currently not installed. You can install it by typing:
  sudo apt-get install selinux-utils
  </code></pre>
  
  <p>My CNI conf is also pretty simple, generated by the install-cni calico container:</p>
  
  <pre><code>ubuntu@r172-16-6-39:~$ cat /etc/cni/net.d/10-calico.conf
  {
      "name": "k8s-pod-network",
      "cniVersion": "0.1.0",
      "type": "calico",
      "log_level": "debug",
      "datastore_type": "kubernetes",
      "nodename": "172.16.6.39",
      "mtu": 1500,
      "ipam": {
          "type": "host-local",
          "subnet": "usePodCidr"
      },
      "policy": {
          "type": "k8s",
          "k8s_auth_token": "****"
      },
      "kubernetes": {
          "k8s_api_root": "https://168.16.0.1:443",
          "kubeconfig": "/etc/kubernetes/kubeconfig"
      }
  }
  </code></pre>
  
  <p>Has anyone hit something similar?</p>