title: Kube flannel in CrashLoopBackOff status
body: <p>We just start to create our cluster on kubernetes.</p>
  
  <p>Now we try to deploy tiller but we have en error:</p>
  
  <blockquote>
    <p>NetworkPlugin cni failed to set up pod
    "tiller-deploy-64c9d747bd-br9j7_kube-system" network: open
    /run/flannel/subnet.env: no such file or directory</p>
  </blockquote>
  
  <p>After that I call:</p>
  
  <pre><code>kubectl get pods --all-namespaces -o wide
  </code></pre>
  
  <p>And got response:</p>
  
  <pre><code>NAMESPACE     NAME                                   READY     STATUS              RESTARTS   AGE       IP              NODE          NOMINATED NODE
  kube-system   coredns-78fcdf6894-ksdvt               1/1       Running             2          7d        192.168.0.4     kube-master   &lt;none&gt;
  kube-system   coredns-78fcdf6894-p4l9q               1/1       Running             2          7d        192.168.0.5     kube-master   &lt;none&gt;
  kube-system   etcd-kube-master                       1/1       Running             2          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kube-apiserver-kube-master             1/1       Running             2          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kube-controller-manager-kube-master    1/1       Running             2          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-42rl7            0/1       CrashLoopBackOff    2135       7d        10.168.209.17   node5         &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-5fx2p            0/1       CrashLoopBackOff    2164       7d        10.168.209.14   node2         &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-6bw5g            0/1       CrashLoopBackOff    2166       7d        10.168.209.15   node3         &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-hm826            1/1       Running             1          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-thjps            0/1       CrashLoopBackOff    2160       7d        10.168.209.16   node4         &lt;none&gt;
  kube-system   kube-flannel-ds-amd64-w99ch            0/1       CrashLoopBackOff    2166       7d        10.168.209.13   node1         &lt;none&gt;
  kube-system   kube-proxy-d6v2n                       1/1       Running             0          7d        10.168.209.13   node1         &lt;none&gt;
  kube-system   kube-proxy-lcckg                       1/1       Running             0          7d        10.168.209.16   node4         &lt;none&gt;
  kube-system   kube-proxy-pgblx                       1/1       Running             1          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kube-proxy-rnqq5                       1/1       Running             0          7d        10.168.209.14   node2         &lt;none&gt;
  kube-system   kube-proxy-wc959                       1/1       Running             0          7d        10.168.209.15   node3         &lt;none&gt;
  kube-system   kube-proxy-wfqqs                       1/1       Running             0          7d        10.168.209.17   node5         &lt;none&gt;
  kube-system   kube-scheduler-kube-master             1/1       Running             2          7d        10.168.209.20   kube-master   &lt;none&gt;
  kube-system   kubernetes-dashboard-6948bdb78-97qcq   0/1       ContainerCreating   0          7d        &lt;none&gt;          node5         &lt;none&gt;
  kube-system   tiller-deploy-64c9d747bd-br9j7         0/1       ContainerCreating   0          45m       &lt;none&gt;          node4         &lt;none&gt;
  </code></pre>
  
  <p>We have some flannel pods in CrashLoopBackOff status. For example <code>kube-flannel-ds-amd64-42rl7</code>.</p>
  
  <p>When I call:</p>
  
  <pre><code>kubectl describe pod -n kube-system kube-flannel-ds-amd64-42rl7
  </code></pre>
  
  <p>I've got status <code>Running</code>:</p>
  
  <pre><code>Name:               kube-flannel-ds-amd64-42rl7
  Namespace:          kube-system
  Priority:           0
  PriorityClassName:  &lt;none&gt;
  Node:               node5/10.168.209.17
  Start Time:         Wed, 22 Aug 2018 16:47:10 +0300
  Labels:             app=flannel
                      controller-revision-hash=911701653
                      pod-template-generation=1
                      tier=node
  Annotations:        &lt;none&gt;
  Status:             Running
  IP:                 10.168.209.17
  Controlled By:      DaemonSet/kube-flannel-ds-amd64
  Init Containers:
    install-cni:
      Container ID:  docker://eb7ee47459a54d401969b1770ff45b39dc5768b0627eec79e189249790270169
      Image:         quay.io/coreos/flannel:v0.10.0-amd64
      Image ID:      docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Command:
        cp
      Args:
        -f
        /etc/kube-flannel/cni-conf.json
        /etc/cni/net.d/10-flannel.conflist
      State:          Terminated
        Reason:       Completed
        Exit Code:    0
        Started:      Wed, 22 Aug 2018 16:47:24 +0300
        Finished:     Wed, 22 Aug 2018 16:47:24 +0300
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /etc/cni/net.d from cni (rw)
        /etc/kube-flannel/ from flannel-cfg (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)
  Containers:
    kube-flannel:
      Container ID:  docker://521b457c648baf10f01e26dd867b8628c0f0a0cc0ea416731de658e67628d54e
      Image:         quay.io/coreos/flannel:v0.10.0-amd64
      Image ID:      docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Command:
        /opt/bin/flanneld
      Args:
        --ip-masq
        --kube-subnet-mgr
      State:          Waiting
        Reason:       CrashLoopBackOff
      Last State:     Terminated
        Reason:       Error
        Exit Code:    1
        Started:      Thu, 30 Aug 2018 10:15:04 +0300
        Finished:     Thu, 30 Aug 2018 10:15:08 +0300
      Ready:          False
      Restart Count:  2136
      Limits:
        cpu:     100m
        memory:  50Mi
      Requests:
        cpu:     100m
        memory:  50Mi
      Environment:
        POD_NAME:       kube-flannel-ds-amd64-42rl7 (v1:metadata.name)
        POD_NAMESPACE:  kube-system (v1:metadata.namespace)
      Mounts:
        /etc/kube-flannel/ from flannel-cfg (rw)
        /run from run (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:
    run:
      Type:          HostPath (bare host directory volume)
      Path:          /run
      HostPathType:
    cni:
      Type:          HostPath (bare host directory volume)
      Path:          /etc/cni/net.d
      HostPathType:
    flannel-cfg:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      kube-flannel-cfg
      Optional:  false
    flannel-token-9wmch:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  flannel-token-9wmch
      Optional:    false
  QoS Class:       Guaranteed
  Node-Selectors:  beta.kubernetes.io/arch=amd64
  Tolerations:     node-role.kubernetes.io/master:NoSchedule
                   node.kubernetes.io/disk-pressure:NoSchedule
                   node.kubernetes.io/memory-pressure:NoSchedule
                   node.kubernetes.io/not-ready:NoExecute
                   node.kubernetes.io/unreachable:NoExecute
  Events:
    Type     Reason   Age                  From            Message
    ----     ------   ----                 ----            -------
    Normal   Pulled   51m (x2128 over 7d)  kubelet, node5  Container image "quay.io/coreos/flannel:v0.10.0-amd64" already present on machine
    Warning  BackOff  1m (x48936 over 7d)  kubelet, node5  Back-off restarting failed container
  </code></pre>
  
  <p>here <code>kube-controller-manager.yaml</code>:</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      scheduler.alpha.kubernetes.io/critical-pod: ""
    creationTimestamp: null
    labels:
      component: kube-controller-manager
      tier: control-plane
    name: kube-controller-manager
    namespace: kube-system
  spec:
    containers:
    - command:
      - kube-controller-manager
      - --address=127.0.0.1
      - --allocate-node-cidrs=true
      - --cluster-cidr=192.168.0.0/24
      - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
      - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
      - --controllers=*,bootstrapsigner,tokencleaner
      - --kubeconfig=/etc/kubernetes/controller-manager.conf
      - --leader-elect=true
      - --node-cidr-mask-size=24
      - --root-ca-file=/etc/kubernetes/pki/ca.crt
      - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
      - --use-service-account-credentials=true
      image: k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 8
        httpGet:
          host: 127.0.0.1
          path: /healthz
          port: 10252
          scheme: HTTP
        initialDelaySeconds: 15
        timeoutSeconds: 15
      name: kube-controller-manager
      resources:
        requests:
          cpu: 200m
      volumeMounts:
      - mountPath: /etc/ssl/certs
        name: ca-certs
        readOnly: true
      - mountPath: /etc/kubernetes/controller-manager.conf
        name: kubeconfig
        readOnly: true
      - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
        name: flexvolume-dir
      - mountPath: /etc/pki
        name: etc-pki
        readOnly: true
      - mountPath: /etc/kubernetes/pki
        name: k8s-certs
        readOnly: true
    hostNetwork: true
    priorityClassName: system-cluster-critical
    volumes:
    - hostPath:
        path: /etc/ssl/certs
        type: DirectoryOrCreate
      name: ca-certs
    - hostPath:
        path: /etc/kubernetes/controller-manager.conf
        type: FileOrCreate
      name: kubeconfig
    - hostPath:
        path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
        type: DirectoryOrCreate
      name: flexvolume-dir
    - hostPath:
        path: /etc/pki
        type: DirectoryOrCreate
      name: etc-pki
    - hostPath:
        path: /etc/kubernetes/pki
        type: DirectoryOrCreate
      name: k8s-certs
  status: {}
  </code></pre>
  
  <p>OS is CentOS Linux release 7.5.1804</p>
  
  <p>logs from one of pods:</p>
  
  <pre><code># kubectl logs --namespace kube-system kube-flannel-ds-amd64-5fx2p
  
  main.go:475] Determining IP address of default interface
  main.go:488] Using interface with name eth0 and address 10.168.209.14
  main.go:505] Defaulting external address to interface address (10.168.209.14)
  kube.go:131] Waiting 10m0s for node controller to sync
  kube.go:294] Starting kube subnet manager
  kube.go:138] Node controller sync successful
  main.go:235] Created subnet manager: Kubernetes Subnet Manager - node2
  main.go:238] Installing signal handlers
  main.go:353] Found network config - Backend type: vxlan
  vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
  main.go:280] Error registering network: failed to acquire lease: node "node2" pod cidr not assigned
  main.go:333] Stopping shutdownHandler...
  </code></pre>
  
  <p>Where error is?</p>