title: Unable to start flannel network on worker nodes in Kubernetes
body: <p>When joining worker node to cluster I am getting following error on worker node :</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-js lang-js prettyprint-override"><code>Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.218433   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
  Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.219198   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
  Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220195   43868 conversion.go:110] Could not get instant cpu stats: different number of cpus
  Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.220712   43868 eviction_manager.go:332] eviction manager: attempting to reclaim nodefs
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.220749   43868 helpers.go:1118] eviction manager: attempting to delete unused containers
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.223497   43868 helpers.go:1128] eviction manager: attempting to delete unused images
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.231231   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" to free 274688064 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234577   43868 remote_image.go:130] RemoveImage "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.234646   43868 kuberuntime_image.go:126] Remove image "sha256:b949fa39c923232fc9dc5521d908cc44047d9a951339af5633b057f6a4878dce" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.234666   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" to free 79601765 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238566   43868 remote_image.go:130] RemoveImage "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.238605   43868 kuberuntime_image.go:126] Remove image "sha256:30ee439fbdcf8fdd4bb9b517983bcc386651ccfe20c93e3aaeb3f7f040057b79" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.238624   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" to free 946975195 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251674   43868 remote_image.go:130] RemoveImage "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.251712   43868 kuberuntime_image.go:126] Remove image "sha256:5e89f9aa5754594e83665a15d8ff0faee9c5991f48c143067a53a0e8dcbcef8b" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.251730   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" to free 1076462680 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.266988   43868 remote_image.go:130] RemoveImage "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.267030   43868 kuberuntime_image.go:126] Remove image "sha256:f5bc15a74665edd153f657b9b84fda59b02e3840733abf4d23413e9dc011e202" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.267048   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" to free 1840 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269050   43868 remote_image.go:130] RemoveImage "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.269078   43868 kuberuntime_image.go:126] Remove image "sha256:05a3bd381fc2470695a35f230afefd7bf978b566253199c4ae5cc96fafa29b37" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.269096   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" to free 88610277 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.269742   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-qrl8j_kube-system(a62b5c5b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271892   43868 remote_image.go:130] RemoveImage "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.271934   43868 kuberuntime_image.go:126] Remove image "sha256:637ee73c524b815fcb40b6d4056af8ad92cf4adc1e9ecedbb4f98964fae8c9b1" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.271953   43868 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" to free 965091970 bytes
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274876   43868 remote_image.go:130] RemoveImage "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" from image service failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274910   43868 kuberuntime_image.go:126] Remove image "sha256:7b9d24c898ae4ec4882eb8f60f75b9ae3fb753b5e078ecd7b310df8bfbfb9f11" failed: rpc error: code = Unknown desc = Error response from daemon: reference does not exist
  Nov 07 14:39:58 xyz.com kubelet[43868]: W1107 14:39:58.274962   43868 eviction_manager.go:435] eviction manager: unexpected error when attempting to reduce nodefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]
  Nov 07 14:39:58 xyz.com kubelet[43868]: I1107 14:39:58.274980   43868 eviction_manager.go:346] eviction manager: must evict pod(s) to reclaim nodefs
  Nov 07 14:39:58 xyz.com kubelet[43868]: E1107 14:39:58.274995   43868 eviction_manager.go:357] eviction manager: eviction thresholds have been met, but no pods are active to evict
  Nov 07 14:39:59 xyz.com kubelet[43868]: W1107 14:39:59.467713   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-pnmbb_kube-system(a6e253dc-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.072307   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-f6rws_kube-system(a73e9d11-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:00 xyz.com kubelet[43868]: W1107 14:40:00.670554   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-s2pvk_kube-system(a799778d-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.268440   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-nln6k_kube-system(a7f4fd2b-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:01 xyz.com kubelet[43868]: W1107 14:40:01.868511   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-gvnk4_kube-system(a85090f6-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:02 xyz.com kubelet[43868]: W1107 14:40:02.466371   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-s5th5_kube-system(a8abcf73-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.068779   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-qpjf9_kube-system(a9079eef-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:03 xyz.com kubelet[43868]: W1107 14:40:03.668663   43868 eviction_manager.go:142] Failed to admit pod kube-proxy-gbvk7_kube-system(a963390c-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]
  Nov 07 14:40:04 xyz.com kubelet[43868]: W1107 14:40:04.269624   43868 eviction_manager.go:142] Failed to admit pod kube-flannel-ds-zzxh8_kube-system(a9bee8fb-c39c-11e7-861d-9418820aee50) - node has conditions: [DiskPressure]</code></pre>
  </div>
  </div>
  </p>
  
  <p>Also when I run command <code>kubectl get pods --all-namespaces</code> to get system status. Following things are displayed </p>
  
  <pre><code>NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE
  kube-system   etcd-xyz.com                              1/1       Running   0          17m
  kube-system   kube-apiserver-xyz.com                    1/1       Running   0          16m
  kube-system   kube-controller-manager-xyz.com           1/1       Running   0          16m
  kube-system   kube-dns-545bc4bfd4-nfhxw                 3/3       Running   0          17m
  kube-system   kube-flannel-ds-hwdx8                     0/1       Pending   0          2s
  kube-system   kube-flannel-ds-x8ppr                     1/1       Running   0          16m
  kube-system   kube-proxy-p65z8                          0/1       Pending   0          1s
  kube-system   kube-proxy-t52cr                          1/1       Running   0          17m
  kube-system   kube-scheduler-xyz.com                    1/1       Running   0          16m
  </code></pre>
  
  <p>Here I can see that network initialization pods <code>kube-flannel-ds-hwdx8</code> and <code>kube-proxy-p65z8</code> on worker node are not getting started at all. What might be the issue. Also similar things happen when I use Weave-net Network in cluster instead of flannel.
  Following are the logs I got by running <code>kubectl describe nodes</code></p>
  
  <pre><code>Events:
    Type     Reason                   Age                 From                       Message
    ----     ------                   ----                ----                       -------
    Normal   Starting                 15m                 kubelet, xyz.com  Starting kubelet.
    Normal   NodeHasSufficientDisk    15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasSufficientDisk
    Normal   NodeHasSufficientMemory  15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasSufficientMemory
    Normal   NodeHasNoDiskPressure    15m (x2 over 15m)   kubelet, xyz.com  Node xyz.com status is now: NodeHasNoDiskPressure
    Normal   NodeAllocatableEnforced  15m                 kubelet, xyz.com  Updated Node Allocatable limit across pods
    Normal   NodeHasDiskPressure      15m                 kubelet, xyz.com  Node xyz.com status is now: NodeHasDiskPressure
    Normal   NodeReady                15m                 kubelet, xyz.com  Node xyz.com status is now: NodeReady
    Warning  EvictionThresholdMet     11m (x15 over 15m)  kubelet, xyz.com  Attempting to reclaim nodefs
    Warning  ImageGCFailed            10m                 kubelet, xyz.com  wanted to free 6135907123 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist, rpc error: code = Unknown desc = Error response from daemon: reference does not exist]
  </code></pre>