title: Kube-proxy fails to retrieve node info - invalid nodeIP
body: <p>I've been trying to setup a Kubernetes cluster for a few months now, but I have no luck so far.</p>
  
  <p>I'm trying to set it up on 4 <strong>bare metal</strong> PCs running <strong>coreOS</strong>. I've just clean installed everything again, but I get to the same problem as before. I'm following <a href="https://coreos.com/kubernetes/docs/latest/getting-started.html" rel="nofollow noreferrer">this</a> tutorial. I think I've configured everything correctly, but am not 100% sure. When I reboot any of the machines, kubelet and flanneld services are running, but I see the following errors for them when checking service status with <code>systemctl status</code>:</p>
  
  <p><strong>kubelet error:</strong> <code>Process: 1246 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid (code=exited, status=254)</code></p>
  
  <p><strong>flanneld error</strong>: <code>Process: 1057 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wrapper.uuid (code=exited, status=254)</code></p>
  
  <p>If I restart both services, they work, or at least look like they work - I get no errors.</p>
  
  <p>Everything else seems to work fine, so the only problem (I think) left are the kube-proxy service on all nodes.</p>
  
  <p>If I run <code>kubectl get pods</code> I see all pods running:</p>
  
  <pre><code>$ kubectl get pods
  NAME                                   READY     STATUS    RESTARTS   AGE
  kube-apiserver-kubernetes-4            1/1       Running   4          6m
  kube-controller-manager-kubernetes-4   1/1       Running   6          6m
  kube-proxy-kubernetes-1                1/1       Running   4          18h
  kube-proxy-kubernetes-2                1/1       Running   5          26m
  kube-proxy-kubernetes-3                1/1       Running   4          19m
  kube-proxy-kubernetes-4                1/1       Running   4          18h
  kube-scheduler-kubernetes-4            1/1       Running   6          18h
  </code></pre>
  
  <p><a href="https://stackoverflow.com/questions/43286091/kubernetes-kube-proxy-failed-to-retrieve-node-info">The answer to this question</a> suggest to check if <code>kubectl get node</code> returns same names that are registered on kubelet. As far as I checked the logs, nodes are registered correctly, and this is the output of <code>kubectl get node</code></p>
  
  <pre><code>$ kubectl get node
  NAME           STATUS                        AGE       VERSION
  kubernetes-1   Ready                         18h       v1.6.1+coreos.0
  kubernetes-2   Ready                         36m       v1.6.1+coreos.0
  kubernetes-3   Ready                         29m       v1.6.1+coreos.0
  kubernetes-4   Ready,SchedulingDisabled      18h       v1.6.1+coreos.0
  </code></pre>
  
  <p>The tutorial I've used (linked above) suggest I use <code>--hostname-override</code> but I couldn't get node info on master node (kubernetes-4) If i tried to curl it locally. So I removed it and I can get node info normally now.</p>
  
  <p>Someone suggested it might be a flannel problem and that I should check the flannel ports. Using <code>netstat -lntu</code> I get the following output:</p>
  
  <pre><code>Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State      
  tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN     
  tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN     
  tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN     
  tcp        0      0 MASTER_IP:2379          0.0.0.0:*               LISTEN     
  tcp        0      0 MASTER_IP:2380          0.0.0.0:*               LISTEN     
  tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN     
  tcp6       0      0 :::4194                 :::*                    LISTEN     
  tcp6       0      0 :::10250                :::*                    LISTEN     
  tcp6       0      0 :::10251                :::*                    LISTEN     
  tcp6       0      0 :::10252                :::*                    LISTEN     
  tcp6       0      0 :::10255                :::*                    LISTEN     
  tcp6       0      0 :::22                   :::*                    LISTEN     
  tcp6       0      0 :::443                  :::*                    LISTEN     
  udp        0      0 0.0.0.0:8472            0.0.0.0:*                     
  </code></pre>
  
  <p>So I assume the ports are fine?</p>
  
  <p>Also etcd2 works, etcdctl <code>cluster-health</code> shows that all nodes are healthy  </p>
  
  <p>This is the part of cloud-config that starts the etcd2 on reboot, besides that I only store ssh keys and node username/password/groups in it:</p>
  
  <pre><code>#cloud-config
  
  coreos:
    etcd2:
      name: "kubernetes-4"
      initial-advertise-peer-urls: "http://NODE_IP:2380"
      listen-peer-urls: "http://NODE_IP:2380"
      listen-client-urls: "http://NODE_IP,http://127.0.0.1:2379"
      advertise-client-urls: "http://NODE_IP:2379"
      initial-cluster-token: "etcd-cluster-1"
      initial-cluster: "kubernetes-4=http://MASTER_IP:2380,kubernetes-1=http://WORKER_1_IP:2380,kubernetes-2=http://WORKER_2_IP:2380,kubernetes-3=http://WORKER_3_IP:2380"
      initial-cluster-state: "new"
    units:
      - name: etcd2.service
        command: start
  </code></pre>
  
  <p>This is the content of <code>/etc/flannel/options.env</code> file:</p>
  
  <pre><code>FLANNELD_IFACE=NODE_IP
  FLANNELD_ETCD_ENDPOINTS=http://MASTER_IP:2379,http://WORKER_1_IP:2379,http://WORKER_2_IP:2379,http://WORKER_3_IP:2379
  </code></pre>
  
  <p>The same endpoints are under <code>--etcd-servers</code> in <code>kube-apiserver.yaml</code> file</p>
  
  <p>Any ideas/suggestion what could be the problem? Also if there are some details missing let me know, I'll add them to the post.</p>
  
  <p><strong>Edit:</strong> I forgot to include kube-proxy logs.</p>
  
  <p>Master node kube-proxy log:</p>
  
  <pre><code>$ kubectl logs kube-proxy-kubernetes-4
  I0615 07:47:45.250631       1 server.go:225] Using iptables Proxier.
  W0615 07:47:45.286923       1 server.go:469] Failed to retrieve node info: Get http://127.0.0.1:8080/api/v1/nodes/kubernetes-4: dial tcp 127.0.0.1:8080: getsockopt: connection refused
  W0615 07:47:45.303576       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
  W0615 07:47:45.303593       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
  I0615 07:47:45.303646       1 server.go:249] Tearing down userspace rules.
  E0615 07:47:45.357276       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://127.0.0.1:8080/api/v1/endpoints?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
  E0615 07:47:45.357278       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
  </code></pre>
  
  <p>Worker nodes kube-proxy log:</p>
  
  <pre><code>$ kubectl logs kube-proxy-kubernetes-1
  I0615 07:47:33.667025       1 server.go:225] Using iptables Proxier.
  W0615 07:47:33.697387       1 server.go:469] Failed to retrieve node info: Get https://MASTER_IP/api/v1/nodes/kubernetes-1: dial tcp MASTER_IP:443: getsockopt: connection refused
  W0615 07:47:33.712718       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
  W0615 07:47:33.712734       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
  I0615 07:47:33.712773       1 server.go:249] Tearing down userspace rules.
  E0615 07:47:33.787122       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get https://MASTER_IP/api/v1/endpoints?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused
  E0615 07:47:33.787144       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get https://MASTER_IP/api/v1/services?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused
  </code></pre>