title: Pods on different nodes can't ping each other
body: <p>I set up 1 master 2 nodes k8s cluster in according to documentation. A pod can ping the other pod on the same node but can't ping the pod on the other node.</p>
  
  <p>To demonstrate the problem I deployed below deployments which has 3 replica. While two of them sits on the same node, the other pod sits on the other node.</p>
  
  <pre>
  
      $ cat nginx.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-deployment
        labels:
          app: nginx
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
            - name: nginx
              image: nginx
              ports:
              - containerPort: 80
      ---
      kind: Service
      apiVersion: v1
      metadata:
        name: nginx-svc
      spec:
        selector:
          app: nginx
        ports:
        - protocol: TCP
          port: 80
  
      $ kubectl get nodes
      NAME                                          STATUS    ROLES     AGE       VERSION
      ip-172-31-21-115.us-west-2.compute.internal   Ready     master    20m       v1.11.2
      ip-172-31-26-62.us-west-2.compute.internal    Ready         19m       v1.11.2
      ip-172-31-29-204.us-west-2.compute.internal   Ready         14m       v1.11.2
  
      $ kubectl get pods -o wide
      NAME                               READY     STATUS    RESTARTS   AGE       IP           NODE                                          NOMINATED NODE
      nginx-deployment-966857787-22qq7   1/1       Running   0          11m       10.244.2.3   ip-172-31-29-204.us-west-2.compute.internal   
      nginx-deployment-966857787-lv7dd   1/1       Running   0          11m       10.244.1.2   ip-172-31-26-62.us-west-2.compute.internal    
      nginx-deployment-966857787-zkzg6   1/1       Running   0          11m       10.244.2.2   ip-172-31-29-204.us-west-2.compute.internal   
  
      $ kubectl get svc
      NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
      kubernetes   ClusterIP   10.96.0.1               443/TCP   21m
      nginx-svc    ClusterIP   10.105.205.10           80/TCP    11m
  
  </pre>
  
  <p>Everything looks fine.</p>
  
  <p>Let me show you containers.</p>
  
  <pre>
  
      # docker exec -it 489b180f512b /bin/bash
      root@nginx-deployment-966857787-zkzg6:/# ifconfig
      eth0: flags=4163  mtu 8951
              inet 10.244.2.2  netmask 255.255.255.0  broadcast 0.0.0.0
              inet6 fe80::cc4d:61ff:fe8a:5aeb  prefixlen 64  scopeid 0x20
  
      root@nginx-deployment-966857787-zkzg6:/# ping 10.244.2.3
      PING 10.244.2.3 (10.244.2.3) 56(84) bytes of data.
      64 bytes from 10.244.2.3: icmp_seq=1 ttl=64 time=0.066 ms
      64 bytes from 10.244.2.3: icmp_seq=2 ttl=64 time=0.055 ms
      ^C
  
  </pre>
  
  <p>So it pings its neighbor pod on the same node.</p>
  
  <pre>
  
      root@nginx-deployment-966857787-zkzg6:/# ping 10.244.1.2
      PING 10.244.1.2 (10.244.1.2) 56(84) bytes of data.
      ^C
      --- 10.244.1.2 ping statistics ---
      2 packets transmitted, 0 received, 100% packet loss, time 1059ms
  
  </pre>
  
  <p>And can't ping its replica on the other node.</p>
  
  <p>Here is host interfaces:</p>
  
  <pre>
  
      # ifconfig
      cni0: flags=4163  mtu 8951
              inet 10.244.2.1  netmask 255.255.255.0  broadcast 0.0.0.0
  
      docker0: flags=4099  mtu 1500
              inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
  
      eth0: flags=4163  mtu 9001
              inet 172.31.29.204  netmask 255.255.240.0  broadcast 172.31.31.255
  
      flannel.1: flags=4163  mtu 8951
              inet 10.244.2.0  netmask 255.255.255.255  broadcast 0.0.0.0
  
      lo: flags=73  mtu 65536
              inet 127.0.0.1  netmask 255.0.0.0
  
      veth09fb984a: flags=4163  mtu 8951
              inet6 fe80::d819:14ff:fe06:174c  prefixlen 64  scopeid 0x20
  
      veth87b3563e: flags=4163  mtu 8951
              inet6 fe80::d09c:d2ff:fe7b:7dd7  prefixlen 64  scopeid 0x20
  
      # ifconfig
      cni0: flags=4163  mtu 8951
              inet 10.244.1.1  netmask 255.255.255.0  broadcast 0.0.0.0
  
      docker0: flags=4099  mtu 1500
              inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
  
      eth0: flags=4163  mtu 9001
              inet 172.31.26.62  netmask 255.255.240.0  broadcast 172.31.31.255
  
      flannel.1: flags=4163  mtu 8951
              inet 10.244.1.0  netmask 255.255.255.255  broadcast 0.0.0.0
  
      lo: flags=73  mtu 65536
              inet 127.0.0.1  netmask 255.0.0.0
  
      veth9733e2e6: flags=4163  mtu 8951
              inet6 fe80::8003:46ff:fee2:abc2  prefixlen 64  scopeid 0x20
  
  </pre>
  
  <p>Processes on the nodes:</p>
  
  <pre>
  
      # ps auxww|grep kube
      root      4059  0.1  2.8  43568 28316 ?        Ssl  00:31   0:01 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
      root      4260  0.0  3.4 358984 34288 ?        Ssl  00:31   0:00 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
      root      4455  1.1  9.6 760868 97260 ?        Ssl  00:31   0:14 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
  
  </pre>
  
  <p>Because of this network problem clusterIP is also unreachable:</p>
  
  <p>$ curl 10.105.205.10:80</p>
  
  <p>Any suggestion?</p>
  
  <p>Thanks.</p>