title: Docker Swarm load balancing or mesh routing issue
body: <p>I have a single docker node configured as a docker swarm.</p>
  
  <p>I have created an overlay network called main_net which uses encryption.</p>
  
  <p>I have mutiple containers connected to this network. One of which is called kong and runs kong. I have a service called saas_thumbsum_0_0_13 - this has a single container running inside it. </p>
  
  <p>Kong is configured as a reverse proxy for saas_thumbsum_0_0_13, and uses the address saas_thumbsum_0_0_13. This setup was working this morning when I had a service called saas_thumbsum_0_0_12 running, Kong was sending requests to <a href="http://saas_thumbsum_0_0_12:80/public" rel="nofollow noreferrer">http://saas_thumbsum_0_0_12:80/public</a> without a problem.</p>
  
  <p>I have now created another service 0_0_13 and connected it to the network but kong can not access <a href="http://saas_thumbsum_0_0_13:80/public" rel="nofollow noreferrer">http://saas_thumbsum_0_0_13:80/public</a></p>
  
  <p>I have logged into the console of the kong container and ran the following commands:</p>
  
  <pre><code>/ # nslookup saas_thumbsum_0_0_13
  nslookup: can't resolve '(null)': Name does not resolve
  
  Name:      saas_thumbsum_0_0_13
  Address 1: 10.0.0.39
  / # nslookup tasks.saas_thumbsum_0_0_13
  nslookup: can't resolve '(null)': Name does not resolve
  
  Name:      tasks.saas_thumbsum_0_0_13
  Address 1: 10.0.0.40 d4c3034462d1.main_net
  </code></pre>
  
  <p>so the service resolves to 10.0.0.39 which I guess is the load balanced ip, and the service has one task with an ip of 10.0.0.40. Seems ok so far.
  Now I try and ping the two ip's from inside the kong container:</p>
  
  <pre><code>/ # ping 10.0.0.39
  PING 10.0.0.39 (10.0.0.39): 56 data bytes
  64 bytes from 10.0.0.39: seq=0 ttl=64 time=0.089 ms
  64 bytes from 10.0.0.39: seq=1 ttl=64 time=0.083 ms
  ^C
  --- 10.0.0.39 ping statistics ---
  2 packets transmitted, 2 packets received, 0% packet loss
  round-trip min/avg/max = 0.083/0.086/0.089 ms
  / # ping 10.0.0.40
  PING 10.0.0.40 (10.0.0.40): 56 data bytes
  64 bytes from 10.0.0.40: seq=0 ttl=64 time=0.146 ms
  64 bytes from 10.0.0.40: seq=1 ttl=64 time=0.091 ms
  ^C
  --- 10.0.0.40 ping statistics ---
  2 packets transmitted, 2 packets received, 0% packet loss
  round-trip min/avg/max = 0.091/0.118/0.146 ms
  </code></pre>
  
  <p>I can successfully ping both IP addresses. This seems fine to me as from my beginner level knowldege they should load balance to the same endpoint and the results should not differ.</p>
  
  <p>Now if I try and get the webpage from the load balanced ip it fails:</p>
  
  <pre><code>/ # wget 10.0.0.39/public/web/frontend
  Connecting to 10.0.0.39 (10.0.0.39:80)
  wget: can't connect to remote host (10.0.0.39): Connection refused
  </code></pre>
  
  <p>Same if I use the service name</p>
  
  <pre><code>/ # wget saas_thumbsum_0_0_13/public/web/frontend
  Connecting to saas_thumbsum_0_0_13 (10.0.0.39:80)
  wget: can't connect to remote host (10.0.0.39): Connection refused
  </code></pre>
  
  <p>same with and without http://</p>
  
  <p>but it works if I go to the task ip:</p>
  
  <pre><code>/ # wget 10.0.0.40/public/web/frontend
  Connecting to 10.0.0.40 (10.0.0.40:80)
  Connecting to 10.0.0.40 (10.0.0.40:80)
  frontend             100% |*******************************************************|  1769   0:00:00 ETA
  </code></pre>
  
  <p>But I should be using the service endpoint not the task endpoint!</p>
  
  <p>If the address is load balanced then they both should go to the same target so the result should be the same.</p>
  
  <p>The target container is running nginx and I have checked the config files there but I don't see any kind of source ip restriction. Not sure how I would do this anyway because the ip addresses would be different every time anyway!</p>
  
  <p>So this is broken, wget saas_thumbsum_0_0_13/public/web/frontend should be giving me the page but it is not working.</p>
  
  <p>If anyone can spot my mistake or give me any debugging pointers it would be appreciated.</p>
  
  <p>Some other info that might be helpful:</p>
  
  <pre><code>robert@metcaac4:~$ docker --version
  Docker version 18.09.3, build 774a1f4
  robert@metcaac4:~$ docker network ls
  NETWORK ID          NAME                DRIVER              SCOPE
  ac295a4f901c        bridge              bridge              local
  5aa2eea87b33        docker_gwbridge     bridge              local
  3744b0f39195        host                host                local
  lffazc510t7z        ingress             overlay             swarm
  r8z2bb9idtwt        main_net            overlay             swarm
  0ad4a8efa4cc        none                null                local
  
  robert@metcaac4:~$ docker network inspect main_net
  [
      {
          "Name": "main_net",
          "Id": "r8z2bb9idtwtyfr20e6u1p2f8",
          "Created": "2019-03-07T20:24:38.553471996Z",
          "Scope": "swarm",
          "Driver": "overlay",
          "EnableIPv6": false,
          "IPAM": {
              "Driver": "default",
              "Options": null,
              "Config": [
                  {
                      "Subnet": "10.0.0.0/24",
                      "Gateway": "10.0.0.1"
                  }
              ]
          },
          "Internal": false,
          "Attachable": true,
          "Ingress": false,
          "ConfigFrom": {
              "Network": ""
          },
          "ConfigOnly": false,
          "Containers": {
              "2b3b0d5311f23b084a2aa5282f451acb2c6d636b5ceb11926a46ee0a62e3035a": {
                  "Name": "code_site2_code_site2.1.bqcsxy73pczsurit07lxmmoru",
                  "EndpointID": "c9974c99e83338aa07f17eda56342757bce6a247105eb8fa4afab1b71f8d0366",
                  "MacAddress": "02:42:0a:00:00:b7",
                  "IPv4Address": "10.0.0.183/24",
                  "IPv6Address": ""
              },
              "2ed56f6b2b06cbdeed62037cf10e40683b2960b2d53866e22a4080fa7a50a563": {
                  "Name": "saas_user_management_0_0_66.1.0092odmxl4er803jhcrad7xhb",
                  "EndpointID": "90f6cc3368c1fd0e5ff0be990f7cd0b1a854584eea3cdac3a80c0d535aa94c43",
                  "MacAddress": "02:42:0a:00:00:f5",
                  "IPv4Address": "10.0.0.245/24",
                  "IPv6Address": ""
              },
              "7059b7eb266a6ad159fab1606b176c23d70f91841fb2635e5e163afa44f78e32": {
                  "Name": "kong_kong.1.dr5afpgi5me23p96kub2qmjaq",
                  "EndpointID": "0bfc5ece35f5278596b05232cbc5f88deef9c9a6d132d6f54b82b4c7c7e4d4fe",
                  "MacAddress": "02:42:0a:00:00:b8",
                  "IPv4Address": "10.0.0.184/24",
                  "IPv6Address": ""
              },
              "7e0153b62eab821019b8f0757c842de367a61025e2848c0548c3cb7bd28d2df0": {
                  "Name": "code_site2_code_site2_nginx.1.ijxkoaxzrly1b0dkt2ysdot3r",
                  "EndpointID": "d776003ee4a98acd9cb1df2d3fe2861f4c8aca9c655a01dac9b371aebc4a6b2f",
                  "MacAddress": "02:42:0a:00:00:27",
                  "IPv4Address": "10.0.0.39/24",
                  "IPv6Address": ""
              },
              "9cebbcf75312ce906786d722c38c30c85a84ec5b4efbe46656388f49fa50f942": {
                  "Name": "saas_user_management_0_0_50.1.9xkwovfqejqerzi5co2mfb5l4",
                  "EndpointID": "28cfcbd20723b272f787f1ecea7603b65f1bd48df8dcb89474087d3f5ae77ed7",
                  "MacAddress": "02:42:0a:00:00:21",
                  "IPv4Address": "10.0.0.33/24",
                  "IPv6Address": ""
              },
              "a44f86449e19ca3a6fc170c3ed997156f0d3db66fece23883c1a3f844714e504": {
                  "Name": "kong_dbb.1.l4d1xzqtr4t0thjnf39twfo0j",
                  "EndpointID": "a410b8db987fd7eb261865c27dc84905bdfab273ccf20906c7b5a1ba0cbd1ab8",
                  "MacAddress": "02:42:0a:00:00:ad",
                  "IPv4Address": "10.0.0.173/24",
                  "IPv6Address": ""
              },
              "d4c3034462d106496da5df3449d1da26dd50bcf40e4b954aea0b64a33a88d954": {
                  "Name": "saas_thumbsum_0_0_13.1.of39j50gs0d5376nqgtsszx7m",
                  "EndpointID": "a10e51cc60c559f9631a0fb883ca00a654bd1c815cc0a4730222be27847a802f",
                  "MacAddress": "02:42:0a:00:00:28",
                  "IPv4Address": "10.0.0.40/24",
                  "IPv6Address": ""
              },
              "deda51bfbee097093ccb3e5be7ea11aa0648bdc74381f1552fe102b8646343cf": {
                  "Name": "dockerregistry_dockerregistry.1.rty0eoy404m870f6qkpla9c9z",
                  "EndpointID": "ff64d38ae4f932b7567cb0b5417df0cb8eef12e46572636a40d59da37511eff9",
                  "MacAddress": "02:42:0a:00:00:ae",
                  "IPv4Address": "10.0.0.174/24",
                  "IPv6Address": ""
              },
              "e117e90a37cacbdbfc2768e2c77512b1c967e268718884f0372d63c1f6b79cbd": {
                  "Name": "code_site2_code_site2_nginx.1.dihdw4g5n8i20pbb7nfwwi6ud",
                  "EndpointID": "620d3c5de8169a380cb3c2e2caa91e18c91a1d4cf244900e5ea4912941e9c4c8",
                  "MacAddress": "02:42:0a:00:00:ac",
                  "IPv4Address": "10.0.0.172/24",
                  "IPv6Address": ""
              },
              "lb-main_net": {
                  "Name": "main_net-endpoint",
                  "EndpointID": "d4b3747a3701b05f716f47295132939bed6f68c566fbf5ab50b70071e2d5cd0d",
                  "MacAddress": "02:42:0a:00:00:02",
                  "IPv4Address": "10.0.0.2/24",
                  "IPv6Address": ""
              }
          },
          "Options": {
              "com.docker.network.driver.overlay.vxlanid_list": "4097",
              "encrypted": ""
          },
          "Labels": {},
          "Peers": [
              {
                  "Name": "1a2559b7bdb0",
                  "IP": "78.31.105.225"
              }
          ]
      }
  ]
  
  
  
  robert@metcaac4:~$ docker service ls
  ID                  NAME                            MODE                REPLICAS            IMAGE                                         PORTS
  miqimzjmh0qe        code_site2_code_site2           replicated          1/1                 metcarob/phpfpm_for_drupal:0.0.6              
  bgixg86fhaqs        code_site2_code_site2_nginx     replicated          1/1                 nginx:1.13.0-alpine                           
  0d2267ul4iv4        dockerregistry_dockerregistry   replicated          1/1                 registry:2                                    *:5000-&gt;5000/tcp
  4tn5xqbss3i5        kong_dbb                        replicated          1/1                 postgres:9.6                                  
  u9xo3prf53v2        kong_kong                       replicated          1/1                 kong:1.1.2                                    *:80-&gt;8000/tcp, *:443-&gt;8443/tcp
  kgmhthn93qwy        saas_thumbsum_0_0_13            replicated          1/1                 metcarob/saas_thumbsum:0.0.13                 
  zhyna2ktbgw4        saas_user_management_0_0_66     replicated          1/1                 metcarob/saas_user_management_system:0.0.66
  </code></pre>
  
  <p>I have tried removing and recreating the saas_thumbsum service but this made no difference. (Only altered the ip's in question)</p>
  
  <p>I can supply the saas_thumbsum_0_0_13 nginx config but this hasn't changed since the working version and I don't think this is the problem.</p>
  
  <p>Thanks for any help!</p>
  
  <h2>Update 001 - Extra information</h2>
  
  <p>I have exactly the same setup with my saas_user_management_0_0_66 service. This is working without a problem.</p>
  
  <pre><code>robert@metcaac4:~$ docker exec -it kong_kong.1.dr5afpgi5me23p96kub2qmjaq /bin/sh
  / # nslookup saas_user_management_system_0_0_66
  nslookup: can't resolve '(null)': Name does not resolve
  
  nslookup: can't resolve 'saas_user_management_system_0_0_66': Name does not resolve
  / # nslookup saas_user_management_0_0_66
  nslookup: can't resolve '(null)': Name does not resolve
  
  Name:      saas_user_management_0_0_66
  Address 1: 10.0.0.244
  / # nslookup tasks.saas_user_management_0_0_66
  nslookup: can't resolve '(null)': Name does not resolve
  
  Name:      tasks.saas_user_management_0_0_66
  Address 1: 10.0.0.245 saas_user_management_0_0_66.1.0092odmxl4er803jhcrad7xhb.main_net
  / # pint 10.0.0.244
  /bin/sh: pint: not found
  / # ping 10.0.0.244
  PING 10.0.0.244 (10.0.0.244): 56 data bytes
  64 bytes from 10.0.0.244: seq=0 ttl=64 time=0.082 ms
  64 bytes from 10.0.0.244: seq=1 ttl=64 time=0.113 ms
  ^C
  --- 10.0.0.244 ping statistics ---
  2 packets transmitted, 2 packets received, 0% packet loss
  round-trip min/avg/max = 0.082/0.097/0.113 ms
  / # ping 10.0.0.245
  PING 10.0.0.245 (10.0.0.245): 56 data bytes
  64 bytes from 10.0.0.245: seq=0 ttl=64 time=0.177 ms
  64 bytes from 10.0.0.245: seq=1 ttl=64 time=0.112 ms
  ^C
  --- 10.0.0.245 ping statistics ---
  2 packets transmitted, 2 packets received, 0% packet loss
  round-trip min/avg/max = 0.112/0.144/0.177 ms
  / # wget 10.0.0.245
  Connecting to 10.0.0.245 (10.0.0.245:80)
  wget: server returned error: HTTP/1.1 404 Not Found
  / # wget 10.0.0.244
  Connecting to 10.0.0.244 (10.0.0.244:80)
  wget: server returned error: HTTP/1.1 404 Not Found
  </code></pre>
  
  <p>In the above wget to both 10.0.0.245 and 10.0.0.244 give exactly the same 404 not found response from the server.I have no idea why saas_thumbsum gets connection refused.</p>