title: Node cannot join Swarm Cluster
body: <p>I have 3 VM's. They all have docker 1.12 and they are running on centos7.
  All the ports are opened and the vm's are able to ping eachother
  I started my cluster with </p>
  
  <pre><code>docker swarm init --advertise-addr 192.168.140.12
  </code></pre>
  
  <p>Docker info showed me:</p>
  
  <pre><code>Swarm: active
   NodeID: 0drcj2nku1mv8t16fxva48edxx
   Is Manager: true
   ClusterID: cchn0yzospwoe1h9f55d7omxx
   Managers: 1
   Nodes: 1
  </code></pre>
  
  <p>Now I try to join nodes (other vms) to the cluster. I use the command which was recommended after starting my manager.</p>
  
  <pre><code>docker swarm join \
       --token SWMTKN-1-48ythur5k6ckkz90ttlprw37p9z3ldclws51qirw5wdyfmvevr-3sb2t66b2fj6e4dhmfo1vavxx \
       192.168.140.12:2377
  </code></pre>
  
  <p>But I got:</p>
  
  <pre><code>Error response from daemon: Timeout was reached before node was joined. Attempt to join the cluster will continue in the background. Use "docker info" command to see the current swarm status of your node.
  </code></pre>
  
  <p>Docker info showed me:</p>
  
  <pre><code>Swarm: pending
   NodeID:
   Error: rpc error: code = 1 desc = context canceled
   Is Manager: false
   Node Address: 192.168.140.14
  </code></pre>
  
  <p>On manager of cluster:</p>
  
  <pre><code># netstat -tulpn | grep docker
  tcp6       0      0 :::2377                 :::*                    LISTEN      1602/dockerd
  tcp6       0      0 :::7946                 :::*                    LISTEN      1602/dockerd
  tcp6       0      0 :::8080                 :::*                    LISTEN      3398/docker-proxy
  tcp6       0      0 :::32768                :::*                    LISTEN      3199/docker-proxy
  tcp6       0      0 :::32769                :::*                    LISTEN      3219/docker-proxy
  tcp6       0      0 :::32770                :::*                    LISTEN      3341/docker-proxy
  tcp6       0      0 :::32771                :::*                    LISTEN      3436/docker-proxy
  tcp6       0      0 :::2375                 :::*                    LISTEN      1602/dockerd
  udp6       0      0 :::7946                 :::*                                1602/dockerd
  </code></pre>
  
  <p>How can I debug this issue or did I forgot to perform some important step? Do the servers need ssh-access to each other? Thanks</p>
  
  <p>logs on node:</p>
  
  <pre><code>Aug  8 09:50:24 localhost dockerd: time="2016-08-08T09:50:24.393432145-04:00" level=error msg="Handler for POST /v1.24/swarm/leave returned error: This node is not part of swarm"
  Aug  8 09:51:01 localhost su: (to root) worker1 on pts/1
  Aug  8 09:51:34 localhost dockerd: time="2016-08-08T09:51:34.384408514-04:00" level=error msg="Handler for POST /v1.24/swarm/join returned error: Timeout was reached before node was joined. Attempt to join the cluster will continue in the background. Use \"docker info\" command to see the current swarm status of your node."
  Aug  8 09:51:40 localhost su: (to root) worker1 on pts/1
  Aug  8 09:52:47 localhost dhclient[1277]: DHCPREQUEST on eno16777736 to 192.168.140.254 port 67 (xid=0x11f8fba8)
  Aug  8 09:52:47 localhost dhclient[1277]: DHCPACK from 192.168.140.254 (xid=0x11f8fba8)
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    address 192.168.140.13
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    plen 24 (255.255.255.0)
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    gateway 192.168.140.2
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    server identifier 192.168.140.254
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    lease time 1800
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    nameserver '192.168.140.2'
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;    domain name 'localdomain'
  Aug  8 09:52:47 localhost NetworkManager[953]: &lt;info&gt;  (eno16777736): DHCPv4 state changed bound -&gt; bound
  Aug  8 09:52:47 localhost dbus[878]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
  Aug  8 09:52:47 localhost dbus-daemon: dbus[878]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
  Aug  8 09:52:47 localhost systemd: Starting Network Manager Script Dispatcher Service...
  Aug  8 09:52:47 localhost dhclient[1277]: bound to 192.168.140.13 -- renewal in 713 seconds.
  Aug  8 09:52:47 localhost dbus[878]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
  Aug  8 09:52:47 localhost dbus-daemon: dbus[878]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
  Aug  8 09:52:47 localhost nm-dispatcher: Dispatching action 'dhcp4-change' for eno16777736
  Aug  8 09:52:47 localhost systemd: Started Network Manager Script Dispatcher Service.
  </code></pre>
  
  <p>Sometimes warnings:</p>
  
  <pre><code>level=warning msg="failed to retrieve remote root CA certificate: rpc error: code = 1 desc = context canceled
  </code></pre>