title: Recovering a docker swarm cluster
body: <p>We have several docker swarm clusters in production. Each of them has 4-10 nodes. We have 3-5 <code>manager</code> nodes in each of them as per the recommendation to meet raft quorum.  These are hosted in EC2 instances in different AWS regions.</p>
  <p>Our cluster became unstable due to a recent outage in one of the AWS AZs and since then I'm running this production cluster with just one <code>manager</code> which is not recommended.  My attempt to promote other nodes succeeds, but in a few seconds, I lose the cluster and I see the message <code>The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.</code>  Only way to recover the cluster is to reinitialize swarm cluster again on the manager node.  For eg.  <code>swarm init --force-new-cluster</code>.</p>
  <p>Though the cluster continues to work after this, I see inconsistent node list.  The node I attempted to add remain <code>down</code> and I must forcefully leave the swarm and remove the node from manager cluster.  Attempt to remove this node from cluster results in the message <code>the node you are attempting to remove is a manager</code>. <code>docker node inspect &lt;nodeid&gt;</code> shows its <code>Role</code> as <code>manager</code> but any cluster level operations on the node fails stating that the node is not a manager.</p>
  <p>I can't afford to lose this cluster and recreate as there are live customers who expect high uptime.   It's risky leaving the setup as I don't have sufficient <code>manager</code> nodes and any operation related to swarm cluster is leaving the cluster in an inconsistent state.</p>
  <p><code>dockerd </code> logs in <code>/var/log/syslog</code> is not very helpful in identifying the root cause. Following is the log snippet I found when I promote another node as <code>manager</code></p>
  <pre><code> dockerd[24061]: time=&quot;2021-03-03T05:43:12.576569707Z&quot; level=info msg=&quot;dispatcher stopping&quot; method=&quot;(*Dispatcher).Stop&quot; module=dispatcher node.id=t2x1wbrgq616fsx6v7ay2euqn
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.576746632Z&quot; level=info msg=&quot;dispatcher session dropped, marking node ig3w333ffmkdbff9hz1g58d0f down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=ig3w333ffmkdbff9hz1g58d0f node.session=c19degio4wtlcfm2xwkjsr7oq
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.576772083Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=ig3w333ffmkdbff9hz1g58d0f node.session=c19degio4wtlcfm2xwkjsr7oq
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.576858405Z&quot; level=info msg=&quot;dispatcher session dropped, marking node 3zgtdu6meh3ixf8h7vtwpo2ic down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=3zgtdu6meh3ixf8h7vtwpo2ic node.session=1yqzxseeumz6m8cqy4bfpqq9f
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.576878476Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=3zgtdu6meh3ixf8h7vtwpo2ic node.session=1yqzxseeumz6m8cqy4bfpqq9f
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.576986159Z&quot; level=info msg=&quot;dispatcher session dropped, marking node bk5mapetwtri0gax5o8zc29k2 down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=bk5mapetwtri0gax5o8zc29k2 node.session=9ae2u0v2dlyyo8do3tv7e4b8m
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577008040Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=bk5mapetwtri0gax5o8zc29k2 node.session=9ae2u0v2dlyyo8do3tv7e4b8m
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577097253Z&quot; level=info msg=&quot;dispatcher session dropped, marking node znbimzdsduwzzfggl1uno3n9c down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=znbimzdsduwzzfggl1uno3n9c node.session=xz5xvsnc1tr0eb0p5ytd8eu5s
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577127094Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=znbimzdsduwzzfggl1uno3n9c node.session=xz5xvsnc1tr0eb0p5ytd8eu5s
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577268958Z&quot; level=info msg=&quot;dispatcher session dropped, marking node c6fnnzlrcb59ob0mcpwaf52f7 down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=c6fnnzlrcb59ob0mcpwaf52f7 node.session=mt8sd2646xgk9lw549ydri43a
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577326000Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=c6fnnzlrcb59ob0mcpwaf52f7 node.session=mt8sd2646xgk9lw549ydri43a
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577396992Z&quot; level=info msg=&quot;dispatcher session dropped, marking node p6r18tf30q6boy5woygvnfnrv down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=p6r18tf30q6boy5woygvnfnrv node.session=680gvutmvvyf14sy6yjc3sf00
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577426743Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=p6r18tf30q6boy5woygvnfnrv node.session=680gvutmvvyf14sy6yjc3sf00
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577309389Z&quot; level=info msg=&quot;dispatcher session dropped, marking node t2x1wbrgq616fsx6v7ay2euqn down&quot; method=&quot;(*Dispatcher).Session&quot; node.id=t2x1wbrgq616fsx6v7ay2euqn node.session=dii2khkdxlqvs2ckao7hhin0u
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.577502235Z&quot; level=error msg=&quot;failed to remove node&quot; error=&quot;rpc error: code = Aborted desc = dispatcher is stopped&quot; method=&quot;(*Dispatcher).Session&quot; node.id=t2x1wbrgq616fsx6v7ay2euqn node.session=dii2khkdxlqvs2ckao7hhin0u
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.578248017Z&quot; level=info msg=&quot;leadership changed from not yet part of a raft cluster to no cluster leader&quot; module=node node.id=t2x1wbrgq616fsx6v7ay2euqn
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.578520436Z&quot; level=error msg=&quot;agent: session failed&quot; backoff=100ms error=&quot;rpc error: code = Aborted desc = node must disconnect&quot; module=node/agent node.id=t2x1wbrgq616fsx6v7ay2euqn
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.578593468Z&quot; level=info msg=&quot;manager selected by agent for new session: { }&quot; module=node/agent node.id=t2x1wbrgq616fsx6v7ay2euqn
   dockerd[24061]: time=&quot;2021-03-03T05:43:12.578634949Z&quot; level=info msg=&quot;waiting 77.462294ms before registering session&quot; module=node/agent node.id=t2x1wbrgq616fsx6v7ay2euqn
   dockerd[24061]: time=&quot;2021-03-03T05:43:17.656467883Z&quot; level=error msg=&quot;agent: session failed&quot; backoff=300ms error=&quot;session initiation timed out&quot; module=node/agent node.id=t2x1wbrgq616fsx6v7ay2euqn
  </code></pre>
  <p>This is a 7-node cluster with 3 nodes running v20.10.5 and 4 nodes running v19.03.13 on Ubuntu LTS 18.04. Any pointers to troubleshoot this and recover the cluster is appreciated.</p>