title: Apache ZooKeeper Cluster loses connectivity after leader election
body: <p>I am running a ZooKeeper Cluster with five nodes, each of which has the following configuration (plus the correct quorum information at the end):</p>
  
  <pre><code>tickTime=2000
  initLimit=5
  syncLimit=2
  dataDir=/usr/local/zookeeper/data
  clientPort=2181
  minSessionTimeout=4000
  maxSessionTimeout=40000
  4lw.commands.whitelist=*
  </code></pre>
  
  <p>The cluster works fine at first. For testing purposes, I then kill the instance of ZooKeeper that is currently the leader (each ZooKeeper server is running in the foreground of a <code>screen</code> session). Then, a leader election is triggered, as expected, and another node is elected leader.<br>
  However, when then trying to send <code>get</code>/<code>set</code>/<code>create</code> requests to th cluster in a Java client (the process that was running fine initially), as well as interfacing with the cluster via <code>zkCli.sh</code> just gives the state of the client as <code>CONNECTING</code> forever.<br>
  At this time, I have conducted the obvious troubleshooting steps such as  </p>
  
  <ul>
  <li><code>echo stat | nc localhost 2181</code> -- On any one of the servers that are still running this will just give an indication of all being well, such as </li>
  </ul>
  
  <pre><code>Clients:
   /10.0.0.1:35264[1](queued=0,recved=2,sent=1)
   /10.0.0.2:49230[0](queued=0,recved=1,sent=0)
   /127.0.0.1:34162[0](queued=0,recved=1,sent=0)
   /10.0.0.3:49530[0](queued=0,recved=1,sent=0)
   /10.0.0.1:35250[0](queued=0,recved=1,sent=0)
   /10.0.0.4:35406[1](queued=0,recved=2,sent=1)
   /10.0.0.2:49304[1](queued=0,recved=1,sent=1)
  
   Latency min/avg/max: 0/0/0
   Received: 484
   Sent: 140
   Connections: 7
   Outstanding: 343 
   Zxid: 0x200000000
   Mode: leader
   Node count: 15
   Proposal sizes last/min/max: 32/32/75
  </code></pre>
  
  <ul>
  <li><code>echo ruok | nc localhost 2181</code> just ouputs <code>iamok</code></li>
  <li>The following is the contents (abridged) of the log file on the server which is elected leader after I inject the failure:</li>
  </ul>
  
  <pre><code>2019-08-12 12:25:36,799 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 23 MS
  2019-08-12 12:25:37,004 [myid:4] - WARN  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner@282] - Unexpected exception, tries=0, remaining init limit=9798, connecting to /10.0.0.10:2888
  java.net.ConnectException: Connection refused (Connection refused)
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:589)
      at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:233)
      at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:262)
      at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:77)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1271)
  2019-08-12 12:25:39,148 [myid:4] - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
  2019-08-12 12:25:39,174 [myid:4] - INFO  [NIOWorkerThread-5:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
  2019-08-12 12:25:39,175 [myid:4] - INFO  [NIOWorkerThread-5:FourLetterCommands@235] - The list of enabled four letter word commands is : [[wchs, stat, wchp, dirs, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, telnet close, crst, cons]]
  2019-08-12 12:25:39,175 [myid:4] - INFO  [NIOWorkerThread-5:NIOServerCnxn@518] - Processing stat command from /127.0.0.1:59614
  2019-08-12 12:25:39,858 [myid:4] - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
  2019-08-12 12:25:39,906 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner@391] - Getting a diff from the leader 0x0
  2019-08-12 12:25:39,913 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER message
  2019-08-12 12:25:40,253 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner@529] - Learner received UPTODATE message
  2019-08-12 12:25:40,266 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 16 worker threads.
  2019-08-12 12:25:40,536 [myid:4] - WARN  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower@125] - Got zxid 0x100000001 expected 0x1
  2019-08-12 12:25:40,536 [myid:4] - INFO  [SyncThread:4:FileTxnLog@216] - Creating new log file: log.100000001
  2019-08-12 12:25:43,617 [myid:4] - INFO  [NIOWorkerThread-7:NIOServerCnxn@518] - Processing stat command from /127.0.0.1:59638
  2019-08-12 12:25:43,619 [myid:4] - INFO  [NIOWorkerThread-7:StatCommand@53] - Stat command output
  2019-08-12 12:26:06,907 [myid:4] - WARN  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower@96] - Exception when following the leader
  java.io.EOFException
      at java.io.DataInputStream.readInt(DataInputStream.java:392)
      at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
      at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
      at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
      at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)
      at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1271)
  2019-08-12 12:26:06,908 [myid:4] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 5, my id = 4, error = 
  java.io.EOFException
      at java.io.DataInputStream.readInt(DataInputStream.java:392)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
  2019-08-12 12:26:06,909 [myid:4] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker
  2019-08-12 12:26:06,909 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower@201] - shutdown called
  java.lang.Exception: shutdown Follower
      at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1275)
  2019-08-12 12:26:06,910 [myid:4] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue
  java.lang.InterruptedException
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
      at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
  2019-08-12 12:26:06,910 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down
  2019-08-12 12:26:06,911 [myid:4] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@1102] - Send worker leaving thread  id 5 my id = 4
  2019-08-12 12:26:06,911 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@558] - shutting down
  2019-08-12 12:26:06,912 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FollowerRequestProcessor@139] - Shutting down
  2019-08-12 12:26:06,912 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@362] - Shutting down
  2019-08-12 12:26:06,912 [myid:4] - INFO  [FollowerRequestProcessor:4:FollowerRequestProcessor@110] - FollowerRequestProcessor exited loop!
  2019-08-12 12:26:06,914 [myid:4] - INFO  [CommitProcessor:4:CommitProcessor@195] - CommitProcessor exited loop!
  2019-08-12 12:26:06,917 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FinalRequestProcessor@514] - shutdown of request processor complete
  2019-08-12 12:26:06,922 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):SyncRequestProcessor@191] - Shutting down
  2019-08-12 12:26:06,923 [myid:4] - INFO  [SyncThread:4:SyncRequestProcessor@169] - SyncRequestProcessor exited!
  2019-08-12 12:26:06,923 [myid:4] - WARN  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1318] - PeerState set to LOOKING
  2019-08-12 12:26:06,924 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKING
  2019-08-12 12:26:06,925 [myid:4] - INFO  [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id =  4, proposed zxid=0x100000024
  2019-08-12 12:26:07,129 [myid:4] - WARN  [WorkerSender[myid=4]:QuorumCnxManager@677] - Cannot open channel to 5 at election address /10.0.0.10:3888
  java.net.ConnectException: Connection refused (Connection refused)
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:589)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:648)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:705)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:618)
      at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)
      at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)
      at java.lang.Thread.run(Thread.java:748)
  2019-08-12 12:26:07,337 [myid:4] - INFO  [WorkerSender[myid=4]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (5, 4)
  2019-08-12 12:26:07,337 [myid:4] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
  2019-08-12 12:26:07,338 [myid:4] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@679] - Notification: 2 (message format version), 4 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 4 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
  2019-08-12 12:26:07,338 [myid:4] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
  2019-08-12 12:26:07,338 [myid:4] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@679] - Notification: 2 (message format version), 4 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
  2019-08-12 12:26:07,342 [myid:4] - INFO  [/0.0.0.0:3888:QuorumCnxManager$Listener@888] - Received connection request /172.17.0.6:46520
  2019-08-12 12:26:07,345 [myid:4] - INFO  [/0.0.0.0:3888:QuorumCnxManager$Listener@888] - Received connection request /172.17.0.6:46522
  2019-08-12 12:26:07,345 [myid:4] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue
  java.lang.InterruptedException
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
      at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
  2019-08-12 12:44:00,079 [myid:4] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 2, my id = 4, error = 
  java.net.SocketException: Socket closed
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      at java.net.SocketInputStream.read(SocketInputStream.java:171)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
      at java.io.DataInputStream.readInt(DataInputStream.java:387)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
  
  2019-08-12 12:44:43,382 [myid:4] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x4003ead876c0025
  2019-08-12 12:44:45,381 [myid:4] - INFO  [SessionTracker:ZooKeeperServer@398] - Expiring session 0x1003ead86e90025, timeout of 40000ms exceeded
  2019-08-12 12:44:45,381 [myid:4] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1003ead86e90025
  2019-08-12 12:45:00,485 [myid:4] - INFO  [/0.0.0.0:3888:QuorumCnxManager$Listener@888] - Received connection request /10.0.0.7:42596
  2019-08-12 12:45:00,485 [myid:4] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 4, my id = 4, error = 
  java.io.EOFException
      at java.io.DataInputStream.readInt(DataInputStream.java:392)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
  2019-08-12 12:45:00,485 [myid:4] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue
  </code></pre>
  
  <p>I would be extremely grateful for any help as to why the cluster doesn't seem to interconnect (or at least won't serve requests anymore) after the leader election takes place, or any advice what to look at for debugging purposes.</p>