title: Getting error in kafka as Unable to reconnect to Zookeeper, session 0x20000xxxxxxxx
body: <p>We are running confluent kafka (<a href="https://github.com/confluentinc/cp-helm-charts" rel="nofollow noreferrer">https://github.com/confluentinc/cp-helm-charts</a>) on kubernetes(1.14.6). We have log retention of 30mins and storage as 300GB. We have 4 brokers and replication factor of 3. We have approximately 65MBps of throughput. After running after around an hour we observe below error. Kafka broker has 6GB of heap.</p>
  
  <p><code>[2019-09-27 12:32:05,278] WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error when sending leader epoch request for Map(RT-15-7 -&gt; (currentLeaderEpoch=Optional[8], leaderEpoch=6), RT-17-0 -&gt; (currentLeaderEpoch=Optional[5], leaderEpoch=3), RT-19-6 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-02-0 -&gt; (currentLeaderEpoch=Optional[5], leaderEpoch=3), RT-27-4 -&gt; (currentLeaderEpoch=Optional[6], leaderEpoch=4), RT-22-3 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-32-5 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-42-4 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-27-1 -&gt; (currentLeaderEpoch=Optional[9], leaderEpoch=7), _confluent-controlcenter-5-2-0-1-MetricsAggregateStore-repartition-2 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-21-6 -&gt; (currentLeaderEpoch=Optional[8], leaderEpoch=6), _confluent-controlcenter-5-2-0-1-metrics-trigger-measurement-rekey-3 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-30-6 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-06-6 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), _confluent-controlcenter-5-2-0-1-expected-group-consumption-rekey-1 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-17-1 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), _confluent-metrics-10 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-21-0 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), _confluent-monitoring-9 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-17-9 -&gt; (currentLeaderEpoch=Optional[9], leaderEpoch=7), RT-02-9 -&gt; (currentLeaderEpoch=Optional[9], leaderEpoch=7), RT-20-1 -&gt; (currentLeaderEpoch=Optional[8], leaderEpoch=6), RT-30-0 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-12-0 -&gt; (currentLeaderEpoch=Optional[8], leaderEpoch=6), RT-32-9 -&gt; (currentLeaderEpoch=Optional[6], leaderEpoch=4), RT-02-1 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-06-0 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), _confluent-monitoring-3 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), _confluent-metrics-7 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), _confluent-controlcenter-5-2-0-1-MetricsAggregateStore-changelog-0 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-27-5 -&gt; (currentLeaderEpoch=Optional[9], leaderEpoch=7), RT-17-6 -&gt; (currentLeaderEpoch=Optional[6], leaderEpoch=4), RT-32-3 -&gt; (currentLeaderEpoch=Optional[7], leaderEpoch=5), RT-02-6 -&gt; (currentLeaderEpoch=Optional[6], leaderEpoch=4), _confluent-monitoring-0 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9), RT-27-2 -&gt; (currentLeaderEpoch=Optional[6], leaderEpoch=4), _confluent-controlcenter-5-2-0-1-actual-group-consumption-rekey-2 -&gt; (currentLeaderEpoch=Optional[11], leaderEpoch=9)) (kafka.server.ReplicaFetcherThread)
  java.io.IOException: Connection to 3 was disconnected before the response was read
      at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100)
      at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:107)
      at kafka.server.ReplicaFetcherThread.fetchEpochEndOffsets(ReplicaFetcherThread.scala:310)
      at kafka.server.AbstractFetcherThread.truncateToEpochEndOffsets(AbstractFetcherThread.scala:208)
      at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:173)
      at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113)
      at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)</code></p>
  
  <p>Rest of the configuration is default. I'm not sure what is causing this zookeeper to close the socket connection. Also I can see all my pods are healthy.
  Please let me know if more information needs to be added. Appreciate any pointers for debugging.</p>
  
  <p><a href="https://i.stack.imgur.com/IACtO.png" rel="nofollow noreferrer">Grafana Dashbaord depicting brokers and partitions</a>
  <a href="https://i.stack.imgur.com/smWsc.png" rel="nofollow noreferrer">Other useful metrics</a></p>