title: Applications are disconnecting from Kafka cluster
body: <p>Dears,</p>
  <p>Our applications are disconnecting from Kafka cluster. Applications logs show warning like this one:</p>
  <pre><code>14:15:24.445   [kafka-producer-network-thread | producer-1] WARN  o.a.k.c.NetworkClient - [Producer clientId=producer-1] Bootstrap broker A.B.C.D:9093 (id: -2 rack: null) disconnected
  </code></pre>
  <p>On the side of Kafka cluster there is nothing in the logs.</p>
  <p>log4j.properties looks like this one:</p>
  <pre><code>cat log4j.properties|grep -v &quot;#&quot;
  
  log4j.rootLogger=INFO, kafkaAppender
  
  log4j.appender.stdout=org.apache.log4j.ConsoleAppender
  log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
  log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
  
  log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
  log4j.appender.kafkaAppender.MaxFileSize=50MB
  log4j.appender.kafkaAppender.MaxBackupIndex=4
  log4j.appender.kafkaAppender.File=/var/log/kafka/kafka-server.log
  log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
  log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
  
  log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender
  log4j.appender.stateChangeAppender.MaxFileSize=50MB
  log4j.appender.stateChangeAppender.MaxBackupIndex=4
  log4j.appender.stateChangeAppender.File=/var/log/kafka/state-change.log
  log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
  log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
  
  log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender
  log4j.appender.controllerAppender.MaxFileSize=50MB
  log4j.appender.controllerAppender.MaxBackupIndex=4
  log4j.appender.controllerAppender.File=/var/log/kafka/controller.log
  log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
  log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
  
  log4j.logger.kafka=WARN, kafkaAppender
  
  log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender
  log4j.appender.requestAppender.MaxFileSize=50MB
  log4j.appender.requestAppender.MaxBackupIndex=4
  log4j.appender.requestAppender.File=/var/log/kafka/kafka-request.log
  log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
  log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
  
  log4j.logger.kafka.controller=DEBUG, controllerAppender
  log4j.additivity.kafka.controller=false
  
  log4j.logger.state.change.logger=DEBUG, stateChangeAppender
  log4j.additivity.state.change.logger=false
  </code></pre>
  <p>server.properties file look like this one:</p>
  <pre><code>broker.id=2
  delete.topic.enable=true
  auto.create.topics.enable=false
  listeners=INSIDE://:9091,OUTSIDE1://:9093,OUTSIDE2://:9092
  advertised.listeners=INSIDE://kafka-nX:9091,OUTSIDE1://kafka-nX:9093,OUTSIDE2://kafka-nX:9092
  listener.security.protocol.map=INSIDE:SASL_SSL,OUTSIDE1:SASL_SSL,OUTSIDE2:PLAINTEXT
  inter.broker.listener.name=INSIDE
  sasl.enabled.mechanisms=SCRAM-SHA-512
  sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
  authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
  allow.everyone.if.no.acl.found=true
  ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
  ssl.secure.random.implementation=SHA1PRNG
  super.users=User:admin
  ssl.client.auth=none
  ssl.keystore.location=/etc/pki/CA/certs/kafka.jks
  ssl.keystore.password=password
  ssl.key.password=kafkakafka
  ssl.truststore.location=/etc/pki/CA/certs/truststore.p12
  ssl.truststore.password=password
  num.network.threads=3
  num.io.threads=8
  socket.send.buffer.bytes=102400
  socket.receive.buffer.bytes=102400
  socket.request.max.bytes=104857600
  offsets.topic.replication.factor=3
  transaction.state.log.replication.factor=3
  transaction.state.log.min.isr=3
  log.dirs=/kafka_data
  num.partitions=1
  num.recovery.threads.per.data.dir=1
  log.flush.interval.messages=10000
  log.flush.interval.ms=1000
  log.retention.hours=168
  log.retention.bytes=1073741824
  log.segment.bytes=1073741824
  log.retention.check.interval.ms=300000
  zookeeper.connect=A.B.C.D:2181,A.B.C.D+1:2181,A.B.C.D+2:2181
  zookeeper.connection.timeout.ms=6000
  </code></pre>
  <p>Kafka cluster version: 2.8.1
  Kafka client version: 3.0.1</p>
  <p>How to monitor/analyze such cases?</p>
  <p>How to configure log4j.properties file on the Kafka cluster side to see such disconnected sessions?</p>
  <p>Do you have any idea what can be wrong?</p>
  <p>Best Regards,
  Dan</p>