title: Memory Leak in kafka
body: <p>We have built a data ingestion pipeline using Kafka. We have a consumer that reads from a kafka topic and writes to a database continuously. These consumer pods is experiencing issues with OOM events. We run kafka on top of k8s.</p>
  
  <p>We are now seeing that the consumer pods are being restarted every 4-5 days due to OOM error. We are using Kafka 2.2 version of the kafka java libs. </p>
  
  <p>compile group: 'org.apache.kafka', name: 'kafka_2.12', version: '2.2.0'
          compile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.2.0'</p>
  
  <p>Here are the stats from the jvm inside the pod: </p>
  
  <pre><code>  15:          6075         145800  org.apache.kafka.common.requests.ApiVersionsResponse$ApiVersion
    16:          4030         128960  org.apache.kafka.common.MetricName
    17:          4030         128960  org.apache.kafka.common.metrics.KafkaMetric
    22:          1672          80256  org.apache.kafka.common.metrics.stats.SampledStat$Sample
    24:          1174          75136  org.apache.kafka.common.metrics.Sensor
    27:          1261          40352  org.apache.kafka.common.metrics.stats.Meter
    29:          1291          30984  org.apache.kafka.common.metrics.stats.Total
    30:          1261          30264  org.apache.kafka.common.metrics.stats.Rate
    32:           849          27168  org.apache.kafka.common.MetricNameTemplate
    35:           975          23400  org.apache.kafka.common.protocol.types.BoundField
  
  
  
  / # jcmd {app}.jar GC.class_histogram | head
  
   num     #instances         #bytes  class name
  ----------------------------------------------
     1:          1828        5264368  [B
     2:         33683        2949528  [C
     3:          7422         832008  java.lang.Class
     4:         33674         808176  java.lang.String
     5:         10990         764272  [Ljava.lang.Object;
     6:         23875         764000  java.util.HashMap$Node
  
  </code></pre>
  
  <p>When I started analyzing the heap dumps, I thought the issue is with the KafkaAdminClient threads which are open in 100's on the jvms and never closed. I closed them properly and tried to look the heap after that. I still see that the issue persists. </p>
  
  <p>Can I know what operation in kafka produces the above mentioned classes? Is it the consumer.poll() or the background health check or something else. </p>
  
  <p>Any pointer in the right direction would be great help. I am stuck with this for more than a week now. I tried many different Heap dump analyzers. All of them point to JMX and NIO classes. </p>
  
  <p>Here is one leak suspect report from Eclipse MAT:</p>
  
  <p><code>One instance of "com.sun.jmx.mbeanserver.JmxMBeanServer" loaded by "&lt;system class loader&gt;" occupies 61,571,856 (36.76%) bytes. The instance is referenced by org.apache.commons.dbcp2.PoolableConnection @ 0xf0647668 , loaded by "sun.misc.Launcher$AppClassLoader @ 0xf09a3bd8". The memory is accumulated in one instance of "java.util.HashMap$Node[]" loaded by "&lt;system class loader&gt;".</code></p>
  
  <p>and</p>
  
  <pre><code>sun.nio.ch.EPollArrayWrapper```
  </code></pre>