title: Kafka - Setting high linger.ms and batch.size not helping
body: <p>I am trying to improve Kafka producer throughput, we have CSV reports which are getting process and publish to Kafka topic. using default Kafka settings we are getting on avg <code>300-500 kbps</code> Kafka throughput. to improve the throughput I have tried testing some combinations with <code>linger.ms</code> and <code>batch.size</code> but it's not helping.</p>
  <p>tried with</p>
  <pre><code>&quot;linger.ms= 30000&quot;,&quot;batch.size= 1000000&quot;,&quot;buffer.memory=16777216&quot;
  &quot;linger.ms= 40000&quot;,&quot;batch.size= 1500000&quot;,&quot;buffer.memory=16777216&quot;
  </code></pre>
  <p>even tried with lesser linger.ms and batch.size
  <code>linger.ms = 200, batch.size=65000</code></p>
  <p>but still, throughput is around 150-200 kbps</p>
  <p>but throughput is just decreasing to 100-150kbps.</p>
  <p>Kafka topic has 12 partitions.</p>
  <pre><code>ack is all, and compression is snappy
  </code></pre>
  <p>any suggestions are welcome.</p>