title: KafkaSpout very high latency
body: <p>I am new to apache storm and kafka, as part of POC I am trying to process a message stream using Kafka and apache storm. I am using storm-kafka source from <a href="https://github.com/apache/storm/tree/master/external/storm-kafka" rel="nofollow noreferrer">https://github.com/apache/storm/tree/master/external/storm-kafka</a>, I am able to create a sample program which reads messages from kafka topic using KafkaSpout and output it on another kafka topic. I have 3 node kafka(all three running on same server) cluster and created the topics with 8 partitions. I setup the KafkaSpout parallelism as 8 and bolt's parallelism as 8 as well, tried with 8 executors as well as task. I have tried setting up lot of tunnig parameters both at kafka level, SpoutConfig level and storm level but i am getting very high overall latency issue. I need message process garuntee so acking is really required. Storm cluster has one supervisor and zookeeper has 3 noed, it is shared between kafka and storm. It is running on Red Hat Linux machine with 144MB RAM with 16CPU. With below parameters, i ma getting very high spout process latency about 40Sec, I need to get around 50K msg/sec level, can you please help me with the configuration to achieve it. I have gone through lot of posts on various site and tried lot of tuning options with no results.</p>
  
  <pre><code>Storm config
  topology.receiver.buffer.size=16
  topology.transfer.buffer.size=4096
  topology.executor.receive.buffer.size=16384
  topology.executor.send.buffer.size=16384
  topology.spout.max.batch.size=65536
  topology.max.spout.pending=10000
  topology.acker.executors=20
  
  Kafka config
  fetch.size.bytes=1048576
  socket.timeout.ms=10000
  fetch.max.wait=10000
  buffer.size.bytes=1048576
  </code></pre>
  
  <p>Thanks in advance.</p>
  
  <p>Storm UI screenshot</p>
  
  <p><a href="https://i.stack.imgur.com/MVpjM.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MVpjM.jpg" alt="enter image description here"></a></p>