title: Slow Consumers on Kafka
body: <p>I use Kafka to maintain a Python service which should be working on parallel to handle the slow API requests for each message efficiently.</p>
  
  <p>I used multiprocessing module on Python and kafka-python for the consumers.</p>
  
  <p>ZooKeeper and Kafka 2.11 runs on the same Ubuntu server with mostly defult configurations.</p>
  
  <p>The topic is auto-created with another kafka-python producer and set to have 10 partitions in order to use up to 10 consumers at the same time.</p>
  
  <p>When I check, I see that the queue is really long, so the producer sends so many requests:</p>
  
  <pre><code>$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic usrReq --time -1
  usrReq:8:1157
  usrReq:2:1185
  usrReq:5:1167
  usrReq:4:1115
  usrReq:7:1164
  usrReq:10:1150
  usrReq:1:1149
  usrReq:9:1138
  usrReq:3:1186
  usrReq:6:1220
  usrReq:0:6264
  </code></pre>
  
  <p>However; although working 10 cores in parallel, consumers take very long time (117 seconds on a sample log below) to get the next message from the queue. </p>
  
  <pre><code>thread 7, consumer: 117.485 sec
  api1:0.412 sec
  api2:0.752 sec
  db_insert:0.132 sec
  </code></pre>
  
  <p>This is how each Process creates its own consumer, fetches messages and runs the analysis on the code:</p>
  
  <pre><code>consumer = KafkaConsumer(group_id='my-group', 
                           bootstrap_servers='localhost', 
                           value_deserializer=lambda m: json.loads(m.decode('ascii'))
  consumer.subscribe(topics='usrReq')
  while True:
      msg = next(consumer).value['id']
      method(msg)
  </code></pre>
  
  <p>Where could be the problem in this setup?</p>