title: Is Kafka able to have a dynamic number of consumers?
body: <p>We are looking for a new messaging platform, and have narrowed our choices down to RabbitMQ or Kafka.</p>
  
  <p>Right now, I am leaning toward Kafka, but I have some doubts that it is a good choice given one of our requirements.</p>
  
  <p>We need to have a queue that is consumed by an unknown number of consumers.  That is, we need to dynamically add and remove consumers as "workers" come online to do the processing.  Also, workers may drop off at any time.</p>
  
  <p>So for example, we may start a queue that has no consumers at all, and then the number of consumers may grow to 30.  Later it may grow to 5000 or more, and then drop back off to 3.</p>
  
  <p>We do not care about message ordering for this particular use case.  Is Kafka a good fit for this?</p>
  
  <p>Also, we were planning on maintaining a pool of consumer threads so that the workers could grab a single message and process it.  So there may be 100 consumers in the pool and only 20 workers.  Is it possible that we end up with messages in the other 80 consumers which are not utilized in the workers due to message send buffering?  In other words, does Kafka pre-deliver messages to consumers before they are requested like some messaging systems do?</p>