title: Can Kafka be used as a distribute work queue
body: <p>I'm considering Kafka to use as a distributed work queue multiple workers can retrieve tasks from. My original design looks as:</p>
  <pre><code>Work Producer ---&gt; Kafka topic ------worker 1
                                    |
                                    |__worker 2
                                    ...
                                    |__worker n
  </code></pre>
  <p>The problems with this design is this:</p>
  <ol>
  <li><p>If some worker takes a task from the topic and immediately commits offset then in case of failure the task may not be reprocessed.</p>
  </li>
  <li><p>If some worker takes a task from the topic and commits offset only on finish then other workers may also takes this task and process it. If the task is pretty long lasting then almost all workers will take the same task and process it completely inhibiting the distributing nature.</p>
  </li>
  </ol>
  <hr />
  <p>I'm looking for a way &quot;mark&quot; a task in a queue as &quot;in progress&quot; so it's not consumed by anyone else, but offset is not committed (because it may fail and needs reprocessing). Is it possible to implement?</p>