title: message queue for processing streams of data
body: <p>I have streams of input data that I process. Each stream is sent in chunks of data. I can only process the N+1st chunk of data of a stream <em>i</em> after I <em>finished</em> processing the Nth chunk of data of the same stream <em>i</em>. Therefore, parallelization can happen by processing multiple streams at once, but I can never split one stream on multiple workers.</p>
  
  <p>Chunks of one stream are added to the queue in order (although chunks from several streams can be added at the same time).</p>
  
  <p>Most message queues, like RabbitMQ, guarantee ordered delivery when multiple workers operate on one queue. However, to achieve the behaviour I would like, I'd need to restrict the number of workers to 1 for each queue, so that the next chunk is always only processed when the previous chunk was finished. To parallelize, I could create a queue for each stream, or a queue for each worker, and have another process that redirects the streams to the worker queues. In fact, the one-queue-per-worker approach is what I do right now, using RabbitMQ's consistent-hashing and shovels. Of course, in terms of load balancing and dynamic scaling of the number of workers, that is far from ideal.</p>
  
  <p>I've read a lot about Kafka, and how it is designed for time-series data (like logs). Yet, I couldn't figure out how I could apply Kafka - or any other message queue out there - to solve my problem.</p>
  
  <p>I would greatly appreciate some hints on how to best use a message queue for my problem.</p>