title: Distributed Job Queues Per Customer
body: <p>Requirements:
  - Distributed 
  - At most one job being processed for a single customer
  - Parallelize work amongst customers
  - Support 10k+ customers
  - Client libraries need to be available for the JVM</p>
  
  <p>My company currently uses Quartz as a distributed job store. One of the nice
  features about it, is the ability to "DisallowConcurrentExecution" in which
  each customer can have their own job and its triggers are processes in the order
  received. This works nicely, however, as we continue to add more customers the
  number of jobs isn't scaling as we'd hoped (Postgres backed) and the volume at
  which we are reading/writing/vacumming postgres quartz tables is growing too fast.</p>
  
  <p>We're doing some research and trying to find an architecture pattern where we can
  have a job queue per customer where there is at most a single job being processed per 
  customer at a time (work can be parallelized across customers).</p>
  
  <p>We've explored different approaches in RabbitMQ/SQS/Kafka/Distributed Locks etc 
  and none seem great to meet the "one job at a time per customer" and scalable 
  requirement. One architecture that would solve what we need is Akka but our 
  company has shied away from it due to the many deployment/maintenance issues 
  we've run into in the past with it.</p>
  
  <p>One issue with not just processing all jobs in FIFO is some jobs may take a
  while to complete (30+s) and we don't want other customers waiting for another
  customer's job to complete.</p>