title: Solution for user-specific background job queues
body: <p>I have been researching how to efficiently solve the following use case and I am struggling to find the best solution.</p>
  <p>Basically I have a Node.js REST API which handles requests for users from a mobile application. We want some requests to launch background tasks outside of the req/res flow because they are CPU intensive or might just take a while to execute. We are trying to implement or use any existing frameworks which are able to handle different job queues in the following way (or at least compatible with the use case):</p>
  <ul>
  <li>Every user has their own set job queues (there are different kind of jobs).</li>
  <li>The jobs within one specific queue have to be executed sequentially and only one job at a time but everything else can be executed in parallel (it would be preferable if there are no queues hogging the workers or whatever is actually consuming the tasks so all queues get more or less the same priority).</li>
  <li>Some queues might fill up with hundreds of tasks at a given time but most likely they will be empty a lot of the time.</li>
  <li>Queues need to be persistent.</li>
  </ul>
  <p>We currently have a solution with RabbitMQ with one queue for every kind of task which all the users share. The users dump tasks into the same queues which results in them filling up with tasks from a specific user for a long time and having the rest of users wait for those tasks to be done before their own start to be consumed. We have looked into priority queues but we don't think that's the way to go for our own use case.</p>
  <p>The first somewhat logical solution we thought of is to create temporary queues whenever a user needs to run background jobs and have them be deleted when empty. Nevertheless we are not sure if having that many queues is scalable and we are also struggling with dynamically creating RabbitMQ queues, exchanges, etc (we have even read somewhere that it might be an anti-pattern?).</p>
  <p>We have been doing some more research and maybe the way to go would be with other stuff such as Kafka or Redis based stuff like BullMQ or similar.</p>
  <p>What would you recommend?</p>