title: Bulls Queue Performance and Scalability: Queue.add(), Queue.getJob(jobId), Job.remove()
body: <p>My use case is to create <strong>dynamic delayed</strong> job. (I am Using <a href="https://optimalbits.github.io/bull/" rel="nofollow noreferrer">Bulls Queue</a> which can be used to create delayed Jobs.)</p>
  <p>Based on <strong>some event</strong> add some more delay to the <strong>delayed interval</strong> (further delay the job).</p>
  <p>Since I could not find any <code>function</code> to update the Delayed Interval for a Job I came up with the following steps:</p>
  <pre><code>onEvent(jobId):
    // queue is of Type Bull.Queue
    // job is of type bull.Job
    job = queue.getJob(jobId)
    data = job.data
    delay = job.toJSON().delay
    job.remove()
    queue.add(&quot;jobName&quot;, {value: 1}, {jobId: jobId, delayed: delay + someValue})
  </code></pre>
  <p>This pretty much solves my problem.</p>
  <p>But I am worried about the <strong>SCALE</strong> at which these operations will happen.</p>
  <p>I am expecting nearly <em><strong>50K</strong></em> events per minute or even more in near future.</p>
  <p>My <strong>Queue</strong> size is expected to grow based on unique <strong>JobId</strong>.</p>
  <p>I am expecting more than:</p>
  <ul>
  <li>1 million daily entry</li>
  <li>around 4-5 million weekly entry</li>
  <li>10-12 million monthly entry.</li>
  </ul>
  <p>Also, after <em><strong>60-70 days</strong></em> delayed interval for jobs will reach, and older jobs will be removed one by one.</p>
  <p>I can run multiple processor to handle these delayed job which is not an issue.</p>
  <p>My queue size will be stabilise after 60-70 days and more or less my queue will have around 10 million jobs.</p>
  <p>I can vertically scale my <em><strong>REDIS</strong></em> as required.</p>
  <p>But I want to understand the time complexity for below queries:</p>
  <pre><code> queue.getJob(jobId)  // Get Job By Id
   job.remove() // remove job from queue
   queue.add(name, data, opts) // add a delayed job to this queue
  </code></pre>
  <p>If any of these operations are O(N) OR the <em><strong>QUEUE</strong></em> can keep some max number of <em><strong>Jobs</strong></em> which is less than 10 million.</p>
  <p>Then I might have to discard this design and come up with something entirely different.</p>
  <p>Need advice from experienced folks who can guide me on how solve this problem.</p>
  <p>Any kind of help is appreciated.</p>