title: Celery: how to limit number of tasks in queue and stop feeding when full?
body: <p>I am very new to Celery and here is the question I have:</p>
  
  <p>Suppose I have a script that is constantly supposed to fetch new data from DB and send it to workers using Celery.</p>
  
  <p>tasks.py</p>
  
  <pre><code># Celery Task
  from celery import Celery
  
  app = Celery('tasks', broker='amqp://guest@localhost//')
  
  @app.task
  def process_data(x):
      # Do something with x
      pass
  </code></pre>
  
  <p>fetch_db.py</p>
  
  <pre><code># Fetch new data from DB and dispatch to workers.
  from tasks import process_data
  
  while True:
      # Run DB query here to fetch new data from DB fetched_data
  
      process_data.delay(fetched_data)
  
      sleep(30);
  </code></pre>
  
  <p>Here is my concern: the data is being fetched every 30 seconds. process_data() function could take much longer and depending on the amount of workers (especially if too few) the queue might get throttled as I understand. </p>
  
  <ol>
  <li>I cannot increase number of workers.</li>
  <li>I can modify the code to refrain from feeding the queue when it is full.</li>
  </ol>
  
  <p>The question is how do I set queue size and how do I know it is full? In general, how to deal with this situation?</p>