title: Celery - workers not picking up new tasks
body: <p>I'm building a web catalog archiver, using <a href="http://docs.celeryproject.org/en/latest/index.html" rel="nofollow noreferrer">celery</a> and <a href="https://www.httrack.com/" rel="nofollow noreferrer">httrack</a> asynchronous mirroring.  I currently have a cluster of three servers, each with five workers per server.  Each task can take between two to six hours to execute.  RabbitMQ is used as the broker, with some backtalk over the rpc channel.</p>
  
  <p>I have the following configuration for each celery server:</p>
  
  <pre><code># Celery init script for ingestion workers
  # Set _BIN, _CHDIR, _LOGFILE and _OPTS for the enviroment used.
  
  # This specific configuration is for ingest PROD 2 only
  # Define worker nodes to start
  CELERYD_NODES="ingest_prod_2_1 ingest_prod_2_2 ingest_prod_2_3 ingest_prod_2_4 ingest_prod_2_5"
  
  # Absolute path to celery executable
  CELERY_BIN="/home/ubuntu/.virtualenvs/proj2.7/bin/celery"
  
  CELERYD_CHDIR="/home/ubuntu/project/proj"
  
  # App instance used
  CELERY_APP="ingest.tasks"
  
  #Log file location
  CELERYD_LOG_FILE="/home/ubuntu/project/logs/django/%N.log"
  CELERYD_LOG_LEVEL="DEBUG"
  CELERYD_PID_FILE="/home/ubuntu/project/celery_%N.pid"
  
  CELERYD_ACKS_LATE=True
  CELERYD_PREFETCH_MULTIPLIER=1
  
  # Celery options, define broker backtalk
  CELERYD_OPTS="-b amqp://user:password@host/vhost -Q long_task_queue -c 1 -Ofair"
  
  # Workers should run as an unprivileged user and create dirs if missing
  CELERYD_USER="ubuntu"
  CELERYD_GROUP="ubuntu"
  CELERY_CREATE_DIRS=1
  </code></pre>
  
  <p>For a stress test exercise, I put in 200 tasks in the task queue.  At first, all workers were active, but soon after all the workers on two servers did not pick up a new task after completing the initial task.  Only one server continued on with all five workers running.  Now, two days later, that one server only has one active worker, all others are idle. As a result, instead of in the neighborhood of ~150 tasks executed, I only see 40.</p>
  
  <p>This is the result of `celery -A ingest.tasks inspect active':</p>
  
  <pre><code>-&gt; ingest_prod_3_1@server_1: OK
      - empty -
  -&gt; ingest_prod_2_4@server_2: OK
      - empty -
  -&gt; ingest_prod_4_3@server_3: OK
      - empty -
  -&gt; ingest_prod_3_2@server_1: OK
      - empty -
  -&gt; ingest_prod_2_5@server_2: OK
      - empty -
  -&gt; ingest_prod_4_5@server_3: OK
      - empty -
  -&gt; ingest_prod_3_3@server_1: OK
      - empty -
  -&gt; ingest_prod_2_2@server_2: OK
      - empty -
  -&gt; ingest_prod_4_1@server_3: OK
      - empty -
  -&gt; ingest_prod_3_4@server_1: OK
      - empty -
  -&gt; ingest_prod_2_3@server_2: OK
      - empty -
  -&gt; ingest_prod_4_2@server_3: OK
      - empty -
  -&gt; ingest_prod_3_5@server_1: OK
      * {u'args': u"[u'url', 35637]", u'time_start': 464085.760026547, u'name': u'ingest.tasks.html.download', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'backlog.#', u'exchange': u'celery'}, u'hostname': u'ingest_prod_3_5@server_3', u'acknowledged': True, u'kwargs': u'{}', u'id': u'083b575c-8e52-4426-90a5-e9a6ceb117a4', u'worker_pid': 28807}
  -&gt; ingest_prod_2_1@server_2: OK
      - empty -
  -&gt; ingest_prod_4_4@server_3: OK
      - empty -
  </code></pre>
  
  <p>This is a screenshot of rabbitmq management plugin's queue tab</p>
  
  <p><a href="https://i.stack.imgur.com/bZAdX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bZAdX.png" alt="enter image description here"></a></p>
  
  <p>The queues I've set up are <code>backlog</code> and <code>ingest</code>.  All others are set up by rabbitmq.  I'm not sure how to interpret those at this time.</p>
  
  <p>So I'm not sure what's going on here.</p>