<p dir="auto">I'm running into an issue where some of my worker processes will stop processing new tasks, causing them to pile up on the queue server. Left alone it will sometimes clear, and sometimes grow to impact all workers, effectively blocking processing.</p>
<p dir="auto">However, if I issue a management command, such as "celery inspect scheduled" it will cause workers to resume operation immediately. Although sometimes only briefly before halting again.</p>
<p dir="auto"><strong>The setup:</strong></p>
<ul dir="auto">
<li>celery 3.1.9</li>
<li>kombu 3.0.12</li>
<li>amqp 1.4.4</li>
<li>RabbitMQ (3.1.3) / ampq for all communication and results.</li>
<li>Mid-sized config: 7 worker nodes, 13 queues for service differentiation, and about 40 total worker processes per worker node.</li>
<li>Nodes are running the prefork worker, with 1-8 processes per queue, depending on the queue.</li>
<li>CELERYD_PREFETCH_MULTIPLIER = 1</li>
<li>CELERY_ACKS_LATE = True</li>
<li>CELERYD_MAX_TASKS_PER_CHILD = 16</li>
<li>I'm using -Ofair, as even within a given queue some tasks may take very different times to execute. Overall task per second throughput has never been an issue.</li>
<li>Example command line:
<blockquote>
<p dir="auto">python /home/celeryworker/web/manage.py celery worker --without-gossip --loglevel=INFO -Q transcode -Ofair -c 2 -n celery-worker-7-prod_transcode</p>
</blockquote>
</li>
</ul>
<p dir="auto">I believe it may have to do with scheduled tasks, as everytime I've been able to catch it in action there have been a fairly large number of tasks waiting for a countdown to expire. On the order of the total number of task slots available for that queue (I'm counting 7 workers, listening with -c4 on a queue, and a prefetch multiplier of 1 to be 28 task slots)</p>
<p dir="auto">I have not been able to confirm if the rabbitmq channel prefetch value is being correctly incremented when this issue is occurring and there are scheduled tasks; though it is being correctly incremented by scheduled tasks during normal operation. (I'll double check that the next time this recurs.)</p>
<p dir="auto">In general the above config is doing exactly what I expect; one task per worker process, multiple tasks per node, and the next task coming out of the queue as soon as any given worker is no longer busy. It's only occasionally that things go off the rails and, one node at a time, new tasks will no longer be executed. (Other processes listening on other queues on that node will continue to work, or fail, independently.)</p>
<p dir="auto">See test case here:<br>
<a href="https://github.com/sabw8217/celery_test">https://github.com/sabw8217/celery_test</a></p>
<p dir="auto">If I enqueue a number of tasks, and then start a worker with --without-mingle and --without-gossip, the worker seems to hang and not actually run any of the tasks I enqueued. I'm running this on debian. On my dev box I have pretty consistently been able to cause the worker to "wake up" and start consuming tasks by issuing a 'celery inspect' command. 'celery inspect active' will return something like:<br>
-&gt; <a href="mailto:celery@aaron-dev.localdomain">celery@aaron-dev.localdomain</a>: OK<br>
- empty -<br>
And then worker the will go through the tasks.</p>
<p dir="auto">Debug output from the worker looks like:<br>
[2014-02-04 13:53:47,628: DEBUG/MainProcess] | Worker: Starting Hub<br>
[2014-02-04 13:53:47,628: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,628: DEBUG/MainProcess] | Worker: Starting Pool<br>
[2014-02-04 13:53:47,633: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,634: DEBUG/MainProcess] | Worker: Starting Consumer<br>
[2014-02-04 13:53:47,635: DEBUG/MainProcess] | Consumer: Starting Connection<br>
[2014-02-04 13:53:47,644: INFO/MainProcess] Connected to amqp://guest@localhost:5672//<br>
[2014-02-04 13:53:47,645: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,645: DEBUG/MainProcess] | Consumer: Starting Events<br>
[2014-02-04 13:53:47,659: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,661: DEBUG/MainProcess] | Consumer: Starting Heart<br>
[2014-02-04 13:53:47,663: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,663: DEBUG/MainProcess] | Consumer: Starting Tasks<br>
[2014-02-04 13:53:47,666: DEBUG/MainProcess] basic.qos: prefetch_count-&gt;4<br>
[2014-02-04 13:53:47,667: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,667: DEBUG/MainProcess] | Consumer: Starting Control<br>
[2014-02-04 13:53:47,669: DEBUG/MainProcess] ^-- substep ok<br>
[2014-02-04 13:53:47,670: DEBUG/MainProcess] | Consumer: Starting event loop<br>
[2014-02-04 13:53:47,670: WARNING/MainProcess] <a href="mailto:celery@aaron-dev.localdomain">celery@aaron-dev.localdomain</a> ready.<br>
[2014-02-04 13:53:47,670: DEBUG/MainProcess] | Worker: Hub.register Pool...</p>