<p dir="auto">Tested with latest pypi version (4.0.2) and against master branch.</p>
<p dir="auto">Upgraded recently from 3.1.25 as I had some issues with tasks being received but executed after a short delay (as in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="101742841" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/2771" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/2771/hovercard" href="https://github.com/celery/celery/issues/2771">#2771</a>). Now it got worse.</p>
<p dir="auto">Tried using librabbitmq, but encountered: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="194837417" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/3675" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/3675/hovercard" href="https://github.com/celery/celery/issues/3675">#3675</a> when I was queueing task inside a task.</p>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto">Reproduced it with example project from: <a href="https://github.com/celery/celery/blob/master/examples/app/myapp.py">https://github.com/celery/celery/blob/master/examples/app/myapp.py</a></p>
<ol dir="auto">
<li>Installed rabbitmq locally ({rabbit,"RabbitMQ","3.6.9"})</li>
<li>Ran <code class="notranslate">celery worker -A myapp -l debug</code></li>
<li>In another process, sent task a few times: <code class="notranslate">app.send_task('myapp.add', args=(1,2))</code></li>
</ol>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Tasks should be processed asap.</p>
<h2 dir="auto">Actual behavior</h2>
<p dir="auto">Sometimes there is a pretty big delay between receiving and processing a task.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2017-04-14 02:18:08,889: INFO/MainProcess] Received task: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52]
[2017-04-14 02:18:08,889: DEBUG/MainProcess] TaskPool: Apply &lt;function _fast_trace_task at 0x1064d01b8&gt; (args:('myapp.add', '59816272-ca82-4eee-ba95-8d71da5dca52', {'origin': 'gen84425@RK-MacBook.local', 'lang': 'py', 'task': 'myapp.add', 'group': None, 'root_id': '59816272-ca82-4eee-ba95-8d71da5dca52', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'celery', u'exchange': u''}, 'expires': None, u'correlation_id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(1, 2)', 'eta': None, 'parent_id': None, u'reply_to': '51344e5c-09df-31a0-bbfd-e64295f6db67', 'id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'kwargsrepr': '{}'}, u'[[1, 2], {}, {&quot;chord&quot;: null, &quot;callbacks&quot;: null, &quot;errbacks&quot;: null, &quot;chain&quot;: null}]', 'application/json', 'utf-8') kwargs:{})
[2017-04-14 02:18:10,746: DEBUG/MainProcess] Task accepted: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] pid:84171
[2017-04-14 02:18:10,747: INFO/PoolWorker-8] Task myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] succeeded in 0.000706910970621s: 3"><pre class="notranslate"><code class="notranslate">[2017-04-14 02:18:08,889: INFO/MainProcess] Received task: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52]
[2017-04-14 02:18:08,889: DEBUG/MainProcess] TaskPool: Apply &lt;function _fast_trace_task at 0x1064d01b8&gt; (args:('myapp.add', '59816272-ca82-4eee-ba95-8d71da5dca52', {'origin': 'gen84425@RK-MacBook.local', 'lang': 'py', 'task': 'myapp.add', 'group': None, 'root_id': '59816272-ca82-4eee-ba95-8d71da5dca52', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'celery', u'exchange': u''}, 'expires': None, u'correlation_id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(1, 2)', 'eta': None, 'parent_id': None, u'reply_to': '51344e5c-09df-31a0-bbfd-e64295f6db67', 'id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'kwargsrepr': '{}'}, u'[[1, 2], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2017-04-14 02:18:10,746: DEBUG/MainProcess] Task accepted: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] pid:84171
[2017-04-14 02:18:10,747: INFO/PoolWorker-8] Task myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] succeeded in 0.000706910970621s: 3
</code></pre></div>
<p dir="auto">Above, between  <code class="notranslate">TaskPool: Apply</code> and <code class="notranslate">Task accepted:</code> there is a 2s delay. Usually it's smaller, like 0.5s. But it's very inconsistent.</p>
<p dir="auto">Tried with redis broker - no delays and overall much faster. But I am hesitant to make the jump on production and I don't believe this behavior is to be expected anyway.</p>
<p dir="auto">Tested librabbitmq - looks good, also no delays. But I am getting <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="194837417" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/3675" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/3675/hovercard" href="https://github.com/celery/celery/issues/3675">#3675</a> (which is closed for some reason).</p>
<p dir="auto">I have a service using Celery 4.0.2 to dispatch tasks to workers. I observed that, periodically, Celery workers hang altogether for a while without doing any tasks, and then suddenly, they revive and resume execution again.  Looks like Celery 4.0.2 buffers those tasks and release to workers later.<br>
Celery 3 (I tried 3.1.23 and 3.1.25) don't have those problems and tasks are processed smoothly &amp; evenly.<br>
I use Python Celery module. The command is something like "-A module_name --loglevel=info -n str(uuid4())[:8] -P solo --autoscale=16,2";<br>
Some other configs are like:<br>
BROKER_CONNECTION_TIMEOUT=3600 * 24,<br>
BROKER_CONNECTION_MAX_RETRIES=None,<br>
CELERY_ACCEPT_CONTENT=['application/json'],<br>
CELERY_TASK_SERIALIZER='json',<br>
CELERY_RESULT_SERIALIZER='json',<br>
CELERY_ACKS_LATE=False</p>
<p dir="auto">Some screenshots are as follows;<br>
Celery 4<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/11428198/22787304/c75dcffe-ef16-11e6-966c-1010c738ae2b.png"><img width="950" alt="screen shot 2017-02-09 at 10 19 21 pm" src="https://cloud.githubusercontent.com/assets/11428198/22787304/c75dcffe-ef16-11e6-966c-1010c738ae2b.png" style="max-width: 100%;"></a><br>
Celery 3.0.25<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/11428198/22787353/f1fb3756-ef16-11e6-9342-1471424c4c78.png"><img width="961" alt="screen shot 2017-02-09 at 10 18 43 pm" src="https://cloud.githubusercontent.com/assets/11428198/22787353/f1fb3756-ef16-11e6-9342-1471424c4c78.png" style="max-width: 100%;"></a></p>