<p dir="auto">We are experiencing an issue (since we upgraded from celery 2.5 to 3.0) where our workers mysteriously stop consuming messages from the Q's they are configured on.</p>
<p dir="auto">It happens periodically on each worker node independently, and i cannot get to the root of what is causing it.</p>
<p dir="auto"><code class="notranslate">celery inspect -d appserver-i-82df38f8.prod active</code> always returns empty.<br>
I did find something slightly interesting, in that, the list of reserved tasks seems to stay static for a long time. Since these tasks do not have an eta set, why don't they execute? I cant think of any reason why they are not being executed?</p>
<p dir="auto">Does the reserved Q have any relationship to the prefetch Q? i.e. if there are a lot of reserved tasks which do not execute would that prevent the node from retrieving more tasks from the server?</p>
<p dir="auto">When looking at the rabbit MQ management interface I can see this node is listed as a consumer.</p>
<p dir="auto">I have collected some info from the node which may shine some light on this, but am at a loss as to why this could happen. Could you provide some pointers for things to look into?</p>
<p dir="auto">Worker machine task inspections<br>
<a href="https://gist.github.com/declanshanaghy/5103891">https://gist.github.com/declanshanaghy/5103891</a></p>
<p dir="auto">Worker machine report<br>
<a href="https://gist.github.com/declanshanaghy/5103838">https://gist.github.com/declanshanaghy/5103838</a></p>
<p dir="auto">Hi,</p>
<p dir="auto">I'm encountering a strange issue for which I set up a simple test case.<br>
I'm using djcelery with RabbitMQ (3.0.4) as broker. Here are the first lines from python manage.py celery report :</p>
<p dir="auto">software -&gt; celery:3.0.17 (Chiastic Slide) kombu:2.5.9 py:2.7.3<br>
billiard:2.7.3.26 librabbitmq:1.0.1<br>
platform -&gt; system:Linux arch:64bit, ELF imp:CPython<br>
loader   -&gt; djcelery.loaders.DjangoLoader<br>
settings -&gt; transport:librabbitmq results:database</p>
<p dir="auto">This is the code in the tasks.py module :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from celery import task
from celery.task.control import revoke

import memcache

mc = memcache.Client(['127.0.0.1:11211'], debug=0)


@task(name='tasks.homeioTestTask')
def testTask(feedUid):
    print &quot;Within test task for feedUid : %s&quot; % feedUid
    return

@task(name='tasks.homeioProcessEvent')
def processEvent(event):
    print &quot;Got an event from the views&quot;

    feedUid = event.get('feedUid')

    print &quot;Event occurred for feed %s&quot; % feedUid

    # Checking if we have a task id in memcached
    mcKey = str('taskHomeIoTest_feed_%s' % feedUid)

    taskId = mc.get(mcKey)

    if taskId:
        print &quot;We have a task in the cache, revoke it&quot;
        revoke(taskId)
        mc.delete(mcKey)
    else:
        print &quot;Scheduling a task&quot;
        task = testTask.apply_async((feedUid,), countdown=60)
        mc.set(mcKey,task.id, time=0)

    return"><pre class="notranslate"><code class="notranslate">from celery import task
from celery.task.control import revoke

import memcache

mc = memcache.Client(['127.0.0.1:11211'], debug=0)


@task(name='tasks.homeioTestTask')
def testTask(feedUid):
    print "Within test task for feedUid : %s" % feedUid
    return

@task(name='tasks.homeioProcessEvent')
def processEvent(event):
    print "Got an event from the views"

    feedUid = event.get('feedUid')

    print "Event occurred for feed %s" % feedUid

    # Checking if we have a task id in memcached
    mcKey = str('taskHomeIoTest_feed_%s' % feedUid)

    taskId = mc.get(mcKey)

    if taskId:
        print "We have a task in the cache, revoke it"
        revoke(taskId)
        mc.delete(mcKey)
    else:
        print "Scheduling a task"
        task = testTask.apply_async((feedUid,), countdown=60)
        mc.set(mcKey,task.id, time=0)

    return
</code></pre></div>
<p dir="auto">This code is being called from my views.py file as follows :<br>
from tasks import processEvent</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def event(request):
    event = simplejson.loads(request.raw_post_data)
    processEvent.delay(event)

    return HttpResponse(status=200)"><pre class="notranslate"><code class="notranslate">def event(request):
    event = simplejson.loads(request.raw_post_data)
    processEvent.delay(event)

    return HttpResponse(status=200)
</code></pre></div>
<p dir="auto">When I run celery using python manage.py celery worker --loglevel=debug I get :<br>
<a href="https://gist.github.com/franckb/e6e5db51b41b74b3fbce">https://gist.github.com/franckb/e6e5db51b41b74b3fbce</a> (things go wrong around line 257)</p>
<p dir="auto">Looks like the tasks don't get to the workers anymore but there is no indication that the workers died or that anything is going wrong.</p>
<p dir="auto">I've tried several things when tracking down the issue, but I'm having a hard time knowing where to look. Could anyone provide some guidance ?</p>
<p dir="auto">Thanks</p>