<p dir="auto">When using a chord - even the sample given at <a href="http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords" rel="nofollow">http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords</a> - the callback will never be executed, the task <code class="notranslate">celery.chord_unlock</code> will just retry indefinitely.</p>
<p dir="auto">None if the tasks encounter any issues ( as demonstrated <a href="http://pastebin.com/AGZmWWZM" rel="nofollow">http://pastebin.com/AGZmWWZM</a> )</p>
<p dir="auto">Using Celery 3.1.18, Python 2.7.6, Ubuntu 14.04.2 LTS</p>
<p dir="auto">Duscussion/troubleshooting from irc:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;ItsDanielB&gt; Hey, I'm having a problem using a chord. Once the main group of tasks finishes executing, 'celery.chord_unlock' just keeps retrying forever.
&lt;ItsDanielB&gt; Been doing research, but the only other occurences I can find are happening when there's an error in the main group, but all my tasks finish successfully
&lt;ItsDanielB&gt; Also have `CELERY_IGNORE_RESULT = False`, `CELERY_RESULT_BACKEND = 'rpc'`, `CELERY_CHORD_PROPAGATES = True` and `CELERY_RESULT_PERSISTENT = True`
&lt;ItsDanielB&gt; Even if I make all of the tasks do nothing (just immediatly return) I still get the same result
&lt;ItsDanielB&gt; Pastebin showing the &quot;problem function&quot; and the worker log http://pastebin.com/AGZmWWZM
&lt;amcm_&gt; ItsDanielB 1) why recalculate_bouts_task.si() - I think that should be .s() if you expect the return values from the chord to get passed in.
&lt;amcm_&gt; Are you on 3.1 or master?
&lt;amcm_&gt; I've seen a number of problems with chords completing on 3.1
* amcm_ is now known as amcm
&lt;ItsDanielB&gt; amcm using si because I don't want any of the results - they are talking to an external API and all return None
&lt;ItsDanielB&gt; using celery 3.1.18
&lt;amcm&gt; ItsDanielB then why use a chord? I suppose group | task is a chord anyway...
&lt;amcm&gt; I'd try it on master just to see
&lt;amcm&gt; Also I don't know how the rpc backend works. redis can be smarter about chords when using new_joins=1 for the result backend
&lt;amcm&gt; Actually, that is what I had to do to get working chords (new_joins), but I was already using redis for my results
&lt;ItsDanielB&gt; Installing from source (from github) throws an error when launching the worker - `ImportError: cannot import name timedelta_seconds`
&lt;ItsDanielB&gt; am I missing a requirement / should be using a particular branch/tag/etc?
&lt;amcm&gt; That's an odd one. You have to manually install the dev requirements though to use master
&lt;amcm&gt; pip install -r requirements/dev.txt
&lt;ItsDanielB&gt; `pip install -r requirements/dev.txt` already done
&lt;ItsDanielB&gt; ^
&lt;ItsDanielB&gt; :)
&lt;amcm&gt; Ah'kay
&lt;ItsDanielB&gt; Looks like I'll need to install django-celery from source too
&lt;amcm&gt; Ah, maybe? We're not using django-celery right now, just regular celery
&lt;amcm&gt; You using the DB beat schedual?
&lt;ItsDanielB&gt; Not sure what that is :P
&lt;ItsDanielB&gt; Just queueing taks right from django views
&lt;ItsDanielB&gt; tasks*
&lt;ItsDanielB&gt; Hm, maybe not. Now the worker hangs before it finishes launching.
&lt;ItsDanielB&gt; Also, reading up on django-celery, I wasn't using that before either :|
&lt;ItsDanielB&gt; rolling back, will try again :3
&lt;ItsDanielB&gt; The master branch of celery seems to be incompatiable with any py-amqp version
&lt;ItsDanielB&gt; 1.4.5, 1.4.6 give `from amqp.five import __all__ as _all_five` `ImportError: cannot import name __all__`
&lt;ItsDanielB&gt; wait, hold up - may be may fault here
&lt;ItsDanielB&gt; ok, yea
&lt;ItsDanielB&gt; Seems I need some combination of py-amqp, billiard and kombu but I can't figure out which ones work with each other and master celery
&lt;ItsDanielB&gt; amcm, can I get a `pip freeze` from you so I can just grab the same versions? Everytime I get one requirement working another one breaks :P
&lt;amcm&gt; ItsDanielB pip install -r requirements/dev.txt from inside celery should get you the right versions of billard, kombu and py-amqp
&lt;amcm&gt; See: https://github.com/celery/celery/issues/2083
&lt;ItsDanielB&gt; gives: `pkg_resources.DistributionNotFound: The 'billiard&lt;3.4,&gt;=3.3.0.17' distribution was not found and is required by celery`
&lt;ItsDanielB&gt; then when I install matching version, a different error
&lt;ItsDanielB&gt; requirements/dev.txt currently gives billard billiard-3.4.0.0b1
&lt;amcm&gt; That's what I have
&lt;amcm&gt; The requirement looks like it's for 3.1 not master
&lt;amcm&gt; billiard-3.4.0.0b1 is &gt; 3.4, and it is what you should have
&lt;ItsDanielB&gt; pkg_resources wants me to have &lt;3.4 for some reason
&lt;amcm&gt; Where is that coming from?
&lt;ItsDanielB&gt; starts in `&lt;venv&gt;/bin/celery`
&lt;ItsDanielB&gt; amcm full error: http://pastebin.com/d7S8KLys
&lt;ItsDanielB&gt; Tried on a fresh install, same issue
&lt;amcm&gt; Did you pip uninstall celery (to remove the 3.1 from pypi), then python setup.py install  The git clone?
&lt;amcm&gt; check pip freeze for your celery version
&lt;ItsDanielB&gt; @amcm https://asciinema.org/a/0mmmovl1fmwjji5rtg2bndqwi
&lt;amcm&gt; which celery
&lt;ItsDanielB&gt; `/home/danielb/.venv/celery-test/bin/celery`
&lt;amcm&gt; That screen recorder is pretty slick
&lt;amcm&gt; :( I'm on OSX but I can't reproduce. I get
&lt;ItsDanielB&gt; Was just looking for a way to share without it being an unwieldy log file
&lt;amcm&gt; (deleteme):~/src/celery{celery:(detached from upstream/master)}:$ celery --version
&lt;amcm&gt; 3.2.0a2 (DEV)
&lt;ItsDanielB&gt; :?
&lt;amcm&gt; same steps, new venv, python setup.py install; pip install -r requirements/dev.txt"><pre class="notranslate"><code class="notranslate">&lt;ItsDanielB&gt; Hey, I'm having a problem using a chord. Once the main group of tasks finishes executing, 'celery.chord_unlock' just keeps retrying forever.
&lt;ItsDanielB&gt; Been doing research, but the only other occurences I can find are happening when there's an error in the main group, but all my tasks finish successfully
&lt;ItsDanielB&gt; Also have `CELERY_IGNORE_RESULT = False`, `CELERY_RESULT_BACKEND = 'rpc'`, `CELERY_CHORD_PROPAGATES = True` and `CELERY_RESULT_PERSISTENT = True`
&lt;ItsDanielB&gt; Even if I make all of the tasks do nothing (just immediatly return) I still get the same result
&lt;ItsDanielB&gt; Pastebin showing the "problem function" and the worker log http://pastebin.com/AGZmWWZM
&lt;amcm_&gt; ItsDanielB 1) why recalculate_bouts_task.si() - I think that should be .s() if you expect the return values from the chord to get passed in.
&lt;amcm_&gt; Are you on 3.1 or master?
&lt;amcm_&gt; I've seen a number of problems with chords completing on 3.1
* amcm_ is now known as amcm
&lt;ItsDanielB&gt; amcm using si because I don't want any of the results - they are talking to an external API and all return None
&lt;ItsDanielB&gt; using celery 3.1.18
&lt;amcm&gt; ItsDanielB then why use a chord? I suppose group | task is a chord anyway...
&lt;amcm&gt; I'd try it on master just to see
&lt;amcm&gt; Also I don't know how the rpc backend works. redis can be smarter about chords when using new_joins=1 for the result backend
&lt;amcm&gt; Actually, that is what I had to do to get working chords (new_joins), but I was already using redis for my results
&lt;ItsDanielB&gt; Installing from source (from github) throws an error when launching the worker - `ImportError: cannot import name timedelta_seconds`
&lt;ItsDanielB&gt; am I missing a requirement / should be using a particular branch/tag/etc?
&lt;amcm&gt; That's an odd one. You have to manually install the dev requirements though to use master
&lt;amcm&gt; pip install -r requirements/dev.txt
&lt;ItsDanielB&gt; `pip install -r requirements/dev.txt` already done
&lt;ItsDanielB&gt; ^
&lt;ItsDanielB&gt; :)
&lt;amcm&gt; Ah'kay
&lt;ItsDanielB&gt; Looks like I'll need to install django-celery from source too
&lt;amcm&gt; Ah, maybe? We're not using django-celery right now, just regular celery
&lt;amcm&gt; You using the DB beat schedual?
&lt;ItsDanielB&gt; Not sure what that is :P
&lt;ItsDanielB&gt; Just queueing taks right from django views
&lt;ItsDanielB&gt; tasks*
&lt;ItsDanielB&gt; Hm, maybe not. Now the worker hangs before it finishes launching.
&lt;ItsDanielB&gt; Also, reading up on django-celery, I wasn't using that before either :|
&lt;ItsDanielB&gt; rolling back, will try again :3
&lt;ItsDanielB&gt; The master branch of celery seems to be incompatiable with any py-amqp version
&lt;ItsDanielB&gt; 1.4.5, 1.4.6 give `from amqp.five import __all__ as _all_five` `ImportError: cannot import name __all__`
&lt;ItsDanielB&gt; wait, hold up - may be may fault here
&lt;ItsDanielB&gt; ok, yea
&lt;ItsDanielB&gt; Seems I need some combination of py-amqp, billiard and kombu but I can't figure out which ones work with each other and master celery
&lt;ItsDanielB&gt; amcm, can I get a `pip freeze` from you so I can just grab the same versions? Everytime I get one requirement working another one breaks :P
&lt;amcm&gt; ItsDanielB pip install -r requirements/dev.txt from inside celery should get you the right versions of billard, kombu and py-amqp
&lt;amcm&gt; See: https://github.com/celery/celery/issues/2083
&lt;ItsDanielB&gt; gives: `pkg_resources.DistributionNotFound: The 'billiard&lt;3.4,&gt;=3.3.0.17' distribution was not found and is required by celery`
&lt;ItsDanielB&gt; then when I install matching version, a different error
&lt;ItsDanielB&gt; requirements/dev.txt currently gives billard billiard-3.4.0.0b1
&lt;amcm&gt; That's what I have
&lt;amcm&gt; The requirement looks like it's for 3.1 not master
&lt;amcm&gt; billiard-3.4.0.0b1 is &gt; 3.4, and it is what you should have
&lt;ItsDanielB&gt; pkg_resources wants me to have &lt;3.4 for some reason
&lt;amcm&gt; Where is that coming from?
&lt;ItsDanielB&gt; starts in `&lt;venv&gt;/bin/celery`
&lt;ItsDanielB&gt; amcm full error: http://pastebin.com/d7S8KLys
&lt;ItsDanielB&gt; Tried on a fresh install, same issue
&lt;amcm&gt; Did you pip uninstall celery (to remove the 3.1 from pypi), then python setup.py install  The git clone?
&lt;amcm&gt; check pip freeze for your celery version
&lt;ItsDanielB&gt; @amcm https://asciinema.org/a/0mmmovl1fmwjji5rtg2bndqwi
&lt;amcm&gt; which celery
&lt;ItsDanielB&gt; `/home/danielb/.venv/celery-test/bin/celery`
&lt;amcm&gt; That screen recorder is pretty slick
&lt;amcm&gt; :( I'm on OSX but I can't reproduce. I get
&lt;ItsDanielB&gt; Was just looking for a way to share without it being an unwieldy log file
&lt;amcm&gt; (deleteme):~/src/celery{celery:(detached from upstream/master)}:$ celery --version
&lt;amcm&gt; 3.2.0a2 (DEV)
&lt;ItsDanielB&gt; :?
&lt;amcm&gt; same steps, new venv, python setup.py install; pip install -r requirements/dev.txt
</code></pre></div>
<p dir="auto">When using a chord - even the sample given at <a href="http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords" rel="nofollow">http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords</a> - the callback will never be executed, the task <code class="notranslate">celery.chord_unlock</code> will just retry indefinitely.</p>
<p dir="auto">None if the tasks encounter any issues ( as demonstrated <a href="http://pastebin.com/AGZmWWZM" rel="nofollow">http://pastebin.com/AGZmWWZM</a> )</p>
<p dir="auto">Using Celery 3.1.18, Python 2.7.6, Ubuntu 14.04.2 LTS</p>
<p dir="auto">Duscussion/troubleshooting from irc:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;ItsDanielB&gt; Hey, I'm having a problem using a chord. Once the main group of tasks finishes executing, 'celery.chord_unlock' just keeps retrying forever.
&lt;ItsDanielB&gt; Been doing research, but the only other occurences I can find are happening when there's an error in the main group, but all my tasks finish successfully
&lt;ItsDanielB&gt; Also have `CELERY_IGNORE_RESULT = False`, `CELERY_RESULT_BACKEND = 'rpc'`, `CELERY_CHORD_PROPAGATES = True` and `CELERY_RESULT_PERSISTENT = True`
&lt;ItsDanielB&gt; Even if I make all of the tasks do nothing (just immediatly return) I still get the same result
&lt;ItsDanielB&gt; Pastebin showing the &quot;problem function&quot; and the worker log http://pastebin.com/AGZmWWZM
&lt;amcm_&gt; ItsDanielB 1) why recalculate_bouts_task.si() - I think that should be .s() if you expect the return values from the chord to get passed in.
&lt;amcm_&gt; Are you on 3.1 or master?
&lt;amcm_&gt; I've seen a number of problems with chords completing on 3.1
* amcm_ is now known as amcm
&lt;ItsDanielB&gt; amcm using si because I don't want any of the results - they are talking to an external API and all return None
&lt;ItsDanielB&gt; using celery 3.1.18
&lt;amcm&gt; ItsDanielB then why use a chord? I suppose group | task is a chord anyway...
&lt;amcm&gt; I'd try it on master just to see
&lt;amcm&gt; Also I don't know how the rpc backend works. redis can be smarter about chords when using new_joins=1 for the result backend
&lt;amcm&gt; Actually, that is what I had to do to get working chords (new_joins), but I was already using redis for my results
&lt;ItsDanielB&gt; Installing from source (from github) throws an error when launching the worker - `ImportError: cannot import name timedelta_seconds`
&lt;ItsDanielB&gt; am I missing a requirement / should be using a particular branch/tag/etc?
&lt;amcm&gt; That's an odd one. You have to manually install the dev requirements though to use master
&lt;amcm&gt; pip install -r requirements/dev.txt
&lt;ItsDanielB&gt; `pip install -r requirements/dev.txt` already done
&lt;ItsDanielB&gt; ^
&lt;ItsDanielB&gt; :)
&lt;amcm&gt; Ah'kay
&lt;ItsDanielB&gt; Looks like I'll need to install django-celery from source too
&lt;amcm&gt; Ah, maybe? We're not using django-celery right now, just regular celery
&lt;amcm&gt; You using the DB beat schedual?
&lt;ItsDanielB&gt; Not sure what that is :P
&lt;ItsDanielB&gt; Just queueing taks right from django views
&lt;ItsDanielB&gt; tasks*
&lt;ItsDanielB&gt; Hm, maybe not. Now the worker hangs before it finishes launching.
&lt;ItsDanielB&gt; Also, reading up on django-celery, I wasn't using that before either :|
&lt;ItsDanielB&gt; rolling back, will try again :3
&lt;ItsDanielB&gt; The master branch of celery seems to be incompatiable with any py-amqp version
&lt;ItsDanielB&gt; 1.4.5, 1.4.6 give `from amqp.five import __all__ as _all_five` `ImportError: cannot import name __all__`
&lt;ItsDanielB&gt; wait, hold up - may be may fault here
&lt;ItsDanielB&gt; ok, yea
&lt;ItsDanielB&gt; Seems I need some combination of py-amqp, billiard and kombu but I can't figure out which ones work with each other and master celery
&lt;ItsDanielB&gt; amcm, can I get a `pip freeze` from you so I can just grab the same versions? Everytime I get one requirement working another one breaks :P
&lt;amcm&gt; ItsDanielB pip install -r requirements/dev.txt from inside celery should get you the right versions of billard, kombu and py-amqp
&lt;amcm&gt; See: https://github.com/celery/celery/issues/2083
&lt;ItsDanielB&gt; gives: `pkg_resources.DistributionNotFound: The 'billiard&lt;3.4,&gt;=3.3.0.17' distribution was not found and is required by celery`
&lt;ItsDanielB&gt; then when I install matching version, a different error
&lt;ItsDanielB&gt; requirements/dev.txt currently gives billard billiard-3.4.0.0b1
&lt;amcm&gt; That's what I have
&lt;amcm&gt; The requirement looks like it's for 3.1 not master
&lt;amcm&gt; billiard-3.4.0.0b1 is &gt; 3.4, and it is what you should have
&lt;ItsDanielB&gt; pkg_resources wants me to have &lt;3.4 for some reason
&lt;amcm&gt; Where is that coming from?
&lt;ItsDanielB&gt; starts in `&lt;venv&gt;/bin/celery`
&lt;ItsDanielB&gt; amcm full error: http://pastebin.com/d7S8KLys
&lt;ItsDanielB&gt; Tried on a fresh install, same issue
&lt;amcm&gt; Did you pip uninstall celery (to remove the 3.1 from pypi), then python setup.py install  The git clone?
&lt;amcm&gt; check pip freeze for your celery version
&lt;ItsDanielB&gt; @amcm https://asciinema.org/a/0mmmovl1fmwjji5rtg2bndqwi
&lt;amcm&gt; which celery
&lt;ItsDanielB&gt; `/home/danielb/.venv/celery-test/bin/celery`
&lt;amcm&gt; That screen recorder is pretty slick
&lt;amcm&gt; :( I'm on OSX but I can't reproduce. I get
&lt;ItsDanielB&gt; Was just looking for a way to share without it being an unwieldy log file
&lt;amcm&gt; (deleteme):~/src/celery{celery:(detached from upstream/master)}:$ celery --version
&lt;amcm&gt; 3.2.0a2 (DEV)
&lt;ItsDanielB&gt; :?
&lt;amcm&gt; same steps, new venv, python setup.py install; pip install -r requirements/dev.txt"><pre class="notranslate"><code class="notranslate">&lt;ItsDanielB&gt; Hey, I'm having a problem using a chord. Once the main group of tasks finishes executing, 'celery.chord_unlock' just keeps retrying forever.
&lt;ItsDanielB&gt; Been doing research, but the only other occurences I can find are happening when there's an error in the main group, but all my tasks finish successfully
&lt;ItsDanielB&gt; Also have `CELERY_IGNORE_RESULT = False`, `CELERY_RESULT_BACKEND = 'rpc'`, `CELERY_CHORD_PROPAGATES = True` and `CELERY_RESULT_PERSISTENT = True`
&lt;ItsDanielB&gt; Even if I make all of the tasks do nothing (just immediatly return) I still get the same result
&lt;ItsDanielB&gt; Pastebin showing the "problem function" and the worker log http://pastebin.com/AGZmWWZM
&lt;amcm_&gt; ItsDanielB 1) why recalculate_bouts_task.si() - I think that should be .s() if you expect the return values from the chord to get passed in.
&lt;amcm_&gt; Are you on 3.1 or master?
&lt;amcm_&gt; I've seen a number of problems with chords completing on 3.1
* amcm_ is now known as amcm
&lt;ItsDanielB&gt; amcm using si because I don't want any of the results - they are talking to an external API and all return None
&lt;ItsDanielB&gt; using celery 3.1.18
&lt;amcm&gt; ItsDanielB then why use a chord? I suppose group | task is a chord anyway...
&lt;amcm&gt; I'd try it on master just to see
&lt;amcm&gt; Also I don't know how the rpc backend works. redis can be smarter about chords when using new_joins=1 for the result backend
&lt;amcm&gt; Actually, that is what I had to do to get working chords (new_joins), but I was already using redis for my results
&lt;ItsDanielB&gt; Installing from source (from github) throws an error when launching the worker - `ImportError: cannot import name timedelta_seconds`
&lt;ItsDanielB&gt; am I missing a requirement / should be using a particular branch/tag/etc?
&lt;amcm&gt; That's an odd one. You have to manually install the dev requirements though to use master
&lt;amcm&gt; pip install -r requirements/dev.txt
&lt;ItsDanielB&gt; `pip install -r requirements/dev.txt` already done
&lt;ItsDanielB&gt; ^
&lt;ItsDanielB&gt; :)
&lt;amcm&gt; Ah'kay
&lt;ItsDanielB&gt; Looks like I'll need to install django-celery from source too
&lt;amcm&gt; Ah, maybe? We're not using django-celery right now, just regular celery
&lt;amcm&gt; You using the DB beat schedual?
&lt;ItsDanielB&gt; Not sure what that is :P
&lt;ItsDanielB&gt; Just queueing taks right from django views
&lt;ItsDanielB&gt; tasks*
&lt;ItsDanielB&gt; Hm, maybe not. Now the worker hangs before it finishes launching.
&lt;ItsDanielB&gt; Also, reading up on django-celery, I wasn't using that before either :|
&lt;ItsDanielB&gt; rolling back, will try again :3
&lt;ItsDanielB&gt; The master branch of celery seems to be incompatiable with any py-amqp version
&lt;ItsDanielB&gt; 1.4.5, 1.4.6 give `from amqp.five import __all__ as _all_five` `ImportError: cannot import name __all__`
&lt;ItsDanielB&gt; wait, hold up - may be may fault here
&lt;ItsDanielB&gt; ok, yea
&lt;ItsDanielB&gt; Seems I need some combination of py-amqp, billiard and kombu but I can't figure out which ones work with each other and master celery
&lt;ItsDanielB&gt; amcm, can I get a `pip freeze` from you so I can just grab the same versions? Everytime I get one requirement working another one breaks :P
&lt;amcm&gt; ItsDanielB pip install -r requirements/dev.txt from inside celery should get you the right versions of billard, kombu and py-amqp
&lt;amcm&gt; See: https://github.com/celery/celery/issues/2083
&lt;ItsDanielB&gt; gives: `pkg_resources.DistributionNotFound: The 'billiard&lt;3.4,&gt;=3.3.0.17' distribution was not found and is required by celery`
&lt;ItsDanielB&gt; then when I install matching version, a different error
&lt;ItsDanielB&gt; requirements/dev.txt currently gives billard billiard-3.4.0.0b1
&lt;amcm&gt; That's what I have
&lt;amcm&gt; The requirement looks like it's for 3.1 not master
&lt;amcm&gt; billiard-3.4.0.0b1 is &gt; 3.4, and it is what you should have
&lt;ItsDanielB&gt; pkg_resources wants me to have &lt;3.4 for some reason
&lt;amcm&gt; Where is that coming from?
&lt;ItsDanielB&gt; starts in `&lt;venv&gt;/bin/celery`
&lt;ItsDanielB&gt; amcm full error: http://pastebin.com/d7S8KLys
&lt;ItsDanielB&gt; Tried on a fresh install, same issue
&lt;amcm&gt; Did you pip uninstall celery (to remove the 3.1 from pypi), then python setup.py install  The git clone?
&lt;amcm&gt; check pip freeze for your celery version
&lt;ItsDanielB&gt; @amcm https://asciinema.org/a/0mmmovl1fmwjji5rtg2bndqwi
&lt;amcm&gt; which celery
&lt;ItsDanielB&gt; `/home/danielb/.venv/celery-test/bin/celery`
&lt;amcm&gt; That screen recorder is pretty slick
&lt;amcm&gt; :( I'm on OSX but I can't reproduce. I get
&lt;ItsDanielB&gt; Was just looking for a way to share without it being an unwieldy log file
&lt;amcm&gt; (deleteme):~/src/celery{celery:(detached from upstream/master)}:$ celery --version
&lt;amcm&gt; 3.2.0a2 (DEV)
&lt;ItsDanielB&gt; :?
&lt;amcm&gt; same steps, new venv, python setup.py install; pip install -r requirements/dev.txt
</code></pre></div>