Celery: how to separate different environments with different workers?
Question: I need to route all tasks of a certain django site instance to a certain
queue. My setup is as following:

  * several webservers running a Django project (1.7)
  * one server running celery workers (3.1.7)
  * Three environments: production, staging, development. Each environment runs with a different `DJANGO_SETTINGS_MODULE`, with a different `CELERY_DEFAULT_QUEUE` setting.
  * One redis instance as broker (everything in the same database)

On the "celery server", I run multiple worker instances through supervisor
(simplified conf):

    
    
    [program:production_queue]
    environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.production
    command=/pth/to/python celery -A website.celery worker --events --queues myserver --loglevel WARNING --concurrency 4 -n production@celery.myserver.nl
    
    [program:staging_queue]
    environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.staging
    command=/pth/to/python celery -A website.celery worker --events --queues myserver_staging --loglevel WARNING --concurrency 1 -n staging@celery.myserver.nl
    
    [program:development_queue]
    environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.development
    command=/pth/to/python celery -A website.celery worker --events --queues myserver_development --loglevel INFO --concurrency 1 -n development@celery.myserver.nl
    

This works, with inspection:

    
    
    $ celery -A website.celery inspect activeues
    -> production@celery.myserver.nl: OK
        * {u'exclusive': False, u'name': u'myserver', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}
    -> staging@celery.myserver.nl: OK
        * {u'exclusive': False, u'name': u'myserver_staging', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}
    -> development@celery.myserver.nl: OK
        * {u'exclusive': False, u'name': u'myserver_development', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}
    

(Names accord with the CELERY_DEFAULT_QUEUE settings)

`website/celery.py` contains the basics (imports skipped):

    
    
    app = Celery('proj')
    app.config_from_object('django.conf:settings')
    app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
    

I would therefore expect tasks generated by a webserver running with the
development settings, to end up only in the `development_queue`, and so on.
However, I see tasks being processed by a different queue, or by all three,
which is problematic.

**Are my expectations wrong in that this would be a good way to separate these
tasks?** All [documentation on
routing](http://docs.celeryproject.org/en/latest/userguide/routing.html#manual-
routing) is about routing different tasks to different queues, which I don't
need. I need to route all tasks of a certain site (environment) to a certain
queue. **What can I do to separate these environments?**


Answer: I got an answer from the developer of Celery here:
<https://github.com/celery/celery/issues/2508>, which is:

> You have to set all of `CELERY_DEFAULT_QUEUE`, `CELERY_DEFAULT_EXCHANGE` and
> `CELERY_DEFAULT_ROUTING_KEY`. Otherwise you will end up with three queues
> all bound to the same exchange and routing key.
>
> Or use the method here which will set it up explicitly:
> <http://docs.celeryproject.org/en/latest/userguide/routing.html#changing-
> the-name-of-the-default-queue>

That works!

