Using Celery with SQLAlchemy and Pyramid
Question: I'm creating the web app using Pyramid-1.2.1 with SQLAlchemy as database
backend.  
Now I need to do some periodic tasks along with this app and I want to use
Celery as a task processor and SQLAlchemy as the message queue and the result
backend.

1) I've installed Celery-2.1.4 but I can not figure out how to make it work.
Here is my `celeryconfig.py` file:

    
    
    # List of modules to import when celery starts.
    CELERY_IMPORTS = ("my_app.tasks", )
    
    ## Result store settings.
    CELERY_RESULT_BACKEND = "database"
    CELERY_RESULT_DBURI = "sqlite:///MyDataBase.db"
    CELERY_RESULT_SERIALIZER = "json"
    
    ## Broker settings.
    BROKER_TRANSPORT = "sqlakombu.transport.Transport"
    BROKER_HOST = "sqlite:///MyDataBase.db"
    

The `my_app.tasks` contain a simple addition task from the celery examples.  
Now when I run

    
    
    $ celeryd -l info
    

I see the following:

    
    
    [2011-11-11 20:22:50,750: WARNING/MainProcess] celery@OrangeMac.local v2.1.4 is starting.
    [2011-11-11 20:22:50,765: WARNING/MainProcess]  
    Configuration ->
        . broker -> sqlakombu.transport.Transport://guest@sqlite:///MyDataBase.db/
        . queues ->
            . celery -> exchange:celery (direct) binding:celery
        . concurrency -> 4
        . loader -> celery.loaders.default.Loader
        . logfile -> [stderr]@INFO
        . events -> OFF
        . beat -> OFF
        . tasks ->
            . chatrooms.task.add
    [2011-11-11 20:22:50,787: INFO/PoolWorker-1] child process calling self.run()
    [2011-11-11 20:22:50,789: INFO/PoolWorker-2] child process calling self.run()
    [2011-11-11 20:22:50,791: INFO/PoolWorker-3] child process calling self.run()
    [2011-11-11 20:22:50,796: INFO/PoolWorker-4] child process calling self.run()
    [2011-11-11 20:22:50,802: WARNING/MainProcess] celery@OrangeMac.local has started.
    [2011-11-11 20:22:50,804: WARNING/MainProcess] Traceback (most recent call last):
    [2011-11-11 20:22:50,805: WARNING/MainProcess] File "/Users/shashkin/python_v_env/bin/celeryd", line 8, in <module>
    [2011-11-11 20:22:50,805: WARNING/MainProcess] load_entry_point('celery==2.1.4', 'console_scripts', 'celeryd')()
    [2011-11-11 20:22:50,805: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/celeryd.py", line 166, in main
    [2011-11-11 20:22:50,805: WARNING/MainProcess] worker.execute_from_commandline()
    [2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/base.py", line 40, in execute_from_commandline
    [2011-11-11 20:22:50,806: WARNING/MainProcess] return self.run(*args, **vars(options))
    [2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/celeryd.py", line 85, in run
    [2011-11-11 20:22:50,806: WARNING/MainProcess] return Worker(**kwargs).run()
    [2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/apps/worker.py", line 121, in run
    [2011-11-11 20:22:50,807: WARNING/MainProcess] self.run_worker()
    [2011-11-11 20:22:50,807: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/apps/worker.py", line 219, in run_worker
    [2011-11-11 20:22:50,807: WARNING/MainProcess] worker.start()
    [2011-11-11 20:22:50,807: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/__init__.py", line 217, in start
    [2011-11-11 20:22:50,808: WARNING/MainProcess] component.start()
    [2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 238, in start
    [2011-11-11 20:22:50,808: WARNING/MainProcess] self.reset_connection()
    [2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 416, in reset_connection
    [2011-11-11 20:22:50,808: WARNING/MainProcess] self.connection = self._open_connection()
    [2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 480, in _open_connection
    [2011-11-11 20:22:50,809: WARNING/MainProcess] max_retries=conf.BROKER_CONNECTION_MAX_RETRIES)
    [2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/utils/__init__.py", line 276, in retry_over_time
    [2011-11-11 20:22:50,809: WARNING/MainProcess] retval = fun(*args, **kwargs)
    [2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 472, in _establish_connection
    [2011-11-11 20:22:50,809: WARNING/MainProcess] conn.connect()                              # evaluate connection
    [2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 170, in connect
    [2011-11-11 20:22:50,810: WARNING/MainProcess] return self.connection
    [2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 135, in connection
    [2011-11-11 20:22:50,810: WARNING/MainProcess] self._connection = self._establish_connection()
    [2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 148, in _establish_connection
    [2011-11-11 20:22:50,810: WARNING/MainProcess] return self.create_backend().establish_connection()
    [2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 161, in create_backend
    [2011-11-11 20:22:50,810: WARNING/MainProcess] return backend_cls(connection=self)
    [2011-11-11 20:22:50,811: WARNING/MainProcess] TypeError
    [2011-11-11 20:22:50,811: WARNING/MainProcess] :
    [2011-11-11 20:22:50,811: WARNING/MainProcess] __init__() takes exactly 2 arguments (1 given)
    [2011-11-11 20:22:50,811: INFO/MainProcess] process shutting down
    

What am I missing?

2) Later I want to start Celery inside my app. What should I do after celery-
pylons package installation? Should I put celery's settings in app's `.ini`
file? Could someone provide me any example, because those I found in the net
did not help me much.

Thanks.


Answer: This looks like an issue with version compatibility between `celery` packages
and its dependencies. With the latest version of celery, 2.4.5 your example
just works. With 2.1.4 I get the same error.

If for any reason you need 2.1.4 version of celery only, you have to find out
which dependencies you need to downgrade in order to make it work. By looking
at the traceback, most likely, it'll be the carrot `dependency` but you may
get into a dependency hell if other dependencies need certain version of
`carrot`.

I strongly suggest to use virtualenv with pip to be able to experiment with
the versions of the packages easily. Trying to develop something using Python
packages from Linux repositories can be very painful.

[buildout versions](http://packages.python.org/buildout-versions/use.html) or
[pip requirements](http://www.pip-installer.org/en/latest/requirements.html)
address this kind of problems but AFAIK there is no set of pinned dependency
versions for celery.

