title: Running Celery as Daemon with Supervisor and Django on Elastic Beanstalk
body: <p>I am attempting to get Celery to run on an EB environment with Django and I have gotten super close to getting it all running. I have this config file for supervisor</p>
  
  <pre><code>#!/usr/bin/env bash
  
  # Get django environment variables
  celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
  celeryenv=${celeryenv%?}
  
  # Create celery configuraiton script
  celeryconf="[program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO
  
  directory=/opt/python/current/app
  user=ec2-user
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10
  
  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600
  
  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true
  
  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998
  
  environment=$celeryenv,USER="ec2-user"
  
  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django
  
  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10
  
  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600
  
  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true
  
  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998
  
  environment=$celeryenv"
  
  # Create the celery supervisord conf script
  echo "$celeryconf" | tee /opt/python/etc/celery.conf
  
  # Add configuration script to supervisord conf (if not there already)
  if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
    then
    echo "[include]" | tee -a /opt/python/etc/supervisord.conf
    echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
  fi
  
  # Reread the supervisord config
  echo reread
  supervisorctl -c /opt/python/etc/supervisord.conf reread
  
  echo reload
  # supervisorctl -c /opt/python/etc/supervisord.conf reload
  
  # Update supervisord in cache without restarting all services
  echo update
  supervisorctl -c /opt/python/etc/supervisord.conf update
  
  # Start/Restart celeryd through supervisord
  echo restart
  supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
  supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
  </code></pre>
  
  <p>As it stands right now when I deploy it celery beat will run (verified by sshing in and using ps -aux | less to see the running processes) but no celeryd-worker. In my activity.log I have this.</p>
  
  <pre><code>[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
    + chmod 0755 /var/run/httpd
    + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
    + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
  [2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
    Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
  [2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook] : Starting activity...
  [2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
  [2017-05-30T17:23:37.675Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
    [program:celeryd-worker]
    ; Set full path to celery program if using virtualenv
    command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO
  
    directory=/opt/python/current/app
    user=ec2-user
    numprocs=1
    stdout_logfile=/var/log/celery-worker.log
    stderr_logfile=/var/log/celery-worker.log
    autostart=true
    autorestart=true
    startsecs=10
  
    ; Need to wait for currently executing tasks to finish at shutdown.
    ; Increase this if you have very long running tasks.
    stopwaitsecs = 600
  
    ; When resorting to send SIGKILL to the program to terminate it
    ; send SIGKILL to its whole process group instead,
    ; taking care of its children as well.
    killasgroup=true
  
    ; if rabbitmq is supervised, set its priority higher
    ; so it starts first
    priority=998
  
    environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  
    [program:celeryd-beat]
    ; Set full path to celery program if using virtualenv
    command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django
  
    directory=/opt/python/current/app
    user=nobody
    numprocs=1
    stdout_logfile=/var/log/celery-beat.log
    stderr_logfile=/var/log/celery-beat.log
    autostart=true
    autorestart=true
    startsecs=10
  
    ; Need to wait for currently executing tasks to finish at shutdown.
    ; Increase this if you have very long running tasks.
    stopwaitsecs = 600
  
    ; When resorting to send SIGKILL to the program to terminate it
    ; send SIGKILL to its whole process group instead,
    ; taking care of its children as well.
    killasgroup=true
  
    ; if rabbitmq is supervised, set its priority higher
    ; so it starts first
    priority=998
  
    environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
    reread
    No config updates to processes
    reload
    update
    restart
    celeryd-beat: stopped
    celeryd-beat: started
    celeryd-worker: ERROR (not running)
    celeryd-worker: ERROR (abnormal termination)
  </code></pre>
  
  <p>so there is something wrong with when I try and start the worker but not with beat.</p>
  
  <p>If I SSH into the EB and look at the /var/log/celery-worker.log file I see this.</p>
  
  <pre><code>[2017-05-30 12:30:21,370: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
  Traceback (most recent call last):
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 20, in get_client
      return hub._current_http_client
  AttributeError: 'Hub' object has no attribute '_current_http_client'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
      self.blueprint.start(self)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
      step.start(parent)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
      return self.obj.start()
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
      blueprint.start(self)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
      step.start(parent)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
      c.loop(*c.loop_args())
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
      next(loop)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/hub.py", line 282, in create_loop
      item()
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 139, in __call__
      return self.throw()
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 136, in __call__
      retval = fun(*final_args, **final_kwargs)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
      queue, callback=promise(self._loop1, (queue,)),
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
      return self._get_async(queue, maxcount, callback=callback)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 315, in _get_async
      q, count=count, connection=self.asynsqs,
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 412, in asynsqs
      AsyncSQSConnection, _asynsqs.regions(),
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 400, in _aws_connect_to
      port=port)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/sqs/connection.py", line 39, in __init__
      https_connection_factory=https_connection_factory, **kwargs
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 223, in __init__
      AsyncConnection.__init__(self, http_client, **http_client_params)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 172, in __init__
      self._httpclient = http_client or get_client()
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 22, in get_client
      client = hub._current_http_client = Client(hub, **kwargs)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 13, in Client
      return CurlClient(hub, **kwargs)
    File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/curl.py", line 43, in __init__
      raise ImportError('The curl client requires the pycurl library.')
  ImportError: The curl client requires the pycurl library.
  </code></pre>
  
  <p>Which I also encountered when doing this on my local machine. Although I have made sure that curl is installed using nss with this container command.</p>
  
  <pre><code>container_commands:
    01_libcurl_ssl_backend:
      command: "export PYCURL_SSL_LIBRARY=nss"
      leader_only: true
  </code></pre>
  
  <p>Which is what pycurl is looking for. I also have pycurl in my requirements file and if I pip freeze I can see that pycurl is installed. I fixed this problem locally by recreating my virtual environment. But not quite sure what to do here.</p>
  
  <p><strong>Edit 1</strong>
  I added in </p>
  
  <pre><code>PYCURL_SSL_LIBRARY: nss
  </code></pre>
  
  <p>to my options settings in my config files and was able to deploy and have a slightly different behaviour show up. When looking at the activity log after. </p>
  
  <p>Instead of</p>
  
  <pre><code>update
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)
  </code></pre>
  
  <p>I got:</p>
  
  <pre><code>    reread
    celeryd-beat: changed
    celeryd-worker: changed
    reload
    update
    celeryd-beat: stopped
    celeryd-beat: updated process group
    celeryd-worker: stopped
    celeryd-worker: updated process group
    restart
    celeryd-beat: stopped
    celeryd-beat: started
    celeryd-worker: stopped
    celeryd-worker: ERROR (abnormal termination)
  </code></pre>
  
  <p>but the celery-worker.log still has the same error in it about pycurl</p>