title: How to avoid SECRET_KEY error when starting Celery using Supervisor?
body: <p>I have a Django 1.62 application running on Debian 7.8 with Nginx 1.2.1 as my proxy server and Gunicorn 19.1.1 as my application server.  I've installed Celery 3.1.7 and RabbitMQ 2.8.4 to handle asynchronous tasks.  I'm trying to manage my various applications, particularly Celery, using Supervisor 3.0a8.  The problem is that when I try to start Celery via Supervisor, I get this error:</p>
  
  <pre><code>ImproperlyConfigured: The SECRET_KEY setting must not be empty.
  </code></pre>
  
  <p>(I show the entire stacktrace at the bottom.)</p>
  
  <p>All of my configuration files are kept in a "conf" directory that sits just below my "myproj" project directory like this:</p>
  
  <pre><code>conf
  ├── celeryconfig.py
  ├── celeryconfig.pyc
  ├── celery.py
  ├── __init__.py
  ├── middleware.py
  ├── settings
  │   ├── base.py
  │   ├── dev.py
  │   ├── __init__.py
  │   ├── prod.py
  ├── urls.py
  ├── wsgi.py
  </code></pre>
  
  <p>My production Django settings are kept in prod.py settings file which inherits my base.py base settings.  I keep the secret key in my virtual environment's postactivate file and I read it into my production settings as shown below.  I've verified via the Python interpreter that the secret key is present in my production settings.</p>
  
  <pre><code># conf/settings/prod.py
  from conf.settings.base import *
  ...
  # get_env_variable is defined in base.py
  SECRET_KEY = get_env_variable("SECRET_KEY")
  </code></pre>
  
  <p>This is the get_env_variable function which reads the secret key in:</p>
  
  <pre><code># conf/settings/base.py
  def get_env_variable(var_name):
      try:
          return os.environ[var_name]
      except KeyError:
          error_msg = "Set the %s environment variable" % var_name
          raise ImproperlyConfigured(error_msg)
  </code></pre>
  
  <p>This is my Supervisor configuration file.  It's based on the sample file show in the <a href="https://github.com/celery/celery/blob/3.1/extra/supervisord/celeryd.conf" rel="nofollow noreferrer">Celery documentation</a>:</p>
  
  <pre><code># /etc/supervisor/conf.d/myproj.conf
  [program:myproj]
  command = /www/myproj/bin/start-gunicorn
  user = root
  stdout_logfile = /var/log/gunicorn/supervisor.log
  redirect_stderr = true
  
  [program:celery]
  directory=/www/myproj
  command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery/celery.log
  stderr_logfile=/var/log/celery/celery.log
  autostart=true
  autorestart=true
  startsecs=10
  stopwaitsecs=600
  killasgroup=true
  priority=998
  </code></pre>
  
  <p>Here's how I loaded the new Celery configuration using Supervisor:</p>
  
  <pre><code>sudo service supervisor stop
  sudo service supervisor start
  </code></pre>
  
  <p>This is my Celery application file:</p>
  
  <pre><code># conf/celery.py
  from __future__ import absolute_import
  import os
  from celery import Celery
  from django.conf import settings
  from conf import celeryconfig
  
  os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
  app = Celery('conf')
  app.config_from_object(celeryconfig)
  app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
  </code></pre>
  
  <p>This is my Celery configuration file:</p>
  
  <pre><code># conf/celeryconfig.py
  BROKER_URL = 'amqp://guest@localhost:5672//'
  CELERY_RESULT_BACKEND = 'amqp'
  CELERY_ACCEPT_CONTENT = ['json', ]
  CELERY_TASK_SERIALIZER = 'json'
  CELERY_RESULT_SERIALIZER = 'json'
  CELERY_TASK_RESULT_EXPIRES = 3600
  CELERY_SEND_TASK_ERROR_EMAILS = True
  </code></pre>
  
  <p>Per the Celery documentation, I've modified __init__.py:</p>
  
  <pre><code># conf/__init__.py
  from __future__ import absolute_import
  from .celery import app as celery_app
  </code></pre>
  
  <p>I can start Celery by hand using the following command and it starts up just fine:</p>
  
  <pre><code>workon myproj  # Activate project's virtual environment
  celery worker -A conf -l info
  </code></pre>
  
  <p>However, when I try to start it via Supervisor, I get the error I've described.  Thinking that maybe Supervisor couldn't access the Django secret key because it's an environment variable, I tried hard-coding the key in my prod.py settings file (instead of reading it in via the get_env_variable function) but that didn't fix the problem.</p>
  
  <p>I tried combining all my settings into the prod.py settings file that contained the actual secret key but that didn't help.</p>
  
  <p>I also tried adding this environment parameter to the Supervisor config file as show below but this didn't fix the problem:</p>
  
  <pre><code># /etc/supervisor/conf.d/myproj.conf
  ...
  [program:celery]
  environment=SECRET_KEY="(my secret key)"
  directory=/www/myproj
  command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
  user=nobody
  ...
  </code></pre>
  
  <p>I tried setting the "user" to the name of the user that I start the website as but that didn't help either.</p>
  
  <pre><code># /etc/supervisor/conf.d/myproj.conf
  ...
  [program:celery]
  environment=SECRET_KEY="(my secret key)"
  directory=/www/myproj
  command=/home/myproj/venv/myproj/bin/celery worker --app=conf -l debug
  user=myproj
  ...
  </code></pre>
  
  <p>I read about a library called <a href="https://github.com/rfk/django-supervisor" rel="nofollow noreferrer">django-supervisor</a> that is supposed to ease the integration between Django and Supervisor but I get the same error if I use that library.</p>
  
  <p>Finally I read <a href="https://stackoverflow.com/questions/19054008/how-to-use-environment-variables-with-supervisor-gunicorn-and-django-1-6">here on SO</a> that if your secret key contains a "%" sign, Supervisor won't like it.  I noticed that my key did contain one "%" sign so I escaped it like this "%%" but that didn't fix the problem either.</p>
  
  <p>Can anyone see what I'm doing wrong?  I've seen other questions here from users that were encountering this same error under different circumstances but I tried implementing the various solutions that were discussed and none of them fixed the problem.</p>
  
  <p>Thanks very much for your ideas.  And sorry for the long question but this problem has a lot of moving parts.</p>
  
  <p>Here is the entire stacktrace:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/myproj/venv/myproj/bin/celery", line 11, in &lt;module&gt;
      sys.exit(main())
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
      main()
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
      cmd.execute_from_commandline(argv)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
      super(CeleryCommand, self).execute_from_commandline(argv)))
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/base.py", line 307, in execute_from_commandline
      return self.handle_argv(self.prog_name, argv[1:])
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
      return self.execute(command, argv)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
      ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
      return self(*args, **options)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/base.py", line 270, in __call__
      ret = self.run(*args, **kwargs)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
      state_db=self.node_format(state_db, hostname), **kwargs
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 95, in __init__
      self.app.loader.init_worker()
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in init_worker
      self.import_default_modules()
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/loaders/base.py", line 116, in import_default_modules
      signals.import_modules.send(sender=self.app)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 166, in send
      response = receiver(signal=self, sender=sender, **named)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 69, in on_import_modules
      self.worker_fixup.validate_models()
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 322, in __get__
      value = obj.__dict__[self.__name__] = self.__get(obj)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 64, in worker_fixup
      self._worker_fixup = DjangoWorkerFixup(self.app)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/fixups/django.py", line 99, in __init__
      self._cache = import_module('django.core.cache')
    File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
      __import__(name)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/core/cache/__init__.py", line 69, in &lt;module&gt;
      if DEFAULT_CACHE_ALIAS not in settings.CACHES:
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 54, in __getattr__
      self._setup(name)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in _setup
      self._wrapped = Settings(settings_module)
    File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 151, in __init__
      raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
  django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
  </code></pre>