title: Slow Celery Task Times
body: <p>I'm using <a href="https://www.djangoproject.com/" rel="nofollow">Django</a>, <a href="http://www.celeryproject.org/" rel="nofollow">Celery</a> and <a href="http://www.rabbitmq.com/" rel="nofollow">RabbitMQ</a>. I have a simple task that sends emails. This task works, but its very slow. </p>
  
  <p>For example, I send <strong>5000 emails</strong>, all 5000 emails go straight to <strong>RabbitMQ</strong> as normal but once in the message broker it then proceeds to takes around 30 minutes to complete and clear all tasks. </p>
  
  <p>Without Celery these same tasks would take just a few minutes to process all 5000 tasks.</p>
  
  <p>Have I missed configured something? It would be very helpful if someone could spot my speed issue.</p>
  
  <p>task.py</p>
  
  <pre><code>class SendMessage(Task):
      name = "Sending SMS"
      max_retries = 10
      default_retry_delay = 3
  
      def run(self, message_id, gateway_id=None, **kwargs):
          logging.debug("About to send a message.")
  
  
          try:
              message = Message.objects.get(pk=message_id)
          except Exception as exc:
              raise SendMessage.retry(exc=exc)
  
          if not gateway_id:
              if hasattr(message.billee, 'sms_gateway'):
                  gateway = message.billee.sms_gateway
              else:
                  gateway = Gateway.objects.all()[0]
          else:
              gateway = Gateway.objects.get(pk=gateway_id)
  
          account = Account.objects.get(user=message.sender)
          if account._balance() &gt;= message.length:
              response = gateway._send(message)
  
              if response.status == 'Sent':
                  # Take credit from users account.
                  transaction = Transaction(
                      account=account,
                      amount=- message.charge,
  
                  )
                  transaction.save()
                  message.billed = True
                  message.save()
          else:
              pass
  </code></pre>
  
  <p>settings.py</p>
  
  <pre><code># Celery
  BROKER_URL = 'amqp://admin:xxxxxx@xx.xxx.xxx.xxx:5672//'
  CELERY_SEND_TASK_ERROR_EMAILS = True
  </code></pre>
  
  <p>Apache config</p>
  
  <pre><code>&lt;VirtualHost *:80&gt;
  ServerName www.domain.com
  
  DocumentRoot /srv/project/domain
  
  
  WSGIDaemonProcess domain.com processes=2 threads=15 display-name=%{GROUP}
  WSGIProcessGroup domain.com
  
  WSGIScriptAlias / /srv/project/domain/apache/django.wsgi
  ErrorLog /srv/project/logs/error.log
  &lt;/VirtualHost&gt;
  </code></pre>
  
  <p>conf</p>
  
  <pre><code># Name of nodes to start, here we have a single node
  #CELERYD_NODES="w1"
  # or we could have three nodes:
  CELERYD_NODES="w1 w2 w3"
  
  # Where to chdir at start.
  CELERYD_CHDIR="/srv/project/domain"
  
  # How to call "manage.py celeryd_multi"
  CELERYD_MULTI="$CELERYD_CHDIR/manage.py celeryd_multi"
  
  # How to call "manage.py celeryctl"
  CELERYCTL="$CELERYD_CHDIR/manage.py celeryctl"
  
  # Extra arguments to celeryd
  CELERYD_OPTS="--time-limit=900 --concurrency=8"
  
  # %n will be replaced with the nodename.
  CELERYD_LOG_FILE="/srv/project/logs/celery/%n.log"
  CELERYD_PID_FILE="/srv/project/celery/%n.pid"
  
  # Workers should run as an unprivileged user.
  CELERYD_USER="root"
  CELERYD_GROUP="root"
  
  # Name of the projects settings module.
  export DJANGO_SETTINGS_MODULE="domain.settings"
  
  # Celery Beat Settings.
  
  # Where to chdir at start.
  CELERYBEAT_CHDIR="/srv/project/domain"
  
  # Path to celerybeat
  CELERYBEAT="$CELERYBEAT_CHDIR/manage.py celerybeat"
  </code></pre>