title: Celery, Redis and ConnectionPool
body: <p>I recently had a problem with Celery and Redis : I was using too many connections to my cloud Redis account.</p>
  
  <p>I seem to have fixed the problem for now with better settings and a bigger Redis plan.</p>
  
  <p>However, it lead me to some experimentation on how Redis and Celery work together, and there is something I don't understand : the number of Redis <code>ConnectionPool</code> that are created by the Python Redis module.</p>
  
  <p>I configured Celery to use Redis as a broker and a result backend.</p>
  
  <p>Here is my Celery config :</p>
  
  <pre class="lang-py prettyprint-override"><code>CELERY_TIMEZONE = 'Europe/Paris'
  CELERY_BROKER_URL = REDIS_URL
  CELERY_RESULT_BACKEND = REDIS_URL
  CELERY_TASK_SERIALIZER = 'pickle'
  CELERY_SEND_EVENTS = False
  CELERY_IMPORTS = ('task1', 'task2')
  CELERY_ACCEPT_CONTENT = ['pickle']
  CELERY_REDIS_MAX_CONNECTIONS = 2
  BROKER_POOL_LIMIT = 1
  BROKER_HEARTBEAT = None
  BROKER_TRANSPORT_OPTIONS = {
     'max_connections': 30
  }
  </code></pre>
  
  <p>I run Celery with the following command :</p>
  
  <p><code>celery worker --without-gossip --without-mingle --without-heartbeat --concurrency=1 --app=backZest.celery</code></p>
  
  <p>Right after Celery has booted, I already have <code>10</code> connections to my Redis instance : is that normal?</p>
  
  <p>I checked how many <code>redis</code> <code>ConnectionPool</code> were created : to do this, I just modified the Python Redis module <code>connection.py</code> file, in the <code>__init__</code> method of the <code>ConnectionPool</code> class:</p>
  
  <pre class="lang-py prettyprint-override"><code>import sys
  print "ConnectionPool instance %s with %s max_connections" % (id(self), self.max_connections)
  sys.stdout.flush()
  </code></pre>
  
  <p>As you can see, there are many (8) pools created, all of them with the same max_connections, 30, which is my setting in <code>BROKER_TRANSPORT_OPTIONS</code> :</p>
  
  <pre><code>ConnectionPool instance 4412957392 with 30 max_connections
  ConnectionPool instance 4412959888 with 30 max_connections
  ConnectionPool instance 4420369616 with 30 max_connections
  ConnectionPool instance 4420370320 with 30 max_connections
  ConnectionPool instance 4420367056 with 30 max_connections
  ConnectionPool instance 4420491792 with 30 max_connections
  ConnectionPool instance 4422318224 with 30 max_connections
  ConnectionPool instance 4422319504 with 30 max_connections
  </code></pre>
  
  <p>I don't understand why there are so many of them. I want to control the number of connections to my Redis. I can control the number of max_connections per <code>ConnectionPool</code> but isn't it a bit useless if I don't control the number of <code>ConnectionPool</code> created ?</p>
  
  <p>Thanks a lot for your help !</p>