title: Celery difference between concurrency, workers and autoscaling
body: <p>In my <code>/etc/defaults/celeryd</code> config file, I've set:</p>
  
  <pre><code>CELERYD_NODES="agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8"
  CELERYD_OPTS="--autoscale=10,3 --concurrency=5"
  </code></pre>
  
  <p>I understand that the daemon spawns 8 celery workers, but I'm fully not sure what <code>autoscale</code> and <code>concurrency</code> do together. I thought that concurrency was a way to specify the max number of threads that a worker can use and autoscale was a way for the worker to scale up and down child workers, if necessary. </p>
  
  <p>The tasks have a largish payload (some 20-50kB) and there are like 2-3 million such tasks, but each task runs in less than a second. I'm seeing memory usage spike up because the broker distributes the tasks to every worker, thus replicating the payload multiple times. </p>
  
  <p>I think the issue is in the config and that the combination of workers + concurrency + autoscaling is excessive and I would like to get a better understanding of what these three options do.</p>