title: Celery + RabbitMQ becomes slow
body: <p>I'm here with a performance issue that I can't seem to figure it out.</p>
  
  <p>The problem is that executing tasks is too slow. Based on the celery log most of the tasks are finished under 0.3 seconds. </p>
  
  <p>I noticed that if I stop the workers and start them again the performance increases, almost up to 200 acks / second, then, after a while it becomes much slower, around 40/s. </p>
  
  <p>I'm not sure but I think it might be a broker issue rather than a celery issue.
  Looking at the log of a couple of workers I noticed that they all seem to execute tasks, then stop for a bit and start again.</p>
  
  <p>It feels like receiving tasks is slow.</p>
  
  <p>Any ideas about what might cause this ? Thanks !</p>
  
  <hr>
  
  <p>A log example:</p>
  
  <p><code>Task drones.tasks.blue_drone_process_task[64c0a826-aa18-4226-8a39-3a455e0916a5] succeeded in 0.18421914400005335s: None</code></p>
  
  <p>10 seconds break</p>
  
  <p><code>Received task: drones.tasks.blue_drone_process_task[924a1b99-670d-492e-94a1-91d5ff7142b9]
  Received task: drones.tasks.blue_drone_process_task[74a9a1d3-aa2b-40eb-9e5a-1420ea8b13d1]
  Received task: drones.tasks.blue_drone_process_task[99ae3ca1-dfa6-4854-a624-735fe0447abb]
  Received task: drones.tasks.blue_drone_process_task[dfbc0d65-c189-4cfc-b6f9-f363f25f2713]</code></p>
  
  <p>IMO those tasks should execute so fast that I shouldn't be able to read the log.</p>
  
  <hr>
  
  <p>My setup is: </p>
  
  <ul>
  <li>celery 4.2.1</li>
  <li>RabbitMQ 3.7.8</li>
  <li>Erlang 21.1</li>
  </ul>
  
  <p>I use this setup for web scraping, have 2 queue. Let's call them Requests and Process.</p>
  
  <p>In the Requests queue I URLs that need to be scraped and in the Process queue will find the URL + source code of that page. (max 2.5 MB / source page, I drop it in case it's bigger than that), so all messages in the Process queue are max 2.5MB ± 1KB.</p>
  
  <p>To execute tasks from the Requests queue I use celery with the gevent pool, concurrency 300. (-P gevent -c 300 --without-gossip --without-mingle --without-heartbeat). 4-8 workers like this. </p>
  
  <p>To execute tasks from the Process queue I use the prefork pool (default). (-c 4 --without-gossip --without-mingle --without-heartbeat). 30 workers like this.</p>
  
  <hr>
  
  <p>Other setup info:</p>
  
  <ul>
  <li>disabled heartbeats in celery and RabbitMQ, use TCP keep-alive</li>
  <li>everything is in AWS</li>
  <li>c4.xlarge instances for workers</li>
  <li>i3.xlarge for RabbitMQ (30GB RAM, 765 NVMe SSD, 4 cores)</li>
  <li>haproxy for load balancing (I had 2 x RabbitMQ clustered for HA, fully replicated, stopped one thinking that might cause the issue, but I left the load balancer in case I decide to recreate the cluster)</li>
  </ul>
  
  <hr>
  
  <p>RabbitMQ config:</p>
  
  <ul>
  <li>hearbeat = 0</li>
  <li>lazy_queue_explicit_gc_run_operation_threshold = 500</li>
  <li>proxy-protocol = true</li>
  <li>vm_memory_high_watermark = 0.6</li>
  <li>vm_memory_high_watermark_paging_ratio = 0.1</li>
  <li>queue_index_embed_msgs_below = 4096</li>
  </ul>
  
  <hr>
  
  <p>Celery config:</p>
  
  <ul>
  <li>CELERY_TASK_ACKS_LATE = false (tried both ways)</li>
  <li>CELERY_RESULT_BACKEND = None</li>
  <li>CELERY_WORKER_ENABLE_REMOTE_CONTROL = True</li>
  <li>BROKER_HEARTBEAT = 0</li>
  <li>CELERY_CONTROL_QUEUE_EXPIRES = 60</li>
  <li>CELERY_BROKER_CONNECTION_TIMEOUT = 30</li>
  <li>CELERY_WORKER_PREFETCH_MULTIPLIER = 1</li>
  <li>workers running with Ofair</li>
  <li>max-tasks-per-child = 10 (tried without it as well)</li>
  </ul>
  
  <p>Tried using a higher prefetch, like 5, 10 and 20 and it did not work.</p>
  
  <p><a href="https://i.stack.imgur.com/c12nc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/c12nc.png" alt="In case this helps"></a></p>