title: celery worker thread hangs indefinitely
body: <p>I am running my spiders into celery worker. Spider scrape a website and then bunch of follow-up links. after some time spider stop processing any further.</p>
  <p>lsof output shows that for thread , connection are in CLOSE_WAIT state</p>
  <pre><code># lsof -i -n
  COMMAND PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
  celery   10 root   32u  IPv4 105621511      0t0  TCP 127.0.0.1:6023 (LISTEN)
  celery   10 root   33u  IPv4 105603949      0t0  TCP 10.1.195.250:38162-&gt;104.17.38.150:http (ESTABLISHED)
  celery   10 root   34u  IPv4 105610494      0t0  TCP 10.1.195.250:41864-&gt;185.230.61.195:https (CLOSE_WAIT)
  celery   10 root   35u  IPv4 105614120      0t0  TCP 10.1.195.250:39742-&gt;185.230.61.195:http (CLOSE_WAIT)
  celery   10 root   36u  IPv4 105603950      0t0  TCP 10.1.195.250:52672-&gt;185.230.61.96:http (CLOSE_WAIT)
  celery   10 root   37u  IPv4 105620542      0t0  TCP 10.1.195.250:38200-&gt;209.236.228.178:http (CLOSE_WAIT)
  celery   10 root   38u  IPv4 105603948      0t0  TCP 10.1.195.250:51848-&gt;35.208.181.87:http (CLOSE_WAIT)
  celery   10 root   39u  IPv4 105614124      0t0  TCP 10.1.195.250:56290-&gt;185.230.61.96:https (CLOSE_WAIT)
  celery   10 root   40u  IPv4 105604983      0t0  TCP 10.1.195.250:43118-&gt;216.185.90.112:http (CLOSE_WAIT)
  celery   10 root   41u  IPv4 105618465      0t0  TCP 10.1.195.250:55006-&gt;209.59.212.167:http (CLOSE_WAIT)
  celery   10 root   45u  IPv4 105600888      0t0  TCP 10.1.195.250:34572-&gt;23.227.38.74:http (ESTABLISHED)
  celery   10 root   46u  IPv4 105620539      0t0  TCP 10.1.195.250:35846-&gt;205.178.189.129:http (CLOSE_WAIT)
  celery   10 root   48u  IPv4 105620541      0t0  TCP 10.1.195.250:39674-&gt;185.230.61.195:http (CLOSE_WAIT)
  celery   10 root   49u  IPv4 105610495      0t0  TCP 10.1.195.250:49450-&gt;178.128.150.108:http (CLOSE_WAIT)
  celery   10 root   51u  IPv4 105614122      0t0  TCP 10.1.195.250:53770-&gt;23.227.38.74:https (ESTABLISHED)
  celery   10 root   52u  IPv4 105614123      0t0  TCP 10.1.195.250:52930-&gt;54.86.91.237:https (CLOSE_WAIT)
  celery   10 root   53u  IPv4 105614125      0t0  TCP 10.1.195.250:37998-&gt;209.236.228.178:https (CLOSE_WAIT)
  celery   10 root   54u  IPv4 105614126      0t0  TCP 10.1.195.250:59992-&gt;35.208.181.87:https (CLOSE_WAIT)
  celery   10 root   55u  IPv4 105605002      0t0  TCP 10.1.195.250:39692-&gt;192.124.249.18:http (CLOSE_WAIT)
  celery   10 root   56u  IPv4 105612653      0t0  TCP 10.1.195.250:41912-&gt;185.230.61.195:https (CLOSE_WAIT)
  celery   10 root   57u  IPv4 105612657      0t0  TCP 10.1.195.250:47560-&gt;104.197.82.118:http (CLOSE_WAIT)
  celery   10 root   58u  IPv4 105612656      0t0  TCP 10.1.195.250:33926-&gt;209.59.212.167:https (CLOSE_WAIT)
  celery   10 root   59u  IPv4 105614129      0t0  TCP 10.1.195.250:41614-&gt;178.128.150.108:https (CLOSE_WAIT)
  celery   10 root   62u  IPv4 105614131      0t0  TCP 10.1.195.250:37534-&gt;34.66.87.174:http (CLOSE_WAIT)
  celery   10 root   63u  IPv4 105600910      0t0  TCP 10.1.195.250:47682-&gt;166.62.115.136:https (CLOSE_WAIT)
  celery   10 root   64u  IPv4 105614141      0t0  TCP 10.1.195.250:43222-&gt;216.185.90.112:http (CLOSE_WAIT)
  celery   10 root   65u  IPv4 105600912      0t0  TCP 10.1.195.250:41060-&gt;50.63.7.227:http (CLOSE_WAIT)
  celery   10 root   66u  IPv4 105600913      0t0  TCP 10.1.195.250:41254-&gt;104.197.82.118:https (CLOSE_WAIT)
  celery   10 root   69u  IPv4 105614695      0t0  TCP 10.1.195.250:42766-&gt;104.112.162.8:https (ESTABLISHED
  </code></pre>
  <p>ps -aux shows that thread is in sleep and waiting for an event</p>
  <pre><code># ps -aux
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root         1  0.1  0.0  80024 62700 ?        Ss   17:23   0:05 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root         8  0.0  0.0 118892 76360 ?        S    17:23   0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root        10  0.0  0.0 902592 100916 ?       Sl   17:23   0:01 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root       485  0.0  0.0 121900 79376 ?        S    18:07   0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root       486 10.0  0.1 950312 144056 ?       Sl   18:07   1:19 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root       501  0.4  0.0 455868 62432 ?        Sl   18:11   0:02 /usr/local/bin/python /usr/local/bin/celery flower -A 
  root       508  0.3  0.0 121916 79388 ?        S    18:17   0:00 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root       509 22.4  0.1 958724 154876 ?       Sl   18:17   0:42 /usr/local/bin/python /usr/local/bin/celery -A data_ex
  root       520  0.5  0.0   2388   700 pts/0    Ss   18:20   0:00 /bin/sh
  root       526  0.0  0.0   9392  3048 pts/0    R+   18:20   0:00 ps -aux
  </code></pre>
  <p>Starce shows that thread is waiting on fd 69</p>
  <pre><code># strace -p 10  
  strace: Process 10 attached
  read(69, 
  </code></pre>
  <p>Seems like spider are not closing connection properly.</p>
  <p>how do i solve this?</p>
  <ul>
  <li>I thought of adding timeouts to celery task , but all thread will hit HARD_LIMIT eventually.</li>
  <li>how to make sure that scrapy is closing each connection propery?</li>
  </ul>