title: Scrapy - set delay to retry middleware
body: <p>I'm using <code>Scrapy-splash</code> and I have a problem with memory. I can clearly see that memory used by <code>docker</code> <code>python3</code> is gradually increasing until PC freezes. </p>
  
  <p>Can't figure out why it behaves this way because I have <code>CONCURRENT_REQUESTS=3</code> and there is no way 3 <code>HTML</code> consumes 10GB RAM.</p>
  
  <p>So there is a workaround to set <code>maxrss</code> to some reasonable value. When RAM usage has this value, docker is restarted so RAM is flushed. </p>
  
  <p>But the problem is that for the time <code>docker</code> is down, <code>scrapy</code> continues sending requests so there is a couple of <code>urls</code> not scraped. Retry middleware is trying to retry these requests right now and then give up.</p>
  
  <pre><code>[scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://www.ex.com/eiB3t/ via http://127.0.0.1:8050/execute&gt; (failed 2 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.&gt;]
  2019-03-30 14:28:33 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;GET https://www.ex.com/eiB3t/
  </code></pre>
  
  <p>So I have two questions</p>
  
  <ol>
  <li>Do you know a better solution?</li>
  <li>If not, how can I set <code>Scrapy</code> to <code>retry</code> request after some time (let's say on minute so <code>docker</code> has time to restart)?</li>
  </ol>