title: Scrapy applying download_delay to selenium
body: <p>I am using scrapy with Selenium, here's the spider outline</p>
  
  <pre><code>class MySpider(spider.Spider):
  
      name = "myname"
      start_urls = ["example.com"]
  
      def __init__(self, *args, *kwargs):
           self.driver = webdriver.PhantomJS()
           super(MySpider, self).__init__(*args, **kwargs)
  
      def parse(self, response):
          # ignore response, work with selenium instead
          source = self.driver.get(response.url).page_source
          sel = Selector(text=source)
          # now use xpath/css selectors on sel and yield item
          ...
  </code></pre>
  
  <p>The problem is I was getting 111 Connection Error after a varying number of pages were crawled (sometimes 100, sometimes 200) so I implemented download_delay and autothrottling which is pretty cool. </p>
  
  <p>But selenium doesn't talk to the middleware and I'm not really using the response returned from the middleware. So I will have to implement my own download_delay. </p>
  
  <p>Unfortunately, I know very little about twisted, concurrent programming and the like, so I'm not sure is something which is the equivalent of <code>time.sleep(secs)</code> before calling <code>self.driver.get</code> would be reasonable (blocking code?).</p>
  
  <p>How do you implement download delays when using scrapy with selenium to parse the responses? </p>