title: Scrapy - bug in custom DownloaderMiddleware
body: <p>I have list of thousands of <code>URL</code> which I scrape using one <code>Spider</code>. Some <code>URL</code>s has the same domain. I want to count a number of Timeout errors per domain. If for domain x, is a number of Timeouts higher than LIMIT, I want to avoid scraping all <code>URL</code>s of this domain. Instead, I want to treat them as it were normal Timeout (without real requesting) - my spider then handle Timeouts in <code>errback</code> function.</p>
  
  <p>So the point is that I don't have to keep trying to scrape URLs from domain X if this domain raised 5 Timeouts already. I just want to fake those Timeouts without actually sending requests.</p>
  
  <p>I've created a middleware for this purpose, which seems to be a good way but it has some bugs.</p>
  
  <p>For example, if an <code>URL</code> has to be automatically Timeouted (without real request), it seems that scrapy tries to repeat this request multiple times. </p>
  
  <p>LOG:</p>
  
  <pre><code>2017-05-22 20:37:16 [engineapp.engine.scrapy.scrapytest.scrapytest.middlewares] INFO: Arbitrary Timeout! OCID: 12751
  2017-05-22 20:37:16 [engineapp.engine.scrapy.scrapytest.scrapytest.middlewares] INFO: Arbitrary Timeout! OCID: 12751
  2017-05-22 20:37:16 [engineapp.engine.scrapy.scrapytest.scrapytest.middlewares] INFO: Arbitrary Timeout! OCID: 12751
  </code></pre>
  
  <p>Maybe the problem is that I'm not returning anything from <code>process_request</code> and <code>process_exception</code> functions.</p>
  
  <p>This is the middleware:</p>
  
  <pre><code>class TimeoutProcessMiddleware():
      timeouts = defaultdict(int)
  
      def process_request(self,request, spider):
          logger.debug(self.timeouts)
          occ = request.meta['occ']
          domain = Site.parse_site_from_url(request.url)
  
          if self.timeouts[domain] &gt;= settings.TIMEOUT_COUNT_LIMIT: # If we detected timeout for more than x URLs with the same domain
              logger.info('Arbitrary Timeout! OCID: {}'.format(occ.id))
              raise TimeoutError()
  
      def process_exception(self,request, exception, spider):
          logger.debug('Processing exception: EXC: {}'.format(exception))
          if exception.__class__ in [TimeoutError,TCPTimedOutError]:
              domain = Site.parse_site_from_url(request.url)
              self.timeouts[domain]+=1
  </code></pre>
  
  <p>This is a part of Spider errback:</p>
  
  <pre><code>class MainSpider(scrapy.Spider):
       ...
  
       def err(self, failure):
              if failure.check(HttpError):
                  ....
              elif failure.check(TimeoutError, TCPTimedOutError): # HERE I WANT TO PROCESS ALL TIMEOUTS 
                  self.process_timeout_error(occ)
  </code></pre>
  
  <p>Do you know what to do?</p>