title: Scrapy Request callbacks not firing
body: <p>I am using scrapy 0.24 to scrape data from a website. However, I am unable to make any requests from my callback method <code>parse_summary</code>.</p>
  
  <pre><code>class ExampleSpider(scrapy.Spider):
      name = "tfrrs"
      allowed_domains = ["example.org"]
      start_urls = (
          'http://www.example.org/results_search.html?page=0&amp;sport=track&amp;title=1&amp;go=1',
      )
  
      def __init__(self, *args, **kwargs):
          super(TfrrsSpider, self).__init__(*args, **kwargs)
          self.start_urls = ['http://www.example.org/results_search.html?page=0&amp;sport=track'&amp;title=1&amp;go=1',]
              pass
  
      # works without issue
      def parse(self, response):
          races = response.xpath("//table[@width='100%']").xpath(".//a[starts-with(@href, 'http://www.tfrrs.org/results/')]/@href").extract()
          callback = self.parse_trackfieldsummary
          for race in races:
              yield scrapy.Request(race, callback=self.parse_summary)
          pass
  
      # works without issue
      def parse_summary(self, response):
          baseurl = 'http://www.example.org/results/'
          results = response.xpath("//div[@class='data']").xpath('.//a[@style="margin-left: 20px;"]/@href').extract()
          for result in results:
              print(baseurl+result)  # shows that url is correct everytime
              yield scrapy.Request(baseurl+result, callback=self.parse_compiled)
  
      # is never fired or shown in terminal
      def parse_compiled(self, response): 
          print('test')
          results = response.xpath("//table[@style='width: 935px;']")
          print(results)
  </code></pre>
  
  <p>When I intentially make the request in <code>parse_summary</code> fail (due to domain errors, etc), I am able to see the error in the prompt, but when I use the correct url, its as if I am not even calling it. I have also tested the urls being requested in <code>parse_summary</code> in the <code>parse</code> method, where they work as expected. What could be causing them to not be fired in the <code>parse_summary</code> method but successfully in the <code>parse method</code>?  Thank you for your help in advance.</p>
  
  <h2>Edit</h2>
  
  <p>After making some changes to my <code>Spider</code>, I still have the same result. However, it works if I use an entirely new project. So I am guessing it has to do with my project settings.</p>
  
  <p>Here are my project settings (where <code>raceretrieval</code> is the name of my project):</p>
  
  <pre><code>BOT_NAME = 'raceretrieval'
  DOWNLOAD_DELAY= 1
  CONCURRENT_REQUESTS = 100
  SPIDER_MODULES = ['raceretrieval.spiders']
  NEWSPIDER_MODULE = 'raceretrieval.spiders'
  ITEM_PIPELINES = {
          'raceretrieval.pipelines.RaceValidationPipeline':1,
          'raceretrieval.pipelines.RaceDistanceValidationPipeline':2,
  #        'raceretrieval.pipelines.RaceUploadPipeline':9999
      }
  </code></pre>
  
  <p>If I comment out <strong>both</strong> <code>DOWNLOAD_DELAY= 1</code> and <code>CONCURRENT_REQUESTS = 100</code>, the spider works as expected.  Why could this be? I don't understand how they would effect this.</p>