title: Scrapy Crawl an spider on scheduled time
body: <p>I want to crawl an spider multiple times on the scheduled time. The next crawl time will be determined after the first crawl finished. Here is my code to do that but the code will be blocked at the first <code>crawler.start()</code> line:</p>
  
  <pre><code>spidersQ = collections.OrderedDict()
  
  class QuotesSpider(scrapy.Spider):
      name = "quotes"
      global spidersQ
      start_urls = [
          "https://www.amazon.com",
      ]
  
      def parse(self, response):
          root = lxml.html.fromstring(response.body)
          lxml_result = root.xpath("(//div[contains(@class,'a-section')]/div[contains(@class,'olpOffer')])[1]")
  
          price = lxml_result[0].text.strip()
          # Now schedule this spider to run again after 5 seconds
          spidersQ[datetime.datetime.now() + datetime.timedelta(seconds=5)] = QuotesSpider
  
  
  def main():
      process = CrawlerProcess({
          'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
      })
  
      process.crawl(QuotesSpider)
      process.start(stop_after_crawl=False)  # the script will block here forever
  
      while True:
          if datetime.datetime.now() &gt; first(spidersQ):
              schedTime, spider = spidersQ.popitem(last=False)
              process.crawl(spider)
              process.start(stop_after_crawl=False)
          else:
              time.sleep(1)
  </code></pre>