title: Repeatedly scrape data from same url with a time delay
body: <p>I am trying to consistently scrape a single webpage over a period of time to monitor some values and how they are changing within that period</p>
  
  <p>I have no experience in web-scraping and I'm a little confused as to why my scrapy spider runs once and hangs.</p>
  
  <p>I am attempting to run the spider through a 'main.py' script as follows:</p>
  
  <pre><code>timecheck = timeit.default_timer()
  
  while (timecheck) &lt; 30:
      if __name__ == "__main__":
          settings = get_project_settings()
          crawler = CrawlerProcess(settings)
          crawler.crawl(mySpider())
          crawler.start(stop_after_crawl=False)
      time.sleep(5)
  
  print("Finished crawl, time elapsed: %s" % str(timecheck))
  </code></pre>
  
  <p>My spider script is as follows:</p>
  
  <pre><code>class poeSpider(scrapy.Spider):
      name = "spidername"
  
      def start_requests(self):
          urls = ["myurl.com/page1"]
          allowed_domains = ["myurl.com"]
  
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse, dont_filter=True)
  
  
      def parse(self,response):
  
          datavar = response.xpath("/mydata").extract()
          print(datavar[:3])
  </code></pre>
  
  <p>PS: A different issue, but related:</p>
  
  <p>I'm also looking to store the data in a list or variable of some kind between scrapes so that I am able to compare the values while the program is running, ideally without any file I/O, which is my reasoning for using a 'main.py' though I'm also at a loss for how to pass the data in the <code>parse</code> function of the spider to anywhere where I can make such comparisons.</p>