title: Is it possible to run the python script automatically from the same position where the connection lost?
body: <p>How do I make a web scraping script that will run automatically from the same position where the connection lost?.</p>
  <p>Let's suppose we run a scrapy web scraping script and we have to scrape 100 pages and after 50 pages suddenly I lost my internet connection.</p>
  <p>So how can I modify my code to run again for the remaining  50 pages?</p>
  <pre><code># -*- coding: utf-8 -*-
  from scrapy import Spider
  from scrapy.http import Request
  from scrapy.loader import ItemLoader
  from books_crawler.items import BooksCrawlerItem
  
  
  class BooksSpider(Spider):
      name = 'books'
      allowed_domains = ['books.toscrape.com']
      start_urls = ['http://books.toscrape.com']
  
      def parse(self, response):
          books = response.xpath('//h3/a/@href').extract()
          for book in books:
              absolute_url = response.urljoin(book)
              yield Request(absolute_url, callback=self.parse_book)
  
          # process next page
          next_page_url = response.xpath('//a[text()=&quot;next&quot;]/@href').extract_first()
          absolute_next_page_url = response.urljoin(next_page_url)
          yield Request(absolute_next_page_url)
  
      def parse_book(self, response):
          l = ItemLoader(item=BooksCrawlerItem(), response=response)
  
          title = response.css('h1::text').extract_first()
          price = response.xpath('//*[@class=&quot;price_color&quot;]/text()').extract_first()
  
          image_urls = response.xpath('//img/@src').extract_first()
          image_urls = image_urls.replace('../..', 'http://books.toscrape.com/')
  
          l.add_value('title', title)
          l.add_value('price', price)
          l.add_value('image_urls', image_urls)
  
          return l.load_item()
  </code></pre>