title: Scrapy crawled but not scraped
body: <p>I'm scraping data from <a href="https://www.openingstijden.nl/" rel="nofollow noreferrer">this website</a> and have found their ajax source links to have the format
  <code>f&quot;https://www.openingstijden.nl/ajax-resultaten/?page={page_num}&amp;business_id={business_id}&amp;no-limit=1&quot;</code> with page_num being page number and business id being the id of the store. Example: <a href="https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1" rel="nofollow noreferrer">https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1</a>.</p>
  <p>This is my code:</p>
  <pre><code>class SupermarketScraper(scrapy.Spider):
  name = &quot;supermarkets&quot;
  
  def __init__(self):
      self.images = ast.literal_eval(pkg_resources.read_text(resources, 'times.json'))
      self.geodata = set()
      sentry_sdk.init(dsn=SENTRY_SDK_DSN)
      self.session = requests.Session()
  
  def start_requests(self):
      b_ids = ['5688'] #[item['b_id'] for item in business_ids]
      page_num = 1
      for b_id in b_ids:
          url = f'https://www.openingstijden.nl/ajax-resultaten/?page={page_num}&amp;business_id={b_id}&amp;no-limit=1'
          yield scrapy.Request(url=url, callback=self.parse, cb_kwargs=dict(b_id=b_id, page_num=page_num))
              
  def parse(self, response, b_id, page_num):
      items = response.css('li')
      if items: 
          for item in items:
              link = item.css('a.btn::attr(href)').get()
              geodata = item.css('li::attr(data-geo)').get().split(',')
              if (geodata[0], geodata[1]) not in self.geodata:
                  self.geodata.add((geodata[0], geodata[1]))
              else:
                  return
  
              yield response.follow(link, callback=self.parse_store, cb_kwargs=dict(geodata=geodata, b_id=b_id, page_num=page_num))
  
          new_url = f'http://www.openingstijden.nl/ajax-resultaten/?page={page_num+1}&amp;business_id={b_id}&amp;no-limit=1'
          yield response.follow(new_url, callback=self.parse, cb_kwargs=dict(b_id=b_id, page_num=page_num+1))
  
  def parse_store(self, response, geodata, b_id, page_num):
      name = response.css('#est-name::text')[0].get()
      location = response.css('span.locate span::text').getall()
      street, postal_code, city = location[0:3]
  
      location = LocationItem()
      location['street'] = street
      location['postal_code'] = postal_code
      location['city'] = city
      location['lat'] = geodata[0]
      location['lng'] = geodata[1]
  
      item = SupermarketItem()
      item['business_id'] = b_id
      item['name'] = name
      item['location'] = location
      item['page_num'] = page_num
      yield item
  </code></pre>
  <p>The code runs without error, however, it does not scrape all the data and some pages appear to be crawled but no data were scraped, like this:</p>
  <pre><code>2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/Den-Haag/Prinses-Beatrixlaan-590/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/De-Meern/Mereveldplein-13/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/Voorschoten/Schoolstraat-1/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/Den-Haag/Stationsplein-75/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/Brielle/Thoelaverweg-2/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.openingstijden.nl/Albert-Heijn/Oude-Tonge/Dabbehof-34/&gt; (referer: https://www.openingstijden.nl/ajax-resultaten/?page=20&amp;business_id=5688&amp;no-limit=1)
  2020-11-11 09:23:55 [scrapy.core.scraper] DEBUG: Scraped from &lt;200 https://www.openingstijden.nl/Albert-Heijn/Den-Haag/Johanna-Westerdijkplein-64/&gt;
  {'business_id': '5688',
   'location': {'city': 'Den Haag',
   'lat': '52.0674',
   'lng': '4.32342',
   'postal_code': '2521EN',
   'street': 'Johanna Westerdijkplein 64'},
   'name': 'Albert Heijn',
   'page_num': 20}
  </code></pre>
  <p>The last link was properly scraped, however, all the previous links were crawled but nothing returned. No error were thrown and the spider closed without scraping all data.</p>
  <p>I've tried changing my USER_AGENT to <code>USER_AGENT='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'</code> but it didnt work. All these pages also have identical web structure and can be scraped individually at ease.</p>
  <p>Is the server actively blocking scraper? Why are these urls' data missed by the scraper?</p>