title: How to simulate xhr request using Scrapy when trying to crawl data from an ajax-based webstie?
body: <p>I am new to crawl the webpage using Scrapy and unfortunately chose a dynamic one to start...</p>
  
  <p>I've successfully crawled part (120 links), thanks to someone helping me <a href="https://stackoverflow.com/questions/35304470/scrapy-cant-crawl-all-links-in-a-page">here</a>, but not links in <a href="https://play.google.com/store/apps/collection/topselling_new_free" rel="nofollow noreferrer">target website</a></p>
  
  <p>After doing some research, I know that crawling ajax web is nothing different from those simple ideas:</p>
  
  <p>•open browser developer tools, network tab</p>
  
  <p>•go to the target site</p>
  
  <p>•click submit button and see what XHR request is going to the server</p>
  
  <p>•simulate this XHR request in your spider</p>
  
  <p>The last one sounds obscure to me though---How to simulate XHR request? </p>
  
  <p>I've seen someone using 'headers' or 'formdata' and other parameters to simulate. Can't figure out what does that mean.</p>
  
  <p>Here is part of my code:</p>
  
  <pre><code>class googleAppSpider(scrapy.Spider):
  name = "googleApp"
  allowed_domains = ['play.google.com']
  start_urls = ['https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0']
  
  def start_request(self,response):
      for i in range(0,10): 
          yield FormRequest(url="https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0", method="POST", formdata={'start':str(i+60),'num':'60','numChildren':'0','ipf':'1','xhr':'1','token':'m1VdlomIcpZYfkJT5dktVuqLw2k:1455483261011'}, callback=self.parse)
  
  def parse(self,response):
      links = response.xpath("//a/@href").extract()
      crawledLinks = [ ]
      LinkPattern = re.compile("^/store/apps/details\?id=.")
      for link in links:
          if LinkPattern.match(link) and not link in crawledLinks:
              crawledLinks.append("http://play.google.com"+link+"#release")
      for link in crawledLinks:
              yield scrapy.Request(link, callback=self.parse_every_app)
  
  def parse_every_app(self,response):
  </code></pre>
  
  <p>The start_request seems to not play any role here. If I delete them, the spider would still crawl the same amount of links.</p>
  
  <p>I've worked on this problem for a week... Highly appreciate it if you could help me out...</p>