title: Scrape websites using scrapy
body: <p>I am trying to scrape a <a href="http://www.jabong.com/men/shoes/" rel="nofollow noreferrer">website</a> with scrapy but I am having problem with scraping the all products from this site as it is using endless scrolling...</p>
  
  <p>I can scrape only below data for 52 items only but their are 3824 items.</p>
  
  <pre><code>hxs.select("//span[@class='itm-Catbrand strong']").extract()
  hxs.select("//span[@class='itm-price ']").extract()
  hxs.select("//span[@class='itm-title']").extract()
  </code></pre>
  
  <p>If I use <code>hxs.select("//div[@id='content']/div/div/div").extract()</code> Then it extracts whole items list but it won't filter further....How do I scrape all the items?</p>
  
  <p>I have tried this but same result. Where am I wrong?</p>
  
  <pre><code>def parse(self, response):
      filename = response.url.split("/")[-2]
      open(filename, 'wb').write(response.body
      for n in [2,3,4,5,6]:            
      req = Request(url="http://www.jabong.com/men/shoes/?page=" + n,
                        headers = {"Referer": "http://www.jabong.com/men/shoes/",
                                   "X-Requested-With": response.header['X-Requested-With']})
      return req 
  </code></pre>