title: Scrapy crawl category links till product page
body: <p>Hello i am trying to scrape a ecommerce website which loads data with scroll and load more button I followed this <a href="https://stackoverflow.com/questions/19709086/how-to-scrape-website-with-infinte-scrolling">How to scrape website with infinte scrolling?</a> link but when I tried the code it's closing the spider without an products may be the structure has changed and I would like some help to get me starting I am quite new to webscraping.</p>
  
  <p>edit question 
  ok i am scraping this website [link]<a href="http://www.jabong.com/women/" rel="nofollow noreferrer">http://www.jabong.com/women/</a> which has subcategories , i am trying to scrape all the subcategories products i tried above code but that didnt work for so after doing some research i created a code which works but doesnt satisfy my goal .so far i have tried this  </p>
  
  <pre><code>` import scrapy
  #from scrapy.exceptions import CloseSpider
  
  from scrapy.spiders import Spider
  #from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.http import Request
  from koovs.items import product
  #from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import Selector    
  
  class propubSpider(scrapy.Spider):
  name = 'koovs'
  allowed_domains = ['jabong.com']
  max_pages = 40
  
  def start_requests(self):
      for i in range(self.max_pages):
           yield scrapy.Request('http://www.jabong.com/women/clothing/tops-tees-shirts/tops/?page=%d' % i, callback=self.parse)
  
  def parse(self, response):
      for sel in response.xpath('//*[@id="catalog-product"]/section[2]'):
          item = product()
          item['price'] = sel.xpath('//*[@class="price"]/span[2]/text()').extract()
          item['image'] = sel.xpath('//*[@class="primary-image thumb loaded"]/img').extract()
          item['title'] = sel.xpath('//*[@data-original-href]/@href').extract()
       `
  </code></pre>
  
  <p>the above code works for one category and also if i specify the number of pages, the above website has a lot of products for given category and i dont know in how many pages they reside , so i decided to use crawl spider to go through all the categories and product pages and fetch data but i am very new to scrapy any help would be highly appreciated</p>