title: scrapy speed up crawling
body: <p>I am currently crawling a website to extract product information. The site consists of millions[around 20 millions] of product information and i need to extract each of those . I need to speed up the crawling speed as my crawler is currently scraping only few hundred pages per minute. </p>
  
  <p>Here is my sample code of crawler.</p>
  
  <pre><code>class XYSspider(CrawlSpider):
  name = "XYZ"
  
  allowed_domains = ["www.xyz.info"]
  start_urls = ['http://www.xyz.info/']
  
  rules = [
      Rule(LinkExtractor(allow=r'\.kat$'), follow=True),
      Rule(LinkExtractor(allow=r'/id_\d+/'), callback='parse_product'),
  ]
  
  def parse_product(self, response):
      item = DmozItem()
      item['title'] = response.xpath('//title/text()').extract()[0].encode('utf-8')
      yield item
  </code></pre>
  
  <p>Is there any way i can speed up the crawling process. At the current speed it would take me days to scrape all data. Any insights will be really helpful. Thankyou</p>