title: How to scrape tens of thousands urls every night using scrapy
body: <p>I am using scrapy to scrape some big brands to import the sale data for my site. Currently I am using </p>
  
  <pre><code>DOWNLOAD_DELAY = 1.5
  
  CONCURRENT_REQUESTS_PER_DOMAIN = 16
  
  CONCURRENT_REQUESTS_PER_IP = 16
  </code></pre>
  
  <p>I am using Item loader to specify css/xpath rules and Pipeline to write the data into csv. The data that I collect is original price, sale price, colour, sizes, name, image url and brand.</p>
  
  <p>I have written the spider for only one merchant who has around 10k urls and it takes me about 4 hours.</p>
  
  <p>My question is, does 4 hours sounds alright for 10k urls or it should be faster than that. If so, what else do I need to do to speed it up. </p>
  
  <p>I am using only one SPLASH instance locally to test. But in production I am planning to use 3 SPLASH instances. </p>
  
  <p>Now the main issue, I have about 125 merchants and on avg 10k product for each. couple of them has more than 150k urls to scrape.</p>
  
  <p>I need to scrape all their data every night to update my site. 
  Since my single spider takes 4 hours to scrape 10k urls, I am wondering if it is even valid dream to achieve 125 x 10k urls every night</p>
  
  <p>I will really appreciate your experienced input to my problem.</p>