title: 500 Internal server error scrapy
body: <p>I am using scrapy to crawl a product website which over 4 million products. However after crawling around 50k products it starts throwing 500 HTTP error. I have set Auto throttling to false as after enabling its very slow and will take around 20-25 days to complete the scraping. I think the server starts blocking the crawler temporarily after sometime. Any solutions what can be done ? 
  I am using sitemap crawler - I want to extract some information from the url itself if the server is not responding and proceed with next url instead of finishing the crawling and closing the spider, for that I was looking at the errback parameter in Request. However, since I am using sitemap crawler so I don't explicitly create a Request Object. Is there any default errback function that I can override or where can I define it. </p>
  
  <p>One more way to do it is defined here-<a href="https://stackoverflow.com/questions/16909106/scrapyin-a-request-fails-eg-404-500-how-to-ask-for-another-alternative-reque">Scrapy:In a request fails (eg 404,500), how to ask for another alternative request?</a></p>