title: Error 503 while scraping cdiscount (scrapy) Python
body: <p>I've made a spider which work to collect data on cdiscount website. However, each time I scrape more than 320 pages of a category, I have the 503 error and the spider close.</p>
  
  <p>How to deal with that problem ? I've tried changing user agent and using a proxy pool like this :</p>
  
  <pre><code>def __init__(self, *args, **kwargs):
      super(CdiscountSpider, self).__init__(*args, **kwargs)
      self.proxy_pool = ['49.236.220.238:52840',  '181.112.41.50:33381', '50.235.111.161:45126']
  
  (...)
  
         request = scrapy.Request(url, callback=self.parse_dir_contents) #on accède au contenu des catégories
         request.meta["proxy"] = random.choice(self.proxy_pool)
         yield request
  </code></pre>
  
  <p>but it didn't work. Please, any help greatly appreciated :)</p>