title: Error 403 : HTTP status code is not handled or not allowed in scrapy
body: <p>This is the code, I have written to scrape justdial website.</p>
  
  <pre><code>import scrapy
  from scrapy.http.request import Request
  
  class JustdialSpider(scrapy.Spider):
      name = 'justdial'
      # handle_httpstatus_list = [400]
      # headers={'User-Agent': "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"}
      # handle_httpstatus_list = [403, 404]
      allowed_domains = ['justdial.com']
      start_urls = ['https://www.justdial.com/Delhi-NCR/Chemists/page-1']
      # def  start_requests(self):
      #     # hdef start_requests(self):
      #     headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
      #     for url in self.start_urls:
      #         self.log("I just visited :---------------------------------- "+url)
      #         yield Request(url, headers=headers)
      def parse(self,response):
          self.log("I just visited the site:---------------------------------------------- "+response.url)
           urls = response.xpath('//a/@href').extract()
           self.log("Urls-------: "+str(urls))
  </code></pre>
  
  <p>This is Error is showing in Terminal:</p>
  
  <pre><code>2017-08-18 18:32:25 [scrapy.core.engine] INFO: Spider opened
  2017-08-18 18:32:25 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pag
  es/min), scraped 0 items (at 0 items/min)
  2017-08-18 18:32:25 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache
  storage in D:\scrapy\justdial\.scrapy\httpcache
  2017-08-18 18:32:25 [scrapy.extensions.telnet] DEBUG: Telnet console listening o
  n 127.0.0.1:6023
  2017-08-18 18:32:25 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://www.j
  ustdial.com/robots.txt&gt; (referer: None) ['cached']
  2017-08-18 18:32:25 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://www.j
  ustdial.com/Delhi-NCR/Chemists/page-1&gt; (referer: None) ['cached']
  2017-08-18 18:32:25 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response
   &lt;403 https://www.justdial.com/Delhi-NCR/Chemists/page-1&gt;: HTTP status code is n
  ot handled or not allowed 
  </code></pre>
  
  <p>I have seen the similar questions on stackoverflow i tried everything like,
  You can see in Code with comment what i tried,</p>
  
  <ul>
  <li><p>changed the UserAgents</p></li>
  <li><p>Setting handle_httpstatus_list = [400]</p></li>
  </ul>
  
  <p>Note: This (<a href="https://www.justdial.com/Delhi-NCR/Chemists/page-1" rel="noreferrer">https://www.justdial.com/Delhi-NCR/Chemists/page-1</a>) website not even blocked in my system. When i open the website in chrome/mozilla, it's opening. This is same error with (<a href="https://www.practo.com/bangalore#doctor-search" rel="noreferrer">https://www.practo.com/bangalore#doctor-search</a>) site also.</p>