title: Scrapy: Debug Redirecting (301)
body: <p>Before I was getting the error &quot;HTTP status code is not handled or not allowed&quot;, I modified the USER_AGENT that was in default mode and now I am getting this error:</p>
  <pre><code>import scrapy
   
   
  class OlxSpider(scrapy.Spider):
      name = &quot;olx&quot;
      allowed_domains = [&quot;pe.olx.com.br&quot;]
      start_urls = (
          'http://pe.olx.com.br/imoveis/aluguel',
      )
   
      def parse(self, response):
          items =  response.xpath(
              '//div[contains(@class,&quot;section_OLXad-list&quot;)]//li[contains'
              '(@class,&quot;item&quot;)]'
          )
          for item in items:
              url = item.xpath(
                  &quot;.//a[contains(@class,'OLXad-list-link')]/@href&quot;
              ).extract_first()
              yield scrapy.Request(url=url, callback=self.parse_detail)
   
          next_page = response.xpath(
              '//li[contains(@class,&quot;item next&quot;)]//a/@href'
          ).extract_first()
          if next_page:
              self.log('Next Page: {0}'.format(next_page))
              yield scrapy.Request(url=next_page, callback=self.parse)
   
      def parse_detail(self, response):
          self.log(u'Imóvel URL: {0}'.format(response.url))
          item = {}
          item['photos'] = response.xpath(
              '//div[contains(@class,&quot;photos&quot;)]//a/@href'
          ).extract()
          item['url'] = response.url
          item['address'] = response.xpath(
              'normalize-space(//div[contains(@class,&quot;OLXad-location&quot;)]'
              '//.)'
          ).extract_first()
          item['title'] = response.xpath(
              'normalize-space(//h1[contains(@id,&quot;ad_title&quot;)]//.)'
          ).extract_first()
          item['price'] = response.xpath(
              'normalize-space(//div[contains(@class,&quot;OLXad-price&quot;)]'
              '//span[contains(@class,&quot;actual-price&quot;)]//.)'
          ).extract_first()
          item['details'] = response.xpath(
              'normalize-space(//div[contains(@class,&quot;OLXad-description&quot;)]'
              '//.)'
          ).extract_first()
          item['source_id'] = response.xpath(
              'normalize-space(//div[contains(@class,&quot;OLXad-id&quot;)]//strong//.)'
          ).extract_first()
          date = response.xpath(
              'normalize-space(//div[contains(@class,&quot;OLXad-date&quot;)]//.)'
          ).re(&quot;Inserido em: (.*).&quot;)
          item['date'] = (date and date[0]) or ''
          yield item
  
  
  </code></pre>
  <p>trying to execute the .py file in the terminal, I get the following message:</p>
  <pre><code>2022-01-13 12:36:36 [scrapy.core.engine] INFO: Spider opened
  2022-01-13 12:36:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2022-01-13 12:36:36 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
  2022-01-13 12:36:37 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to &lt;GET https://pe.olx.com.br/robots.txt&gt; from &lt;GET http://pe.olx.com.br/robots.txt&gt;
  2022-01-13 12:36:37 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://pe.olx.com.br/robots.txt&gt; (referer: None)
  2022-01-13 12:36:37 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to &lt;GET https://pe.olx.com.br/imoveis/aluguel&gt; from &lt;GET http://pe.olx.com.br/imoveis/aluguel&gt;
  </code></pre>
  <p>Does anyone know what might be causing this problem?</p>
  <p>P.s.: I have tried these solutions <a href="https://stackoverflow.com/questions/38658247/python-scrapy-301-redirects">Python Scrapy 301 redirects</a></p>