title: How do I fix 404 HTTP status code is not handled or not allowed in scrapy?
body: <p>The spider will scrape but the website but I keep on getting the 404 HTTP status code is not handled or not allowed. Is my code fully correct?</p>
  
  <p>I've changed the user-agent in settings.py but it won't fix the problem. </p>
  
  <pre><code>import scrapy
  
  # Creating a new class to implement Spide
  class QuuickSpider(scrapy.Spider):
  
      # Spider name
      name = 'quick'
  
      # Domain names to scrape
      allowed_domains = ['trustpilot.com']
  
      # Base URL for the MacBook air reviews
      myBaseUrl = "https://www.trustpilot.com/review/www.quickenloans.com"
      start_urls=[]
  
      # Creating list of urls to be scraped by appending page number a the end of base url
      for i in range(1,121):
          start_urls.append(myBaseUrl+str(i))
  
      # Defining a Scrapy parser
      def parse(self, response):
              data = response.css('#cm_cr-review_list')
  
              # Collecting product star ratings
              star_rating = data.css('.review-rating')
  
              # Collecting user reviews
              comments = data.css('.review-text')
              count = 0
  
              # Combining the results
              for review in star_rating:
                  yield{'stars': ''.join(review.xpath('.//text()').extract()),
                        'comment': ''.join(comments[count].xpath(".//text()").extract())
                       }
                  count=count+1
  
  </code></pre>