title: Getting Error 403 in Scrapy while crawling Cloudflare protected website
body: <p>My goal is to scrape the reviews on a product page from <a href="https://www.g2.com/products/solidworks-product-designer/reviews" rel="nofollow noreferrer">this website</a>. I am using Scrapy with Python 3.7.1.</p>
  <p>Since this is a cloudflare protected website, I set up User Agent: <code>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.74 Safari/537.36 Edg/79.0.309.43' </code>, set <code>ROBOTSTXT_OBEY = True </code>, and the <code>DOWNLOAD_DELAY = 5</code>.</p>
  <p>I still get 403 for the first  request, and the response received by scrapy is hCaptcha page that wants me to prove that a human is accessing the page.</p>
  <p>I tried cfscrape module but the issue persists.</p>
  <p>Following is the cmd output:</p>
  <pre><code>2021-01-19 13:52:03 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
  2021-01-19 13:52:04 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.g2.com/robots.txt&gt; (referer: None)
  2021-01-19 13:52:04 [protego] DEBUG: Rule at line 1 without any user agent to enforce it on.
  2021-01-19 13:52:04 [protego] DEBUG: Rule at line 2 without any user agent to enforce it on.
  2021-01-19 13:52:09 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://www.g2.com/products/solidworks-product-designer/reviews/&gt; (referer: None)
  2021-01-19 13:52:09 [scrapy.core.engine] INFO: Closing spider (finished)
  </code></pre>
  <p><a href="https://codepen.io/utkashtyagix/pen/QWKYerz" rel="nofollow noreferrer">Here</a> is the codepen with response output as seen by scrapy.</p>
  <p><strong>Edit1</strong>: Added Codepen link for response text</p>