title: I got blocked when I used scrapy and not when I used selenium and request 403 error code
body: <p>I am trying to scrape this website using scrapy
  <a href="https://www.dnb.com/business-directory/company-profiles.pfizer_inc.140f48fa0b37556f925afcaec7b5c566.html" rel="nofollow noreferrer">https://www.dnb.com/business-directory/company-profiles.pfizer_inc.140f48fa0b37556f925afcaec7b5c566.html</a></p>
  <p>I tried</p>
  <p>1-crawling using scrapy shell</p>
  <p>2-rotating user agent and proxies</p>
  <p>3-enable COOKIES</p>
  <p>4-disable Javascript from the browser to try what happens</p>
  <p>this my settings</p>
  <pre><code>    'DOWNLOAD_DELAY': 0.25,
      'CONCURRENT_REQUESTS': 10,
      'CONCURRENT_REQUESTS_PER_DOMAIN': 10,
      'CONCURRENT_ITEMS': 100,
      'DOWNLOAD_TIMEOUT': 100,
      'RANDOMIZE_DOWNLOAD_DELAY': True,
      # proxy setting
      'RETRY_TIMES': 10,
      'RETRY_HTTP_CODES': [500, 503, 504, 400, 403, 408],
      'PROXY_MODE': 0,
      'PROXY_LIST': r'https_proxies_ip.txt',
      'RANDOM_UA_FILE': r'default_uas.txt',
      'RANDOM_UA_PER_PROXY': True,  # for only proxy
  
      'DOWNLOADER_MIDDLEWARES':
          {
              'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
              'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
  
              'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
              'scrapy_proxies.RandomProxy': 100,
              'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
          }
  </code></pre>
  <p>but when I tried selenium and requests without user agents and proxies it worked fine!
  my questions are what should I do? and why did this happen?</p>
  <p>thanks</p>