title: Can you not get detected as a robot when using selenium in Python?
body: <p>I've been trying to scrape this website for 2 days now. I'm completely stuck. The problem is that it detects me as a bot.</p>
  <p>I have a list of urls that I need to crawl. and in the results folder, every file says that <code>Access to this page has been denied... To continue, please prove you are not a robot... etc.</code></p>
  <p>Below is my current code</p>
  <pre><code>import time
  from selenium import webdriver
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.chrome.options import Options
  from fake_useragent import UserAgent
  import sys
  reload(sys)
  sys.setdefaultencoding('utf-8')
  
  CHROMEDRIVER_PATH = './chromedriver'
  
  chrome_options = Options()
  chrome_options.add_argument(&quot;--headless&quot;)
  chrome_options.add_argument(&quot;start-maximized&quot;)
  chrome_options.add_argument(&quot;disable-infobars&quot;)
  chrome_options.add_argument(&quot;--disable-extensions&quot;)
  chrome_options.add_argument(&quot;--disable-blink-features=AutomationControlled&quot;)
  ua = UserAgent()
  userAgent = ua.random
  chrome_options.add_argument('user-agent={userAgent}')
  
  LOGIN_PAGE = &quot;https://www.seekingalpha.com/login&quot;
  ACCOUNT = &quot;Account&quot;
  PASSWORD = &quot;Password&quot;
  
  driver = webdriver.Chrome(executable_path=CHROMEDRIVER_PATH, chrome_options=chrome_options)
  driver.execute_script(&quot;Object.defineProperty(navigator, 'webdriver', {get: () =&gt; undefined})&quot;)
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {&quot;userAgent&quot;: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  wait = WebDriverWait(driver, 30)
  driver.get(&quot;https://www.seekingalpha.com/login&quot;)
  time.sleep(1)
  wait.until(EC.element_to_be_clickable((By.NAME, &quot;email&quot;))).send_keys(ACCOUNT)
  wait.until(EC.element_to_be_clickable((By.ID, &quot;signInPasswordField&quot;))).send_keys(PASSWORD)
  wait.until(EC.element_to_be_clickable((By.XPATH, &quot;//button[text()='Sign in']&quot;))).click()
  
  time.sleep(1)
  
  with open(&quot;links.txt&quot;, &quot;r&quot;) as inArticle:
      articles = inArticle.read().splitlines()
  
  for article in articles:
      outName = article.split(&quot;/&quot;)[-1]
      outName = outName.split(&quot;-&quot;)[0]
  
      driver.get(article)
      time.sleep(1)
      html_source = driver.page_source
  
      out_text = str(html_source).encode(&quot;utf8&quot;)
  
      with open(&quot;./results/&quot;+outName, &quot;w&quot;) as outFile:
          outFile.write(out_text)
  
  driver.quit()
  </code></pre>
  <p>Is there a better way to do this? and is there a way to pass this bot check?</p>