title: Unable to scrape via selenium in python because of infinite page load
body: <p>I am trying to extract the contents of some of the news articles. Some of the urls required logging in in order to access the full content. I decided to use selenium to automate logging in. However, I am not able to extract contents because the first url takes forever to load and never reaches the point where actual text extraction is done. It ends up throwing timeout exception.</p>
  <p>Here is my code</p>
  <pre><code>for url in url_list:
      chrome_options = Options()
      ua = UserAgent()
      userAgent = ua.random
      options.add_argument(f'user-agent={userAgent}')
      driver = webdriver.Chrome(ChromeDriverManager().install(), options = chrome_options)
      driver.get(url)
      time.sleep(5)
      frame = driver.find_elements_by_xpath('//iframe[@id=&quot;wallIframe&quot;]')
      #Some articles require going through a paywall and some don't
      if len(frame)==0:
          text_element = driver.find_elements_by_xpath('//section[@id=&quot;main-content&quot;]//article//p')
          text = &quot; &quot;.join(x.text for x in element)
      else:
          text = log_in(frame)
      driver.quit()
  </code></pre>
  <p>Although the code never reaches to it, here is my log_in method</p>
  <pre><code>def log_in(frame):
      driver.switch_to.frame(frame[0])
      driver.find_element_by_id(&quot;PAYWALL_V2_SIGN_IN&quot;).click()
      time.sleep(2)
      driver.find_elements_by_id(&quot;username&quot;)[0].send_keys(username)
      time.sleep(2)
      driver.find_elements_by_xpath('//button[text()=&quot;Continue&quot;]')[0].click()
      time.sleep(1)
      driver.find_elements_by_id(&quot;password&quot;)[0].send_keys(password)
      time.sleep(1)
      element = driver.find_elements_by_xpath('//button[@type=&quot;submit&quot;]')[0].click()
      time.sleep(1)
      text = parse_text(element)
  </code></pre>
  <p>How can I get around this?</p>