title: Selenium scrolling to bottom of page occasionally stops
body: <p>I have a page that can be scrolled down "infinitely" (for illustration, I will use the page <a href="https://www.reuters.com/news/us" rel="nofollow noreferrer">https://www.reuters.com/news/us</a>, which reveals more older news items every time you scroll to the bottom of the page). I want to scroll down to the bottom of the page to load older items many times using selenium. The following code works in principle:</p>
  
  <pre><code>driver = webdriver.Chrome(executable_path=r"/Users/bernhardclemm/Desktop/chromedriver")
  driver.get('https://www.reuters.com/news/us')
  
  # Accept Cookies
  wait = WebDriverWait(driver, 5)
  cookies = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@id='_evidon-banner-acceptbutton']")))
  cookies.click()
  
  # Switch of "Photos" to speed up loading
  wait = WebDriverWait(driver, 5)
  photos_off = wait.until(EC.element_to_be_clickable((By.XPATH, "//label[contains(text(),'Photos')]")))
  photos_off.click()
  
  # Scroll down 100 times
  i  = 1
  while i &lt; 500:
      print(i)
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      i += 1
      time.sleep(1)
  </code></pre>
  
  <p>As you can see in the following GIF, for every turn in the loop (<code>i</code> printed in the Python console to the right), the Selenium driver scrolls down once (Google Chrome in the left).</p>
  
  <p><a href="https://i.stack.imgur.com/P5Xs2.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P5Xs2.gif" alt="enter image description here"></a></p>
  
  <p>However, sometimes the Selenium driver has reached the bottom of the page but no new content loads, like this GIF shows. Note how the loop is still running in Python:</p>
  
  <p><a href="https://i.stack.imgur.com/DA8hE.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DA8hE.gif" alt="enter image description here"></a></p>
  
  <p>When I scroll up a bit in Chrome <em>manually</em>, the automatic scrolling starts again. So this is <strong>not</strong> because the site has reached the end of its contents.</p>
  
  <p>Any idea whether this is due to the web site or a flaw in my code? If it's a something about the web site, what would be a way to avoid the problem with Selenium?</p>