title: Scrolling down a webpage using selenium chromedriver
body: <p>I am trying to retrieve all relevant URLs from a site, however in order to render them all I have to scroll down the webpage, otherwise it returns 500 URLs. </p>
  
  <p>I have two key functions.
  One that gets all od the relevant URLs:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  
  from selenium import webdriver 
  
  def scrapeCategory(url):
      url1 = url + "?max=10000"
      html = getHtmlHeadless(url1)
      site = htmlParser(html)
      links = site.findAll('a', {'class':'itemImage', 'data-e2e':'product-listing'}, href=True)
      url_list = []
      for link in links:
          url_list.append("https://www.size.co.uk"+link['href'])
      return url_list
  </code></pre>
  
  <p>By specifying that max=10000 I ensure that all of the listings are on 1 page (rather than having to jump from page to page).</p>
  
  <pre><code>url1 = url + "?max=10000"
  </code></pre>
  
  <p>And a function that retrieves the HTML, employing a headless chromedriver:</p>
  
  <pre><code>def getHtmlHeadless(url):
      user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'
      options = webdriver.ChromeOptions()
  
      # specify headless mode
      options.add_argument('headless')
  
      # specify the desired user agent
      options.add_argument(f'user-agent={user_agent}')
      driver = webdriver.Chrome(executable_path='./chromedriver',options=options)
  
      # Ensure it is a string
      if ( type (url)!= str):
          print("The input must be a string or list of strings")
      driver.get(url)
  #     driver.send_keys(Keys.PAGE_DOWN)
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      html = driver.page_source
      return html
  </code></pre>
  
  <p>Following advice given in other similar queries I have tried applying </p>
  
  <pre><code>driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  </code></pre>
  
  <p>and</p>
  
  <pre><code>driver.send_keys(Keys.PAGE_DOWN)
  </code></pre>
  
  <p>However it does not seem to be doing the job as I either get a maximum of 500 URLs, or in the latter case I get an error. </p>
  
  <p>Error:</p>
  
  <pre><code>&lt;AttributeError: 'WebDriver' object has no attribute 'send_keys'&gt;
  </code></pre>
  
  <p>I am suspecting that I am not placing the</p>
  
  <pre><code>driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  </code></pre>
  
  <p>in the correct place. However I do not know where else to place it.</p>