title: Turn table in selenium into a pandas data frame?
body: <p>I am trying to a scrape a <strong>table consisting og 45 columns and 7 rows</strong>. The table is loaded using ajax and I can't access the API. Thus I needed to use selenium in Python. I am close to get what I want but I don't know how I can turn my 'selenium find elements' into a Pandas DataFrame. So far, my code looks like this:</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  import pandas as pd
  import time
  
  driver = webdriver.Chrome()
  url = &quot;http://www.hctiming.com/myphp/resources/login/browse_results.php?live_action=yes&amp;smartphone_action=no&quot; #a redirect to a login page occurs
  driver.get(url)
  driver.find_element_by_id(&quot;open&quot;).click()
  
  user = driver.find_element_by_name(&quot;username&quot;)
  password = driver.find_element_by_name(&quot;password&quot;)
  user.clear()
  user.send_keys(&quot;MyUserNameWhichIWillNotShare&quot;)
  password.clear()
  password.send_keys(&quot;myPasswordWhicI willNotShare&quot;)
  driver.find_element_by_name(&quot;submit&quot;).click()
  
  try:
      element = WebDriverWait(driver, 10).until(
          EC.presence_of_element_located((By.LINK_TEXT, &quot;Results Services&quot;)) # I must first click in this line
      )
      element.click()
  
      element = WebDriverWait(driver, 10).until(
          EC.presence_of_element_located((By.LINK_TEXT, &quot;View Live&quot;)) # Then I must click in this link. Now I have access to the result database
      )
      element.click()
  
  except:
      driver.quit()
  
  time.sleep(5) #I have set a timesleep to 5 secunds. There must be a better way to accomplish this. I just want to make sure that the table is loaded when I try to scrape it
  
  columns = len(driver.find_elements_by_xpath(&quot;/html/body/div[2]/div/form[3]/div[2]/div[1]/div/div/div/div[2]/div[4]/section[1]/div[2]/div/div/table/thead/tr[2]/th&quot;))
  rows = len(driver.find_elements_by_xpath(&quot;/html/body/div[2]/div/form[3]/div[2]/div[1]/div/div/div/div[2]/div[4]/section[1]/div[2]/div/div/table/tbody/tr&quot;))
  print(columns, rows)
  
  </code></pre>
  <p>The last code line prints 45 and 7. Thus, this seems to work. However, I don't understand how I can make a dataframe of it? Thank you.</p>