title: Why is BeautifulSoup Returning Duplicate Table Rows and Missing Others Altogether?
body: <p>I'm trying to scrape MLB player stats from mlb.com using Selenium and BeautifulSoup and am returning duplicates of many players, and am also missing several others just by giving it the eye test. It's unclear where the duplicates are coming from or why I'm missing others.</p>
  
  <p>I chose to use Selenium instead of Requests because the data is stored in a table, but not visible in the HTML until the browser loads the page. Any ideas where the data integrity errors are coming from?</p>
  
  <p>Thanks in advance! </p>
  
  <p>Highlights from my script below</p>
  
  <pre><code># Navigate to stats page
  def getToStats():
      url = ("http://mlb.mlb.com/stats/sortable.jsp?c_id=mlb#elem=%5Bobject+Object%5D&amp;tab_level=child&amp;click_text=Sortable+Player+hitting&amp;game_type='R'&amp;season=2019&amp;season_type=ANY&amp;league_code='MLB'&amp;sectionType=sp&amp;statType=hitting&amp;page=1&amp;ts=1586832481387&amp;playerType=ALL")
      driver.get(url)
      html = driver.page_source
  
  # Get source HTML/CSS
  def getCurrentURL():
      url = driver.current_url
      html = driver.page_source
      soup = BeautifulSoup(html, 'html.parser')
      new_soup = soup.find("tbody")
      return new_soup
  
  def getPlayers(html):
      players = []
      for player in html.find_all('tr'):
          players.append(player)
      return players
  
  def getPlayerStats(player):
      playerRecord = []
      for data in player.find_all('td'):
          stat = data.text.replace('\xa0','') # Removes empty non-ASCII character attached to player last name
          playerRecord.append(stat)
      playerRecord.pop(0) # Removes MLB determined player rank
      playerRecord.pop(2) # Removes empty non-ASCII element
      return playerRecord
  
  
  ##################FUNCTIONS COMPLETE CODE EXECUTION BEGINS######################
  playerData = []
  driver = webdriver.Chrome()
  getToStats()
  html = getCurrentURL()
  players = getPlayers(html)
  for player in players:
     playerData.append(getPlayerStats(player))
  
  print('Page 1 retrieved')
  pageNum = 2
  for page in range(25):
      element = driver.find_elements_by_class_name('paginationWidget-next')
      driver.execute_script("arguments[0].click();", element[0])
      next_html = getCurrentURL()
      playerRecords = getPlayers(next_html)
      for player in playerRecords:
          playerData.append(getPlayerStats(player))
      print('Page ' + str(pageNum) + ' retrieved')
      pageNum = pageNum + 1
  print('Found the data')
  
  my_df = pd.DataFrame(playerData)
  my_df.to_csv('mlb_stats_2019.csv', index = False, header = False)
  print('Enjoy your CSV file')
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/e3HiK.png" rel="nofollow noreferrer">Duplicate Output Sample</a></p>