title: How to fix BeautifulSoup/selenium working on same website for some pages but not all?
body: <p>I'm trying to scrape each page of: <a href="https://www.prosportstransactions.com/football/Search/SearchResults.php?Player=&amp;Team=&amp;BeginDate=&amp;EndDate=&amp;PlayerMovementChkBx=yes&amp;submit=Search&amp;start=0" rel="nofollow noreferrer">https://www.prosportstransactions.com/football/Search/SearchResults.php?Player=&amp;Team=&amp;BeginDate=&amp;EndDate=&amp;PlayerMovementChkBx=yes&amp;submit=Search&amp;start=0</a></p>
  
  <p>Right now I have code that changes the URL iteratively. The URL is then passed into a selenium driver to grab the HTML content. The content is then put into BeautifulSoup to process. My problem is I get the following message randomly(Happens randomly on different pages which causes the program to crash. There's no consistent page that it fails on):</p>
  
  <pre><code>Traceback (most recent call last):
    File "scrape.py", line 89, in &lt;module&gt;
     i, i + 5000)
    File "scrape.py", line 37, in scrapeWebsite
      extractedInfo = info.findAll("td")
  AttributeError: 'NoneType' object has no attribute 'findAll'
  </code></pre>
  
  <p>The i, i + 5000 is used to loop to update the pages iteratively so that's not important.</p>
  
  <p>Here's the code that's doing the HTML grabbing:</p>
  
  <pre><code>driver = webdriver.Chrome(executable_path='/Users/Downloads/chromedriver')
  print(start, stop)
  
  
  madeDict = {"Date": [], "Team": [], "Name": [], "Relinquished": [], "Notes": []}
  
  #for i in range(0, 214025, 25):
  for i in range(start, stop, 25):
      print("Current Page: " + str(i))
      currUrl = url + str(i)
      driver.get(currUrl)
      driver.implicitly_wait(100
      soupPage = BeautifulSoup(driver.page_source, 'html.parser')
      #page = urllib2.urlopen(currUrl)
      #soupPage = BeautifulSoup(page, 'html.parser')
  
      # #Sleep the program to ensure page is fully loaded
      # time.sleep(1)
  
      info = soupPage.find("table", attrs={'class': 'datatable center'})
      extractedInfo = info.findAll("td")
  </code></pre>
  
  <p>My guess is the page doesn't finish loading so when it tries to grab the content, the tags may not be there. However, I thought Selenium prevented that issue with dynamic loading webpages to ensure the page is fully loaded before BeautifulSoup grabs the info. I was looking at other posts and some said I needed to wait the program for the page to dynamically load but I tried that and still the same error.</p>