title: Selenium Webdriver Python- Page loads incompletely/ sometimes freezes on refresh
body: <p>I am scraping a website with a lot of javascript that is generated when the page is called. As a result, traditional web scraping methods (beautifulsoup, ect.) are not working for my purposes (at least I have been unsuccessful in getting them to work, all of the important data is in the javascript parts). As a result I have started using selenium webdriver. I need to scrape a few hundred pages, each of which has between 10 and 80 data points (each with about 12 fields), so it is important that this script (is that the right terminology?) can run for quite awhile without me having to babysit it. </p>
  
  <p>I have the code working for a single page, and I have a controlling section that tells the scraping section what page to scrape. The problem is that sometimes the javascript portions of  the page load, and sometimes they don't when they don't(~1/7), a refresh fixes things, but occasionally the refresh will freeze webdriver and thus the python runtime environment as well. Annoyingly, when it freezes like this, the code fails to time out. What is going on?</p>
  
  <p>Here is a stripped down version of my code:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.support.ui import Select
  from selenium.common.exceptions import NoSuchElementException, TimeoutException
  import time, re, random, csv
  from collections import namedtuple
  
  def main(url_full):
  driver = webdriver.Firefox()
  driver.implicitly_wait(15)
  driver.set_page_load_timeout(30)   
  
  
  #create HealthPlan namedtuple
  HealthPlan = namedtuple( "HealthPlan", ("State, County, FamType, Provider, PlanType,      Tier,") +
                         (" Premium, Deductible, OoPM, PrimaryCareVisitCoPay, ER, HospitalStay,") +
                         (" GenericRx, PreferredPrescription, RxOoPM, MedicalDeduct, BrandDrugDeduct"))
  
  
  #check whether the page has loaded and handle page load and time out errors
  pageNotLoaded= bool(True)
  while pageNotLoaded:
      try:
          driver.get(url_full)
          time.sleep(6+ abs(random.normalvariate(1.8,3)))
      except TimeoutException:
          driver.quit()
          time.sleep(3+ abs(random.normalvariate(1.8,3)))
          driver.get(url_full)
          time.sleep(6+ abs(random.normalvariate(1.8,3)))
      # Handle page load error by testing presence of showAll, 
      # an important feature of the page, which only appears if everything else loads
  
      try:
          driver.find_element_by_xpath('//*[@id="showAll"]').text
      # catch NoSuchElementException=&gt;refresh page
      except NoSuchElementException:
          try:
              driver.refresh()
  
              # catch TimeoutException =&gt; quit and load the page 
              # in a new instance of firefox,
              # I don't think the code ever gets here, because it freezes in the refresh
              # and will not throw the timeout exception like I would like
          except TimeoutException:
              driver.quit()
              time.sleep(3+ abs(random.normalvariate(1.8,3)))
              driver.get(url_full)
              time.sleep(6+ abs(random.normalvariate(1.8,3)))
  
      pageNotLoaded= False
  
      scrapePage() # this is a dummy function, everything from here down works fine, 
  </code></pre>
  
  <p>I have looked extensively for similar problems, and I do not think anyone else has posted about this on so, or anywhere else that I have looked. I am using python 2.7, selenium 2.39.0 and I am trying to scrape Healthcare.gov 's get premium estimate's pages </p>
  
  <p>EDIT: (as an example,<a href="https://www.healthcare.gov/find-premium-estimates/#results/&amp;aud=indv&amp;type=med&amp;state=AL&amp;county=Autauga&amp;age0=27&amp;employerCoverage=no&amp;householdSize=1&amp;income=100000" rel="nofollow">this page</a>) It may also be worth mentioning that the page fails to load completely more often when the computer has been on/ doing this for awhile (i'm guessing that the free ram is getting full, and it glitches while loading) this is kind of beside the point though, because this should be handled by the try/except.</p>
  
  <p>EDIT2: I should also mention that this is being run on windows7 64bit, with firefox 17 (which I believe is the newest supported version)</p>