title: Scraping a website using selenium error on square-space site
body: <p>I am trying to scrape a website hosted on Squarespace using the python bindings for selenium that contains a web serial, I am copying the content of a div into a text file. Here is the code I was using to do that:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  driver = webdriver.Firefox()
  i = 1
  while i&lt;250:
      driver.get("http://www.drewhayesnovels.com/spy4/{0}".format(i))
      content = driver.find_element_by_css_selector('p') #Get content
      content_text = content.get_attribute('innerHTML')
      file = open("output/ch{0}.txt".format(i), 'w')
      file.write(str(content_text))
      file.close()
      print(i)
      i = i + 1
  
  print("Complete")
  </code></pre>
  
  <p>For some reason, on the third chapter, it stops being able to find the element. So I wrote this little test: </p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  driver = webdriver.Opera()
  i = 1
  while i&lt;250:
      driver.get("http://www.drewhayesnovels.com/spy4/{0}".format(i))
      try:
          content = driver.find_element_by_css_selector('p') #Get content
          content_text = content.get_attribute('innerHTML')
          file = open("output/ch{0}.txt".format(i), 'w')
          file.write(str(content_text))
          file.close()
          print(i)
          i = i + 1
      except:
          print("Error, keep running.")
          i = i + 1
  
  print("Complete")
  </code></pre>
  
  <p>Out of this, only 5 or 6 seemingly random pages were successful. I tried it with another browser (Opera) and it still didn't work. Any ideas?</p>