title: Is there a way to speed up web scraping with selenium?
body: <p>I am scraping the Twitter website. (API is not used to retrieve historical data.) .Smaller ranges will come up sooner, but if you make a larger range, you'll have to spend a lot of time.</p>
  
  <p>Also, there is an error that the connection is stopped after a long period of time.</p>
  
  <p>So my query is, Is there any way to speed up web scraping with Selenium ? Or do I need to modify my code?</p>
  
  <pre><code>#python3
  import requests
  import time  
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  from bs4 import BeautifulSoup
  
  browser = webdriver.PhantomJS('C:\phantomjs-2.1.1-windows/bin/phantomjs')
  url =u'https://twitter.com/search?f=tweets&amp;vertical=default&amp;q=%EC%84%B8%EC%9B%94%ED%98%B8%20since%3A2014-04-16%20until%3A2014-04-17&amp;src=typd&amp;lang=ko'
  
  
  browser.get(url)
  time.sleep(1)
  
  body = browser.find_element_by_tag_name('body')
  
  
  for _ in range(10000):
      browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      time.sleep(0.1)
  
  tweets=browser.find_elements_by_class_name('tweet-text')
  
  wfile = open("140416.txt", mode='w', encoding='utf8')
  data={}
  i = 1
  for i, tweet in enumerate(tweets):
      data['text'] = tweet.text
      print(i, ":", data)
      wfile.write(str(data) +'\n')
      i += 1
  wfile.close()
  </code></pre>