title: How can I make web scraping faster in Python using Selenium and Beautiful Soup?
body: <p>I wrote a script to <strong>scrape</strong> the website Vivino, using the Beautiful Soup and Selenium libraries.</p>
  <p>In this website, I want to store information of a certain wine's reviews.</p>
  <p>I have to use Selenium to do dynamic scraping since the reviews can only be accessed pressing the &quot;Show more reviews&quot; button in the webpage, which appears after scrolling down to the top of the page.</p>
  <p>I adapted the code for just one wine so you can see, if needed, how long it takes:</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  import time
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  import pandas as pd
  
  
  def scroll_to_bottom_wine_page(driver):
  
      #driver = self.browser
      scroll_pause_time = 0.01 #Change time?
      # Get scroll height
      last_height = driver.execute_script(&quot;return document.body.scrollHeight&quot;)
   
      while True:
          # Scroll down to bottom
          driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)
   
          # Wait to load page
          time.sleep(scroll_pause_time)
   
          # Calculate new scroll height and compare with last scroll height
          new_height = driver.execute_script(&quot;return document.body.scrollHeight&quot;)
          if new_height == last_height:
              break
          last_height = new_height
  
          
  def scroll_to_bottom_review_page(driver, rating_count):
  
      stuck_counter = 0
      current_reviews_now = 0
      current_reviews_previous = 0
      scroll_review_pause_time = 0.8 #Change time?
      stop_indicator = rating_count 
      
      time.sleep(scroll_review_pause_time)
      element_inside_popup = driver.find_element_by_xpath('//*[@id=&quot;baseModal&quot;]/div/div[2]/div[3]//a')  #Reviews path
  
  
  
      while True:
          time.sleep(scroll_review_pause_time)
          element_inside_popup.send_keys(Keys.END)
          results_temp = driver.execute_script(&quot;return document.documentElement.outerHTML&quot;)
          soup = BeautifulSoup(results_temp, 'lxml')    
          reviews = soup.findAll(&quot;div&quot;, {&quot;class&quot;: &quot;card__card--2R5Wh reviewCard__reviewCard--pAEnA&quot;})
          current_reviews_now = len(reviews)
  
          #In case there actually are less reviews than what the rating_count states, we avoid scrolling down forever
          if(current_reviews_now == current_reviews_previous):
              stuck_counter += 1
          
          if (current_reviews_now &gt; (stop_indicator)) or (stuck_counter &gt; 2):
              break
          
          current_reviews_previous = current_reviews_now
          
      return reviews
  
  
  
  def get_reviews(wine_ids, wine_urls, rating_counts):
  
      #Create a dataframe
      review_info = pd.DataFrame()
     
      #Create a driver
      driver = webdriver.Chrome()
      
      for wine_url in wine_urls:
  
          #Pass URL to driver
          driver.get(wine_url)
  
          #We scroll down to the bottom of the wine webpage
          scroll_to_bottom_wine_page(driver)
  
          #Search for the &quot;Show more reviews button and click it
          wait = WebDriverWait(driver,40)
          wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Show more reviews')))
          more_reviews_button = driver.find_element_by_link_text('Show more reviews')
          more_reviews_button.click()
  
          #Scroll till we reach the number of reviews 
          reviews = scroll_to_bottom_review_page(driver, rating_counts)
          length = len(reviews)
          
          wine_ids_list = [wine_ids] * length
          review_user_links = []
          review_ratings = []
          review_usernames = []
          review_dates = []
          review_texts = []
          review_likes_count = []
          review_comments_count = []
  
          for review in reviews:
              
              
              review_user_links.append([a['href'] for a in review.find_all('a', href=True)][0])
              review_ratings.append(float((review.find(&quot;div&quot;, class_=&quot;rating__rating--ZZb_x&quot;)[&quot;aria-label&quot;]).split()[1]))
              review_usernames.append(str((review.find('a', {&quot;class&quot; : 'anchor__anchor--3DOSm reviewCard__userName--2KnRl'})).string))
              review_dates.append(&quot;&quot;.join(((review.find('div', {&quot;class&quot; : 'reviewCard__ratingsText--1LU2T'})).text).rsplit((str(review_usernames[-1])))))
              
              if (review.find('p', {&quot;class&quot; : 'reviewCard__reviewNote--fbIdd'})) is not None:
                  review_texts.append(str((review.find('p', {&quot;class&quot; : 'reviewCard__reviewNote--fbIdd'})).string))
                  review_texts = [item.strip() for item in review_texts]  
              else:
                  review_texts.append('None')
  
              if (review.find(&quot;div&quot;, class_=&quot;likeButton__likeCount--82au4&quot;)) is not None:
                  review_likes_count.append(int(review.find(&quot;div&quot;, class_=&quot;likeButton__likeCount--82au4&quot;).text))
              else:
                  review_likes_count.append(int(0))
  
              if (review.find(&quot;div&quot;, class_=&quot;commentsButton__commentsCount--1_Ugm&quot;)) is not None:
                  review_comments_count.append(int(review.find(&quot;div&quot;, class_=&quot;commentsButton__commentsCount--1_Ugm&quot;).text))
              else:
                  review_comments_count.append(int(0))
                  
          #We put the information in a dataframe
          review_info_temp = pd.DataFrame()
          
          review_info_temp.loc[:,'wine_id'] = wine_ids_list
          review_info_temp.loc[:,'review_user_links'] = review_user_links
          review_info_temp.loc[:,'review_ratings'] = review_ratings
          review_info_temp.loc[:,'review_usernames'] = review_usernames
          review_info_temp.loc[:,'review_dates'] = review_dates
          review_info_temp.loc[:,'review_texts'] = review_texts
          review_info_temp.loc[:,'review_likes_count'] = review_likes_count
          review_info_temp.loc[:,'review_comments_count'] = review_comments_count
  
          #We update the total dataframe
          review_info = pd.concat([review_info,review_info_temp], axis=0, ignore_index=True)
         
      #We close the driver
      driver.quit()
          
      return review_info
  
  
  wine_id = ['123']
  wine_url = ['https://www.vivino.com/vinilourenco-pai-horacio-grande-reserva/w/5154081?year=2015&amp;price_id=21118981']
  wine_rating_count = 186 
  
  start_time = time.time()
  reviews_info = get_reviews(wine_id, wine_url, wine_rating_count)
  elapsed_time = time.time() - start_time
  print('The scrape took: ', elapsed_time) #For this particular wine, the code took 38 seconds to run
  
  </code></pre>
  <p>The script I wrote do the following steps:</p>
  <ol>
  <li><p>With a certain wine link (ie: <a href="https://www.vivino.com/vinilourenco-pai-horacio-grande-reserva/w/5154081?year=2015&amp;price_id=21118981" rel="nofollow noreferrer">https://www.vivino.com/vinilourenco-pai-horacio-grande-reserva/w/5154081?year=2015&amp;price_id=21118981</a>), I can access that webpage with Selenium driver.</p>
  </li>
  <li><p>Then, I scroll down to the bottom of the web page.</p>
  </li>
  <li><p>I find and click the button &quot;Show more reviews&quot;</p>
  </li>
  <li><p>After pressing this button, a pop-up page appears with the wine reviews</p>
  </li>
  <li><p>I scroll down in these pop-up window until it reaches a certain amount of reviews</p>
  </li>
  <li><p>I extract the information I need from the reviews (each review is a Beautiful Soup's soup object)</p>
  </li>
  </ol>
  <p>The problem is that, if I want to <strong>scrape</strong> the reviews information of thousands of wines, it would take forever. For a single wine with 99 reviews, it takes 35 seconds to do this.</p>
  <p>Is there any way I can speed up this process?</p>