title: performance improvements for web scraping?
body: <p>As part of a personal project, I tried coding a web scraper which accesses my instagram account to scrape through all the DMs in a given conversation. </p>
  
  <p>It works quite well, to some extent; my problem is that the group conversation I'm trying to scrape is quite active and goes back to 2017 (so it has A LOT of messages), and at some point, the chromeengine just lags so much that the entire thing times out and crashes.<br>
  Is there any way that I can improve the performance? Maybe there's an entirely different way I should be doing this altogether?</p>
  
  <pre><code>def userlist():
      #create my selenium instance
      options = webdriver.ChromeOptions()
      #options.add_argument('headless')
      options.add_argument('window-size=1200x600')
      driver = webdriver.Chrome(chrome_options=options)
  
      #Log into instagram
      driver.get("https://instagram.com")
      WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.NAME, "username")))
      driver.find_element_by_name("username").send_keys("###############")
      driver.find_element_by_name("password").send_keys("###############")
      driver.find_element_by_xpath("//button[contains(@class, 'sqdOP  L3NKy   y3zKF     ')]").click()
      sleep(5)
      driver.find_element_by_xpath("//button[contains(@class, 'sqdOP yWX7d    y3zKF     ')]").click()
      sleep(3)
      driver.find_element_by_xpath('//button[contains(@class, "aOOlW   HoLwm ")]').click()
      sleep(1)
  
      #go to the group convo
      driver.get("https://instagram.com/direct/inbox")
      sleep(3)
      driver.find_element_by_xpath(
          '//a[contains(@href, "/direct/t/###################")]').click()
      sleep(1)
      print("Verbindung zur Gruppe hergestellt!")
      print("")
      print("Beginne Auszählung, bitte etwas Geduld haben...")
  
      #scroll to the top to load older messages, until that isn't possible anymore - do this by checking for scrollTop &gt; 0 every 3 seconds
      i = 1
      while int(driver.find_element_by_xpath('//div[contains(@class, "frMpI  -sxBV")]').get_attribute("scrollTop")) &gt; 0:
          driver.execute_script("document.getElementsByClassName('frMpI  -sxBV')[0].scrollTop = 0")
          sleep(3)
          print(f"Schritt {i}...")
          print("")
          i += 1
  
      #add author names to a list
      for element in driver.find_elements_by_xpath("//div[contains(@class, '_7UhW9  PIoXz       MMzan   _0PwGv            fDxYl')]"):
          text = element.get_attribute("innerHTML")
          if not re.match(".*[0-9]:.*", text):
              userslist.append(element.get_attribute("innerHTML"))
  
      print("")
  </code></pre>