title: Selenium scrolling and scraping with BeautifulSoup produces duplicate results
body: <p>I have this script to download images from Instagram. The only issue I am having is that when Selenium starts scrolling down to the bottom of the webpage, BeautifulSoup starts grabbing the same <code>img src</code> links after requests is being looped. </p>
  
  <p>Although it will continue to scroll down and download pictures, after all that is done, I end up having 2 or 3 duplicates. So my question is is there a way of preventing this duplication from happening? </p>
  
  <pre class="lang-py prettyprint-override"><code>import requests
  from bs4 import BeautifulSoup
  import selenium.webdriver as webdriver
  
  
  url = ('https://www.instagram.com/kitties')
  driver = webdriver.Firefox()
  driver.get(url)
  
  scroll_delay = 0.5
  last_height = driver.execute_script("return document.body.scrollHeight")
  counter = 0
  
  print('[+] Downloading:\n')
  
  def screens(get_name):
      with open("/home/cha0zz/Desktop/photos/img_{}.jpg".format(get_name), 'wb') as f:
          r = requests.get(img_url)
          f.write(r.content)
  
  while True:
  
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      time.sleep(scroll_delay)
      new_height = driver.execute_script("return document.body.scrollHeight")
  
      soup = BeautifulSoup(driver.page_source, 'lxml')
      imgs = soup.find_all('img', class_='_2di5p')
      for img in imgs:
          img_url = img["src"]
          print('=&gt; [+] img_{}'.format(counter))
          screens(counter)
          counter = counter + 1
  
      if new_height == last_height:
          break
      last_height = new_height
  </code></pre>
  
  <p><br/></p>
  
  <p><strong>Update:</strong>
  So I placed this part of the code outside of <code>while True</code> and let selenium load the whole page first in order to hopefully have bs4 scrape all the images. It works to number 30 only and then stops.</p>
  
  <pre class="lang-py prettyprint-override"><code>soup = BeautifulSoup(driver.page_source, 'lxml')
  imgs = soup.find_all('img', class_='_2di5p')
  for img in imgs:
      #tn = datetime.now().strftime('%H:%M:%S')
      img_url = img["src"]
      print('=&gt; [+] img_{}'.format(counter))
      screens(counter)
      counter = counter + 1
  </code></pre>