title: How to access source from a dropdown menu
body: <p>I'm really new into python and programming, I want to scrape data with <code>find_all(class_=&quot;randomname&quot;)</code> which class name can be found with using &quot;view page source&quot; from a website, but the thing is, I just realize that I want to scrape data of all of the region name that didn't show up in view page source. I mean, it supposes to be in view page source right? even it is a drop-down list when I'm searching again, I know that this menu listed with tableau or whatever its name, for sure I can right-click on one of region name and can inspect element, but I can't access it through view page source, which I think the only way to tells python to <code>find_all(class_=&quot;randomname&quot;)</code> from viewpagesource. I hope u understand what I mean. How can I access this element?</p>
  <p>this is my code, so far I can only do click using mouse position :</p>
  <pre><code>import time
  from pynput.mouse import Button, Controller as mouseC
  from pynput.keyboard import Key, Controller as keyC
  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  
  # I'm trying to use mouse click, and keyboard input later
  mouse, keyboard = mouseC(),keyC()
  
  driver = webdriver.Chrome()
  driver.get(&quot;https://bpjs-kesehatan.go.id/bpjs/&quot;)
  driver.maximize_window()
  wait = WebDriverWait(driver, 4)
  
  # what i'm trying to do here, the button that I want to click later
  # that is &quot;download button&quot; which doesnt have element like 
  # &quot;inspect element&quot;, I cant right click on it, they say cause it's tableau 
  # the only thing I can do to access this download, by using mouse click 
  # so I need to make the position visible on the screen, first thing I move
  # the screen to the nearest element that can be found to this &quot;tableau&quot;
  # then I'm doing scrolling a bit till the &quot;tableau&quot; visible
  element = driver.find_element_by_xpath(&quot;/html/body/section[2]/div&quot;)
  element.location_once_scrolled_into_view
  driver.execute_script(&quot;window.scrollTo(0, 500)&quot;)
  
  # code to take out the unneeded blue pop up
  time.sleep(4)
  mouse.position = (1017,313)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  # click Pembayaran klaim, this tab that I want to download the data
  mouse.position = (337,269)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  # Click section provinsi, this tab that I want to access/scrape all of available option
  mouse.position = (1143,322)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  # typing the provinsi name, what I planned is if I can access/scrape all of this element, I can do the &quot;looping download&quot; as many as this items
  for char in &quot;BALI&quot;:
      time.sleep(0.12)
      keyboard.press(char)
      keyboard.release(char)
  time.sleep(4)
  mouse.position = (1036,362)    
  mouse.click(Button.left,1)
  time.sleep(4)
      
  #klik blok RS
  mouse.position = (1134,369)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  #pilih RS
  for char in &quot;RS BALIMED&quot;:
      time.sleep(0.12)
      keyboard.press(char)
      keyboard.release(char)
  time.sleep(4)
  mouse.position = (1055,418)    
  mouse.click(Button.left,1)
  time.sleep(4)
  
  #klik Download
  mouse.position = (1092,645)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  #klik PDF
  mouse.position = (729,491)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  #klik Create PDF
  mouse.position = (776,570)
  mouse.click(Button.left,1)
  time.sleep(4)
  </code></pre>
  <p>I'm trying to add this, to prove that it can't access this &quot;tableau&quot; dropdown element of Province, there are some other code bfore this, but the target is to actiavate this code below, so it can detect all of province element which I find on class FIItem FISimpleDropdownItem, but it returns [], so I assume, this phyton cant access this element due it's not in &quot;viewpagesource&quot;.</p>
  <pre><code>x=[]
  x=link_soup.find_all(class_=&quot;FIItem FISimpleDropdownItem&quot;)
  print(x)
  </code></pre>
  <p>after searching this is my code to not click by using mouse.position, or to not typing by keyboard, but to select element with using &quot;switch frame&quot; :</p>
  <pre><code>#klik blok provinsi
  mouse.position = (1143,322)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  wait = WebDriverWait(driver, 4)
  frame = driver.find_element_by_class_name('tableauViz')
  driver.switch_to.frame(frame)
  driver.find_element_by_id(&quot;FI_sqlproxy.0m3kdgc0og85y318c97vy111o7p3,none:Nmprop:nk4537390993424018834_13946604467497001081_5&quot;).click()
  driver.switch_to.default_content()
  time.sleep(2)
  
  #klik blok RS
  mouse.position = (1134,369)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  driver.switch_to.frame(frame)
  driver.find_element_by_id(&quot;FI_sqlproxy.0m3kdgc0og85y318c97vy111o7p3,none:Nmppk:nk4537390993424018834_13946604467497001081_3&quot;).click()
  driver.switch_to.default_content()
  time.sleep(2)
  </code></pre>
  <p>it works, and can download with given province, and RUmah sakit (hospital), and the next challenge for me is to list all the items in those dropdowns menu, im trying to use the idea of 'find_all(class_=&quot;randomname&quot;)', which I need to add this to my code :</p>
  <pre><code>page_source = driver.page_source
  link_soup = bs4.BeautifulSoup(page_source,'html.parser')
  </code></pre>
  <p>and I want to check if it can list all the available items in those dropdowns of province by :</p>
  <pre><code>x = []
  x = link_soup.find_all(class_='FIItem FISimpleDropdownItem')
  print(x)
  </code></pre>
  <p>it returns no error, but :</p>
  <pre><code>[]
  </code></pre>
  <p>what's wrong? I mean, I assume I'm doing the right thing, cause I finally can select the items by element in both dropdowns menu, but whn I'm trying to list all availabe option, for looping needs later, it returns []. Can you give me advice? Thanks.</p>
  <p>Well finally my program runs, and it can download the files, by choosing the 1st dropdown menu which is &quot;Province&quot;, and for each province, it will download the file for each &quot;Hospital&quot; which is 2nd dropdown menu. Please kindly test this code, it works on my PC, but I believe u guys knows how to modify it.</p>
  <pre><code>import time, bs4, requests
  from pynput.mouse import Button, Controller as mouseC
  from pynput.keyboard import Key, Controller as keyC
  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  
  mouse, keyboard = mouseC(),keyC()
  
  driver = webdriver.Chrome()
  driver.get(&quot;https://bpjs-kesehatan.go.id/bpjs/&quot;)
  driver.maximize_window()
  wait = WebDriverWait(driver, 4)
  element = driver.find_element_by_xpath(&quot;/html/body/section[2]/div&quot;)
  element.location_once_scrolled_into_view
  driver.execute_script(&quot;window.scrollTo(0, 500)&quot;)
  
  #hilangkan window biru
  time.sleep(4)
  mouse.position = (1017,313)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  #klik Pembayaran klaim
  mouse.position = (337,269)
  mouse.click(Button.left,1)
  time.sleep(4)
  
  ###########################################
  provinsi = []
  RS = []
  frame = driver.find_element_by_class_name('tableauViz')
  driver.switch_to.frame(frame)
  
  #list provinsi
  mouse.position = (1143,322)
  mouse.click(Button.left,1)
  time.sleep(2)
  
  page_source = driver.page_source
  link_soup = bs4.BeautifulSoup(page_source,'html.parser')
  
  for e in link_soup.find_all(class_='FIItem FISimpleDropdownItem'):
      provinsi.append(e.text)
  driver.switch_to.default_content()
  time.sleep(2)
  mouse.position = (1143,322)
  mouse.click(Button.left,1)
  time.sleep(2)
  
  for i in range(len(provinsi)):
      mouse.position = (1143,322)
      mouse.click(Button.left,1)
      time.sleep(2)
      driver.switch_to.frame(frame)
      driver.find_element_by_id(&quot;FI_sqlproxy.0m3kdgc0og85y318c97vy111o7p3,none:Nmprop:n    k4537390993424018834_13946604467497001081_&quot;+str(i)).click()
      driver.switch_to.default_content()
      time.sleep(5)
  
  mouse.position = (1134,369)
  mouse.click(Button.left,1)
  time.sleep(2)
  driver.switch_to.frame(frame)
  for j in link_soup.find_all(class_='FIItem FISimpleDropdownItem'):
      RS.append(j.text)
  driver.switch_to.default_content()
  mouse.position = (1134,369)
  mouse.click(Button.left,1)
  
  #klik blok RS
  for k in range(len(RS)):
      print(len(RS))
      mouse.position = (1134,369)
      mouse.click(Button.left,1)
      time.sleep(4)    
      driver.switch_to.frame(frame)
      driver.find_element_by_id(&quot;FI_sqlproxy.0m3kdgc0og85y318c97vy111o7p3,none:Nmppk:nk4537390993424018834_13946604467497001081_&quot;+str(k)).click()
      driver.switch_to.default_content()
      time.sleep(10)
  
      #klik Download
      mouse.position = (1092,645)
      mouse.click(Button.left,1)
      time.sleep(4)
  
      #klik PDF
      mouse.position = (729,491)
      mouse.click(Button.left,1)
      time.sleep(4)
  
      #klik Create PDF
      mouse.position = (776,570)
      mouse.click(Button.left,1)
      time.sleep(10)
  </code></pre>
  <p>summary of my latest code, is im combining these :</p>
  <ol>
  <li>selenium selecting element class</li>
  <li>requests bs4 to list all the available province and hospital for each dropdowns, which I can only access if I automating click the dropdowns first by using &quot;3rd point&quot; which is ..</li>
  <li>mouse event</li>
  </ol>
  <p>it works, but the next challenge is to prevent the program running or like stop for a while, if it loads or someting, thats why I add delay like 10 seconds for each download, but sometimes, it delays more than that, which makes my programs seems cant run well, cause it exceeds the given delay. The question how to tell the program to &quot;wait this line to run completed first, then continue with the next line&quot;.</p>