title: Python Web Scraping: Clicking links one by one on Ajax Site
body: <p>I have a list of search criteria saved in a csv file. I'd like to loop through each search criteria to generate the corresponding search results on a website. For each set of search results generated (which are links), I'd like to click into the link and then grab the data from the new page generated. Unfortunately, I am experiencing problems going into each link. If anyone could please kindly provide some insight, it would be <strong>much appreciated</strong>. </p>
  
  <pre><code>import pandas as pd
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  from bs4 import BeautifulSoup
  
  # read list of CAS Numbers to be searched
  data = pd.read_csv("NPRI CACs.csv", names=["CAS Number", "Chemical Name"])
  data.dropna()
  CAS = data["CAS Number"]
  
  # Parameters to be called
  url = 'http://www.lifelabs.msdss.com/Login.aspx?ReturnUrl=%2fMainMenu.aspx%3ffm%3d0%26tb%3d0'
  
  # Sign into SafeTec
  browser = webdriver.Firefox()
  browser.get(url)
  browser.find_element_by_class_name("text").click()
  
  # Conduct MSDS Searches on SafeTec
  for i in range(10):
      try:
          Ingredient_CAS_Number = browser.find_element_by_id("placeBody_dynField48_txtTextBox")
          Ingredient_CAS_Number.send_keys(CAS[i])
          browser.find_element_by_id("placeBody_linkSearchBottom").click()
  
          list_links = browser.find_elements_by_css_selector("a[href*='MSDSDetail']")
          links = []
          for j in range(len(list_links)):
              links.append(list_links[j].get_attribute('href'))
  
          Product_Name = []
          for link in links:
              browser.get(link)
              product = browser.find_element_by_id("placeBody_dynField1_txtTextBox")
              Product_Name.append(product)
          print(Product_Name)
  
          browser.get(url)
      except:
          print(CAS[i])
          continue
  </code></pre>