title: Navigating through links using Selenium in Python
body: <p>I'm trying to scrape data from a site with multiple pages linked via a <strong>NEXT</strong> button </p>
  
  <p>The successive page URL has no correspondence with the previous page URL as one might assume</p>
  
  <p>(In that case modifying the path would've solved the problem)</p>
  
  <p>This is what I plan to do -</p>
  
  <p>1.Start with an initial URL </p>
  
  <p>2.Extract information</p>
  
  <p>3.Click <strong>NEXT</strong></p>
  
  <p>Repeat 2 and 3 <em>n</em> times</p>
  
  <p>Specifically, I wanted to know how to get the new page URL on clicking </p>
  
  <p>This is what I've come up with so far</p>
  
  <pre><code>def startWebDriver():
      global driver
      options = Options()
      options.add_argument("--disable-extensions")
      driver = webdriver.Chrome(executable_path = '/path/to/driver/chromedriver_linux64/chromedriver',options=options)
  
  #URL of the initial page
  driver.get('https://openi.nlm.nih.gov/detailedresult.php?img=CXR1_1_IM-0001-3001&amp;query=&amp;coll=cxr&amp;req=4&amp;npos=1')
  
  time.sleep(4)
  
  #XPATH of the "NEXT" button
  element = driver.find_element_by_xpath('//*[@id="imageClassM"]/div/a[2]/img').click()
  </code></pre>
  
  <p>Any help would be appreciated</p>