title: Iterate through all links of a website using Selenium
body: <p>I'm new to Selenium and I would like to download all the <code>pdf</code>, <code>ppt(x)</code> and <code>doc(x)</code> files from a website. I have written the following code. But I'm confused how to get the inner links:</p>
  <pre><code>import java.io.*;
  import java.util.ArrayList;
  import java.util.List;
  
  import org.apache.commons.io.FileUtils;
  import org.openqa.selenium.By;
  import org.openqa.selenium.OutputType;
  import org.openqa.selenium.TakesScreenshot;
  import org.openqa.selenium.WebDriver;
  import org.openqa.selenium.WebElement;
  import org.openqa.selenium.firefox.FirefoxDriver;
  
  public class WebScraper {
  
      String loginPage = &quot;https://blablah/login&quot;;
      static String userName = &quot;11&quot;;
      static String password = &quot;11&quot;;
     static String mainPage = &quot;https://blahblah&quot;;
  
      public WebDriver driver = new FirefoxDriver();
      ArrayList&lt;String&gt; visitedLinks = new ArrayList&lt;&gt;();
  
      public static void main(String[] args) throws IOException {
  
          System.setProperty(&quot;webdriver.gecko.driver&quot;, &quot;E:\\geckodriver.exe&quot;);
  
          WebScraper webSrcaper = new WebScraper();
          webSrcaper.openTestSite();
          webSrcaper.login(userName, password);
  
          webSrcaper.getText(mainPage);
          webSrcaper.saveScreenshot();
          webSrcaper.closeBrowser();
      }
  
          /**
       * Open the test website.
       */
      public void openTestSite() {
  
          driver.navigate().to(loginPage);
      }
  
      /**
       * @param username
       * @param Password Logins into the website, by entering provided username and password
       */
      public void login(String username, String Password) {
  
          WebElement userName_editbox = driver.findElement(By.id(&quot;IDToken1&quot;));
          WebElement password_editbox = driver.findElement(By.id(&quot;IDToken2&quot;));
          WebElement submit_button = driver.findElement(By.name(&quot;Login.Submit&quot;));
  
          userName_editbox.sendKeys(username);
          password_editbox.sendKeys(Password);
          submit_button.click();
  
      }
  
      /**
       * grabs the status text and saves that into status.txt file
       *
       * @throws IOException
       */
      public void getText(String website) throws IOException {
  
          driver.navigate().to(website);
  
          try {
              Thread.sleep(10000);
          } catch (InterruptedException e) {
              e.printStackTrace();
          }
  
          List&lt;WebElement&gt; allLinks = driver.findElements(By.tagName(&quot;a&quot;));
  
          System.out.println(&quot;Total no of links Available: &quot; + allLinks.size());
  
          for (int i = 0; i &lt; allLinks.size(); i++) {
  
              String fileAddress = allLinks.get(i).getAttribute(&quot;href&quot;);
  
              System.out.println(allLinks.get(i).getAttribute(&quot;href&quot;));
              if (fileAddress.contains(&quot;download&quot;)) {
                  driver.get(fileAddress);
              } else {
  //                getText(allLinks.get(i).getAttribute(&quot;href&quot;));
              }
          }
          
      }
  
      /**
       * Saves the screenshot
       *
       * @throws IOException
       */
      public void saveScreenshot() throws IOException {
          File scrFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
          FileUtils.copyFile(scrFile, new File(&quot;screenshot.png&quot;));
      }
  
      public void closeBrowser() {
          driver.close();
      }
      
  }
  </code></pre>
  <p>I have an if clause which checks if the current link is a downloadable file (with an address including the word &quot;download&quot;). If it is, I will get it, if not, what to do? That part is my problem. I tried to implement a recursive function to retrieve the nested links and repeat the steps for the nested links, but no success.</p>
  <p>In the meantime, the first link which is found when giving <code>https://blahblah</code> as the input, is <code>https://blahblah/#</code> which refers to the same page as <code>https://blahblah</code>. It can also cause a problem, but currently, I'm trapped in another problem, namely the implementation of the recursion function. Could you please help me?</p>