title: StaleElementReferenceException when crawling a website using Selenium
body: <p>I am using <code>Selenium</code> to crawl a web page with lots of inner links.
  I have the following code:</p>
  <pre><code>import org.openqa.selenium.*;
  import org.openqa.selenium.firefox.FirefoxDriver;
  import java.util.ArrayList;
  import java.util.List;
  
  public class WebScraper {
      //list to save visited links
      static List&lt;String&gt; linkAlreadyVisited = new ArrayList&lt;String&gt;();
      static String userName = &quot;user&quot;;
      static String password = &quot;passw&quot;;
      static String mainPage = &quot;https://web/&quot;;
      WebDriver driver;
      //    public WebDriver driver = new FirefoxDriver();
      String loginPage = &quot;https://web/Login&quot;;
  
      public WebScraper(WebDriver driver) {
          this.driver = driver;
      }
  
      public static void main(String[] args) throws InterruptedException {
  
          System.setProperty(&quot;webdriver.gecko.driver&quot;, &quot;E:\\geckodriver.exe&quot;);
  
          WebDriver driver = new FirefoxDriver();
  
          WebScraper webScraper = new WebScraper(driver);
          webSrcapper.openTestSite();
          webSrcapper.login(userName, password);
          driver.navigate().to(mainPage);
  
          driver.get(mainPage);
          // start recursive linkText
          new WebScraper(driver).linkTest();
      }
  
      public static boolean isElementStale(WebElement e) {
          try {
              e.isDisplayed();
              return false;
          } catch (StaleElementReferenceException ex) {
              return true;
          }
      }
  
      public void linkTest() {
          // loop over all the a elements in the page
          for (WebElement link : driver.findElements(By.tagName(&quot;a&quot;))) {
              // Check if link is displayed and not previously visited
              if (!isElementStale(link)
                      &amp;&amp; !linkAlreadyVisited.contains(link.getText())) {
                  // add link to list of links already visited
                  linkAlreadyVisited.add(link.getText());
                  System.out.println(link.getText());
                  try {
                      Thread.sleep(200);
                  } catch (InterruptedException e) {
                      e.printStackTrace();
                  }
  
                  try {
                      link.click();
  
                  } catch (Exception ex) {
  
  //                    String id = link.getAttribute(&quot;id&quot;);
                      ((JavascriptExecutor) driver).executeScript(&quot;$('#id').click();&quot;);
  
                  }
                  // call recursiveLinkTest on the new page
                  new WebScraper(driver).linkTest();
  
              } else {
  continue;
              }
          }
          driver.navigate().back();
      }
  
      /**
       * //     * Open the test website. //
       */
      public void openTestSite() {
  
          driver.navigate().to(loginPage);
      }
  
      public void login(String username, String Password) {
  
          WebElement userName_editbox = driver.findElement(By.id(&quot;IDToken1&quot;));
          WebElement password_editbox = driver.findElement(By.id(&quot;IDToken2&quot;));
          WebElement submit_button = driver.findElement(By.name(&quot;Login.Submit&quot;));
  
          userName_editbox.sendKeys(username);
          password_editbox.sendKeys(Password);
          submit_button.click();
  
      }
  }
  </code></pre>
  <p>This code works for around one hour and then I get a <code>StaleElementReferenceException</code>. As there are lots of links on the webpage, I can simply ignore that link and it's not that important to me to follow every single link.</p>
  <p>So, I tried to escape this exception with the <code>else clause</code> including <code>continue;</code> command. But it doesn't work. My question is why? I just want to go to the next link.</p>
  <p>In the meantime, as the code takes a long time to run, I don't know where exactly it happens (also, I have seen that sometimes the code runs differently, that is the order of the links are not always the same, so I can't debug easily) and I can't check that specific link.</p>
  <p>I tried different solutions that I found on the websites. For example, <code>Thread.sleep</code> after and before clicking on a link and <code>new WebDriverWait(driver, 10)</code> .</p>
  <p>Does any one have any solution for this problem?</p>