Improving performance in a Webdriver method I'm testing an application which is not exactly an e-commerce application but which behaves enough like one that you can think of it as an e-commerce application for the purposes of understanding this code :) 

This code relies heavily on the Selenium Webdriver library.

The code comes from the Product page, where you have searched and found a list of products. This is a utility method to get the list of products being displayed so I can verify correctness along any axis. This method only gets those products that are visible, which may differ from the total list of products if filtering is involved. 

I maintain a list of WebElements representing the container for the product display, one for each product on the page, which is initialized in initLists() (along with some other non-important lists) so that we don't have stale references. That's what's stored in allProducts.

ProductView is an inner class that encapsulates a specific Product record in the list; interactions with a given product (such as clicking any of the links for more information) are delegated to the ProductView for that Product. 

public ArrayList&lt;Product&gt; getAllVisibleProducts() {
    initLists(); //Reinitialize lists

    Utils.WaitForAjax(driver);
    ArrayList&lt;Product&gt; retval = new ArrayList&lt;Product&gt;();
    for (WebElement rProduct : allProducts) {
        try {
            if (rProduct.isDisplayed()) retval.add(new ProductView(rProduct).getProduct());
        } catch (StaleElementReferenceException e) {
            continue; //We just ignore that element
        }
    }
    return retval;
}


I'm getting hangs of sometimes multiple minutes where the test appears to do nothing on the product page. I'm currently still trying to reproduce the super-long lag, but at least part of my problem is that this method takes quite a long time to return. I put log statements at various portions of the app, and between starting the loop and returning, it was something like 44 seconds for 8 products. Am I doing too much web scraping here? 

(ProductView stores references to the elements that are interactable for the given product and attempts to build an in-memory representation of that product by scraping the page using Webdriver. The code to build the product is the vast majority of the constructor and clocks in about 70-odd lines, so I omitted it). 

Ideally I think I need to hear from someone who has used WebDriver before, since this probably boils down to WebDriver best practices.
Bunch of small things not related to your performance problem :


Utils.WaitForAjax should be Utils.waitForAjax. Method in Java start with a lower letter.
Don't return ArrayList, it's an implementation. You should use the interface List. The day you want to change to an LinkedList, it will be easier.
I recommend to always use {} for if, even if you only have just one line. But that is up to you.
I like to name ignoredException an exception that I'm ignoring. Make it more clear.




By looking at your code, I can only guess that your performance bottleneck probably is new ProductView(rProduct) or rProduct.isDisplayed(). If in ProductView your scrapping the page, it could take a lot of time depending on the driver and the method you're doing it.

But with the code your showing us right now, there is nothing particularly wrong.