Class structure to handle API exchange I'm developing a simple application that crawls web pages to obtain some information. For this I used and tested some libraries, like crawler4j, jsoup, jaunt and htmlunit. I change the crawler APIs several times because sometimes perceived that one served me better in a certain feature than the one I was using. I may have to do it again, and every time I do so I have to go around the code making various refactorings.

So I decide to separate the calls for these APIs in a kind of encapsulated class that holds all the operations I have to do with these API. So if I decide to change the crawler API I just point to another implementation of 
CrawlerApiHandler.

I have troubles to identify one design pattern to mitigate this problem.

Below the 2 classes and the interface I created so far:

I named with suffix handler, but I don't know if it is it's appropriate.

/**
 * Created by alexandre on 29/01/2016.
 */
public interface CrawlerApiHandler {
    String visit (String url);
}

public class JauntApiHandlerImpl implements CrawlerApiHandler {
    UserAgent userAgent;

    public JauntApiHandlerImpl(UserAgent userAgent) {
        this.userAgent = userAgent;
    }


    @Override
    public String visit(String url) {
        try {
            return userAgent.visit(url).toString();
        } catch (ResponseException e) {
            return "";
        }
    }
}

public class JsoupCrawlerApiHandler implements CrawlerApiHandler {
    public static final String USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36";

    @Override
    public String visit(String url) {
        try {
            return Jsoup.connect(url).timeout(20000)
                    .userAgent(USER_AGENT).get().toString();
        } catch (IOException e) {
            //LOG
            return "";
        }
    }
}
Honestly, I think you have it just right with your current code: simple enough, yet powerful enough.

In your case, you need to support multiple implementations of Web crawlers API. To be able to abstract your code from all those implementations, an interface needs to be introduced. This interface will define the contract that your Web crawlers will satisfy. When this interface is introduced, the rest of your code can depend on that interface only, and not on the implementation.

Right now, your contract defines a methid visit(url) that takes a URL and returns the HTML content at that URL. This could be changed at a later time (e.g. adding time-outs or error handlers) but it's a good start.



The next problem to tackle is how to use the right implementation? This is typically done by using dependency injection. To put it simply, this is a design pattern where a class lets its dependencies be injected through constructor or setters instead of constructing them itself. For your example, consider a class MyClass that would depend on a Web crawler:

public class MyClass {

    private CrawlerApiHandler crawler;

    // the crawler to use in being injected into this class
    public BodyExtractor(CrawlerApiHandler crawler) {
        this.crawler = crawler;
    }

    // later on, use crawler to do some fancy things

}


You could then centralize the real implementation in a configuration class / file. The great advantage is that you now need to define the real implementation class in a single place and the rest of the code is completely decoupled from it by just depending on the interface.