Class structure to handle API exchange
Honestly, I think you have it just right with your current code: simple enough, yet powerful enough.

In your case, you need to support multiple implementations of Web crawlers API. To be able to abstract your code from all those implementations, an interface needs to be introduced. This interface will define the contract that your Web crawlers will satisfy. When this interface is introduced, the rest of your code can depend on that interface only, and not on the implementation.

Right now, your contract defines a methid visit(url) that takes a URL and returns the HTML content at that URL. This could be changed at a later time (e.g. adding time-outs or error handlers) but it's a good start.



The next problem to tackle is how to use the right implementation? This is typically done by using dependency injection. To put it simply, this is a design pattern where a class lets its dependencies be injected through constructor or setters instead of constructing them itself. For your example, consider a class MyClass that would depend on a Web crawler:

public class MyClass {

    private CrawlerApiHandler crawler;

    // the crawler to use in being injected into this class
    public BodyExtractor(CrawlerApiHandler crawler) {
        this.crawler = crawler;
    }

    // later on, use crawler to do some fancy things

}


You could then centralize the real implementation in a configuration class / file. The great advantage is that you now need to define the real implementation class in a single place and the rest of the code is completely decoupled from it by just depending on the interface.