Improve performance for stanford-nlp in a "homemade" search engine I am working on a javaproject for the Information Retrieval subject in my graduation. What I'm trying to do is to reduce the lines of code I have to write due to the lack of time. We're currently on the second checkpoint, where we have to implement the engine collector component. What I'm trying to do is to use the Jsoup lib to crawl the web. That part works at the moment, but now I must parse the fetched HTML documents to retrive entites (people and places) and store the relevant pages on the disk. For that, I'm trying to use Stanford's NLP. What I'm doing to test it on one single page, at the moment is this:

// assuming all needed imports are done

String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) "
        + "AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1";
String url = "https://www.example.com";
Document htmlDocument = Jsoup.connect(url).userAgent(userAgent).get();
String text = htmlDocument.body();
String model = "english.all.3class.distsim.crf.ser.gz";
AbstractSequenceClassifier&lt;CoreLabel&gt; classifier = CRFClassifier.getClassifier(model);
System.out.println(classifier.classifyToString(text));


This works rather fine, it prints |word in the text/category|, but that execution is very slow (around 6 minutes, more or less). Is there a way to speed that up? Do I have to use other classes rather than CRFClassifier? Is Jsoup the real dragger and not the NLP process?

My internet connection is already a bottleneck to the project. The parser can't become another one, considering that the engine must collect and store thousand of pages.
Most of the time is going in loading the CRFClassifier model. You want to put the model loading outside the page processing:

// assuming all needed imports are done

String model = "english.all.3class.distsim.crf.ser.gz";
AbstractSequenceClassifier&lt;CoreLabel&gt; classifier = CRFClassifier.getClassifier(model);
String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) "
        + "AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1";
String[] pages = { "https://www.example.com" };

for (String url : pages) {
  Document htmlDocument = Jsoup.connect(url).userAgent(userAgent).get();
  String text = htmlDocument.body();
  System.out.println(classifier.classifyToString(text));
}