1. A method comprising: establishing a website visitation policy according to a previous crawling cycle and vocabulary gaps in a language model, wherein the website visitation policy identifies, according to a pattern of links, a likelihood of web pages to have information capable of filling the vocabulary gaps, and wherein the website visitation policy comprises a crawling schedule according to perplexity of the web pages with respect to the language model; crawling, via a processor, the web-pages according to the crawling schedule, to yield new vocabulary words; and generating a diverse language model according to the language model and the new vocabulary words.
1. A method comprising: establishing a website visitation policy according to a previous crawling cycle and vocabulary gaps in a language model, wherein the website visitation policy identifies, according to a pattern of links, a likelihood of web pages to have information capable of filling the vocabulary gaps, and wherein the website visitation policy comprises a crawling schedule according to perplexity of the web pages with respect to the language model; crawling, via a processor, the web-pages according to the crawling schedule, to yield new vocabulary words; and generating a diverse language model according to the language model and the new vocabulary words. 4. The method of claim 1 , wherein the likelihood of the web pages is further according to an information theoretic measure.