10. A system comprising: a processor; and a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform instructions comprising: establishing a website visitation policy according to a previous crawling cycle and vocabulary gaps in a language model, wherein the website visitation policy identifies, according to a pattern of links, a likelihood of web pages to have information capable of filling the vocabulary gaps, and wherein the website visitation policy comprises a crawling schedule according to perplexity of the web pages with respect to the language model; crawling the web-pages according to the crawling schedule, to yield new vocabulary words; and generating a diverse language model according to the language model and the new vocabulary words.
10. A system comprising: a processor; and a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform instructions comprising: establishing a website visitation policy according to a previous crawling cycle and vocabulary gaps in a language model, wherein the website visitation policy identifies, according to a pattern of links, a likelihood of web pages to have information capable of filling the vocabulary gaps, and wherein the website visitation policy comprises a crawling schedule according to perplexity of the web pages with respect to the language model; crawling the web-pages according to the crawling schedule, to yield new vocabulary words; and generating a diverse language model according to the language model and the new vocabulary words. 13. The system of claim 10 , wherein the likelihood of the web pages is further according to an information theoretic measure.