6. The method of claim 1 , further comprising updating the website visitation policy for the crawling once a specified number of pages is crawled.
6. The method of claim 1 , further comprising updating the website visitation policy for the crawling once a specified number of pages is crawled. 7. The method of claim 6 , wherein updating the website visitation policy is according to an expected perplexity value of novelty regions.