5. The method of claim 1 , wherein the bidirectional web crawl considers back links and forward links.
1. A method comprising: performing a generic web crawl to identify a first webpage in a first language having a link thereon which points to a second webpage in a second language, wherein the first webpage and the second webpage comprise a bilingual website comprising the first webpage and the second webpage in respective languages; based on an analysis of parameters on the first webpage comprising at least two of: the link pointing to the second webpage, a title, a link neighborhood, a link context and data indicating a separate version of the first webpage, classifying the first webpage as a root page and as an entry point for the bilingual website via the link to the second webpage; identifying, using a visitation policy which constrains web-crawling to a graph neighborhood of bilingual websites, a pattern of links within between the first webpage and the second webpage, to yield a bipartite graph; ranking a relevance of candidate links which point to parallel text in the first webpage and the second webpage, to yield classifications of links based on the bipartite graph and the relevance; performing, based on the relevance, a bidirectional web crawl of the candidate links, to identify the first webpage and the second webpage as a bilingual website, the bidirectional web crawl utilizing the classifications of links to avoid links having a low respective relevance; analyzing the first webpage and the second webpage to identify information pairs in the first language and the second language; extracting the information pairs from the first webpage and the second webpage for use in a language translation model, the information pairs comprising at least one of a sentence pair and a paragraph pair; and updating a statistical model with domain representative data using the information pairs.