title: Designing a web crawler
body: <p>I have come across an interview question "If you were designing a web crawler, how would you avoid getting into infinite loops? " and I am trying to answer it.</p>
  
  <p>How does it all begin from the beginning.
  Say Google started with some hub pages say hundreds of them (How these hub pages were found in the first place is a different sub-question).
  As Google follows links from a page and so on, does it keep making  a hash table to make sure that it doesn't follow the earlier visited pages.</p>
  
  <p>What if the same page has 2 names (URLs) say in these days when we have URL shorteners etc..</p>
  
  <p>I have taken Google as an example. Though Google doesn't leak how its web crawler algorithms and page ranking etc work, but any guesses?</p>