Writing a Generalized Web Crawler to map all the URLs on web sites?
I want to crawl web sites to find all all the pages on each site (domain). My simplistic attempts were quickly foiled by weird URL structures of sites. Many sites add dynamic query parameters to URLs (session ID, refering page, etc) effectively creating millions of unique URLs.<p>Is there a good strategy to find every page without getting caught in accidental tarpits like this?<p>Ideally I'd love to find some existing code that solves the problem of crawling web sites in a generalized way. Second best would be finding some great texts on the subject that I can work from while creating my own code.<p>Thanks in advance for any help.