links link_regex findall(pagelinks self normalize_url(urlparse(urlpathlinkfor link in links self collected_links[urllinks for link in linksself collected_links setdefault(linkset()unvisited_links links difference(self visited_linksfor link in unvisited_linksif link startswith(self url)queue put(linkdef normalize_url(selfpathlink)if link startswith("return link rstrip('/'elif link startswith("/")return self url link rstrip('/'elsereturn self url path rpartition('/')[ '/link rstrip('/'if __name__ ="__main__"collector linkcollector(sys argv[ ]collector collect_links(for linkitem in collector collected_links items()print("% % (linkitem) had to manually strip any trailing forward slashes in the normalize_url method to remove duplicates in this version of the code because the end result is an unsorted dictionarythere is no restriction on what order the links should be processed in thereforewe could just as easily have used lifoqueue instead of queue here priority queue probably wouldn' make lot of sense since there is no obvious priority to attach to link in this case exercises the best way to learn how to choose the correct data structure is to do it wrong few times take some code you've recently writtenor write some new code that uses list try rewriting it using some different data structures which ones make more sensewhich ones don'twhich have the most elegant code