so now we have collection of all the links in the first page what can we do with itwe can' just pop the links into set to remove duplicates because links may be relative or absolute for examplecontact html and /contact html point to the same page so the first thing we should do is normalize all the links to their full urlincluding hostname and relative path we can do this by adding normalize_url method to our objectdef normalize_url(selfpathlink)if link startswith("return link elif link startswith("/")return self url link elsereturn self url path rpartition'/')[ '/link this method converts each url to complete address that includes protocol and hostname now the two contact pages have the same value and we can store them in set we'll have to modify __init__ to create the setand collect_links to put all the links into it thenwe'll have to visit all the non-external links and collect them too but wait minuteif we do thishow do we keep from revisiting link when we encounter the same page twiceit looks like we're actually going to need two setsa set of collected linksand set of visited links this suggests that we were wise to choose set to represent our datawe know that sets are most useful when we're manipulating more than one of them let' set these upclass linkcollectordef __init__(selfurl)self url "self collected_links set(self visited_links set(def collect_links(selfpath="/")full_url self url path self visited_links add(full_urlpage str(urlopen(full_urlread()links link_regex findall(pagelinks {self normalize_url(pathlink for link in linksself collected_links links unionself collected_linksunvisited_links links differenceself visited_links