return link elif link startswith("/")return self url link elsereturn self url path rpartition('/)[ '/link if __name__ ="__main__"collector linkcollector(sys argv[ ]collector collect_links(for linkitem in collector collected_links items()print("{}{}format(linkitem)it is surprisingly small changethe line that originally created union of two sets has been replaced with three lines that update the dictionary the first of these simply tells the dictionary what the collected links for that page are the second creates an empty set for any items in the dictionary that have not already been added to the dictionaryusing setdefault the result is dictionary that contains all the links as its keysmapped to sets of links for all the internal linksand empty sets for the external links finallyinstead of recursively calling collect_linkswe can use queue to store the links that haven' been processed yet this implementation won' support itbut this would be good first step to creating multithreaded version that makes multiple requests in parallel to save time from urllib request import urlopen from urllib parse import urlparse import re import sys from queue import queue link_regex re compile("]*href=['\"]([^'\"]+)['\"][^>]*>"class linkcollectordef __init__(selfurl)self url "self collected_links {self visited_links set(def collect_links(self)queue queue(queue put(self urlwhile not queue empty()url queue get(rstrip('/'self visited_links add(urlpage str(urlopen(urlread()