before we get into thatlet' start with the basics what code do we need to connect to page and parse all the links from that pagefrom urllib request import urlopen from urllib parse import urlparse import re import sys link_regex re compile"]*href=['\"]([^'\"]+)['\"][^>]*>"class linkcollectordef __init__(selfurl)self url "urlparse(urlnetloc def collect_links(selfpath="/")full_url self url path page str(urlopen(full_urlread()links link_regex findall(pageprint(linksif __name__ ="__main__"linkcollector(sys argv[ ]collect_links(this is short piece of codeconsidering what it' doing it connects to the server in the argument passed on the command linedownloads the pageand extracts all the links on that page the __init__ method uses the urlparse function to extract just the hostname from the urlso even if we pass in htmlit will still operate on the top level of the host makes sensebecause we want to collect all the links on the sitealthough it assumes every page is connected to the index by some sequence of links the collect_links method connects to and downloads the specified page from the serverand uses regular expression to find all the links in the page regular expressions are an extremely powerful string processing tool unfortunatelythey have steep learning curveif you haven' used them beforei strongly recommend studying any of the entire books or websites on the topic if you don' think they're worth knowingtry writing the preceding code without them and you'll change your mind the example also stops in the middle of the collect_links method to print the value of links this is common way to test program as we're writing itstop and output the value to ensure it is the value we expect here' what it outputs for our example['contact html''blog html''esme html''/hobbies html''/contact html''