Downloading Links with Python
Question: I have two sets of scripts. One to download a webpage and another to download
links from the webpage. They both run but the links script doesn't return any
scripts. Can anyone see or tell me why?

webpage script;

    
    
    import sys, urllib
    def getWebpage(url):
        print '[*] getWebpage()'
        url_file = urllib.urlopen(url)
        page = url_file.read()
        return page
    def main():
        sys.argv.append('http://www.bbc.co.uk')
        if len(sys.argv) != 2:
            print '[-] Usage: webpage_get URL'
            return
        else:
            print getWebpage(sys.argv[1])
    
    if __name__ == '__main__':
        main()
    

Links Script

    
    
    import sys, urllib, re
    import getWebpage
    def print_links(page):
        print '[*] print_links()'
        links = re.findall(r'\<a.*href\=.*http\:.+', page)
        links.sort()
        print '[+]', str(len(links)), 'HyperLinks Found:'
    
        for link in links:
            print link
    
    def main():
        sys.argv.append('http://www.bbc.co.uk')
        if len(sys.argv) != 2:
            print '[-] Usage: webpage_links URL'
            return
        page = webpage_get.getWebpage(sys.argv[1])
        print_links(page)
    


Answer: This will fix most of your problems:

    
    
    import sys, urllib, re
    
    def getWebpage(url):
        print '[*] getWebpage()'
        url_file = urllib.urlopen(url)
        page = url_file.read()
        return page
    
    def print_links(page):
        print '[*] print_links()'
        links = re.findall(r'\<a.*href\=.*http\:.+', page)
        links.sort()
        print '[+]', str(len(links)), 'HyperLinks Found:'
        for link in links:
            print link
    
    def main():
        site = 'http://www.bbc.co.uk'
        page = getWebpage(site)
        print_links(page)
    
    if __name__ == '__main__':
        main()
    

Then you can move on to fixing your regular expression.

While we are on the topic, though, I have two material recommendations:

  * [use python library `requests`](http://stackoverflow.com/questions/22676/how-do-i-download-a-file-over-http-using-python/10744565#10744565) for getting web pages
  * [use a real XML/HTML library for parsing HTML](http://stackoverflow.com/questions/2782097/python-is-there-a-built-in-package-to-parse-html-into-dom/2782492#2782492) (recommend `lxml`)

