Downloading all links on a webpage using Mechanize in Python
Question: I was trying to follow the following thread which seemed to answer my
question. It serves as a great example that shows how to download all links on
a webpage using Mechanize:

[Download all the links(related documents) on a webpage using
Python](http://stackoverflow.com/questions/5974595/download-all-the-
linksrelated-documents-on-a-webpage-using-python)

I followed the code that was posted (i.e.):

    
    
    import mechanize
    from time import sleep
    #Make a Browser (think of this as chrome or firefox etc)
    br = mechanize.Browser()
    
    #visit http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
    #for more ways to set up your br browser object e.g. so it look like mozilla
    #and if you need to fill out forms with passwords.
    
    # Open your site
    br.open('http://pypi.python.org/pypi/xlwt')
    
    f=open("source.html","w")
    f.write(br.response().read()) #can be helpful for debugging maybe
    
    filetypes=[".zip",".exe",".tar.gz"] #you will need to do some kind of pattern matching on your files
    myfiles=[]
    for l in br.links(): #you can also iterate through br.forms() to print forms on the page!
        for t in filetypes:
            if t in str(l): #check if this link has the file extension we want (you may choose to use reg expressions or something)
                myfiles.append(l)
    
    
    def downloadlink(l):
        f=open(l.text,"w") #perhaps you should ensure that file doesn't already exist.
    
        br.click_link(l)
        f.write(br.response().read())
        print l.text," has been downloaded"
        #br.back()
    
    for l in myfiles:
        sleep(1) #throttle so you dont hammer the site
        downloadlink(l)
    

i only changed:

    
    
    f=open(l.text,"w") #perhaps you should open in a better way & ensure that file doesn't already exist.
    

To:

    
    
    f=open('C:\\l.text',"w") #perhaps you should open in a better way & ensure that file doesn't already exist.
    

That made the code work for me, else it was giving me an error. When i run the
code, i get the following output:

    
    
    Download> xlwt-0.7.5.tar.gz has been downloaded 
    xlwt-0.7.5.tar.gz has been downloaded
    

So it worked. But i have no idea where this file was downloaded to? Any ideas?
I have searched my C drive, and could not find it.

If the code is run as:

    
    
    f=open(l.text,"w")
    

It raises the following exception:

    
    
    Traceback (most recent call last):
      File "C:\Python27\mech.py", line 33, in <module>
    downloadlink(l)
      File "C:\Python27\mech.py", line 25, in downloadlink
    f=open(l.text,"w") #perhaps you should ensure that file doesn't already exist.
    IOError: [Errno 22] invalid mode ('w') or filename: 'Download> <span style="font-size: 75%">xlwt-0.7.5.tar.gz<span>'
    


Answer: The Python code you quoted uses the `text` attribute of the link `l` (hence
the expression `l.text`) as the filename. Consequently (since each link should
hopefully have a different `text` attribute value) the code should produce a
number of files, one for each link.

Your change replaces a variable expression (one which has a different value
for each link) with a constant. So each file is being written to the `C:\`
directory as `l.text`. Consequently when you look at this file you should see
the contexts of the last link on the page.

(By the way, not your fault I know, but `l` is a very bad name for a variable
due to its potential for confusion with the digit one).

The correct way to run this program is inside an empty directory (otherwise
the individual files will be hard to track down) on which you have write
permission. If any of the filenames contain slashes then you will have to take
special pains to either create the necessary directory structure or transform
them somehow into acceptable Windows filenames.

You may also wish to replace the detection code with something a little more
colloquial.

    
    
    for l in br.links(): #you can also iterate through br.forms() to print forms on the page!
        s = str(l)
        if any(s.endswith(t) for t in filetypes):
            myfiles.append(l)
    

