Python mechanize download file with original filename
Question: Working on a python script to scraping multi files from a website.

The download form html is something like this:

    
    
    <span>
        <a class="tooltip" href="download.php?action=download&amp;id=xxx&amp;authkey=yyy&amp;pass=zzz" title="Download">DL</a>
    </span>
    

What I'm thinking of is:

    
    
    f1 = open('scraping.log', 'a')
    page = br.open(url)
    html = page.read()
    soup = BeautifulSoup(html)
    
    for a in soup.select('a[href^="download.php?action=download"]'):
            link = a.attrs.get('href')
            print >>f1, link
            br.retrieve(url+link, destination)
    

However, for the retrieve I have to state out the output filename. I want to
get the original filename instead of setting it as a random name. Is there any
way to do that? Moreover, as I want to add this script to run frequently in
crontab, is there a way for me to set us check the scraping.log and skip those
that have been downloaded before?


Answer: If you don't like "download.php", check for a [Content-Disposition
header](http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1),
e.g.:

    
    
    Content-Disposition: attachment; filename="fname.ext"
    

And ensure the filename [complies with your
intent](http://www.ietf.org/rfc/rfc2183.txt):

> It is important that the receiving MUA not blindly use the suggested
> filename. The suggested filename SHOULD be checked (and possibly changed) to
> see that it conforms to local filesystem conventions, does not overwrite an
> existing file, and does not present a security problem (see Security
> Considerations below).

Python 2:

    
    
    import re
    import mechanize  # pip install mechanize
    
    br = mechanize.Browser()
    r = br.open('http://yoursite.com')
    #print r.info()['Content-Disposition']
    unsafe_filename = r.info().getparam('filename')  # Could be "/etc/evil".
    filename = re.findall("([a-zA-Z0-9 _,()'-]+[.][a-z0-9]+)$", unsafe_filename)[0]  # "-]" to match "-".
    

As for skipping links you've processed before,

    
    
    f1 = open('scraping.log', 'a')
    processed_links = f1.readlines()
    page = br.open(url)
    html = page.read()
    soup = BeautifulSoup(html)
    
    for a in soup.select('a[href^="download.php?action=download"]'):
        link = a.attrs.get('href')
        if not link in processed_links:
            print >>f1, link
            processed_links += [link]
            br.retrieve(url+link, destination)
    

