How to reach the mediafire direct link that is hidden behind a captcha?
Question: I wrote a python program to download a file from the internet :

    
    
     url = "http://download2163.mediafire.com/icum151v51zg/55rll9s5ioshz5n/Alcohol52_FE_2-0-3-6850.exe"
     file_name ='file'
     u = urllib2.urlopen(url)
     f = open(file_name, 'wb')
     buffer = u.read()
     f.write(buffer)
     f.close()
    

And it work correctly. The problem is that in this program the link that is
used to download the file is not costant ! The file that i want to download
was been uploaded using mediafire. I found out that the link of this page
(<http://www.mediafire.com/download/55rll9s5ioshz5n/Alcohol52_FE_2-0-3-6850.exe>)
is costant, and in this page I found the link that i put in my program. Infact
by clicking on the button "download (6.77 MB)" with the right button of my
mouse and selecting "gain this link" , I gained the direct link that I used in
my program :
<http://download2163.mediafire.com/icum151v51zg/55rll9s5ioshz5n/Alcohol52_FE_2-0-3-6850.exe>

But this second direct link - that is the direct link I really need - is
variable!

I have found the way to gain this variable and important direct link : using
the first and costant
link(<http://www.mediafire.com/download/55rll9s5ioshz5n/Alcohol52_FE_2-0-3-6850.exe>)
I downloaded the HTML page, and inside of this HTML file I found the direct
link that I needed!

The problem is: sometimes when my python program try to download the HTML page
it download the right page that contain the direct link, but sometime it
download wrong one, with the captcha! So the direct link can not be founded.

I' m looking for a way to avoid this captcha and to be sure that my python
program Always download the correct HTML page with the direct link inside !

Any suggestions ?

* * *

If there isn't any way, does anyone know how can I gain the direct link of a
file that I want upload on the internet and that I want to be downloaded by my
python program ?


Answer: You'll need to look for something in the page that you can use to always
identify the link. For example, the download link is in a div element with
class "download-link". You can parse the HTML for that div, then grab the link
from it's child element. There are other possibilities too. For example, you
could look for something unique and constant in the URL of interest and use a
regular expression to select for it after grabbing all links from the page.

I'd highly recommend looking into the BeautifulSoup library, which will allow
you to easily parse HTML.

EDIT: Okay, I didn't notice this because I initially looked at the page in my
browser, but apparently mediafire only populates the download div with
javascript after the page has loaded, which makes scraping the link a lot
harder. Thankfully, they still have to include the download link and using an
ugly, hideous little hack we can grab it:

First, you'll need this regex for URLs:
<http://daringfireball.net/2010/07/improved_regex_for_matching_urls>

Then grab the page contents and parse with beautifulsoup as such:

    
    
    soup = BeautifulSoup(page)
    div_tag = soup.find_all(class_="download_link")[0]
    script_tag = div_tag("script")[0]
    link = re.findall(regex, script_tag.contents[0])[0]
    

Here's my whole working code:

    
    
    import requests
    import re
    from bs4 import BeautifulSoup
    
    pre_regex = r"""(?xi)
    \b
    (                           # Capture 1: entire matched URL
      (?:
        [a-z][\w-]+:                # URL protocol and colon
        (?:
          /{1,3}                        # 1-3 slashes
          |                             #   or
          [a-z0-9%]                     # Single letter or digit or '%'
                                        # (Trying not to match e.g. "URI::Escape")
        )
        |                           #   or
        www\d{0,3}[.]               # "www.", "www1.", "www2." ... "www999."
        |                           #   or
        [a-z0-9.\-]+[.][a-z]{2,4}/  # looks like domain name followed by a slash
      )
      (?:                           # One or more:
        [^\s()<>]+                      # Run of non-space, non-()<>
        |                               #   or
        \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
      )+
      (?:                           # End with:
        \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
        |                                   #   or
        [^\s`!()\[\]{};:'".,<>?]        # not a space or one of these punct chars
      )
    )"""
    regex = re.compile(pre_regex)
    
    url = "http://www.mediafire.com/download/raju14e8aq6azbo/Getting+Started+with+MediaFire.pdf"
    s = requests.session()
    result = s.get(url)
    soup = BeautifulSoup(result.content)
    
    div_tag = soup.find_all(class_="download_link")[0]
    script_tag = div_tag("script")[0]
    link = re.findall(regex, script_tag.contents[0])[0][0]
    
    print link
    

