Python urllib2 with headers not working
Question: I am trying to scrape a page for a certain url and I am using BeautifulSoup to
do so, with a request from urllib2. The problem I am having right now is that
it sends a different url to my python client than my browser. So I went around
and had a look at the header requests and tried my best to imitate them to
make it appear as though my python client is a real browser to no avail.

Basically:

Browser: ThisIsARealUrlToDocuments.doc Python: ThisIsADummyForNonHumans.doc

I should also mention that if I refresh the url in my browser then run the
python script it downloads successfully a single time, then it will give me
the dummy until I refresh the page.

I'm not exactly a veteran at anything web so help would be appreciated.

    
    
    req = urllib2.Request(url)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36     (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36')
    req.add_header('Referer', url_to_open)
    
    soup = BeautifulSoup(urllib2.urlopen(req).read()
    
    # Dummy video url always contains 779de9061
    if '779de9061' in soup.find("div", {'id':'divDownload'}).a['href']:
        return "Failed to get valid video link."
    # Returns the found video link    
    return soup.find("div", {'id':'divDownload'}).a['href']
    

The page I'm trying to scrape is any page with video here:
<http://kissanime.com/Anime/Neon-Genesis-Evangelion/Episode-026?id=5260>

Right now I'm using a hacky method of just loading their page quickly in a
native browser then downloading, as it seems to grant me one single non dummy
download per pageload.


Answer: When I try the URL provided with `curl`, I get

    
    
    $ curl --silent "http://kissanime.com/Anime/Neon-Genesis-Evangelion/Episode-026?id=5260" | grep "divDownload"
    <div style="font-size: 14px; font-weight: bold; padding-top: 15px" id="divDownload">
    

So it doesn't look like anything weird happening from my location...

Why not look at [PyQuery](https://pypi.python.org/pypi/pyquery)?

    
    
    >>> from pyquery import PyQuery
    >>> doc = PyQuery('http://kissanime.com/Anime/Neon-Genesis-Evangelion/Episode-026?id=5260')
    >>> print doc.find('#divDownload a')[0].attrib['href']
    http://redirector.googlevideo.com/videoplayback?id=779de9061bfeb5d6&itag=37&source=picasa&cmo=sensitive_content=yes&ip=0.0.0.0&ipbits=0&expire=1397587987&sparams=id,itag,source,ip,ipbits,expire&signature=45A93E092F1750C81437ED7F0A5EEB5ABFCD5611.D30926273240E4116B91E85D59FF1268D0E5D5A1&key=lh1
    

