Accessing netflix api from python's urllib2 results in 500 error
Question: I'm currently trying to fix a Kodi plugin called NetfliXBMC.

It uses this url to get information on specific movies:

    
    
    http://www.netflix.com/JSON/BOB?movieid=<SOMEID>
    

While trying to build a minimal case to ask this question I discovered that
it's not even necessary to be logged in to access the information, which
simplifies my question a lot.

Querying information about a movie works from wget, from curl, from incognito
chrome etc. It just never works from urllib2:

    
    
    # wget works just fine
    $: wget -q -O- http://www.netflix.com/JSON/BOB?movieid=80021955
        {"contextData":"{\"cookieDisclosure\":{\"data\":{\"showCookieBanner\":false}}}","result":"success","actionErrors":null,"fieldErrors":null,"actionMessages":null,"data":[output omitted for brevity]}
    
    # so does curl
    $: curl http://www.netflix.com/JSON/BOB?movieid=80021955
            {"contextData":"{\"cookieDisclosure\":{\"data\":{\"showCookieBanner\":false}}}","result":"success","actionErrors":null,"fieldErrors":null,"actionMessages":null,"data":[output omitted for brevity}
    
    
    # but python's urllib always gets a 500
    $: python -c "import urllib2; urllib2.urlopen('http://www.netflix.com/JSON/BOB?movieid=80021955').read()"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
        return _opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 410, in open
        response = meth(req, response)
      File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
        'http', request, response, code, msg, hdrs)
      File "/usr/lib/python2.7/urllib2.py", line 448, in error
        return self._call_chain(*args)
      File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 500: Internal Server Error
    
    $: python --version
    Python 2.7.6
    

What I've tried so far: several different user-agent strings, initializing a
urlopener with a cookie jar, plain old urllib (doesn't raise an exception but
receives the same error page).

I'm really curious as to why this might be. Thanks in advance!


Answer: It turned out to be a bug on netflix' side when no `Accept` header is sent.

This doesn't work:

    
    
    opener = urllib2.build_opener()
    opener.open("http://www.netflix.com/JSON/BOB?movieid=80021955")
    

Adding a proper accept header makes it work:

    
    
    opener = urllib2.build_opener()
    mimeAccept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    opener.addheaders = [('Accept', mimeAccept)]
    opener.open("http://www.netflix.com/JSON/BOB?movieid=80021955")
    [...]
    

Of course, there is another bug there: it returns a `500 internal server
error` instead of a `400 bad request` when the problem was clearly on the
request.

