Use python mechanize to log into pages with NTLM authentication
Question: I want to use mechanize to log into a page and retrieve some information. But
however I try to authenticate It just fails with Error code **HTTP 401** , as
you can see below:

    
    
    r = br.open('http://intra')
    File "bui...e\_mechanize.py", line 203, in open
    File "bui...g\mechanize\_mechanize.py", line 255,
    in _mech_openmechanize._response.httperror_seek_wrapper: HTTP Error 401: Unauthorized
    

This is my code so far:

    
    
    import mechanize
    import cookielib
    
    # Browser
    br = mechanize.Browser()
    
    # Cookie Jar
    cj = cookielib.LWPCookieJar()
    br.set_cookiejar(cj)
    
    # Browser options
    br.set_handle_equiv(True)
    # br.set_handle_gzip(True)
    br.set_handle_redirect(True)
    br.set_handle_referer(True)
    br.set_handle_robots(False)
    
    # Follows refresh 0 but not hangs on refresh > 0
    br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
    
    # If the protected site didn't receive the authentication data you would
    # end up with a 410 error in your face
    br.add_password('http://intra', 'myusername', 'mypassword')
    
    # User-Agent (this is cheating, ok?)
    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
    # Open some site, let's pick a random one, the first that pops in mind:
    # r = br.open('http://google.com')
    r = br.open('http://intra')
    html = r.read()
    
    # Show the source
    print html
    

What am I doing wrong? visiting `http://intra` (internal page) with e.g.
chrome, it pops open a windows and asks for username/password once and then
all is good.

The dialogue which pops open looks like this:

![enter image description here](http://i.stack.imgur.com/JTb7M.png)


Answer: After tons of reaserch I managed to find out the reason behind this.

Find of all the site uses a so called [NTLM
authentication](http://hc.apache.org/httpclient-
legacy/authentication.html#Authentication_Schemes), which is not supported by
mechanize. This can help to find out the authentication mechanism of a site:

    
    
    wget -O /dev/null -S http://www.the-site.com/
    

So the code was modified a little bit:

    
    
    import sys
    import urllib2
    import mechanize
    from ntlm import HTTPNtlmAuthHandler
    
    print("LOGIN...")
    user = sys.argv[1]
    password = sys.argv[2]
    url = sys.argv[3]
    
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, user, password)
    # create the NTLM authentication handler
    auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
    
    browser = mechanize.Browser()
    handlersToKeep = []
    
    for handler in browser.handlers:
        if not isinstance(handler,
        (mechanize._http.HTTPRobotRulesProcessor)):
            handlersToKeep.append(handler)
    
    browser.handlers = handlersToKeep
    browser.add_handler(auth_NTLM)
    
    response = browser.open(url)
    response = browser.open("http://www.the-site.com")
    print(response.read())
    

and finally mechanize needs to be patched, as mentioned
[here](http://stackoverflow.com/questions/13649964/python-mechanize-with-ntlm-
getting-attributeerror-httpresponse-instance-has-no):

    
    
    --- _response.py.old    2013-02-06 11:14:33.208385467 +0100
    +++ _response.py    2013-02-06 11:21:41.884081708 +0100
    @@ -350,8 +350,13 @@
                 self.fileno = self.fp.fileno
             else:
                 self.fileno = lambda: None
    -        self.__iter__ = self.fp.__iter__
    -        self.next = self.fp.next
    +
    +        if hasattr(self.fp, "__iter__"):
    +            self.__iter__ = self.fp.__iter__
    +            self.next = self.fp.next
    +        else:
    +            self.__iter__ = lambda self: self
    +            self.next = lambda self: self.fp.readline()
    
         def __repr__(self):
             return '<%s at %s whose fp = %r>' % (
    

