Login and get HTML file using python
Question: Hey I'm trying to login to a website and get the html of the webpage after the
login. And can't figure out how to do it with python. Using python 2.7. Need
to fill out the html forms on this website:

'user'= 'magaleast' and 'password' = '1181' (real login details that are
useless to me). Then the website redirects the user to an authentication page
and when its done it goes to the page i need.

Any ideas?

EDIT: trying this code:

    
    
    from mechanize import Browser
    import cookielib
    br = Browser()
    br.open("http://www.shiftorganizer.com/")
    
    
    cj = cookielib.LWPCookieJar()
    br.set_cookiejar(cj)
    
    # Browser options
    br.set_handle_equiv(True)
    br.set_handle_gzip(True)
    br.set_handle_redirect(True)
    br.set_handle_referer(True)
    
    # You need to spot the name of the form in source code
    br.select_form(name = "user")
    # Spot the name of the inputs of the form that you want to fill, 
    # say "username" and "password"
    br.form["user"] = "magaleast"
    br.form["password"] = "1181"
    
    response = br.submit()
    print response.read()
    

but i get:

    
    
         <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    
    <html xmlns="http://www.w3.org/1999/xhtml">
    
    <head>
    
    <title>ShiftOrganizer סידור עבודה בפחות משניה</title>
    
        <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    
    
    
    <script type="text/javascript">
    
    var emptyCompany=1
    
    function subIfNewApp()
    
    {
    
        if (emptyCompany){
    
            document.authenticationForm.action = document.getElementById('userName').value + "/authentication.asp"
    
        } else {
    
            document.authenticationForm.action = document.getElementById('Company').value + "/authentication.asp"
    
        }
    
        document.authenticationForm.submit()
    
    }
    
    </script>
    
    </head>
    
        <body onload="subIfNewApp()">
    
        <form name="authenticationForm" method="post" action="">
    
            <input type="hidden" name="userName" id="userName" value="magaleast" />
    
            <input type="hidden" name="password" id="password" value="1181" />
    
            <input type="hidden" name="Company" id="Company" value="שם חברה" />
    
            </form>
    
        </body>
    
    </html>
    

is js the problem? because it stops in the authentication part again..?


Answer: It seems that the website requires some JS indeed so the code below won't be
enough. In that particular case, by looking at the source code, it seems that
at the end this url is used :

<http://shifto.shiftorganizer.com/magaleast/welcome.asp?password=1181> which
seems to contain similar information that the page after login (altough I
can't read Hebrew, I may be totally wrong...). If so, you could simply do:

    
    
    import urllib
    url = 'http://shifto.shiftorganizer.com/*username*/welcome.asp?password=*password*'
    print urllib.urlopen(url).read()
    

* * *

For information, code to login to a form which does not require Javascript.

I would use the [mechanize](http://wwwsearch.sourceforge.net/mechanize/)
library (also Requests will work), doing something like

    
    
    from mechanize import Browser
    
    br = Browser()
    br.set_cookiejar(cookielib.LWPCookieJar())
    
    # Browser options
    br.set_handle_equiv(True)
    br.set_handle_redirect(True)
    br.set_handle_referer(True)
    
    br.open("your url")
    
    # You need to spot the name of the form in source code
    br.select_form(name="form_name")  
    
    # Spot the name of the inputs of the form that you want to fill, 
    # say "username" and "password"
    br.form["username"] = "magaleast"
    br.form["password"] = "1181"
    
    response = br.submit()
    print response.read()
    

