How to use Python Requests to login to website, store cookie, then access another page on the website?
Question: I'm trying to login into website using a Python script, store the cookie I
receive, and then use that same cookie to access member-only parts of the
website. I've read several posts and answers about this topic, but none of the
answers have worked for me.

Here is the HTML code for the website login page I'm trying to access.

    
    
    <form action="/login?task=user.login" method="post">
        <fieldset>
            <table border="0" cellspacing="0" cellpadding="0">
            <tbody>
                                                            <tr>
                <td width="70" nowrap="">Username&nbsp;&nbsp;</td>
                <td width="260"><input type="text" name="username" id="username" value="" class="validate-username" size="25"/></td>
                        </tr>
                                                                                    <tr>
                <td width="70" nowrap="">Password&nbsp;&nbsp;</td>
                 <td width="260"><input type="password" name="password" id="password" value="" class="validate-password" size="25"/></td>
             </tr>
                                                            <tr>
                 <td colspan="2"><label style="float: left;width: 70%;" for="modlgn_remember">Remember Me</label>
                 <input style="float: right;width: 20%;"id="modlgn_remember" type="checkbox" name="remember" class="inputbox" value="yes"/></td>
             </tr>
             <tr>
                <td  colspan="2" width="100%"> <a href="/reset-password"> Forgot your password?</a></td>
            </tr>
            <tr>
                <td  colspan="2" width="100%"> <a href="/username-reminder">Forgot your username?</a></td>
            </tr>
            <tr>
                <td colspan="2"><button type="submit" class="button cta">Log in</button></td>
    <!--                            <td colspan="1"><a href="/--><!--">Register Now</a></td>-->
            </tr>
            </tbody>
            </table>
    
            <input type="hidden" name="return"
                   value="aHR0cHM6Ly9maWYuY29tLw=="/>
            <input type="hidden" name="3295f23066f7c6ab53c290c6c022cc4b" value="1" />                    </fieldset>
    </form>
    

Here is my own code that I'm using to attempt a login.

    
    
    from requests import session
    
    payload = {
         'username': 'MY_USERNAME',
         'password': 'MY_PASSWORD'
    }
    
    s = session()
    s.post('https://fif.com/login?task=user.login', data=payload)
    
    response = s.get('https://fif.com/tools/capacity')
    

From everything I have read, this should work, but it doesn't. I've been
struggling with this for two days, so if you know the answer, I would love the
solution.

For reference, here are all the other StackOverflow posts I have looked at in
hopes for an answer:

  1. [Python Requests and Persistent Sessions](http://stackoverflow.com/questions/12737740/python-requests-and-persistent-sessions)
  2. [Logging into a site using Python Reqeusts](http://stackoverflow.com/questions/30605582/logging-in-to-a-site-using-python-requests?lq=1)
  3. [Login to website using python](http://stackoverflow.com/questions/8316818/login-to-website-using-python)
  4. [How to “log in” to a website using Python's Requests module?](http://stackoverflow.com/questions/11892729/how-to-log-in-to-a-website-using-pythons-requests-module/17633072#17633072)
  5. [Python: Requests Session Login Cookies](http://stackoverflow.com/questions/24260149/python-requests-session-login-cookies)
  6. [How to use Python to login to a webpage and retrieve cookies for later usage?](http://stackoverflow.com/questions/189555/how-to-use-python-to-login-to-a-webpage-and-retrieve-cookies-for-later-usage)
  7. [cUrl Login then cUrl Download](http://stackoverflow.com/questions/6987876/curl-login-then-curl-download)


Answer: You should be posting all the required data, you can use _bs4_ to parse the
login page to get the values you need:

    
    
    from requests import session
    from bs4 import BeautifulSoup
    
    data = {
        'username': 'MY_USERNAME',
        'password': 'MY_PASSWORD'
    }
    
    head = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"}
    with  session() as s:
        soup = BeautifulSoup(s.get("https://fif.com/login").content)
        form_data = soup.select("form[action^=/login?task] input")
        data.update({inp["name"]: inp["value"] for inp in form_data if inp["name"] not in data})
        s.post('https://fif.com/login?task=user.login', data=data, headers=head)
        resp = s.get('https://fif.com/tools/capacity')
    

If you make a requests and look in chrome tools or firebug, the form data
looks like:

    
    
    username:foo
    password:bar
    return:aW5kZXgucGhwP29wdGlvbj1jb21fdXNlcnMmdmlldz1wcm9maWxl
    d68a2b40daf7b6c8eaa3a2f652f7ee62:1
    

