use python requests to post a html form to a server and save the reponse to a file
Question: I have exactly the same problem as this post

[Python submitting webform using
requests](https://stackoverflow.com/questions/22407580/python-submitting-
webform-using-requests)

but your answers do not solve it. When I execute this HTML file called api.htm
in the browser, then for a second or so I see its page.

Then the browser shows the data I want with the URL
<https://api.someserver.com/api/> as as per the action below. But I want the
data written to a file so I try the Python 2.7 script below.

But all I get is the source code of api.htm Please put me on the right track!

    
    
    <html>
    <body>
    <form id="ID" method="post" action="https://api.someserver.com/api/ ">
    <input type="hidden" name="key" value="passkey">
    <input type="text" name="start" value ="2015-05-01">
    <input type="text" name="end" value ="2015-05-31">
    <input type="submit" value ="Submit">
    </form>
    <script type="text/javascript">
    document.getElementById("ID").submit();
    </script>
    </body>
    </html>
    

The code:

    
    
    import urllib
    import requests
    
    def main():
        try:
    
        values = {'start' : '2015-05-01',
        'end' : '2015-05-31'}
        req=requests.post("http://my-api-page.com/api.htm",
        data=urllib.urlencode(values))
        filename = "datafile.csv"
        output = open(filename,'wb')
        output.write(req.text)
        output.close()
        return 
    
    main()
    


Answer: I can see several problems:

  * Your post target URL is incorrect. The `form` `action` attribute tells you where to post to:
    
        <form id="ID" method="post" action="https://api.someserver.com/api/ ">
    

  * You are not including all the fields; `type=hidden` fields need to be posted too, but you are ignoring this one:
    
        <input type="hidden" name="key" value="passkey">
    

  * Do not URL-encode your POST variables yourself; leave this to `requests` to do for you. By encoding yourself `requests` won't recognise that you are using an `application/x-www-form-urlencoded` content type as the body. Just pass in the dictionary as the `data` parameters and it'll be encoded for you _and_ the header will be set.

You can also _stream_ the response straight to a file object; this is helpful
when the response is large. Switch on response streaming, make sure the
underlying raw `urllib3` file-like object decodes from transfer encoding and
use `shutil.copyfileobj` to write to disk:

    
    
    import requests
    import shutil
    
    def main():
        values = {
            'start': '2015-05-01',
            'end': '2015-05-31',
            'key': 'passkey',
        }
        req = requests.post("http://my-api-page.com/api.htm",
                            data=values, stream=True)
        if req.status_code == 200:
            with open("datafile.csv", 'wb') as output:
                req.raw.decode_content = True
                shutil.copyfileobj(req.raw, output)
    

There may still be issues with that `key` value however; perhaps the server
sets a new value for each session, coupled with a cookie, for example. In that
case you'd have to use a [`Session()` object](http://docs.python-
requests.org/en/latest/user/advanced/#session-objects) to preserve cookies,
first do a `GET` request to the `api.htm` page, parse out the `key` hidden
field value and only **then** post. If that is the case then using a tool like
[`robobrowser`](http://robobrowser.readthedocs.org/en/latest/) might just be
easier.

