Python module for HTTP: fill in forms, retrieve result
Question: I'd like to use Python to access an HTTP website, fill out a form, submit the
form, and retrieve the result.

What modules are suitable for the task?


Answer: We cannot advise you with detailed instructions since you never gave us
details of your problem. However, most probably you want to use urllib2 to
fetch an HTML page:

    
    
    import urllib2
    response = urllib2.urlopen('http://python.org/')
    html = response.read()
    

You should then parse the form, find out all the data fields you need to send
with their names , and then create your own POST or GET request, depending on
the form type.

To send a POST request:

    
    
    import urllib
    import urllib2
    
    url = 'http://www.someserver.com/cgi-bin/register.cgi'
    values = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }
    
    data = urllib.urlencode(values)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read()
    

To send a GET request:

    
    
    import urllib2
    import urllib
    data = {}
    data['name'] = 'Somebody Here'
    data['location'] = 'Northampton'
    data['language'] = 'Python'
    url_values = urllib.urlencode(data)
    url = 'http://www.example.com/example.cgi'
    full_url = url + '?' + url_values
    data = urllib2.urlopen(full_url)
    

