Filling a strange webform in python
Question: I am trying build a simple program that fills a webform and then extracts the
a data from the resulted website, it should be pretty straight-forward and
after a short web research (mostly from this website) I came to the conclusion
that python would be my best choice.(with urllib)

I would give a specific example of what I am trying to do and hopefully it
will clarify things:

  1. Lets say I want an automatic script that gets the price for hotel, the website would be: the Hilton website 

  2. In that webpage I want to fill the query as follows: the "where are you going" should have N.Y.C and departure and arrival date.

  3. If used from the browser, I will get a the next page with the results for the query I just filled and from that webpage I want to scrape the prices.

Lets assume that scraping the data can be done from the html source code.

Well, that's how I thought to do it (very general description....)

    
    
    import urllib
    import urllib.request
    
    url = 'http://www3.hilton.com/en/index.html'
    query_args = { 'searchQuery':' New York, NY ', 'arrivalDate':'31 Oct 2013' ,                         'departureDate': '04 Nov 2013' }
    print (query_args)
    data = urllib.parse.urlencode(query_args)
    print (data)
    request = urllib.request.Request(url);
    response = urllib.request.urlopen(request,data)
    html = response.read()
    print (html)
    

  1. First of all I get an the following error when I am sending the request:" TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str." anyone knows why?

  2. If I got it right I would assume that I would accept the response for the following website. hilton_website with the urlencode segment in the end (searchQuery=+New+York%2C+NY&arrivalDate=31+Oct+2013&departureDate=04+Nov+2013)

But there is no such website ( when I am typing the full address to the
browser url) it looks like all the results (and it doesn't matter what was
your query or how you filled the form ) appear on the same webpage:
hilton_web_site...en_US/hi/search/findhotels/results.htm?view=LIST

So what am I doing wrong? I thought that the way to fill a web form.

Thanks a lot for the help


Answer: Here's a dump of hiltons search form (which, i instructed you how to get but
can not paste it all in the comment section so here you go), i forgot to
mention that you should check the "Form data" raw data in Chrome tho which was
my bad.. anyway..

RAW data (so you can understand how a POST request works)

## Request Header

    
    
    POST /en_US/hi/search/findhotels/index.htm?WT.bid=Home,,,find_button HTTP/1.1
    Host: www3.hilton.com
    Connection: keep-alive
    Content-Length: 475
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
    Origin: http://www3.hilton.com
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
    Content-Type: application/x-www-form-urlencoded
    Referer: http://www3.hilton.com/en/index.html
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: en-US,en;q=0.8
    Cookie: mmcore.tst=0.626; ClrCSTO=T; ClrSSID=1382526492268-11812; ClrOSSID=1382526492268-11812; ClrSCD=1382526492268; mmid=-1914085652%7CAQAAAAq0Q+DZtwkAAA%3D%3D; mmcore.pd=-1914085652%7CAQAAAAq0Q+DZtwkAAA%3D%3D; mmcore.srv=ldnvwcgus01; K3R7=3U24QMiCUbjeh65LoEI31TTjisY8czr4zkIUe06gsA4A5lc0bIKrEhQ; GWSESSIONID=qYDpSnSLNFnJ5CrJrwlSWW7CNBHL7vXSMndGmnxghGns1rLjt2lX!-1490734837; __atuvc=1%7C43; mm_pc_HHonorsPoints=false; mm_pers_storage=loggedin%3Ano%7CStayDuration%3A1-2%20nights%7CDaysToBooking%3A0-1%7CSatStay%3Ano%7CChildren%3Ano%7CBrand%3Ahi%7CHHonorsPoints%3Afalse%7CFlexDates%3Afalse%7CPromoCode%3Ano%7Cproperties8%3Ano%7Chotelcode%3Ano; WT_FPC=id=6f7c5181-8354-430b-b943-9bb95bf2c75c:lv=1382490515002:ss=1382490495203
    

## Form data

    
    
    searchQuery=N.Y.C&arrivalDate=23+Oct+2013&departureDate=24+Oct+2013&_flexibleDates=on&numberOfRooms=1&numberOfAdults%5B0%5D=1&numberOfChildren%5B0%5D=0&numberOfAdults%5B1%5D=1&numberOfChildren%5B1%5D=0&numberOfAdults%5B2%5D=1&numberOfChildren%5B2%5D=0&numberOfAdults%5B3%5D=1&numberOfChildren%5B3%5D=0&promoCode=&groupCode=&corporateId=&_travelAgentRate=on&_aaaRate=on&_aarpRate=on&_seniorRate=on&_governmentRate=on&offerId=&bookButton=false&searchType=ALL&roomKeyEnable=true
    

And these are they keys (not values) that you **need** to send with each
search query:

  * searchQuery
  * arrivalDate
  * departureDate
  * _flexibleDates
  * numberOfRooms
  * numberOfAdults%5B0%5D
  * numberOfChildren%5B0%5D
  * numberOfAdults%5B1%5D
  * numberOfChildren%5B1%5D
  * numberOfAdults%5B2%5D
  * numberOfChildren%5B2%5D
  * numberOfAdults%5B3%5D
  * numberOfChildren%5B3%5D
  * promoCode
  * groupCode
  * corporateId
  * _travelAgentRate
  * _aaaRate
  * _aarpRate
  * _seniorRate
  * _governmentRate
  * offerId
  * bookButton
  * searchType
  * roomKeyEnable

And this is what a python build of the whole thing would look like:

    
    
    from socket import *
    s = socket()
    
    POST = 'searchQuery=N.Y.C&arrivalDate=23+Oct+2013&departureDate=......'
    
    header = ''
    header += 'POST /en_US/hi/search/findhotels/index.htm?WT.bid=Home,,,find_button HTTP/1.1\r\n'
    header += 'Host: www3.hilton.com\r\n'
    header += 'Connection: keep-alive\r\n'
    header += 'Content-Length: ' + str(len(POST)) + '\r\n'
    header += 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n'
    header += 'Origin: http://www3.hilton.com\r\n'
    header += 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) \r\n'
    header += 'Chrome/30.0.1599.101 Safari/537.36\r\n'
    header += 'Content-Type: application/x-www-form-urlencoded\r\n'
    header += 'Referer: http://www3.hilton.com/en/index.html\r\n'
    header += 'Accept-Encoding: gzip,deflate,sdch\r\n'
    header += 'Accept-Language: en-US,en;q=0.8\r\n'
    header += '\r\n'
    
    s.connect(('www3.hilton.com', 80))
    s.send(header+POST)
    print(s.recv(8192))
    

And that would give you what you're looking for. Note that i left out the
cookie field in the header.. normally the server doesn't care to much about it
because it's an ancient technique and there for the server usually assumes
that if no cookies was given, just give the client new ones and assume it's
the first visit even tho (well technically it is but) it is'nt really the
actual first page.

## Also note:

All data in the `POST` variable must be URL encoded as such:
`urlencode(key)=urlencode(val)&` for each pair. Note how i did not url-encode
the `=` or the `&` in this raw POST data.

This is where you wouldn't need to worry about it because urllib does it for
you. But this gives you a understanding of how HTTP requests work.

