Website scraping: python requests not downloading full site?
Question: I'm having problems in scraping a website. The aim will be to scrape prices
for hotels in London for certain days. For that I'm loading the below URL from
booking.com and then try to search for keywords. But for some reason the
requests.get doesn't download the full site. For example the URL below
displays a list of hotels in my browser. Each of them showing 'Total' and the
the price. However, in the below code site.find('Total') shows that no word
'Total' can be found in the string, even though it's visible in the browser.
Any suggestions why this happens are appreciated.

    
    
    import requests
    
    url='http://www.booking.com/searchresults.en-gb.html?label=gen173nr-17CAEoggJCAlhYSDNiBW5vcmVmaFCIAQGYAS64AQTIAQTYAQHoAQH4AQs;sid=1a43e0952558ac0ad0061d5b6523a7bc;dcid=1;checkin_monthday=4;checkin_year_month=2016-2;checkout_monthday=11;checkout_year_month=2016-2;city=-2601889;class_interval=1;csflt=%7B%7D;group_adults=7;group_children=0;highlighted_hotels=1192837;hp_sbox=1;label_click=undef;no_rooms=1;review_score_group=empty;room1=A%2CA%2CA%2CA%2CA%2CA%2CA;sb_price_type=total;score_min=0;si=ai%2Cco%2Cci%2Cre%2Cdi;ss=London;ssafas=1;ssb=empty;ssne=London;ssne_untouched=London&;order=price_for_two'
    r=requests.get(url)
    
    site=r.text
    site.find('Total')
    


Answer: The "Total" information you're talking about is generated with Javascript in
the browser. The Requests library can't generate this HTML for you because it
isn't a browser environment.

To see what I'm talking about, try running that URL in a browser without
Javascript.

[![enter image description
here](http://i.stack.imgur.com/WEVlm.png)](http://i.stack.imgur.com/WEVlm.png)

If you want to scrape HTML that requires Javascript to run, you should look
into a library that binds to a browser environment, like
[Selenium](http://selenium-python.readthedocs.org/api.html).

