Web scraping with lxml and requests
Question: I have a [web
page](http://www.booking.com/searchresults.et.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaEKIAQGYAQu4AQbIAQzYAQHoAQH4AQuoAgM&sid=1bc09296ee139ec3cb0edce87d7fb20a&dcid=1&class_interval=1&dest_id=67&dest_type=country&dtdisc=0&group_adults=2&group_children=0&hlrd=0&hyb_red=0&inac=0&label_click=undef&nha_red=0&no_rooms=1&postcard=0&redirected_from_city=0&redirected_from_landmark=0&redirected_from_region=0&review_score_group=empty&room1=A%2CA&sb_price_type=total&score_min=0&src=index&ss=Eesti&ss_all=0&ss_raw=Eesti&ssb=empty&sshis=0&traveller=other&nflt=ht_id%3D204%3B&lsf=ht_id%7C204%7C221&unchecked_filter=hoteltype)
with hotels, where i want to get all the hotel names. I made a code following
instructions from [this page](http://docs.python-
guide.org/en/latest/scenarios/scrape/), but no success. My code is here:

    
    
    from lxml import html
    import requests
    
    page = requests.get('web page url')
    tree = html.fromstring(page.content)
    
    hotel_name = tree.xpath('//span[@title="sr-hotel__name"]/text()')
    
    print(hotel_name)
    

All I get is an empty list. Any help?


Answer: You need to add a user-agent:

    
    
    from lxml import html
    import requests
    headers= {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"}
    page = requests.get('http://www.booking.com/searchresults.et.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaEKIAQGYAQu4AQbIAQzYAQHoAQH4AQuoAgM&sid=1bc09296ee139ec3cb0edce87d7fb20a&dcid=1&class_interval=1&dest_id=67&dest_type=country&dtdisc=0&group_adults=2&group_children=0&hlrd=0&hyb_red=0&inac=0&label_click=undef&nha_red=0&no_rooms=1&postcard=0&redirected_from_city=0&redirected_from_landmark=0&redirected_from_region=0&review_score_group=empty&room1=A%2CA&sb_price_type=total&score_min=0&src=index&ss=Eesti&ss_all=0&ss_raw=Eesti&ssb=empty&sshis=0&traveller=other&nflt=ht_id%3D204%3B&lsf=ht_id%7C204%7C221&unchecked_filter=hoteltype'
                        , headers=headers)
    tree = html.fromstring(page.content)
    print(page.text)
    hotel_name = tree.xpath('//span[@class="sr-hotel__name"]/text()')
    
    print(hotel_name)
    

Which will give you:

    
    
    ['\nHotel Telegraaf\n', '\nRadisson Blu Hotel Olümpia\n', '\nRadisson Blu Sky Hotel\n', '\nPark Inn by Radisson Central Tallinn\n', '\nPark Inn by Radisson Meriton Conference & Spa Hotel Tallinn\n', '\nMerchants House Hotel\n', '\nSwissotel Tallinn\n', '\nMy City Hotel\n', '\nNordic Hotel Forum\n', '\nHotel Palace by TallinnHotels\n', '\nHotel Ülemiste\n', '\nTallink City Hotel\n', '\nHotel London by Tartuhotels\n', '\nJohan Design & SPA Hotel\n', '\nThe von Stackelberg Hotel Tallinn\n']
    

But you should read their
[TOS](http://www.booking.com/content/terms.et.html?label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGmIAQGYAQu4AQrIAQzYAQPoAQGoAgM;sid=52cedf70cb08d87dee23f69f173a8650;dcid=12):

_Our services are only for personal and non-commercial use. Consequently, you
will not be allowed on our web site available through the content,
information, software, products or services of a commercial or compete for the
purpose of reselling link (deep link) to use, copy, monitor (eg, spiders,
scrape), display, download download or reproduce._

