title: Scraping data from TripAdvisor using R
body: <p>I want to create a crawler that will scrape some data from Trip Advisor. Ideally, it will 
  <strong>(a)</strong> identify the links to all locations to crawl, 
  <strong>(b)</strong> collect links to all attractions in each location and 
  <strong>(c)</strong> will collect the destination names, dates and ratings for all reviews. 
  I'd like to focus on part (a) for now.</p>
  
  <p>Here is the website I'm starting off with:
  <a href="http://www.tripadvisor.co.nz/Tourism-g255104-New_Zealand-Vacations.html">http://www.tripadvisor.co.nz/Tourism-g255104-New_Zealand-Vacations.html</a></p>
  
  <p>There is problem here: the link gives top 10 destinations to begin with, and if you then click on "See more popular destinations" it will expand the list. It appears as though it uses a javascript function to achieve this. Unfortunately, I'm not familiar with javascript, but I think the following chunk may give clues about how it works:</p>
  
  <pre><code>&lt;div class="morePopularCities" onclick="ta.call('ta.servlet.Tourism.showNextChildPage', event, this)"&gt;
  &lt;img id='lazyload_2067453571_25' height='27' width='27' src='http://e2.tacdn.com/img2/x.gif'/&gt;
  See more popular destinations in New Zealand &lt;/div&gt;
  </code></pre>
  
  <p>I've found a few useful webscraping packages for R, such as rvest, RSelenium, XML, RCurl, but of these, only RSelenium appears to be able to resolve this, having said that, I still haven't been able to work it out.</p>
  
  <p>Here is some relevant code:</p>
  
  <pre><code>tu = "http://www.tripadvisor.co.nz/Tourism-g255104-New_Zealand-Vacations.html"
  RSelenium::startServer()
  remDr = RSelenium::remoteDriver(browserName = "internet explorer")
  remDr$open()
  remDr$navigate(tu)
  # remDr$executeScript("JS_FUNCTION")
  </code></pre>
  
  <p>The last line should do the trick here, but I'm not sure what function I need to call here. </p>
  
  <p>Once I manage to expand this list, I will be able to obtain the links for each destination the same way I would solve part (b) and I think I've already solved this (for those interested):</p>
  
  <pre><code>library(rvest)
  tu = "http://www.tripadvisor.co.nz/Tourism-g255104-New_Zealand-Vacations.html"
  tu = html_session(tu)
  tu %&gt;% html_nodes(xpath='//div[@class="popularCities"]/a') %&gt;% html_attr("href")
   [1] "/Tourism-g255122-Queenstown_Otago_Region_South_Island-Vacations.html"                      
   [2] "/Tourism-g255106-Auckland_North_Island-Vacations.html"                                     
   [3] "/Tourism-g255117-Blenheim_Marlborough_Region_South_Island-Vacations.html"                  
   [4] "/Tourism-g255111-Rotorua_Rotorua_District_Bay_of_Plenty_Region_North_Island-Vacations.html"
   [5] "/Tourism-g255678-Nelson_Nelson_Tasman_Region_South_Island-Vacations.html"                  
   [6] "/Tourism-g255113-Taupo_Taupo_District_Waikato_Region_North_Island-Vacations.html"          
   [7] "/Tourism-g255109-Napier_Hawke_s_Bay_Region_North_Island-Vacations.html"                    
   [8] "/Tourism-g612500-Wanaka_Otago_Region_South_Island-Vacations.html"                          
   [9] "/Tourism-g255679-Russell_Bay_of_Islands_Northland_Region_North_Island-Vacations.html"      
  [10] "/Tourism-g255114-Tauranga_Bay_of_Plenty_Region_North_Island-Vacations.html"  
  </code></pre>
  
  <p>As for step (c), I've found some useful links that might be helpful for that:
  <a href="https://github.com/hadley/rvest/blob/master/demo/tripadvisor.R">https://github.com/hadley/rvest/blob/master/demo/tripadvisor.R</a>
  <a href="http://notesofdabbler.github.io/201408_hotelReview/scrapeTripAdvisor.html">http://notesofdabbler.github.io/201408_hotelReview/scrapeTripAdvisor.html</a></p>
  
  <p>If you have any tips on how to expand the list of top destinations or how to go through the other steps in a smarter way, please let me know, I'd be really keen to hear from you.</p>
  
  <p>Many thanks in advance! </p>