title: Using RSelenium to web scrape Google Scholar
body: <p>I'm trying to develop an academic network using information available from Google Scholar. Part of this involves scraping data from a pop-up window (not actually sure what kind of window it is - it doesn't seem to be a regular window or an iframe) that is produced from clicking on an article title on an individual scholar's page.</p>
  
  <p>I've been using RSelenium to perform this task. Below is the code I've developed so far for interacting with Google Scholar.</p>
  
  <pre><code>#Libraries----    
  library(RSelenium)
  
  
  #Functions----
  #Convenience function for simplifying data generated from .$findElements()
  unPack &lt;- function(x, opt = "text"){
    unlist(sapply(x, function(x){x$getElementAttribute(opt)}))  
  }
  
  
  #Analysis----
  #Start up the server for Chrome.
  rD &lt;- rsDriver(browser = "chrome")
  #Start Chrome.
  remDr &lt;- rD[["client"]]
  #Add a test URL.
  siteAdd &lt;- "http://scholar.google.com/citations?user=sc3TX6oAAAAJ&amp;hl=en&amp;oi=ao"
  #Open the site.
  remDr$navigate(siteAdd)
  
  #Create a list of all the article titles
  cite100Elem &lt;- remDr$findElements(using = "css selector", value = "a.gsc_a_at")
  cite100 &lt;- unPack(cite100Elem)
  
  #Start scraping the first article. I will create some kind of loop for all
  # articles later.
  #This opens the pop-up window with additional data I'm interested in.
  citeTitle &lt;- cite100[1]
  citeElem &lt;- remDr$findElement(using = 'link text', value = citeTitle)
  citeElem$clickElement()
  </code></pre>
  
  <p>Here's where I get stuck. Looking at the underlying webpage using Chrome's Developer tools, I can see that the first bit of information I'm interested in, the authors of the article, which is associated with the following HTML:</p>
  
  <pre><code>&lt;div class="gsc_vcd_value"&gt;TR Moore, NT Roulet, JM Waddington&lt;/div&gt;
  </code></pre>
  
  <p>This suggests that I should be able do something like:</p>
  
  <pre><code>#Extract all the information about the article.
  articleElem &lt;- remDr$findElements(value = '//*[@class="gsc_vcd_title"]')
  articleInfo &lt;- unPack(articleElem)
  </code></pre>
  
  <p>However, this solution doesn't seem to work; it returns a value of "NULL".</p>
  
  <p>I'm hoping that someone out there has an R-based solution, because I know very little about Java Script.</p>
  
  <p>Last, if search the resulting text from the following code (parse the page I'm currently on):</p>
  
  <pre><code>htmlOut &lt;- XML::htmlParse(remDr$getPageSource()[[1]])
  htmlOut
  </code></pre>
  
  <p>I can't find the CSS class associated with "gsc_vcd_title", which suggests to me that the page I'm interested in has a more complicated structure that I haven't quite figured out yet.</p>
  
  <p>Any insights you have would be very welcome. Thanks!</p>