title: Scraping several webpages from a website (newspaper archive) using RSelenium
body: <p>I managed to scrape one page from a <a href="https://en.trend.az/archive/2021-09-30" rel="nofollow noreferrer">newspaper archive</a> according to explanations <a href="https://stackoverflow.com/questions/70056380/online-newspaper-data-scraping-with-r-rvest-package">here</a>.</p>
  <p>Now I am trying to automatise the process to access a list of pages by running one code.
  Making a list of URLs was easy as the newspaper's archive has a similar pattern of links:</p>
  <blockquote>
  <p><a href="https://en.trend.az/archive/2021-XX-XX" rel="nofollow noreferrer">https://en.trend.az/archive/2021-XX-XX</a></p>
  </blockquote>
  <p>The problem is with writing a loop to scrape such data as <strong>title, date, time, category</strong>. For simplicity, I tried to work only with article headlines from 2021-09-30 to 2021-10-02.</p>
  <pre><code>## Setting data frames
  
  d1 &lt;- as.Date(&quot;2021-09-30&quot;)
  d2 &lt;- as.Date(&quot;2021-10-02&quot;)
  
  list_of_url &lt;- character()   # or str_c()
  
  ## Generating subpage list 
   
  for (i in format(seq(d1, d2, by=&quot;days&quot;), format=&quot;%Y-%m-%d&quot;))  {
    list_of_url[i] &lt;- str_c (&quot;https://en.trend.az&quot;, &quot;/archive/&quot;, i)
  
  # Launching browser
  
  driver &lt;- rsDriver(browser = c(&quot;firefox&quot;))  #Version 93.0 (64-bit)
  remDr &lt;- driver[[&quot;client&quot;]]
  remDr$errorDetails
  remDr$navigate(list_of_url[i])
     
     remDr0$findElement(using = &quot;xpath&quot;, value = '/html/body/div[1]/div/div[1]/h1')$clickElement()
     
     webElem &lt;- remDr$findElement(&quot;css&quot;, &quot;body&quot;)
  #scrolling to the end of webpage, to load all articles 
  for (i in 1:25){
    Sys.sleep(2)
    webElem$sendKeysToElement(list(key = &quot;end&quot;))
  } 
  
  page &lt;- read_html(remDr$getPageSource()[[1]])
  
  # Scraping article headlines
  
  get_headline &lt;- page %&gt;%
  html_nodes('.category-article') %&gt;% html_nodes('.article-title') %&gt;% 
    html_text()
  get_time &lt;- str_sub(get_time, start= -5)
  
  length(get_time)
     }
  }
  </code></pre>
  <p>In total length should have been 157+166+140=463. In fact, I did not manage to collect all data even from one page (length(get_time) = 126)</p>
  <p>I considered that after the first set of commands in the loop, I obtained three <code>remDr</code> for the 3 dates specified, but they were not recognised later independently.</p>
  <p>Because of that I tried to initiate a second loop inside the initial one before or after <code>page &lt;-</code> by</p>
  <pre><code>  for (remDr0 in remDr) {
  page &lt;- read_html(remDr0$getPageSource()[[1]])
  # substituted all remDr-s below with remDr0
  </code></pre>
  <p>OR</p>
  <pre><code>page &lt;- read_html(remDr$getPageSource()[[1]])
  for (page0 in page)
  # substituted all page-s below with page0
  </code></pre>
  <p>However, these attempts ended with different errors.</p>
  <p>I would appreciate the help of specialists as it is my first time using R for such purposes.</p>
  <p>Hope it will be possible to correct the existing loop that I made or maybe even suggest a shorter pathway, by making a <code>function</code>, for example.</p>