title: R web scraping difficulty--Why can't I get all of the listing prices from a multi-page website?
body: <p>I have been trying to scrape data from a real estate website using R's rvest package. The website that I am attempting to scrape listing prices has 15 pages with 631 total listings. However, when I use the script that follows it results with a data frame with only a little over 360 values (it seems to take listing prices from the first 9 pages and then stops). Additionally, when I try using the exact same script right after the first try, it replaces the previous data frame with 0 values. If I wait 30 min and use the same code again, I get the original data frame with ~369 values again. I will include my code below:</p>
  <pre><code>library(rvest)
  library(purrr)
  library(httr)
  library(stringr)
  
  url &lt;- &quot;https://www.realtor.com/soldhomeprices/Boulder_CO/type-single-family-home,multi-family-home/pg%d
  
  
  boulder_sold &lt;- map_df(1:15, function(i){
    pg &lt;- read_html(sprintf(url, i))
    data.frame(Price = parse_number(html_text(html_nodes(pg, &quot;.data-price&quot;))),
               stringsAsFactors = FALSE)
  })
  </code></pre>
  <p>I thought that perhaps my problem was that the website was timing out and kicking me off, so I also tried another iteration with a for-loop to try to give breaks in between reading groups of pages. The script for this was:</p>
  <pre><code>boulder_sold_break &lt;- map_df(1:15, function(i){
    for(j in i){
      Sys.sleep(5)
      if((i %% 2) == 0){
        message(&quot;taking a break&quot;)
        Sys.sleep(2)
      }
    }
    pg &lt;- read_html(sprintf(url, i))
    data.frame(Price = parse_number(html_text(html_nodes(pg, &quot;.data-price&quot;))),
               stringsAsFactors = FALSE)
  }) 
  </code></pre>
  <p>Therefore, could anyone tell me: 1) Why will my code not give me a data frame with all 631 listing prices? 2) Why does the same script stop giving me any list prices after an initial attempt (and then go back to outputting results after a certain period of time)?</p>