title: Scrape and compile data using JSON API in R: all possible data ID numbers
body: <p>I am trying to compile all the climbing route information from <a href="https://www.mountainproject.com/" rel="nofollow noreferrer">https://www.mountainproject.com/</a> They provide an API for doing this <a href="https://www.mountainproject.com/data" rel="nofollow noreferrer">https://www.mountainproject.com/data</a>, but you need to know the specific route identification number i.e.("<a href="https://www.mountainproject.com/data/get-routes?routeIds=IDNUMBER&amp;key=PERSONAL-API-KEY" rel="nofollow noreferrer">https://www.mountainproject.com/data/get-routes?routeIds=IDNUMBER&amp;key=PERSONAL-API-KEY</a>")</p>
  
  <p>There is no list of the ID numbers, but they all seem to fall within 105,000,000 : 118,000,000. There should be around 200,000 total.</p>
  
  <p>I need a way to run through all of the possible route ID numbers, figure out which ones are real routes, download the data, and add it to a dataset (csv)</p>
  
  <p>I wrote a function and loop that will do this and it works for a small number of possible IDs, but for all 13,000,000 possible ID numbers, I calculated that it would take over a month to run.</p>
  
  <p>I have not been able to do this in a vectorized way, but I think that will possibly be the solution.</p>
  
  <pre><code>    library(jsonlite)
  
      #create the function to download the data
  
      getRouteData&lt;-function(routeID){
        y&lt;-paste0("https://www.mountainproject.com/data/get-routes?routeIds=",routeID,"&amp;key=PERSONAL_API_KEY")
        y&lt;-as.data.frame(fromJSON(y))
  #delete some unnecessary columns 
        y[,-(c(9:13,16))]
  
      }
  
      #Make an empty data frame
  
      df &lt;- data.frame(matrix(ncol = 10, nrow = 0))
      colnames(df) &lt;- c("routes.id","routes.name","routes.type",
                        "routes.rating","routes.stars","routes.starVotes",
                        "routes.pitches","routes.location", 
                        "routes.longitude","routes.latitude" )
  
      #try for just 50 possible IDs
  
      for(i in 105759900:105759950){
  
        try(df&lt;-rbind(getRouteData(i),df))
  
      }
  
      #that works but is slow
      df
  </code></pre>
  
  <p>That works fine, but is really slow. Any help on a different way to sort out the irrelevant numbers or peed up the process will be much appreciated!</p>