title: How to scrape all subreddit posts in a given time period
body: <p>I have a function to scrape all the posts in the Bitcoin subreddit between 2014-11-01 and 2015-10-31.  </p>
  
  <p>However, I'm only able to extract about 990 posts that go back only to October 25.  I don't understand what's happening.  I included a Sys.sleep of 15 seconds between each extract after referring to <a href="https://github.com/reddit/reddit/wiki/API">https://github.com/reddit/reddit/wiki/API</a>, to no avail.</p>
  
  <p>Also, I experimented with scraping from another subreddit (fitness), but it also returned around 900 posts.  </p>
  
  <pre><code>require(jsonlite)
  require(dplyr)
  
  getAllPosts &lt;- function() {
      url &lt;- "https://www.reddit.com/r/bitcoin/search.json?q=timestamp%3A1414800000..1446335999&amp;sort=new&amp;restrict_sr=on&amp;rank=title&amp;syntax=cloudsearch&amp;limit=100"
      extract &lt;- fromJSON(url)
      posts &lt;- extract$data$children$data %&gt;% dplyr::select(name, author,   num_comments, created_utc,
                                               title, selftext)  
      after &lt;- posts[nrow(posts),1]
      url.next &lt;- paste0("https://www.reddit.com/r/bitcoin/search.json?q=timestamp%3A1414800000..1446335999&amp;sort=new&amp;restrict_sr=on&amp;rank=title&amp;syntax=cloudsearch&amp;after=",after,"&amp;limit=100")
      extract.next &lt;- fromJSON(url.next)
      posts.next &lt;- extract.next$data$children$data
  
      # execute while loop as long as there are any rows in the data frame
      while (!is.null(nrow(posts.next))) {
          posts.next &lt;- posts.next %&gt;% dplyr::select(name, author, num_comments, created_utc, 
                                      title, selftext)
          posts &lt;- rbind(posts, posts.next)
          after &lt;- posts[nrow(posts),1]
          url.next &lt;- paste0("https://www.reddit.com/r/bitcoin/search.json?q=timestamp%3A1414800000..1446335999&amp;sort=new&amp;restrict_sr=on&amp;rank=title&amp;syntax=cloudsearch&amp;after=",after,"&amp;limit=100")
          Sys.sleep(15)
          extract &lt;- fromJSON(url.next)
          posts.next &lt;- extract$data$children$data
      }
      posts$created_utc &lt;- as.POSIXct(posts$created_utc, origin="1970-01-01")
      return(posts)
  }
  
  posts &lt;- getAllPosts()
  </code></pre>
  
  <p>Does reddit have some kind of limit that I'm hitting?</p>