Best way to store large data set using R from Twitter? I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R.  have difficulty storing tweets because my machine has only 8 GB of memory. It ran out of memory even before I set it to retrieve for one day. Is there a way where I can store the tweets straight to my disk without storing into RAM? I am not using the streaming API as I need to get old tweets.
Find a way to make your program write to disk periodically. Keep count of the number of tweets you grab and save after that number is high. I don't write R but psuedocode might look like:

$tweets = get_tweets();
$count = 0;
$tweet_array = array();
for each ($tweets as $tweet) {
  $tweet_array += $tweet;
  $count++;
  if ($count &gt; 10000) {
    append_to_file($tweet_array, 'file_name.txt');
    clear_array($tweet_array);
  }
}