title: R: stack overflow error with randomForest on large dataset (48-512 GB RAM)
body: <p>I am attempting an R randomForest analysis in R on a wide genetic dataset​(662 x 35350). All variables except the outcome are numeric and 99% of them are binary 0/1. I am quite familiar with R randomForest(), but have only worked with datasets with 5000-10000 variables previously. The next planned phase of analyses will be on an exceptionally large dataset with millions of variables so I am motivated to find a solution to this problem. </p>
  
  <p>My understanding is that R randomForest has no inherent limits on number of variables and I know I've read published work with variable numbers in the 100,000s.  When I attempt the analysis on the full dataset (setting ntree=100, I get:
  "Error: protect(): protection stack overflow"</p>
  
  <p>This is true whether the dataset is a dataframe (as it was originally provided) or when I convert it to a matrix. When I submit the run to a cluster for parallel processing, I see that all of my cores are working as soon as I execute the code. I also see that, at no point, does my RAM usage approach the machine's limit (48 GB). At most it hits about 16% of RAM during the execution attempt.  (I also had the same problem on my 512 GB RAM machine at the office, where it never used more than about 5%).</p>
  
  <p>I have tried several solutions found online, including one in a previous stackoverflow post (<a href="https://stackoverflow.com/questions/1395229/increasing-or-decreasing-the-memory-available-to-r-processes">Increasing (or decreasing) the memory available to R processes</a>). I tried the instructions provided by BobbyShaftoe in 2009 (adding --max-mem-size=49000M and --max-vsize=49000M within the properties of the Shortcut tab), but this prevented R from opening properly. I also tried running these commands in command line, but these generated: '--max-ppsize'/'--max-vsize=5000M" is not recognized as an internal or external command, operable program or batch file.</p>
  
  <p>I have also read the suggestions made in this post: <a href="https://stackoverflow.com/questions/23075506/how-to-improve-randomforest-performance">How to improve randomForest performance?</a>.  I can't reduce the number of features until I have at least one run with the full feature set. (Plus, I am not sure the problem is RAM, per se.)</p>
  
  <p>I'm on Windows 7 running Revolution R 7.2 (64-bit).
  My memory limit is set at 49807 Mb, but I'm not sure if the memory.limit is specifially addressing the protection stack size allowed. </p>
  
  <p>Breaking the dataset into smaller chunks of variables (which does solve the stack overflow problem) does not solve the analytic problem. Are there any suggestions as to R settings that may permit the analysis on the full dataset?</p>
  
  <pre><code>##########################################
  # input DF
  ##########################################
  object.size(inputDF) # 191083664 bytes (as matrix, size=189391088 bytes, not much smaller)
  dim(inputDF) # 662 x 35350 
  
  ##########################################
  #Load necessary packages into R's memory
  ##########################################
  require(iterators)
  require(foreach)
  require(parallel)
  require(doParallel)
  require(randomForest)
  
  ###########################################
  # Get the number of available logical cores
  ###########################################
  cores &lt;- detectCores()
  cores #12
  
  ###########################################
  # Print info on computer, OS, cores
  ###########################################
  print(paste('Processor: ', Sys.getenv('PROCESSOR_IDENTIFIER')), sep='')
  print(paste('OS: ', Sys.getenv('OS')), sep='')
  print(paste('Cores: ', cores, sep=''))
  
  ###########################################################################
  # Setup clusters via parallel/DoParallel
  ###########################################################################
  cl.spec &lt;- rep("localhost", 10)
  cl &lt;- makeCluster(cl.spec, type="SOCK")
  registerDoParallel(cl, cores=10)
  
  ###########################################################################
  # RUN RANDOM FOREST
  ###########################################################################
  system.time(forestOUT&lt;- randomForest(as.factor(Dx01) ~ .,
  data=inputDF,
  do.trace = 10, 
  ntree=100,
  mtry =  sqrt(ncol(inputDF)), 
  nodesize = 0.1*nrow(inputDF),
  importance=T, 
  proximity=F,
  replace=TRUE,
  keep.forest=TRUE))
  
  stopCluster(cl)
  </code></pre>
  
  <p>See sessionInfo()</p>
  
  <pre><code>#Language: R
  #OS: Windows 7
  sessionInfo()
  #R version 3.0.3 (2014-03-06)
  #Platform: x86_64-w64-mingw32/x64 (64-bit)
  #
  #locale:
  #[1] LC_COLLATE=English_Canada.1252  LC_CTYPE=English_Canada.1252    LC_MONETARY=English_Canada.1252 LC_NUMERIC=C                   
  #[5] LC_TIME=English_Canada.1252    
  #
  #attached base packages:
   #[1] stats4    parallel  splines   grid      stats     graphics  grDevices utils     datasets  methods   base     
  #
  #other attached packages:
   #[1] QuantPsyc_1.5         boot_1.3-13           perturb_2.05          RCurl_1.95-4.5        bitops_1.0-6          car_2.0-22           
   #[7] reprtree_0.6          plotrix_3.5-10        rpart.plot_1.4-5      sqldf_0.4-7.1         RSQLite.extfuns_0.0.1 RSQLite_1.0.0        
  #[13] gsubfn_0.6-6          proto_0.3-10          XML_3.98-1.1          RMySQL_0.9-3          DBI_0.3.1             mlbench_2.1-1        
  #[19] polycor_0.7-8         sfsmisc_1.0-26        quantregForest_0.2-3  tree_1.0-35           maptree_1.4-7         cluster_1.15.3       
  #[25] mice_2.22             VIM_4.0.0             colorspace_1.2-4      randomForest_4.6-10   ROCR_1.0-5            gplots_2.15.0        
  #[31] caret_6.0-37          partykit_0.8-0        biomaRt_2.18.0        NCBI2R_1.4.6          snpStats_1.12.0       betareg_3.0-5        
  #[37] arm_1.7-07            lme4_1.1-7            Rcpp_0.11.3           Matrix_1.1-4          nlme_3.1-118          mvtnorm_1.0-1        
  #[43] taRifx_1.0.6          sos_1.3-8             brew_1.0-6            R.utils_1.34.0        R.oo_1.18.0           R.methodsS3_1.6.1    
  #[49] rattle_3.3.0          jsonlite_0.9.13       httpuv_1.3.2          httr_0.5              gmodels_2.15.4.1      ggplot2_1.0.0        
  #[55] JGR_1.7-16            iplots_1.1-7          JavaGD_0.6-1          party_1.0-18          modeltools_0.2-21     strucchange_1.5-0    
  #[61] sandwich_2.3-2        zoo_1.7-11            pROC_1.7.3            e1071_1.6-4           psych_1.4.8.11        gtools_3.4.1         
  #[67] functional_0.6        modeest_2.1           stringi_0.3-1         languageR_1.4.1       utility_1.3           data.table_1.9.4     
  #[73] xlsx_0.5.7            xlsxjars_0.6.1        rJava_0.9-6           snow_0.3-13           doParallel_1.0.8      iterators_1.0.7      
  #[79] foreach_1.4.2         reshape2_1.4          reshape_0.8.5         plyr_1.8.1            xtable_1.7-4          stringr_0.6.2        
  #[85] foreign_0.8-61        Hmisc_3.14-6          Formula_1.1-2         survival_2.37-7       class_7.3-11          MASS_7.3-35          
  #[91] nnet_7.3-8            Revobase_7.2.0        RevoMods_7.2.0        RevoScaleR_7.2.0      lattice_0.20-27       rpart_4.1-5          
  #
  #loaded via a namespace (and not attached):
   #[1] abind_1.4-0         acepack_1.3-3.3     BiocGenerics_0.8.0  BradleyTerry2_1.0-5 brglm_0.5-9         caTools_1.17.1      chron_2.3-45       
   #[8] coda_0.16-1         codetools_0.2-9     coin_1.0-24         DEoptimR_1.0-2      digest_0.6.4        flexmix_2.3-12      gdata_2.13.3       
  #[15] glmnet_1.9-8        gtable_0.1.2        KernSmooth_2.23-13  latticeExtra_0.6-26 lmtest_0.9-33       minqa_1.2.4         munsell_0.4.2      
  #[22] nloptr_1.0.4        pkgXMLBuilder_1.0   png_0.1-7           RColorBrewer_1.0-5  revoIpe_1.0         robustbase_0.92-2   scales_0.2.4       
  #[29] sp_1.0-16           tcltk_3.0.3         tools_3.0.3         vcd_1.3-2   
  </code></pre>