title: understanding scikit learn Random Forest memory requirement for prediction
body: <p>I have a set of 2000 trained random regression trees (from scikit learn's Random Forest Regressor with <code>n_estimators=1</code>). Training the trees in parallel (50 cores) on a large dataset (~100000*700000 = 70GB @ 8-bit) using <code>multiprocessing</code> and shared memory works like a charm. Note, I am not using RF's inbuilt multicore support since I am doing feature selection beforehand.</p>
  
  <p>The problem: when testing a large matrix (~20000*700000) in parallel I always run out of memory (I have access to a server with 500 GB of RAM).</p>
  
  <p>My strategy is to have the test matrix in memory and share it among all processes. 
  According to a <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4091/scikit-learn-multi-core-processing/24747" rel="noreferrer">statement by one of the developers</a> the memory requirement for testing is 2*n_jobs*sizeof(X), and in my case another factor of *4 is relevant, since the 8bit matrix entries are upcast to float32 internally in RF. </p>
  
  <p>By numbers, I think for testing I need:<br>
  14GB to hold the test matrix in memory + 50(=n_jobs)*20000(n_samples)*700(=n_features)*4(upcasting to float)*2 bytes = 14 gb + 5.6 gb = ~21GB of memory.</p>
  
  <p>Yet it always blows up to several hundreds of GB.
  What am I missing here?
  (I am on the newest version of scikit-learn, so the old memory issues should be ironed out)</p>
  
  <p>An observation:<br>
  Running on one core only memory usage for testing fluctuates between 30 and 100 GB (as measured by <code>free</code>)</p>
  
  <p>My code:</p>
  
  <pre><code>#----------------
  #helper functions
  def initializeRFtest(*args):
      global df_test, pt_test #initialize test data and test labels as globals in shared memory
      df_test, pt_test = args
  
  
  def star_testTree(model_featidx):
      return predTree(*model_featidx)
  
  #end of helper functions
  #-------------------
  
  def RFtest(models, df_test, pt_test, features_idx, no_trees):
      #test trees in parallel
      ncores = 50
      p = Pool(ncores, initializer=initializeRFtest, initargs=(df_test, pt_test))
      args = itertools.izip(models, features_idx)
      out_list = p.map(star_testTree, args)
      p.close()
      p.join()
      return out_list
  
  def predTree(model, feat_idx):
      #get all indices of samples that meet feature subset requirement
      nan_rows = np.unique(np.where(df_test.iloc[:,feat_idx] == settings.nan_enc)[0])
      all_rows = np.arange(df_test.shape[0])
      rows = all_rows[np.invert(np.in1d(all_rows, nan_rows))]    #discard rows with missing values in the given features
  
      #predict
      pred = model.predict(df_test.iloc[rows,feat_idx])
      return predicted
  
  #main program
  out = RFtest(models, df_test, pt_test, features_idx, no_trees)
  </code></pre>
  
  <p><strong>Edit:</strong>
  another observation:
  When chunking the test data the program runs smoothly with much reduced memory usage. This is what I used to make the program run.<br>
  Code snippet for the updated <code>predTree</code> function:</p>
  
  <pre><code>def predTree(model, feat_idx):
      # get all indices of samples that meet feature subset requirement
      nan_rows = np.unique(np.where(test_df.iloc[:,feat_idx] == settings.nan_enc)[0])
      all_rows = np.arange(test_df.shape[0])
      rows = all_rows[np.invert(np.in1d(all_rows, nan_rows))]    #discard rows with missing values in the given features
  
      # predict height per valid sample
      chunksize = 500
      n_chunks = np.int(math.ceil(np.float(rows.shape[0])/chunksize))
  
  
      pred = []
      for i in range(n_chunks):
          if n_chunks == 1:
              pred_chunked = model.predict(test_df.iloc[rows[i*chunksize:], feat_idx])
              pred.append(pred_chunked)
              break
          if i == n_chunks-1:
              pred_chunked = model.predict(test_df.iloc[rows[i*chunksize:], feat_idx])
          else:
              pred_chunked = model.predict(test_df.iloc[rows[i*chunksize:(i+1)*chunksize], feat_idx])
          print pred_chunked.shape
          pred.append(pred_chunked)
      pred = np.concatenate(pred)
  
      # populate matrix
      predicted = np.empty(test_df.shape[0])
      predicted.fill(np.nan)
      predicted[rows] = pred
      return predicted
  </code></pre>