ScikitLearn - RandomForestRegressor score different in and out of grid search I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
                 'min_samples_split' : range( 2 , 11 ) }


  rfr = RandomForestRegressor()

  best_score = 0
  best_param = {}

  for param in ParameterGrid( param_grid ):
      rfr.set_params( **g )
      rfr.fit( X_train , y_train )
      score = rfr.score( X_train , y_train )

  if score &gt; best_score :
      best_score = score
      best_param = param

  print( 'best_score : {0}'.format( best_score ) )
  print( 'best_parameters : {0}'.format( best_param ) )


I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )


I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )


Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )


The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?
I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )


What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter. 

I guess it is something I will have to deal with.