title: How to use RandomState with Sklearn RandomizedSearchCV on multiple cores
body: <p>I am puzzled about the right way to use <code>np.random.RandomState</code> with <code>sklearn.model_selection.RandomizedSearchCV</code> when running on multiple cores. </p>
  
  <p>I use <code>RandomState</code> to generate pseudo-random numbers so that my results are reproducible. I give <code>RandomizedSearchCV</code> an instance of <code>RandomState</code> and set <code>n_jobs=-1</code> so that it uses all six cores. </p>
  
  <p>Running on multiple cores introduces an asynchronous element. I expect that this will cause requests for pseudo-random numbers from the various cores to be made in different orders in different runs. Therefore the different runs should give different results, rather than displaying reproducibility.</p>
  
  <p>But in fact the results are reproducible. For a given value of <code>n_iter</code> (i.e., the number of draws from the parameter space), the best hyper-parameter values found are identical from one run to the next. I also get the same values if <code>n_jobs</code> is a positive number that is smaller than the number of cores.</p>
  
  <p>To be specific, here is code:</p>
  
  <pre><code>import numpy as np
  import scipy.stats as stats
  from sklearn.datasets import load_iris
  from sklearn.ensemble import GradientBoostingClassifier
  from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split
  
  # Use RandomState for reproducibility.
  random_state = np.random.RandomState(42)
  
  # Get data. Split it into training and test sets.
  iris = load_iris()
  X, y = iris.data, iris.target
  X_train, X_test, y_train, y_test = train_test_split(
      X, y, test_size=0.4, random_state=random_state, stratify=y)
  
  # Prepare for hyper-parameter optimization.
  n_iter = 1_000
  
  base_clf = GradientBoostingClassifier(
      random_state=random_state, max_features='sqrt')
  
  param_space = {'learning_rate': stats.uniform(0.05, 0.2),
                 'n_estimators': [50, 100, 200],
                 'subsample': stats.uniform(0.8, 0.2)}
  
  # Generate data folds for cross validation.
  skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=random_state)
  
  # Create the search classifier.
  search_clf = RandomizedSearchCV(
      base_clf, param_space, n_iter=n_iter, scoring='f1_weighted', n_jobs=-1, 
      cv=skf, random_state=random_state, return_train_score=False)
  
  # Optimize the hyper-parameters and print the best ones found.
  search_clf.fit(X_train, y_train)
  print('Best params={}'.format(search_clf.best_params_))
  </code></pre>
  
  <p>I have several questions.</p>
  
  <ol>
  <li><p>Why do I get reproducible results despite the asynchronous aspect?</p></li>
  <li><p>The documentation for <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html" rel="nofollow noreferrer"><code>RandomizedSearchCV</code></a> says about the <code>random_state</code> parameter: "Pseudo random number generator state used for random uniform sampling from lists of possible values instead of scipy.stats distributions." Does this mean that it does not affect the distributions in the parameter space? Is the code above sufficient to ensure reproducibility, or do I need to set <code>np.random.seed()</code>, or perhaps write something like this:</p>
  
  <pre><code>distn_learning_rate = stats.uniform(0.05, 0.2)  
  distn_learning_rate.random_state = random_state  
  distn_subsample = stats.uniform(0.8, 0.2)  
  distn_subsample.random_state = random_state  
  param_space = {'learning_rate': distn_learning_rate,  
                 'n_estimators': [50, 100, 200],  
                 'subsample': distn_subsample}  
  </code></pre></li>
  <li><p>Overall, is this the correct way to set up <code>RandomizedSearchCV</code> for reproducibility?</p></li>
  <li><p>Is using a single instance of <code>RandomState</code> ok, or should I use separate instances for <code>train_test_split</code>, <code>GradientBoostingClassifier</code>, <code>StratifiedKFold</code>, and <code>RandomizedSearchCV</code>? Also, the documentation of <a href="https://docs.scipy.org/doc/numpy-1.15.4/reference/generated/numpy.random.seed.html" rel="nofollow noreferrer"><code>np.random.seed</code></a> says that the seed is set when <code>RandomState</code> is initialized. How does this interact with <code>RandomizedSearchCV</code> setting the seed?</p></li>
  <li><p>When <code>n_jobs</code> is set to use fewer than all the cores, I still see activity on all the cores, though the usage level per core increases and the elapsed time decreases as the number of cores increases. Is this just sklearn and/or macOS optimizing the machine usage?</p></li>
  </ol>
  
  <p>I am using macOS 10.14.2, Python 3.6.7, Numpy 1.15.4, Scipy 1.1.0, and Sklearn 0.20.1.</p>