title: Why is Random Search showing better results than Grid Search?
body: <p>I'm playing with RandomizedSearchCV function from scikit-learn. Some academic paper claims that Randomized Search can provide 'good enough' results comparing with a whole grid search, but saves a lot of time.</p>
  
  <p>Surprisingly, on one occasion, the RandomizedSearchCV provided me better results than GridSearchCV. I think GridSearchCV is suppose to be exhaustive, so the result has to be better than RandomizedSearchCV suppose they search through the same grid.</p>
  
  <p>for the same dataset and mostly same settings, GridsearchCV returned me the following result:<br>
  Best cv accuracy: 0.7642857142857142<br>
  Test set score:   0.725<br>
  Best parameters:  'C': 0.02  </p>
  
  <p>the RandomizedSearchCV returned me the following result:
  Best cv accuracy: 0.7428571428571429<br>
  Test set score:   0.7333333333333333<br>
  Best parameters:  'C': 0.008  </p>
  
  <p>To me the test score of 0.733 is better than 0.725, and the difference between test score and training score for the RandomizedSearchCV is smaller, which to my knowledge means less overfitting.</p>
  
  <p>So why did GridSearchCV return me worse results?</p>
  
  <p>GridSearchCV code:  </p>
  
  <pre><code>def linear_SVC(x, y, param, kfold):
      param_grid = {'C':param}
      k = KFold(n_splits=kfold, shuffle=True, random_state=0)
      grid = GridSearchCV(LinearSVC(), param_grid=param_grid, cv=k, n_jobs=4, verbose=1)
  
      return grid.fit(x, y)
  
  #high C means more chance of overfitting
  
  start = timer()
  param = [i/1000 for i in range(1,1000)]
  param1 = [i for i in range(1,101)]
  param.extend(param1)
  
  #progress = progressbar.bar.ProgressBar()
  clf = linear_SVC(x=x_train, y=y_train, param=param, kfold=3)
  
  print('LinearSVC:')
  print('Best cv accuracy: {}' .format(clf.best_score_))
  print('Test set score:   {}' .format(clf.score(x_test, y_test)))
  print('Best parameters:  {}' .format(clf.best_params_))
  print()
  
  duration = timer() - start
  print('time to run: {}' .format(duration))
  </code></pre>
  
  <p>RandomizedSearchCV code:  </p>
  
  <pre><code>from sklearn.model_selection import RandomizedSearchCV
  
  def Linear_SVC_Rand(x, y, param, kfold, n):
      param_grid = {'C':param}
      k = StratifiedKFold(n_splits=kfold, shuffle=True, random_state=0)
      randsearch = RandomizedSearchCV(LinearSVC(), param_distributions=param_grid, cv=k, n_jobs=4,
                                      verbose=1, n_iter=n)
  
      return randsearch.fit(x, y)
  
  start = timer()
  param = [i/1000 for i in range(1,1000)]
  param1 = [i for i in range(1,101)]
  param.extend(param1)
  
  #progress = progressbar.bar.ProgressBar()
  clf = Linear_SVC_Rand(x=x_train, y=y_train, param=param, kfold=3, n=100)
  
  print('LinearSVC:')
  print('Best cv accuracy: {}' .format(clf.best_score_))
  print('Test set score:   {}' .format(clf.score(x_test, y_test)))
  print('Best parameters:  {}' .format(clf.best_params_))
  print()
  
  duration = timer() - start
  print('time to run: {}' .format(duration))
  </code></pre>