title: H2O R api: retrieving optimal model from grid search
body: <p>I'm using the <code>h2o</code> package (v 3.6.0) in R, and I've built a grid search model. Now, I'm trying to access the model which minimizes MSE on the validation set. In python's <code>sklearn</code>, this is easily achievable when using <code>RandomizedSearchCV</code>:</p>
  
  <pre><code>## Pseudo code:
  grid = RandomizedSearchCV(model, params, n_iter = 5)
  grid.fit(X)
  best = grid.best_estimator_
  </code></pre>
  
  <p>This, unfortunately, does not prove as straightforward in h2o. Here's an example you can recreate:</p>
  
  <pre><code>library(h2o)
  ## assume you got h2o initialized...
  
  X &lt;- as.h2o(iris[1:100,]) # Note: only using top two classes for example 
  grid &lt;- h2o.grid(
      algorithm = 'gbm',
      x = names(X[,1:4]),
      y = 'Species',
      training_frame = X,
      hyper_params = list(
          distribution = 'bernoulli',
          ntrees = c(25,50)
      )
  )
  </code></pre>
  
  <p>Viewing <code>grid</code> prints a wealth of information, including this portion:</p>
  
  <pre><code>&gt; grid
  ntrees distribution status_ok                                                                 model_ids
   50    bernoulli        OK Grid_GBM_file1742e107fe5ba_csv_10.hex_11_model_R_1456492736353_16_model_1
   25    bernoulli        OK Grid_GBM_file1742e107fe5ba_csv_10.hex_11_model_R_1456492736353_16_model_0
  </code></pre>
  
  <p>With a bit of digging, you can access each individual model and view every metric imaginable:</p>
  
  <pre><code>&gt; h2o.getModel(grid@model_ids[[1]])
  H2OBinomialModel: gbm
  Model ID:  Grid_GBM_file1742e107fe5ba_csv_10.hex_11_model_R_1456492736353_18_model_1 
  Model Summary: 
    number_of_trees model_size_in_bytes min_depth max_depth mean_depth min_leaves max_leaves mean_leaves
  1              50                4387         1         1    1.00000          2          2     2.00000
  
  
  H2OBinomialMetrics: gbm
  ** Reported on training data. **
  
  MSE:  1.056927e-05
  R^2:  0.9999577
  LogLoss:  0.003256338
  AUC:  1
  Gini:  1
  
  Confusion Matrix for F1-optimal threshold:
             setosa versicolor    Error    Rate
  setosa         50          0 0.000000   =0/50
  versicolor      0         50 0.000000   =0/50
  Totals         50         50 0.000000  =0/100
  
  Maximum Metrics: Maximum metrics at their respective thresholds
                        metric threshold    value idx
  1                     max f1  0.996749 1.000000   0
  2                     max f2  0.996749 1.000000   0
  3               max f0point5  0.996749 1.000000   0
  4               max accuracy  0.996749 1.000000   0
  5              max precision  0.996749 1.000000   0
  6           max absolute_MCC  0.996749 1.000000   0
  7 max min_per_class_accuracy  0.996749 1.000000   0
  </code></pre>
  
  <p>And with a <em><strong>lot</strong></em> of digging, you can finally get to this:</p>
  
  <pre><code>&gt; h2o.getModel(grid@model_ids[[1]])@model$training_metrics@metrics$MSE
  [1] 1.056927e-05
  </code></pre>
  
  <p>This seems like a lot of kludgey work to get down to a metric that ought to be top-level for model selection. In my situation, I've got a grid with hundreds of models, and my current, hacky solution just doesn't seems very "R-esque":</p>
  
  <pre><code>model_select_ &lt;- function(grid) {
    model_ids &lt;- grid@model_ids
    min = Inf
    best_model = NULL
  
    for(model_id in model_ids) {
      model &lt;- h2o.getModel(model_id)
      mse &lt;- model@model$training_metrics@metrics$MSE
      if(mse &lt; min) {
        min &lt;- mse
        best_model &lt;- model
      }
    }
  
    best_model
  }
  </code></pre>
  
  <p>This seems like overkill for something that is so core to the practice of machine learning, and it just strikes me as odd that h2o would not have a "cleaner" method of extracting the optimal model, or at least model metrics. </p>
  
  <p>Am I missing something? Is there no "out of the box" method for selecting the best model?</p>