class_docstr: PUBDEV-2967: gridsearch sorting metric with cross-validation.
  
  This class is created to test that when cross-validation is enabled, the gridsearch models are returned sorted
  according to the cross-validation metrics.
  
  Test Descriptions:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O random forest models using grid search.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. Check and make sure that the models are returned sorted with the correct cross-validation metrics.
  
  Note that for hyper-parameters containing all legal parameter names and parameter value lists with legal
  and illegal values, grid-models should be built for all combinations of legal parameter values.  For
  illegal parameter values, a warning/error message should be printed out to warn the user but the
  program should not throw an exception;
  
  We will re-use the dataset generation methods for GLM.  There will be only one data set for classification.
class_docstr_tok: ["PUBDEV-2967",":","gridsearch","sorting","metric","with","cross-validation",".","This","class","is","created","to","test","that","when","cross-validation","is","enabled",",","the","gridsearch","models","are","returned","sorted","according","to","the","cross-validation","metrics",".","Test","Descriptions",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","random","forest","models","using","grid","search",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","Check","and","make","sure","that","the","models","are","returned","sorted","with","the","correct","cross-validation","metrics",".","Note","that","for","hyper-parameters","containing","all","legal","parameter","names","and","parameter","value","lists","with","legal","and","illegal","values",",","grid-models","should","be","built","for","all","combinations","of","legal","parameter","values",".","For","illegal","parameter","values",",","a","warning\/error","message","should","be","printed","out","to","warn","the","user","but","the","program","should","not","throw","an","exception",";","We","will","re-use","the","dataset","generation","methods","for","GLM",".","There","will","be","only","one","data","set","for","classification","."]
docstr: test_rf_gridsearch_sorting_metrics performs the following:
  b. build H2O random forest models using grid search.  No model is built for bad hyper-parameters
     values.  We should instead get a warning/error message printed out.
  c. Check and make sure that the models are returned sorted with the correct cross-validation metrics.
docstr_tok: ["test_rf_gridsearch_sorting_metrics","performs","the","following",":","b.","build","H2O","random","forest","models","using","grid","search",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","Check","and","make","sure","that","the","models","are","returned","sorted","with","the","correct","cross-validation","metrics","."]
code: def test_rf_gridsearch_sorting_metrics(self):
      """
      test_rf_gridsearch_sorting_metrics performs the following:
      b. build H2O random forest models using grid search.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. Check and make sure that the models are returned sorted with the correct cross-validation metrics.
      """
  
      if self.possible_number_models > 0:
          print(
              "*******************************************************************************************"
          )
          print("test_rf_gridsearch_sorting_metrics for random forest ")
          h2o.cluster_info()
  
          print(
              "Hyper-parameters used here is {0}".format(
                  self.final_hyper_params
              )
          )
  
          # start grid search
          grid_model = H2OGridSearch(
              H2ORandomForestEstimator(
                  nfolds=self.nfolds,
                  seed=self.seed,
                  score_tree_interval=0,
              ),
              hyper_params=self.final_hyper_params,
          )
          grid_model.train(
              x=self.x_indices,
              y=self.y_index,
              training_frame=self.training1_data,
          )
  
          result_table = grid_model._grid_json["summary_table"]
          model_index = 0
          grid_model_metrics = []
  
          diff = 0  # calculate difference between gridsearch model metrics and manually extracted model.
          diff_train = 0  # calculate difference between training and cross-validation metrics
  
          # grab performance metric for each model of grid_model and collect correct sorting metrics by hand
          for each_model in grid_model:
              grid_model_metric = float(
                  result_table[self.training_metric][model_index]
              )
              grid_model_metrics.append(grid_model_metric)
  
              manual_metric = each_model._model_json["output"][
                  "cross_validation_metrics"
              ]._metric_json["logloss"]
              if not (type(grid_model_metrics) == unicode) and not (
                  type(manual_metric) == unicode
              ):
                  diff += abs(grid_model_metric - manual_metric)
  
              manual_training_metric = each_model._model_json["output"][
                  "training_metrics"
              ]._metric_json["logloss"]
              if not (type(grid_model_metrics) == unicode) and not (
                  type(manual_training_metric) == unicode
              ):
                  diff_train += abs(
                      grid_model_metric - manual_training_metric
                  )
  
              print(
                  "grid model logloss: {0}, grid model training logloss: "
                  "{1}".format(
                      grid_model_metric, manual_training_metric
                  )
              )
  
              model_index += 1
  
          if (
              (diff > self.diff)
              or not (grid_model_metrics == sorted(grid_model_metrics))
              or (diff_train < self.diff)
          ):
              self.test_failed = 1
              print(
                  "test_rf_gridsearch_sorting_metrics for random forest has failed!"
              )
  
          if self.test_failed == 0:
              print(
                  "test_rf_gridsearch_sorting_metrics for random forest has passed!"
              )
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/rf/pyunit_rf_gridsearch_sorting_metrics.py
imports: import sys
  import random
  import os
  from builtins import range
  import time
  import json
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.random_forest import H2ORandomForestEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 4 for calling the Test_rf_gridsearch_sorting_metrics obj's underlying member method code with example usage: obj.test_rf_gridsearch_sorting_metrics() without return types