class_docstr: PUBDEV-1843: Grid testing.  Subtask 2.
  
  This class is created to test the gridsearch for deeplearning algo and make sure it runs.  Only one test is
  performed here.
  
  Test Descriptions:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O deeplearning models using grid search.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.  Due to the multitude of warning
         messages about not returning a model when gridsearch parameters are set to be unreasonable, we will
         not attempt to count the number of gridsearch models built here.  Since we test this feature for all
         other algos, we are skipping it here.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O deeplearning model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
  
  Note that for hyper-parameters containing all legal parameter names and parameter value lists with legal
  and illegal values, grid-models should be built for all combinations of legal parameter values.  For
  illegal parameter values, a warning/error message should be printed out to warn the user but the
  program should not throw an exception.
  
  We will re-use the dataset generation methods for GLM.
class_docstr_tok: ["PUBDEV-1843",":","Grid","testing",".","Subtask","2",".","This","class","is","created","to","test","the","gridsearch","for","deeplearning","algo","and","make","sure","it","runs",".","Only","one","test","is","performed","here",".","Test","Descriptions",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","deeplearning","models","using","grid","search",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","Due","to","the","multitude","of","warning","messages","about","not","returning","a","model","when","gridsearch","parameters","are","set","to","be","unreasonable",",","we","will","not","attempt","to","count","the","number","of","gridsearch","models","built","here",".","Since","we","test","this","feature","for","all","other","algos",",","we","are","skipping","it","here",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","deeplearning","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure",".","Note","that","for","hyper-parameters","containing","all","legal","parameter","names","and","parameter","value","lists","with","legal","and","illegal","values",",","grid-models","should","be","built","for","all","combinations","of","legal","parameter","values",".","For","illegal","parameter","values",",","a","warning\/error","message","should","be","printed","out","to","warn","the","user","but","the","program","should","not","throw","an","exception",".","We","will","re-use","the","dataset","generation","methods","for","GLM","."]
docstr: test_deeplearning_fieldnames performs the following:
  a. build H2O deeplearning models using grid search.  Count and make sure models
     are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
     values.  We should instead get a warning/error message printed out.
  c. For each model built using grid search, we will extract the parameters used in building
     that model and manually build a H2O deeplearning model.  Training metrics are calculated from the
     gridsearch model and the manually built model.  If their metrics
     differ by too much, print a warning message but don't fail the test.
  d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
     for it as well.  If max_runtime_secs was exceeded, declare test failure.
docstr_tok: ["test_deeplearning_fieldnames","performs","the","following",":","a.","build","H2O","deeplearning","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","deeplearning","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure","."]
code: def test_deeplearning_grid_search_over_params(self):
      """
      test_deeplearning_fieldnames performs the following:
      a. build H2O deeplearning models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O deeplearning model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
      """
  
      print(
          "*******************************************************************************************"
      )
      print(
          "test_deeplearning_fieldnames for deeplearning " + self.family
      )
      h2o.cluster_info()
  
      # start grid search
      # grid_model = H2OGridSearch(H2ODeepLearningEstimator(nfolds=self.nfolds, seed=self.seed),
      #                             hyper_params=self.final_hyper_params)
      # grid_model.train(x=self.x_indices, y=self.y_index, training_frame=self.training1_data)
      #
      # self.correct_model_number = len(grid_model)     # store number of models built
  
      try:
          print(
              "Hyper-parameters used here is {0}".format(
                  self.final_hyper_params
              )
          )
  
          # start grid search
          grid_model = H2OGridSearch(
              H2ODeepLearningEstimator(nfolds=self.nfolds),
              hyper_params=self.final_hyper_params,
          )
          grid_model.train(
              x=self.x_indices,
              y=self.y_index,
              training_frame=self.training1_data,
          )
  
          # add parameters into params_dict.  Use this to manually build model
          params_dict = dict()
          params_dict["distribution"] = self.family
          params_dict["nfolds"] = self.nfolds
          total_run_time_limits = (
              0.0  # calculate upper bound of max_runtime_secs
          )
          true_run_time_limits = 0.0
          manual_run_runtime = 0.0
  
          # compare MSE performance of model built by gridsearch with manually built model
          for each_model in grid_model:
              params_list = grid_model.get_hyperparams_dict(
                  each_model._id
              )
              params_list.update(params_dict)
  
              model_params = dict()
  
              # need to taken out max_runtime_secs from model parameters, it is now set in .train()
              if "max_runtime_secs" in params_list:
                  model_params["max_runtime_secs"] = params_list[
                      "max_runtime_secs"
                  ]
                  max_runtime = params_list["max_runtime_secs"]
                  del params_list["max_runtime_secs"]
              else:
                  max_runtime = 0
  
              if "elastic_averaging_moving_rate" in params_list:
                  model_params[
                      "elastic_averaging_moving_rate"
                  ] = params_list["elastic_averaging_moving_rate"]
                  del params_list["elastic_averaging_moving_rate"]
  
              if "validation_frame" in params_list:
                  model_params["validation_frame"] = params_list[
                      "validation_frame"
                  ]
                  del params_list["validation_frame"]
  
              if "elastic_averaging_regularization" in params_list:
                  model_params[
                      "elastic_averaging_regularization"
                  ] = params_list["elastic_averaging_regularization"]
                  del params_list["elastic_averaging_regularization"]
  
              if "hidden" in params_list:
                  temp = params_list["hidden"]
                  params_list["hidden"] = [temp]
  
              if "hidden_dropout_ratios" in params_list:
                  temp = params_list["hidden_dropout_ratios"]
                  params_list["hidden_dropout_ratios"] = [temp]
  
              manual_model = H2ODeepLearningEstimator(**params_list)
              manual_model.train(
                  x=self.x_indices,
                  y=self.y_index,
                  training_frame=self.training1_data,
                  **model_params,
              )
  
              # collect the time taken to manually built all models
              model_runtime = pyunit_utils.find_grid_runtime(
                  [manual_model]
              )  # time taken to build this model
              manual_run_runtime += model_runtime
  
              summary_list = manual_model._model_json["output"][
                  "scoring_history"
              ]
              if len(summary_list.cell_values) < 3:
                  num_iterations = 1
              else:
                  num_iterations = summary_list["iterations"][2]
  
              if max_runtime > 0:
                  # shortest possible time it takes to build this model
                  if (max_runtime < self.min_runtime_per_iteration) or (
                      num_iterations <= 1
                  ):
                      total_run_time_limits += model_runtime
                  else:
                      total_run_time_limits += max_runtime
  
              true_run_time_limits += max_runtime
  
              # compute and compare test metrics between the two models
              grid_model_metrics = (
                  each_model.model_performance()._metric_json[
                      self.training_metric
                  ]
              )
              manual_model_metrics = (
                  manual_model.model_performance()._metric_json[
                      self.training_metric
                  ]
              )
  
              # just compare the mse in this case within tolerance:
              if not (
                  (type(grid_model_metrics) == str)
                  or (type(manual_model_metrics) == str)
              ):
                  if (abs(grid_model_metrics) > 0) and abs(
                      grid_model_metrics - manual_model_metrics
                  ) / grid_model_metrics > self.allowed_diff:
                      print(
                          "test_deeplearning_fieldnames for deeplearning warning: grid search "
                          "model metric ({0}) and manually built H2O model metric ({1}) differ too much"
                          "!".format(
                              grid_model_metrics, manual_model_metrics
                          )
                      )
  
          total_run_time_limits = max(
              total_run_time_limits, true_run_time_limits
          ) * (1 + self.extra_time_fraction)
  
          # make sure the max_runtime_secs is working to restrict model built time
          if not (manual_run_runtime <= total_run_time_limits):
              self.test_failed += 1
              print(
                  "test_deeplearning_fieldnames for deeplearning failed: time taken to manually build"
                  " models is {0}.  Maximum allowed time is {1}".format(
                      manual_run_runtime, total_run_time_limits
                  )
              )
          else:
              print(
                  "time taken to manually build all models is {0}. Maximum allowed time is "
                  "{1}".format(
                      manual_run_runtime, total_run_time_limits
                  )
              )
  
          if self.test_failed == 0:
              print(
                  "test_deeplearning_fieldnames for deeplearning has passed!"
              )
      except Exception as e:
          if len(grid_model) > 0:
              print(
                  "test_deeplearning_fieldnames for deeplearning failed: exception ({0}) was thrown for "
                  "no reason.".format(e)
              )
              self.test_failed += 1
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/deeplearning/pyunit_deeplearning_gridsearch_over_all_params_large.py
imports: import sys
  import random
  import os
  from builtins import range
  import time
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.deeplearning import H2ODeepLearningEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 4 for calling the Test_deeplearning_grid_search obj's underlying member method code with example usage: obj.test_deeplearning_grid_search_over_params() without return types