class_docstr: PUBDEV-1843: Grid testing.  Subtask 7,8.
  
  This class is created to test the gridsearch with the GLM algo using Guassian, Binonmial or
  Multinomial family.  Three tests are written to test the following conditions:
  1. For hyper-parameters containing all legal parameter names and parameter value lists with legal
  and illegal values, grid-models should be built for all combinations of legal parameter values.  For
  illegal parameter values, a warning/error message should be printed out to warn the user but the
  program should not throw an exception;
  2. For hyper-parameters with illegal names, an exception should be thrown and no models should be built;
  3. For parameters that are specified both in the hyper-parameters and model parameters, unless the values
  specified in the model parameters are set to default values, an exception will be thrown since parameters are
  not supposed to be specified in both places.
  
  Test Descriptions:
  test1_glm_grid_search_over_params: test for condition 1 and performs the following:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O GLM models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O GLM model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
  
  test2_illegal_name_value: test for condition 1 and 2.  Randomly go into the hyper_parameters that we
  have specified, either
      a. randomly alter the name of a hyper-parameter name (fatal, exception will be thrown)
      b. randomly choose a hyper-parameter and remove all elements in its list (fatal)
      c. add randomly generated new hyper-parameter names with random list (fatal)
      d: randomly choose a hyper-parameter and insert an illegal type into it (non fatal, model built with
         legal hyper-parameters settings only and error messages printed out for illegal hyper-parameters
         settings)
  
  test3_duplicated_parameter_specification: test for condition 3.  Go into our hyper_parameters list, randomly
  choose some hyper-parameters to specify and specify it as part of the model parameters.  Hence, the same
  parameter is specified both in the model parameters and hyper-parameters.  Make sure the test failed with
  error messages when the parameter values are not set to default if they are specified in the model parameters
  as well as in the hyper-parameters.
class_docstr_tok: ["PUBDEV-1843",":","Grid","testing",".","Subtask","7,8",".","This","class","is","created","to","test","the","gridsearch","with","the","GLM","algo","using","Guassian",",","Binonmial","or","Multinomial","family",".","Three","tests","are","written","to","test","the","following","conditions",":","1",".","For","hyper-parameters","containing","all","legal","parameter","names","and","parameter","value","lists","with","legal","and","illegal","values",",","grid-models","should","be","built","for","all","combinations","of","legal","parameter","values",".","For","illegal","parameter","values",",","a","warning\/error","message","should","be","printed","out","to","warn","the","user","but","the","program","should","not","throw","an","exception",";","2",".","For","hyper-parameters","with","illegal","names",",","an","exception","should","be","thrown","and","no","models","should","be","built",";","3",".","For","parameters","that","are","specified","both","in","the","hyper-parameters","and","model","parameters",",","unless","the","values","specified","in","the","model","parameters","are","set","to","default","values",",","an","exception","will","be","thrown","since","parameters","are","not","supposed","to","be","specified","in","both","places",".","Test","Descriptions",":","test1_glm_grid_search_over_params",":","test","for","condition","1","and","performs","the","following",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","GLM","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","GLM","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure",".","test2_illegal_name_value",":","test","for","condition","1","and","2",".","Randomly","go","into","the","hyper_parameters","that","we","have","specified",",","either","a.","randomly","alter","the","name","of","a","hyper-parameter","name","(","fatal",",","exception","will","be","thrown",")","b.","randomly","choose","a","hyper-parameter","and","remove","all","elements","in","its","list","(","fatal",")","c.","add","randomly","generated","new","hyper-parameter","names","with","random","list","(","fatal",")","d",":","randomly","choose","a","hyper-parameter","and","insert","an","illegal","type","into","it","(","non","fatal",",","model","built","with","legal","hyper-parameters","settings","only","and","error","messages","printed","out","for","illegal","hyper-parameters","settings",")","test3_duplicated_parameter_specification",":","test","for","condition","3",".","Go","into","our","hyper_parameters","list",",","randomly","choose","some","hyper-parameters","to","specify","and","specify","it","as","part","of","the","model","parameters",".","Hence",",","the","same","parameter","is","specified","both","in","the","model","parameters","and","hyper-parameters",".","Make","sure","the","test","failed","with","error","messages","when","the","parameter","values","are","not","set","to","default","if","they","are","specified","in","the","model","parameters","as","well","as","in","the","hyper-parameters","."]
docstr: test1_glm_grid_search_over_params: test for condition 1 and performs the following:
  a. grab all truely griddable parameters and randomly or manually set the parameter values.
  b. Next, build H2O GLM models using grid search.  Count and make sure models
     are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
     values.  We should instead get a warning/error message printed out.
  c. For each model built using grid search, we will extract the parameters used in building
     that model and manually build a H2O GLM model.  Training metrics are calculated from the
     gridsearch model and the manually built model.  If their metrics
     differ by too much, print a warning message but don't fail the test.
  d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
     for it as well.  If max_runtime_secs was exceeded, declare test failure.
docstr_tok: ["test1_glm_grid_search_over_params",":","test","for","condition","1","and","performs","the","following",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","GLM","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","GLM","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure","."]
code: def test1_glm_grid_search_over_params(self):
      """
      test1_glm_grid_search_over_params: test for condition 1 and performs the following:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O GLM models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O GLM model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
      """
  
      print(
          "*******************************************************************************************"
      )
      print("test1_glm_grid_search_over_params for GLM " + self.family)
      h2o.cluster_info()
  
      try:
          print(
              "Hyper-parameters used here is {0}".format(
                  self.final_hyper_params_bad
              )
          )
  
          # start grid search
          grid_model = H2OGridSearch(
              H2OGeneralizedLinearEstimator(
                  family=self.family, nfolds=self.nfolds
              ),
              hyper_params=self.final_hyper_params_bad,
          )
          grid_model.train(
              x=self.x_indices,
              y=self.y_index,
              training_frame=self.training1_data,
          )
  
          self.correct_model_number = len(
              grid_model
          )  # store number of models built
  
          # make sure the correct number of models are built by gridsearch
          if (
              self.correct_model_number - self.possible_number_models
          ) > 0.9:  # wrong grid model number
              self.test_failed += 1
              self.test_failed_array[self.test_num] = 1
              print(
                  "test_glm_search_over_params for GLM failed: number of models built by gridsearch: {0} "
                  "does not equal to all possible combinations of hyper-parameters: "
                  "{1}".format(
                      self.correct_model_number, self.possible_models
                  )
              )
          else:
              # add parameters into params_dict.  Use this to manually build model
              params_dict = dict()
              params_dict["family"] = self.family
              params_dict["nfolds"] = self.nfolds
              total_run_time_limits = (
                  0.0  # calculate upper bound of max_runtime_secs
              )
              true_run_time_limits = 0.0
              manual_run_runtime = 0.0
  
              # compare MSE performance of model built by gridsearch with manually built model
              for each_model in grid_model:
                  # grab parameters used by grid search and build a dict out of it
                  params_list = grid_model.get_hyperparams_dict(
                      each_model._id
                  )
                  params_list.update(params_dict)
  
                  model_params = (
                      dict()
                  )  # some parameters are to be added in .train()
  
                  if "lambda" in list(params_list):
                      params_list["Lambda"] = params_list["lambda"]
                      del params_list["lambda"]
  
                  # need to taken out max_runtime_secs, stopping_rounds, stopping_tolerance
                  # # from model parameters, it is now set in .train()
                  if "max_runtime_secs" in params_list:
                      model_params["max_runtime_secs"] = params_list[
                          "max_runtime_secs"
                      ]
                      del params_list["max_runtime_secs"]
  
                  if "stopping_rounds" in params_list:
                      model_params["stopping_rounds"] = params_list[
                          "stopping_rounds"
                      ]
                      del params_list["stopping_rounds"]
  
                  if "stopping_tolerance" in params_list:
                      model_params["stopping_tolerance"] = params_list[
                          "stopping_tolerance"
                      ]
                      del params_list["stopping_tolerance"]
  
                  manual_model = H2OGeneralizedLinearEstimator(
                      **params_list
                  )
                  manual_model.train(
                      x=self.x_indices,
                      y=self.y_index,
                      training_frame=self.training1_data,
                      **model_params,
                  )
  
                  # collect the time taken to manually built all models
                  model_runtime = pyunit_utils.find_grid_runtime(
                      [manual_model]
                  )  # time taken to build this model
                  manual_run_runtime += model_runtime
  
                  summary_list = manual_model._model_json["output"][
                      "model_summary"
                  ]
                  iteration_num = summary_list.cell_values[
                      "number_of_iterations"
                  ][0]
  
                  if model_params["max_runtime_secs"] > 0:
                      # shortest possible time it takes to build this model
                      if (
                          model_params["max_runtime_secs"]
                          < self.min_runtime_per_epoch
                      ) or (iteration_num <= 1):
                          total_run_time_limits += model_runtime
                      else:
                          total_run_time_limits += model_params[
                              "max_runtime_secs"
                          ]
  
                  true_run_time_limits += model_params[
                      "max_runtime_secs"
                  ]
  
                  # compute and compare test metrics between the two models
                  grid_model_metrics = each_model.model_performance(
                      test_data=self.training2_data
                  )
                  manual_model_metrics = manual_model.model_performance(
                      test_data=self.training2_data
                  )
  
                  # just compare the mse in this case within tolerance:
                  if not (
                      (type(grid_model_metrics.mse()) == str)
                      or (type(manual_model_metrics.mse()) == str)
                  ):
                      mse = grid_model_metrics.mse()
                      if (
                          abs(mse) > 0
                          and abs(mse - manual_model_metrics.mse())
                          / mse
                          > self.allowed_diff
                      ):
                          print(
                              "test1_glm_grid_search_over_params for GLM warning: grid search model metric ({0}) "
                              "and manually built H2O model metric ({1}) differ too much"
                              "!".format(
                                  grid_model_metrics.mse(),
                                  manual_model_metrics.mse(),
                              )
                          )
  
              total_run_time_limits = max(
                  total_run_time_limits, true_run_time_limits
              ) * (1 + self.extra_time_fraction)
  
          # make sure the correct number of models are built by gridsearch
          if not (
              self.correct_model_number == self.possible_number_models
          ):  # wrong grid model number
              self.test_failed += 1
              self.test_failed_array[self.test_num] = 1
              print(
                  "test1_glm_grid_search_over_params for GLM failed: number of models built by gridsearch "
                  "does not equal to all possible combinations of hyper-parameters"
              )
  
          # make sure the max_runtime_secs is working to restrict model built time, GLM does not respect that.
          if not (manual_run_runtime <= total_run_time_limits):
              # self.test_failed += 1
              # self.test_failed_array[self.test_num] = 1
              print(
                  "test1_glm_grid_search_over_params for GLM warning: allow time to build models: {0}, actual "
                  "time taken: {1}".format(
                      total_run_time_limits, manual_run_runtime
                  )
              )
  
          self.test_num += 1
  
          if self.test_failed == 0:
              print(
                  "test1_glm_grid_search_over_params for GLM has passed!"
              )
      except:
          if self.possible_number_models > 0:
              print(
                  "test1_glm_grid_search_over_params for GLM failed: exception was thrown for no reason."
              )
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/glm/pyunit_glm_gridsearch_over_all_params_large.py
imports: import sys
  import random
  import os
  import numpy
  from builtins import range
  import time
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.glm import H2OGeneralizedLinearEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 5 for calling the Test_glm_grid_search obj's underlying member method code with example usage: obj.test1_glm_grid_search_over_params() without return types