class_docstr: PUBDEV-1843: Grid testing.  Subtask 2.
  
  This class is created to test the gridsearch for naivebayes algo and make sure it runs.  Only one test is
  performed here.
  
  Test Descriptions:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O naivebayes models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O naivebayes model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
  
  Note that for hyper-parameters containing all legal parameter names and parameter value lists with legal
  and illegal values, grid-models should be built for all combinations of legal parameter values.  For
  illegal parameter values, a warning/error message should be printed out to warn the user but the
  program should not throw an exception;
  
  We will re-use the dataset generation methods for GLM.  There will be only one data set for classification.
class_docstr_tok: ["PUBDEV-1843",":","Grid","testing",".","Subtask","2",".","This","class","is","created","to","test","the","gridsearch","for","naivebayes","algo","and","make","sure","it","runs",".","Only","one","test","is","performed","here",".","Test","Descriptions",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","naivebayes","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","naivebayes","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure",".","Note","that","for","hyper-parameters","containing","all","legal","parameter","names","and","parameter","value","lists","with","legal","and","illegal","values",",","grid-models","should","be","built","for","all","combinations","of","legal","parameter","values",".","For","illegal","parameter","values",",","a","warning\/error","message","should","be","printed","out","to","warn","the","user","but","the","program","should","not","throw","an","exception",";","We","will","re-use","the","dataset","generation","methods","for","GLM",".","There","will","be","only","one","data","set","for","classification","."]
docstr: test_naivebayes_grid_search_over_params the following:
  a. grab all truely griddable parameters and randomly or manually set the parameter values.
  b. Next, build H2O naivebayes models using grid search.  Count and make sure models
     are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
     values.  We should instead get a warning/error message printed out.
  c. For each model built using grid search, we will extract the parameters used in building
     that model and manually build a H2O naivebayes model.  Training metrics are calculated from the
     gridsearch model and the manually built model.  If their metrics
     differ by too much, print a warning message but don't fail the test.
  d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
     for it as well.  If max_runtime_secs was exceeded, declare test failure.
docstr_tok: ["test_naivebayes_grid_search_over_params","the","following",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","naivebayes","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","naivebayes","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure","."]
code: def test_naivebayes_grid_search_over_params(self):
      """
      test_naivebayes_grid_search_over_params the following:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O naivebayes models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O naivebayes model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
      """
      print(
          "*******************************************************************************************"
      )
      print("test_naivebayes_grid_search_over_params for naivebayes ")
      h2o.cluster_info()
  
      try:
          print(
              "Hyper-parameters used here is {0}".format(
                  self.final_hyper_params
              )
          )
  
          # start grid search
          grid_model = H2OGridSearch(
              H2ONaiveBayesEstimator(nfolds=self.nfolds),
              hyper_params=self.final_hyper_params,
          )
          grid_model.train(
              x=self.x_indices,
              y=self.y_index,
              training_frame=self.training1_data,
          )
  
          self.correct_model_number = len(
              grid_model
          )  # store number of models built
  
          # make sure the correct number of models are built by gridsearch
          if not (
              self.correct_model_number == self.possible_number_models
          ):  # wrong grid model number
              self.test_failed += 1
              print(
                  "test_naivebayes_grid_search_over_params for naivebayes failed: number of models built by "
                  "gridsearch {0} does not equal to all possible combinations of hyper-parameters "
                  "{1}".format(
                      self.correct_model_number,
                      self.possible_number_models,
                  )
              )
          else:
              # add parameters into params_dict.  Use this to manually build model
              params_dict = dict()
              params_dict["nfolds"] = self.nfolds
              total_run_time_limits = (
                  0.0  # calculate upper bound of max_runtime_secs
              )
              true_run_time_limits = 0.0
              manual_run_runtime = 0.0
              gridsearch_runtime = 0.0
  
              # compare performance metric of model built by gridsearch with manually built model
              for each_model in grid_model:
                  params_list = grid_model.get_hyperparams_dict(
                      each_model._id
                  )
                  params_list.update(params_dict)
  
                  model_params = dict()
  
                  # need to taken out max_runtime_secs from model parameters, it is now set in .train()
                  if "max_runtime_secs" in params_list:
                      model_params["max_runtime_secs"] = params_list[
                          "max_runtime_secs"
                      ]
                      max_runtime = params_list["max_runtime_secs"]
                      del params_list["max_runtime_secs"]
                  else:
                      max_runtime = 0
  
                  if "validation_frame" in params_list:
                      model_params["validation_frame"] = params_list[
                          "validation_frame"
                      ]
                      del params_list["validation_frame"]
  
                  if "eps_prob" in params_list:
                      model_params["eps_prob"] = params_list["eps_prob"]
                      del params_list["eps_prob"]
  
                  if "min_prob" in params_list:
                      model_params["min_prob"] = params_list["min_prob"]
                      del params_list["min_prob"]
  
                  # make sure manual model was provided the same max_runtime_secs as the grid model
                  each_model_runtime = pyunit_utils.find_grid_runtime(
                      [each_model]
                  )
                  gridsearch_runtime += each_model_runtime
  
                  manual_model = H2ONaiveBayesEstimator(**params_list)
                  manual_model.train(
                      x=self.x_indices,
                      y=self.y_index,
                      training_frame=self.training1_data,
                      **model_params,
                  )
  
                  # collect the time taken to manually built all models
                  model_runtime = pyunit_utils.find_grid_runtime(
                      [manual_model]
                  )  # time taken to build this model
                  manual_run_runtime += model_runtime
  
                  if max_runtime > 0:
                      # shortest possible time it takes to build this model
                      if max_runtime < self.model_run_time:
                          total_run_time_limits += model_runtime
                      else:
                          total_run_time_limits += max_runtime
  
                  true_run_time_limits += max_runtime
  
                  # compute and compare test metrics between the two models
                  grid_model_metrics = each_model.model_performance(
                      test_data=self.training1_data
                  )._metric_json[self.training_metric]
                  manual_model_metrics = manual_model.model_performance(
                      test_data=self.training1_data
                  )._metric_json[self.training_metric]
  
                  # just compare the mse in this case within tolerance:
                  if not (
                      (type(grid_model_metrics) == str)
                      or (type(manual_model_metrics) == str)
                  ):
                      if (abs(grid_model_metrics) > 0) and (
                          abs(grid_model_metrics - manual_model_metrics)
                          / grid_model_metrics
                          > self.allowed_diff
                      ):
                          print(
                              "test_naivebayes_grid_search_over_params for naivebayes WARNING\ngrid search model "
                              "{0}: {1}, time taken to build (secs): {2}\n and manually built H2O model {3}: {4}, "
                              "time taken to build (secs): {5}\ndiffer too much!"
                              "".format(
                                  self.training_metric,
                                  grid_model_metrics,
                                  each_model_runtime,
                                  self.training_metric,
                                  manual_model_metrics,
                                  model_runtime,
                              )
                          )
  
              print(
                  "Time taken for gridsearch to build all models (sec): {0}\n Time taken to manually build all "
                  "models (sec): {1}, total run time limits (sec): "
                  "{2}".format(
                      gridsearch_runtime,
                      manual_run_runtime,
                      total_run_time_limits,
                  )
              )
              total_run_time_limits = max(
                  total_run_time_limits, true_run_time_limits
              ) * (1 + self.extra_time_fraction)
  
              # make sure the max_runtime_secs is working to restrict model built time
              if not (manual_run_runtime <= total_run_time_limits):
                  self.test_failed += 1
                  print(
                      "test_naivebayes_grid_search_over_params for naivebayes failed: time taken to manually build "
                      "models is {0}.  Maximum allowed time "
                      "is {1}".format(
                          manual_run_runtime, total_run_time_limits
                      )
                  )
  
              if self.test_failed == 0:
                  print(
                      "test_naivebayes_grid_search_over_params for naivebayes has passed!"
                  )
      except Exception as e:
          if self.possible_number_models > 0:
              print(
                  "test_naivebayes_grid_search_over_params for naivebayes failed: exception ({0}) was thrown for "
                  "no reason.".format(e)
              )
              self.test_failed += 1
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/naivebayes/pyunit_naivebayes_gridsearch_over_all_params_large.py
imports: import sys
  import random
  import os
  from builtins import range
  import time
  import json
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.naive_bayes import H2ONaiveBayesEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 5 for calling the Test_naivebayes_grid_search obj's underlying member method code with example usage: obj.test_naivebayes_grid_search_over_params() without return types