class_docstr: PUBDEV-1843: Grid testing.  Subtask 2.
  
  This class is created to test the gridsearch for kmeans algo and make sure it runs.  Only one test is performed
  here.
  
  Test Descriptions:
      a. grab all truely griddable parameters and randomly or manually set the parameter values.
      b. Next, build H2O kmeans models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      c. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O kmeans model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      d. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
  
  Note that for hyper-parameters containing all legal parameter names and parameter value lists with legal
  and illegal values, grid-models should be built for all combinations of legal parameter values.  For
  illegal parameter values, a warning/error message should be printed out to warn the user but the
  program should not throw an exception.
class_docstr_tok: ["PUBDEV-1843",":","Grid","testing",".","Subtask","2",".","This","class","is","created","to","test","the","gridsearch","for","kmeans","algo","and","make","sure","it","runs",".","Only","one","test","is","performed","here",".","Test","Descriptions",":","a.","grab","all","truely","griddable","parameters","and","randomly","or","manually","set","the","parameter","values",".","b",".","Next",",","build","H2O","kmeans","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","c.","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","kmeans","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","d.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure",".","Note","that","for","hyper-parameters","containing","all","legal","parameter","names","and","parameter","value","lists","with","legal","and","illegal","values",",","grid-models","should","be","built","for","all","combinations","of","legal","parameter","values",".","For","illegal","parameter","values",",","a","warning\/error","message","should","be","printed","out","to","warn","the","user","but","the","program","should","not","throw","an","exception","."]
docstr: test_kmeans_grid_search_over_params performs the following:
  a. build H2O kmeans models using grid search.  Count and make sure models
     are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
     values.  We should instead get a warning/error message printed out.
  b. For each model built using grid search, we will extract the parameters used in building
     that model and manually build a H2O kmeans model.  Training metrics are calculated from the
     gridsearch model and the manually built model.  If their metrics
     differ by too much, print a warning message but don't fail the test.
  c. we will check and make sure the models are built within the max_runtime_secs time limit that was set
     for it as well.  If max_runtime_secs was exceeded, declare test failure.
docstr_tok: ["test_kmeans_grid_search_over_params","performs","the","following",":","a.","build","H2O","kmeans","models","using","grid","search",".","Count","and","make","sure","models","are","only","built","for","hyper-parameters","set","to","legal","values",".","No","model","is","built","for","bad","hyper-parameters","values",".","We","should","instead","get","a","warning\/error","message","printed","out",".","b",".","For","each","model","built","using","grid","search",",","we","will","extract","the","parameters","used","in","building","that","model","and","manually","build","a","H2O","kmeans","model",".","Training","metrics","are","calculated","from","the","gridsearch","model","and","the","manually","built","model",".","If","their","metrics","differ","by","too","much",",","print","a","warning","message","but","do","n't","fail","the","test",".","c.","we","will","check","and","make","sure","the","models","are","built","within","the","max_runtime_secs","time","limit","that","was","set","for","it","as","well",".","If","max_runtime_secs","was","exceeded",",","declare","test","failure","."]
code: def test_kmeans_grid_search_over_params(self):
      """
      test_kmeans_grid_search_over_params performs the following:
      a. build H2O kmeans models using grid search.  Count and make sure models
         are only built for hyper-parameters set to legal values.  No model is built for bad hyper-parameters
         values.  We should instead get a warning/error message printed out.
      b. For each model built using grid search, we will extract the parameters used in building
         that model and manually build a H2O kmeans model.  Training metrics are calculated from the
         gridsearch model and the manually built model.  If their metrics
         differ by too much, print a warning message but don't fail the test.
      c. we will check and make sure the models are built within the max_runtime_secs time limit that was set
         for it as well.  If max_runtime_secs was exceeded, declare test failure.
      """
  
      print(
          "*******************************************************************************************"
      )
      print("test_kmeans_grid_search_over_params for kmeans ")
      h2o.cluster_info()
  
      try:
          print(
              "Hyper-parameters used here is {0}".format(
                  self.final_hyper_params
              )
          )
  
          # start grid search
          grid_model = H2OGridSearch(
              H2OKMeansEstimator(), hyper_params=self.final_hyper_params
          )
          grid_model.train(
              x=self.x_indices, training_frame=self.training1_data
          )
  
          self.correct_model_number = len(
              grid_model
          )  # store number of models built
  
          # make sure the correct number of models are built by gridsearch
          if (
              self.correct_model_number - self.possible_number_models
          ) > 0.9:  # wrong grid model number
              self.test_failed += 1
              print(
                  "test_kmeans_grid_search_over_params for kmeans failed: number of models built by gridsearch: {0}"
                  " does not equal to all possible combinations of hyper-parameters: "
                  "{1}".format(
                      self.correct_model_number,
                      self.possible_number_models,
                  )
              )
          else:
              # add parameters into params_dict.  Use this to manually build model
              params_dict = dict()
              total_run_time_limits = (
                  0.0  # calculate upper bound of max_runtime_secs
              )
              true_run_time_limits = 0.0
              manual_run_runtime = 0.0
  
              # compare training metric performance of model built by gridsearch with manually built model
              for each_model in grid_model:
                  params_list = grid_model.get_hyperparams_dict(
                      each_model._id
                  )
                  params_list.update(params_dict)
  
                  model_params = dict()
                  num_iter = 0
  
                  # need to taken out max_runtime_secs from model parameters, it is now set in .train()
                  if "max_runtime_secs" in params_list:
                      model_params["max_runtime_secs"] = params_list[
                          "max_runtime_secs"
                      ]
                      max_runtime = params_list["max_runtime_secs"]
                      del params_list["max_runtime_secs"]
                  else:
                      max_runtime = 0
  
                  # make sure manual model was provided the same max_runtime_secs as the grid model
                  each_model_runtime = pyunit_utils.find_grid_runtime(
                      [each_model]
                  )
  
                  manual_model = H2OKMeansEstimator(**params_list)
                  manual_model.train(
                      x=self.x_indices,
                      training_frame=self.training1_data,
                      **model_params,
                  )
  
                  # collect the time taken to manually built all models
                  model_runtime = pyunit_utils.find_grid_runtime(
                      [manual_model]
                  )  # time taken to build this model
                  manual_run_runtime += model_runtime
  
                  summary_list = manual_model._model_json["output"][
                      "model_summary"
                  ]
                  if summary_list is not None:
                      num_iter = summary_list["number_of_iterations"][0]
  
                      # compute and compare test metrics between the two models
                      if not (
                          each_model._model_json["output"][
                              "model_summary"
                          ]
                          is None
                      ):
                          grid_model_metrics = each_model._model_json[
                              "output"
                          ]["model_summary"]["total_sum_of_squares"][0]
                          manual_model_metrics = (
                              manual_model._model_json["output"][
                                  "model_summary"
                              ]["total_sum_of_squares"][0]
                          )
  
                          # just compare the training metrics in this case within tolerance:
                          if not (
                              (type(grid_model_metrics) == str)
                              or (type(manual_model_metrics) == str)
                          ):
                              if (abs(grid_model_metrics) > 0) and (
                                  abs(
                                      grid_model_metrics
                                      - manual_model_metrics
                                  )
                                  / grid_model_metrics
                                  > self.allowed_diff
                              ):
                                  print(
                                      "test_kmeans_grid_search_over_params for kmeans warning: grid search model "
                                      "metric ({0}) and manually built H2O model metric ({1}) differ too much"
                                      "!".format(
                                          grid_model_metrics,
                                          manual_model_metrics,
                                      )
                                  )
  
                  if max_runtime > 0:
                      # collect allowed max_runtime_secs info
                      if (max_runtime < self.min_runtime_per_iter) or (
                          num_iter <= 1
                      ):
                          total_run_time_limits += model_runtime
                      else:
                          total_run_time_limits += max_runtime
  
                  true_run_time_limits += max_runtime
  
              total_run_time_limits = max(
                  total_run_time_limits, true_run_time_limits
              ) * (1 + self.extra_time_fraction)
  
              # make sure the max_runtime_secs is working to restrict model built time
              if not (manual_run_runtime <= total_run_time_limits):
                  self.test_failed += 1
                  print(
                      "test_kmeans_grid_search_over_params for kmeans failed: time taken to manually build models"
                      " is {0}.  Maximum allowed time is {1}".format(
                          manual_run_runtime, total_run_time_limits
                      )
                  )
              else:
                  print(
                      "time taken to manually build all models is {0}. Maximum allowed time is "
                      "{1}".format(
                          manual_run_runtime, total_run_time_limits
                      )
                  )
  
              if self.test_failed == 0:
                  print(
                      "test_kmeans_grid_search_over_params for kmeans has passed!"
                  )
      except Exception as e:
          if self.possible_number_models > 0:
              print(
                  "test_kmeans_grid_search_over_params for kmeans failed: exception ({0}) was thrown for no reason.".format(
                      e
                  )
              )
              self.test_failed += 1
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/kmeans/pyunit_kmeans_gridsearch_over_all_params_large.py
imports: import sys
  import random
  import os
  from builtins import range
  import time
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.kmeans import H2OKMeansEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 4 for calling the Test_kmeans_grid_search obj's underlying member method code with example usage: obj.test_kmeans_grid_search_over_params() without return types