class_docstr: This class is created to test the three stopping conditions for randomized gridsearch using
  GLM Binomial family.  The three stopping conditions are :
  
  1. max_runtime_secs:
  2. max_models:
  3. metrics.  We will be picking 2 stopping metrics to test this stopping condition with.  One metric
  will be optimized if it increases and the other one should be optimized if it decreases.
  
  I have written 4 tests:
  1. test1_glm_random_grid_search_model_number: this test will not put any stopping conditions
  on randomized search.  The purpose here is to make sure that randomized search will give us all possible
  hyper-parameter combinations.
  2. test2_glm_random_grid_search_max_model: this test the stopping condition of setting the max_model in
  search criteria;
  3. test3_glm_random_grid_search_max_runtime_secs: this test the stopping condition max_runtime_secs
  in search criteria;
  4. test4_glm_random_grid_search_metric: this test the stopping condition of using a metric which can be
  increasing or decreasing.
class_docstr_tok: ["This","class","is","created","to","test","the","three","stopping","conditions","for","randomized","gridsearch","using","GLM","Binomial","family",".","The","three","stopping","conditions","are",":","1.","max_runtime_secs",":","2.","max_models",":","3.","metrics",".","We","will","be","picking","2","stopping","metrics","to","test","this","stopping","condition","with",".","One","metric","will","be","optimized","if","it","increases","and","the","other","one","should","be","optimized","if","it","decreases",".","I","have","written","4","tests",":","1.","test1_glm_random_grid_search_model_number",":","this","test","will","not","put","any","stopping","conditions","on","randomized","search",".","The","purpose","here","is","to","make","sure","that","randomized","search","will","give","us","all","possible","hyper-parameter","combinations",".","2.","test2_glm_random_grid_search_max_model",":","this","test","the","stopping","condition","of","setting","the","max_model","in","search","criteria",";","3.","test3_glm_random_grid_search_max_runtime_secs",":","this","test","the","stopping","condition","max_runtime_secs","in","search","criteria",";","4.","test4_glm_random_grid_search_metric",":","this","test","the","stopping","condition","of","using","a","metric","which","can","be","increasing","or","decreasing","."]
docstr: This function will test the stopping criteria max_runtime_secs.  For each model built, the field
  run_time actually denote the time in ms used to build the model.  We will add up the run_time from all
  models and check against the stopping criteria max_runtime_secs.  Since each model will check its run time
  differently, there is some inaccuracies in the actual run time.  For example, if we give a model 10 ms to
  build.  The GLM may check and see if it has used up all the time for every 10 epochs that it has run.  On
  the other hand, deeplearning may check the time it has spent after every epoch of training.
  
  If we are able to restrict the runtime to not exceed the specified max_runtime_secs by a certain
  percentage, we will consider the test a success.
  
  :return: None
docstr_tok: ["This","function","will","test","the","stopping","criteria","max_runtime_secs",".","For","each","model","built",",","the","field","run_time","actually","denote","the","time","in","ms","used","to","build","the","model",".","We","will","add","up","the","run_time","from","all","models","and","check","against","the","stopping","criteria","max_runtime_secs",".","Since","each","model","will","check","its","run","time","differently",",","there","is","some","inaccuracies","in","the","actual","run","time",".","For","example",",","if","we","give","a","model","10","ms","to","build",".","The","GLM","may","check","and","see","if","it","has","used","up","all","the","time","for","every","10","epochs","that","it","has","run",".","On","the","other","hand",",","deeplearning","may","check","the","time","it","has","spent","after","every","epoch","of","training",".","If","we","are","able","to","restrict","the","runtime","to","not","exceed","the","specified","max_runtime_secs","by","a","certain","percentage",",","we","will","consider","the","test","a","success",".",":","return",":","None"]
code: def test3_glm_random_grid_search_max_runtime_secs(self):
      """
      This function will test the stopping criteria max_runtime_secs.  For each model built, the field
      run_time actually denote the time in ms used to build the model.  We will add up the run_time from all
      models and check against the stopping criteria max_runtime_secs.  Since each model will check its run time
      differently, there is some inaccuracies in the actual run time.  For example, if we give a model 10 ms to
      build.  The GLM may check and see if it has used up all the time for every 10 epochs that it has run.  On
      the other hand, deeplearning may check the time it has spent after every epoch of training.
  
      If we are able to restrict the runtime to not exceed the specified max_runtime_secs by a certain
      percentage, we will consider the test a success.
  
      :return: None
      """
      print(
          "*******************************************************************************************"
      )
      print(
          "test3_glm_random_grid_search_max_runtime_secs for GLM "
          + self.family
      )
      h2o.cluster_info()
  
      if "max_runtime_secs" in list(self.hyper_params):
          del self.hyper_params["max_runtime_secs"]
          # number of possible models being built:
          self.possible_number_models = pyunit_utils.count_models(
              self.hyper_params
          )
  
      # setup_data our stopping condition here
      max_run_time_secs = random.uniform(
          self.one_model_time,
          self.allowed_scaled_time * self.max_grid_runtime,
      )
  
      search_criteria = {
          "strategy": "RandomDiscrete",
          "max_runtime_secs": max_run_time_secs,
          "seed": round(time.time()),
      }
      # search_criteria = {'strategy': 'RandomDiscrete', 'max_runtime_secs': 1/1e8}
  
      print(
          "GLM Gaussian grid search_criteria: {0}".format(
              search_criteria
          )
      )
  
      # fire off random grid-search
      grid_model = H2OGridSearch(
          H2OGeneralizedLinearEstimator(
              family=self.family, nfolds=self.nfolds
          ),
          hyper_params=self.hyper_params,
          search_criteria=search_criteria,
      )
      grid_model.train(
          x=self.x_indices,
          y=self.y_index,
          training_frame=self.training1_data,
      )
  
      actual_run_time_secs = pyunit_utils.find_grid_runtime(grid_model)
  
      print(
          "Maximum time limit is {0}.  Time taken to build all model is "
          "{1}".format(
              search_criteria["max_runtime_secs"], actual_run_time_secs
          )
      )
  
      print(
          "Maximum model number is {0}.  Actual number of models built is {1}".format(
              self.possible_number_models, len(grid_model)
          )
      )
  
      if actual_run_time_secs <= search_criteria["max_runtime_secs"] * (
          1 + self.allowed_diff
      ):
          print(
              "test3_glm_random_grid_search_max_runtime_secs: passed!"
          )
  
          if (
              len(grid_model) > self.possible_number_models
          ):  # generate too many models, something is wrong
              self.test_failed += 1
              self.test_failed_array[self.test_num] = 1
              print(
                  "test3_glm_random_grid_search_max_runtime_secs: failed.  Generated {0} models "
                  " which exceeds maximum possible model number {1}".format(
                      len(grid_model), self.possible_number_models
                  )
              )
      elif len(grid_model) == 1:  # will always generate 1 model
          print(
              "test3_glm_random_grid_search_max_runtime_secs: passed!"
          )
      else:
          self.test_failed += 1
          self.test_failed_array[self.test_num] = 1
          print(
              "test3_glm_random_grid_search_max_runtime_secs: failed.  Model takes time {0}"
              " seconds which exceeds allowed time {1}".format(
                  actual_run_time_secs,
                  max_run_time_secs * (1 + self.allowed_diff),
              )
          )
      self.test_num += 1
      sys.stdout.flush()
  
file_path: h2o-3/h2o-py/dynamic_tests/testdir_algos/glm/pyunit_glm_gaussian_gridsearch_randomdiscrete_large.py
imports: import sys
  import random
  import os
  from builtins import range
  import time
  import json
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.glm import H2OGeneralizedLinearEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 7 for calling the Test_glm_random_grid_search obj's underlying member method code with example usage: obj.test3_glm_random_grid_search_max_runtime_secs() without return types