class_docstr: This class is created to test the GLM algo with Binomial family.  In this case, the relationship
  between the response Y and predictor vector X is assumed to be
  Prob(Y = 1|X) = exp(W^T * X + E)/(1+exp(W^T * X + E)) where E is unknown Gaussian noise.  We
  generate random data set using the exact formula.  To evaluate the H2O GLM Model, we run the sklearn
  logistic regression with the same data sets and compare the performance of the two.  If they are close
  enough within a certain tolerance, we declare the H2O model working.  When regularization and other
  parameters are enabled, we can evaluate H2O GLM model performance by comparing the logloss/accuracy
  from H2O model and to the H2O model generated without regularization.  As long as they do not deviate
  too much, we consider the H2O model performance satisfactory.
  
  In particular, I have written 8 tests in the hope to exercise as many parameters settings of the GLM
  algo with Binomial distribution as possible.  Tomas has requested 2 tests to be added to test his new
  feature of missing_values_handling with predictors with both categorical/real columns.  Here is a list
  of all tests descriptions:
  
  test1_glm_no_regularization(): sklearn logistic regression model is built.
      H2O GLM is built for Binomial family with the same random data sets.  We observe
      the weights, confusion matrices from the two models.  We compare the logloss, prediction
      accuracy from the two models to determine if H2O GLM model shall pass the test.
  test2_glm_lambda_search(): test lambda search with alpha set to 0.5 per Tomas's
      suggestion.  Make sure logloss and prediction accuracy generated here is comparable in
      value to H2O GLM with no regularization.
  test3_glm_grid_search_over_params(): test grid search over
      various alpha values while lambda is set to be the best value obtained
      from test 2.  Cross validation with k=5 and random assignment is enabled
      as well.  The best model performance hopefully will generate logloss and
      prediction accuracies close to H2O with no regularization in test 1.
  test4_glm_remove_collinear_columns(): test parameter remove_collinear_columns=True
      with lambda set to best lambda from test 2, alpha set to best alpha from Gridsearch
      and solver set to the one which generate the smallest validation logloss.  The same dataset
      is used here except that we randomly choose predictor columns to repeat and scale.
      Make sure logloss and prediction accuracies generated here is comparable in value
      to H2O GLM model with no regularization.
  test5_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      only real value predictors.  The same data sets as before is used.  However, we
      go into the predictor matrix and randomly decide to replace a value with
      nan and create missing values.  Sklearn logistic regression model is built using the
      data set where we have imputed the missing values.  This Sklearn model will be used to
      compare our H2O models with.
  test6_enum_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      mixed predictors (categorical/real value columns).  We first generate a data set that
      contains a random number of columns of categorical and real value columns.  Next, we
      encode the categorical columns.  Then, we generate the random data set using the formula
      as before.  Next, we go into the predictor matrix and randomly
      decide to change a value to be nan and create missing values.  Again, we build a Sklearn
      logistic regression model and compare our H2O model with it.
  test7_missing_enum_values_lambda_search(): Test parameter
      missing_values_handling="MeanImputation" with mixed predictors (categorical/real value columns).
      Test parameter missing_values_handling="MeanImputation" with
      mixed predictors (categorical/real value columns) and setting lambda search to be True.
      We use the same prediction data  with missing values from test6.  Next, we encode the categorical columns using
      true one hot encoding since Lambda-search will be enabled with alpha set to 0.5.  Since the encoding
      is different in this case from test6, we will build a brand new Sklearn logistic regression model and
      compare the best H2O model logloss/prediction accuracy with it.
class_docstr_tok: ["This","class","is","created","to","test","the","GLM","algo","with","Binomial","family",".","In","this","case",",","the","relationship","between","the","response","Y","and","predictor","vector","X","is","assumed","to","be","Prob","(","Y","=","1|X",")","=","exp","(","W^T","*","X","+","E",")","\/","(","1+exp","(","W^T","*","X","+","E",")",")","where","E","is","unknown","Gaussian","noise",".","We","generate","random","data","set","using","the","exact","formula",".","To","evaluate","the","H2O","GLM","Model",",","we","run","the","sklearn","logistic","regression","with","the","same","data","sets","and","compare","the","performance","of","the","two",".","If","they","are","close","enough","within","a","certain","tolerance",",","we","declare","the","H2O","model","working",".","When","regularization","and","other","parameters","are","enabled",",","we","can","evaluate","H2O","GLM","model","performance","by","comparing","the","logloss\/accuracy","from","H2O","model","and","to","the","H2O","model","generated","without","regularization",".","As","long","as","they","do","not","deviate","too","much",",","we","consider","the","H2O","model","performance","satisfactory",".","In","particular",",","I","have","written","8","tests","in","the","hope","to","exercise","as","many","parameters","settings","of","the","GLM","algo","with","Binomial","distribution","as","possible",".","Tomas","has","requested","2","tests","to","be","added","to","test","his","new","feature","of","missing_values_handling","with","predictors","with","both","categorical\/real","columns",".","Here","is","a","list","of","all","tests","descriptions",":","test1_glm_no_regularization","(",")",":","sklearn","logistic","regression","model","is","built",".","H2O","GLM","is","built","for","Binomial","family","with","the","same","random","data","sets",".","We","observe","the","weights",",","confusion","matrices","from","the","two","models",".","We","compare","the","logloss",",","prediction","accuracy","from","the","two","models","to","determine","if","H2O","GLM","model","shall","pass","the","test",".","test2_glm_lambda_search","(",")",":","test","lambda","search","with","alpha","set","to","0.5","per","Tomas's","suggestion",".","Make","sure","logloss","and","prediction","accuracy","generated","here","is","comparable","in","value","to","H2O","GLM","with","no","regularization",".","test3_glm_grid_search_over_params","(",")",":","test","grid","search","over","various","alpha","values","while","lambda","is","set","to","be","the","best","value","obtained","from","test","2",".","Cross","validation","with","k=5","and","random","assignment","is","enabled","as","well",".","The","best","model","performance","hopefully","will","generate","logloss","and","prediction","accuracies","close","to","H2O","with","no","regularization","in","test","1.","test4_glm_remove_collinear_columns","(",")",":","test","parameter","remove_collinear_columns=True","with","lambda","set","to","best","lambda","from","test","2",",","alpha","set","to","best","alpha","from","Gridsearch","and","solver","set","to","the","one","which","generate","the","smallest","validation","logloss",".","The","same","dataset","is","used","here","except","that","we","randomly","choose","predictor","columns","to","repeat","and","scale",".","Make","sure","logloss","and","prediction","accuracies","generated","here","is","comparable","in","value","to","H2O","GLM","model","with","no","regularization",".","test5_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","only","real","value","predictors",".","The","same","data","sets","as","before","is","used",".","However",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","replace","a","value","with","nan","and","create","missing","values",".","Sklearn","logistic","regression","model","is","built","using","the","data","set","where","we","have","imputed","the","missing","values",".","This","Sklearn","model","will","be","used","to","compare","our","H2O","models","with",".","test6_enum_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","We","first","generate","a","data","set","that","contains","a","random","number","of","columns","of","categorical","and","real","value","columns",".","Next",",","we","encode","the","categorical","columns",".","Then",",","we","generate","the","random","data","set","using","the","formula","as","before",".","Next",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Again",",","we","build","a","Sklearn","logistic","regression","model","and","compare","our","H2O","model","with","it",".","test7_missing_enum_values_lambda_search","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")","and","setting","lambda","search","to","be","True",".","We","use","the","same","prediction","data","with","missing","values","from","test6",".","Next",",","we","encode","the","categorical","columns","using","true","one","hot","encoding","since","Lambda-search","will","be","enabled","with","alpha","set","to","0.5",".","Since","the","encoding","is","different","in","this","case","from","test6",",","we","will","build","a","brand","new","Sklearn","logistic","regression","model","and","compare","the","best","H2O","model","logloss\/prediction","accuracy","with","it","."]
docstr: This function performs all initializations necessary:
  1. generates all the random values for our dynamic tests like the Binomial
  noise std, column count and row count for training data set;
  2. generate the training/validation/test data sets with only real values;
  3. insert missing values into training/valid/test data sets.
  4. taken the training/valid/test data sets, duplicate random certain columns,
      each duplicated column is repeated for a random number of times and randomly scaled;
  5. generate the training/validation/test data sets with predictors containing enum
      and real values as well***.
  6. insert missing values into the training/validation/test data sets with predictors
      containing enum and real values as well
  
  *** according to Tomas, when working with mixed predictors (contains both enum/real
  value columns), the encoding used is different when regularization is enabled or disabled.
  When regularization is enabled, true one hot encoding is enabled to encode the enum
  values to binary bits.  When regularization is disabled, a reference level plus one hot encoding
  is enabled when encoding the enum values to binary bits.  One data set is generated
  when we work with mixed predictors.
docstr_tok: ["This","function","performs","all","initializations","necessary",":","1.","generates","all","the","random","values","for","our","dynamic","tests","like","the","Binomial","noise","std",",","column","count","and","row","count","for","training","data","set",";","2.","generate","the","training\/validation\/test","data","sets","with","only","real","values",";","3.","insert","missing","values","into","training\/valid\/test","data","sets",".","4.","taken","the","training\/valid\/test","data","sets",",","duplicate","random","certain","columns",",","each","duplicated","column","is","repeated","for","a","random","number","of","times","and","randomly","scaled",";","5.","generate","the","training\/validation\/test","data","sets","with","predictors","containing","enum","and","real","values","as","well","*","*","*",".","6.","insert","missing","values","into","the","training\/validation\/test","data","sets","with","predictors","containing","enum","and","real","values","as","well","*","*","*","according","to","Tomas",",","when","working","with","mixed","predictors","(","contains","both","enum\/real","value","columns",")",",","the","encoding","used","is","different","when","regularization","is","enabled","or","disabled",".","When","regularization","is","enabled",",","true","one","hot","encoding","is","enabled","to","encode","the","enum","values","to","binary","bits",".","When","regularization","is","disabled",",","a","reference","level","plus","one","hot","encoding","is","enabled","when","encoding","the","enum","values","to","binary","bits",".","One","data","set","is","generated","when","we","work","with","mixed","predictors","."]
code: def setup(self):
      """
      This function performs all initializations necessary:
      1. generates all the random values for our dynamic tests like the Binomial
      noise std, column count and row count for training data set;
      2. generate the training/validation/test data sets with only real values;
      3. insert missing values into training/valid/test data sets.
      4. taken the training/valid/test data sets, duplicate random certain columns,
          each duplicated column is repeated for a random number of times and randomly scaled;
      5. generate the training/validation/test data sets with predictors containing enum
          and real values as well***.
      6. insert missing values into the training/validation/test data sets with predictors
          containing enum and real values as well
  
      *** according to Tomas, when working with mixed predictors (contains both enum/real
      value columns), the encoding used is different when regularization is enabled or disabled.
      When regularization is enabled, true one hot encoding is enabled to encode the enum
      values to binary bits.  When regularization is disabled, a reference level plus one hot encoding
      is enabled when encoding the enum values to binary bits.  One data set is generated
      when we work with mixed predictors.
      """
      # clean out the sandbox directory first
      self.sandbox_dir = pyunit_utils.make_Rsandbox_dir(
          self.current_dir, self.test_name, True
      )
  
      # randomly set Binomial noise standard deviation as a fraction of actual predictor standard deviation
      self.noise_std = random.uniform(
          0,
          math.sqrt(pow((self.max_p_value - self.min_p_value), 2) / 12),
      )
      self.noise_var = self.noise_std * self.noise_std
  
      # randomly determine data set size in terms of column and row counts
      self.train_col_count = random.randint(
          3, self.max_col_count
      )  # account for enum columns later
      self.train_row_count = int(
          round(
              self.train_col_count
              * random.uniform(
                  self.min_col_count_ratio, self.max_col_count_ratio
              )
          )
      )
  
      # #  DEBUGGING setup_data, remember to comment them out once done.
      # self.train_col_count = 3
      # self.train_row_count = 500
      # end DEBUGGING
  
      # randomly set number of enum and real columns in the data set
      self.enum_col = random.randint(1, self.train_col_count - 1)
  
      # randomly set number of levels for each categorical column
      self.enum_level_vec = np.random.random_integers(
          2, self.enum_levels - 1, [self.enum_col, 1]
      )
  
      # generate real value weight vector and training/validation/test data sets for GLM
      pyunit_utils.write_syn_floating_point_dataset_glm(
          self.training_data_file,
          self.validation_data_file,
          self.test_data_file,
          self.weight_data_file,
          self.train_row_count,
          self.train_col_count,
          self.data_type,
          self.max_p_value,
          self.min_p_value,
          self.max_w_value,
          self.min_w_value,
          self.noise_std,
          self.family,
          self.train_row_count,
          self.train_row_count,
          class_number=self.class_number,
          class_method=[
              self.class_method,
              self.class_method,
              self.test_class_method,
          ],
          class_margin=[
              self.margin,
              self.margin,
              self.test_class_margin,
          ],
      )
  
      # randomly generate the duplicated and scaled columns
      (
          self.duplicate_col_indices,
          self.duplicate_col_scales,
      ) = pyunit_utils.random_col_duplication(
          self.train_col_count,
          self.duplicate_threshold,
          self.duplicate_col_counts,
          True,
          self.duplicate_max_scale,
      )
  
      # apply the duplication and scaling to training and test set
      # need to add the response column to the end of duplicated column indices and scale
      dup_col_indices = self.duplicate_col_indices
      dup_col_indices.append(self.train_col_count)
      dup_col_scale = self.duplicate_col_scales
      dup_col_scale.append(1.0)
  
      # print out duplication information for easy debugging
      print("duplication column and duplication scales are: ")
      print(dup_col_indices)
      print(dup_col_scale)
  
      # print out duplication information for easy debugging
      print("duplication column and duplication scales are: ")
      print(dup_col_indices)
      print(dup_col_scale)
  
      pyunit_utils.duplicate_scale_cols(
          dup_col_indices,
          dup_col_scale,
          self.training_data_file,
          self.training_data_file_duplicate,
      )
      pyunit_utils.duplicate_scale_cols(
          dup_col_indices,
          dup_col_scale,
          self.test_data_file,
          self.test_data_file_duplicate,
      )
  
      # insert NAs into training/test data sets
      pyunit_utils.insert_nan_in_data(
          self.training_data_file,
          self.training_data_file_nans,
          self.nan_fraction,
      )
      pyunit_utils.insert_nan_in_data(
          self.test_data_file,
          self.test_data_file_nans,
          self.nan_fraction,
      )
  
      # generate data sets with enum as well as real values
      pyunit_utils.write_syn_mixed_dataset_glm(
          self.training_data_file_enum,
          self.training_data_file_enum_true_one_hot,
          self.validation_data_file_enum,
          self.validation_data_file_enum_true_one_hot,
          self.test_data_file_enum,
          self.test_data_file_enum_true_one_hot,
          self.weight_data_file_enum,
          self.train_row_count,
          self.train_col_count,
          self.max_p_value,
          self.min_p_value,
          self.max_w_value,
          self.min_w_value,
          self.noise_std,
          self.family,
          self.train_row_count,
          self.train_row_count,
          self.enum_col,
          self.enum_level_vec,
          class_number=self.class_number,
          class_method=[
              self.class_method,
              self.class_method,
              self.test_class_method,
          ],
          class_margin=[
              self.margin,
              self.margin,
              self.test_class_margin,
          ],
      )
  
      # insert NAs into data set with categorical columns
      pyunit_utils.insert_nan_in_data(
          self.training_data_file_enum,
          self.training_data_file_enum_nans,
          self.nan_fraction,
      )
      pyunit_utils.insert_nan_in_data(
          self.validation_data_file_enum,
          self.validation_data_file_enum_nans,
          self.nan_fraction,
      )
      pyunit_utils.insert_nan_in_data(
          self.test_data_file_enum,
          self.test_data_file_enum_nans,
          self.nan_fraction,
      )
  
      pyunit_utils.insert_nan_in_data(
          self.training_data_file_enum_true_one_hot,
          self.training_data_file_enum_nans_true_one_hot,
          self.nan_fraction,
      )
      pyunit_utils.insert_nan_in_data(
          self.validation_data_file_enum_true_one_hot,
          self.validation_data_file_enum_nans_true_one_hot,
          self.nan_fraction,
      )
      pyunit_utils.insert_nan_in_data(
          self.test_data_file_enum_true_one_hot,
          self.test_data_file_enum_nans_true_one_hot,
          self.nan_fraction,
      )
  
      # only preload data sets that will be used for multiple tests and change the response to enums
      self.training_data = h2o.import_file(
          pyunit_utils.locate(self.training_data_file)
      )
  
      # set indices for response and predictor columns in data set for H2O GLM model to use
      self.y_index = self.training_data.ncol - 1
      self.x_indices = list(range(self.y_index))
  
      # added the round() so that this will work on win8.
      self.training_data[self.y_index] = (
          self.training_data[self.y_index].round().asfactor()
      )
  
      # check to make sure all response classes are represented, otherwise, quit
      if (
          self.training_data[self.y_index].nlevels()[0]
          < self.class_number
      ):
          print(
              "Response classes are not represented in training dataset."
          )
          sys.exit(0)
  
      self.valid_data = h2o.import_file(
          pyunit_utils.locate(self.validation_data_file)
      )
      self.valid_data[self.y_index] = (
          self.valid_data[self.y_index].round().asfactor()
      )
      self.test_data = h2o.import_file(
          pyunit_utils.locate(self.test_data_file)
      )
      self.test_data[self.y_index] = (
          self.test_data[self.y_index].round().asfactor()
      )
  
      # make a bigger training set for grid search by combining data from validation data set
      self.training_data_grid = self.training_data.rbind(
          self.valid_data
      )
  
      # setup_data sklearn class weight of all ones.  Used only to make sure sklearn know the correct number of classes
      for ind in range(self.class_number):
          self.sklearn_class_weight[ind] = 1.0
  
      # save the training data files just in case the code crashed.
      pyunit_utils.remove_csv_files(
          self.current_dir,
          ".csv",
          action="copy",
          new_dir_path=self.sandbox_dir,
      )
  
imports: import sys
  import random
  import os
  import math
  import numpy
  import h2o
  import time
  from builtins import range
  from tests import pyunit_utils
  from h2o.estimators.glm import H2OGeneralizedLinearEstimator
  from h2o.grid.grid_search import H2OGridSearch
  from scipy import stats
  from sklearn.linear_model import LogisticRegression
  from sklearn import metrics
label_desc: Use image node_id 2 for calling the TestGLMBinomial obj's underlying member method code with example usage: obj.setup() without return types