class_docstr: This class is created to test the GLM algo with Multinomial family per request from Tomas as this
  is a new feature for the GLM alo.  In this case, the relationship
  between the response Y (K possible classes) and predictor vector X is assumed to be
  Prob(Y = c|X) = exp(Wc^T * X + E)/(sum k=0 to K-1 (ep(Wk^T *X+E)) where E is
  unknown Gaussian noise.  We generate random data sets using the exact formula.  To evaluate the H2O GLM Model,
  we run the sklearn logistic regression with the same data sets and compare the performance of the two.  If they
  are close enough within a certain tolerance, we declare the H2O model working.  When regularization and other
  parameters are enabled, we can evaluate H2O GLM model performance by comparing the logloss/accuracy
  from H2O model and to the H2O model generated without regularization.  As long as they do not deviate
  too much, we consider the H2O model performance satisfactory.
  
  In particular, I have written 7 tests in the hope to exercise
  as many parameters settings of the GLM algo with Multinomial distribution as possible.  Tomas
  has requested 2 tests to be added to test his new feature of missing_values_handling with
  predictors containing both categorical/real columns.  Here is a list of all tests:
  
  test1_glm_no_regularization(): sklearn logistic regression model is built.
      H2O GLM is built for Multinomial family with the same random data sets.  We observe
      the weights, confusion matrices from the two models.  We compare the logloss, prediction
      accuracy from the two models to determine if H2O GLM model shall pass the test.
  test2_glm_lambda_search(): test lambda search with alpha set to 0.5 per Tomas's
      suggestion.  Make sure loglss and prediction accuracy generated here is comparable in
      value to H2O GLM model with no regularization.
  test3_glm_grid_search_over_params(): test grid search over
      various alpha values while lambda is set to be the best value obtained
      from test 2.  Cross validation with k=5 and random assignment is enabled
      as well.  The best model performance hopefully will generate logloss and
      prediction accuracies close to H2O model with no regularization in test 1.
  test5_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      only real value predictors.  The same data sets as before is used.  However, we
      go into the predictor matrix and randomly decide to replace a value with
      nan and create missing values.  Sklearn logistic regression model is built using the
      data set where we have imputed the missing values.  This Sklearn model will be used to
      compare our H2O models with.
  test6_enum_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      mixed predictors (categorical/real value columns).  We first generate a data set that
      contains a random number of columns of categorical and real value columns.  Next, we
      encode the categorical columns.  Then, we generate the random data set using the formula
      as before.  Next, we go into the predictor matrix and randomly
      decide to change a value to be nan and create missing values.  Again, we build a Sklearn
      logistic regression and compare our H2O model with it.
  test7_missing_enum_values_lambda_search(): Test parameter
      missing_values_handling="MeanImputation" with mixed predictors (categorical/real value columns).
      and setting lambda search to be True.  We use the same predictors from test6.  Next, we encode
      the categorical columns using true one hot encoding since Lambda-search will be enabled with
      alpha set to 0.5.  Since the encoding is different in this case from test6, we will build a brand
      new Sklearn logistic regression model and compare the best H2O model logloss/prediction accuracy with it.
class_docstr_tok: ["This","class","is","created","to","test","the","GLM","algo","with","Multinomial","family","per","request","from","Tomas","as","this","is","a","new","feature","for","the","GLM","alo",".","In","this","case",",","the","relationship","between","the","response","Y","(","K","possible","classes",")","and","predictor","vector","X","is","assumed","to","be","Prob","(","Y","=","c|X",")","=","exp","(","Wc^T","*","X","+","E",")","\/","(","sum","k=0","to","K-1","(","ep","(","Wk^T","*","X+E",")",")","where","E","is","unknown","Gaussian","noise",".","We","generate","random","data","sets","using","the","exact","formula",".","To","evaluate","the","H2O","GLM","Model",",","we","run","the","sklearn","logistic","regression","with","the","same","data","sets","and","compare","the","performance","of","the","two",".","If","they","are","close","enough","within","a","certain","tolerance",",","we","declare","the","H2O","model","working",".","When","regularization","and","other","parameters","are","enabled",",","we","can","evaluate","H2O","GLM","model","performance","by","comparing","the","logloss\/accuracy","from","H2O","model","and","to","the","H2O","model","generated","without","regularization",".","As","long","as","they","do","not","deviate","too","much",",","we","consider","the","H2O","model","performance","satisfactory",".","In","particular",",","I","have","written","7","tests","in","the","hope","to","exercise","as","many","parameters","settings","of","the","GLM","algo","with","Multinomial","distribution","as","possible",".","Tomas","has","requested","2","tests","to","be","added","to","test","his","new","feature","of","missing_values_handling","with","predictors","containing","both","categorical\/real","columns",".","Here","is","a","list","of","all","tests",":","test1_glm_no_regularization","(",")",":","sklearn","logistic","regression","model","is","built",".","H2O","GLM","is","built","for","Multinomial","family","with","the","same","random","data","sets",".","We","observe","the","weights",",","confusion","matrices","from","the","two","models",".","We","compare","the","logloss",",","prediction","accuracy","from","the","two","models","to","determine","if","H2O","GLM","model","shall","pass","the","test",".","test2_glm_lambda_search","(",")",":","test","lambda","search","with","alpha","set","to","0.5","per","Tomas's","suggestion",".","Make","sure","loglss","and","prediction","accuracy","generated","here","is","comparable","in","value","to","H2O","GLM","model","with","no","regularization",".","test3_glm_grid_search_over_params","(",")",":","test","grid","search","over","various","alpha","values","while","lambda","is","set","to","be","the","best","value","obtained","from","test","2",".","Cross","validation","with","k=5","and","random","assignment","is","enabled","as","well",".","The","best","model","performance","hopefully","will","generate","logloss","and","prediction","accuracies","close","to","H2O","model","with","no","regularization","in","test","1.","test5_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","only","real","value","predictors",".","The","same","data","sets","as","before","is","used",".","However",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","replace","a","value","with","nan","and","create","missing","values",".","Sklearn","logistic","regression","model","is","built","using","the","data","set","where","we","have","imputed","the","missing","values",".","This","Sklearn","model","will","be","used","to","compare","our","H2O","models","with",".","test6_enum_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","We","first","generate","a","data","set","that","contains","a","random","number","of","columns","of","categorical","and","real","value","columns",".","Next",",","we","encode","the","categorical","columns",".","Then",",","we","generate","the","random","data","set","using","the","formula","as","before",".","Next",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Again",",","we","build","a","Sklearn","logistic","regression","and","compare","our","H2O","model","with","it",".","test7_missing_enum_values_lambda_search","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","and","setting","lambda","search","to","be","True",".","We","use","the","same","predictors","from","test6",".","Next",",","we","encode","the","categorical","columns","using","true","one","hot","encoding","since","Lambda-search","will","be","enabled","with","alpha","set","to","0.5",".","Since","the","encoding","is","different","in","this","case","from","test6",",","we","will","build","a","brand","new","Sklearn","logistic","regression","model","and","compare","the","best","H2O","model","logloss\/prediction","accuracy","with","it","."]
docstr: Test parameter missing_values_handling="MeanImputation" with mixed predictors (categorical/real value columns).
  We first generate a data set that
  contains a random number of columns of categorical and real value columns.  Next, we
  encode the categorical columns.  Then, we generate the random data set using the formula
  as before.  Next, we go into the predictor matrix and randomly
  decide to change a value to be nan and create missing values.  Lambda-search will be
  enabled with alpha set to 0.5.  Since the encoding is different in this case
  than in test6, we build a Sklearn model and compare the best H2O
  model MSEs with theoretical calculations and hope that they are close.
docstr_tok: ["Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","We","first","generate","a","data","set","that","contains","a","random","number","of","columns","of","categorical","and","real","value","columns",".","Next",",","we","encode","the","categorical","columns",".","Then",",","we","generate","the","random","data","set","using","the","formula","as","before",".","Next",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Lambda-search","will","be","enabled","with","alpha","set","to","0.5",".","Since","the","encoding","is","different","in","this","case","than","in","test6",",","we","build","a","Sklearn","model","and","compare","the","best","H2O","model","MSEs","with","theoretical","calculations","and","hope","that","they","are","close","."]
code: def test7_missing_enum_values_lambda_search(self):
      """
      Test parameter missing_values_handling="MeanImputation" with mixed predictors (categorical/real value columns).
      We first generate a data set that
      contains a random number of columns of categorical and real value columns.  Next, we
      encode the categorical columns.  Then, we generate the random data set using the formula
      as before.  Next, we go into the predictor matrix and randomly
      decide to change a value to be nan and create missing values.  Lambda-search will be
      enabled with alpha set to 0.5.  Since the encoding is different in this case
      than in test6, we build a Sklearn model and compare the best H2O
      model MSEs with theoretical calculations and hope that they are close.
      """
  
      # perform lambda_search, regularization and one hot encoding.
      print(
          "*******************************************************************************************"
      )
      print(
          "Test7: test the GLM with imputation of missing enum/real values under lambda search."
      )
      h2o.cluster_info()
  
      # training result from python sklearn
      (
          p_weights,
          p_logloss_train,
          p_cm_train,
          p_accuracy_training,
          p_logloss_test,
          p_cm_test,
          p_accuracy_test,
      ) = self.sklearn_multinomial_result(
          self.training_data_file_enum_nans_true_one_hot,
          self.test_data_file_enum_nans_true_one_hot,
          True,
          True,
          validation_data_file=self.validation_data_file_enum_nans_true_one_hot,
      )
  
      # import training set and test set with missing values and true one hot encoding
      training_data = h2o.import_file(
          pyunit_utils.locate(
              self.training_data_file_enum_nans_true_one_hot
          )
      )
      validation_data = h2o.import_file(
          pyunit_utils.locate(
              self.validation_data_file_enum_nans_true_one_hot
          )
      )
      test_data = h2o.import_file(
          pyunit_utils.locate(
              self.test_data_file_enum_nans_true_one_hot
          )
      )
  
      # change the categorical data using .asfactor()
      for ind in range(self.enum_col):
          training_data[ind] = training_data[ind].round().asfactor()
          validation_data[ind] = validation_data[ind].round().asfactor()
          test_data[ind] = test_data[ind].round().asfactor()
  
      num_col = training_data.ncol
      y_index = num_col - 1
      x_indices = list(range(y_index))
  
      # change response column to be categorical
      training_data[y_index] = training_data[y_index].round().asfactor()
  
      # check to make sure all response classes are represented, otherwise, quit
      if training_data[y_index].nlevels()[0] < self.class_number:
          print(
              "Response classes are not represented in training dataset."
          )
          sys.exit(0)
  
      validation_data[y_index] = (
          validation_data[y_index].round().asfactor()
      )
      test_data[y_index] = test_data[y_index].round().asfactor()
  
      # train H2O model
      model_h2o_0p5 = H2OGeneralizedLinearEstimator(
          family=self.family,
          lambda_search=True,
          alpha=0.5,
          lambda_min_ratio=1e-20,
          missing_values_handling="MeanImputation",
      )
      model_h2o_0p5.train(
          x=x_indices,
          y=y_index,
          training_frame=training_data,
          validation_frame=validation_data,
      )
  
      h2o_model_0p5_test_metrics = model_h2o_0p5.model_performance(
          test_data=test_data
      )
  
      num_test_failed = self.test_failed
  
      # print out comparison results for our H2O GLM
      self.test_failed = pyunit_utils.extract_comparison_attributes_and_print_multinomial(
          model_h2o_0p5,
          h2o_model_0p5_test_metrics,
          self.family,
          "\nTest7 Done!",
          compare_att_str=[
              "\nComparing intercept and " "weights ....",
              "\nComparing logloss from " "training data set ....",
              "\nComparing logloss from test " "data set ....",
              "\nComparing confusion matrices from"
              "training data set ....",
              "\nComparing confusion matrices from" "test data set ...",
              "\nComparing accuracy from training" " data set ....",
              "\nComparing accuracy from test " "data set ....",
          ],
          h2o_att_str=[
              "H2O with enum/real values, "
              "lamba search and missing values"
              " intercept and weights: \n",
              "H2O with enum/real values, lamba"
              " search and missing values "
              "logloss from training data set: ",
              "H2O with enum/real values, lamba"
              " search and missing values "
              "logloss from test data set",
              "H2O with enum/real values, "
              "lamba search and missing values"
              " confusion matrix from training"
              " data set: \n",
              "H2O with enum/real values, "
              "lamba search and missing values"
              " confusion matrix from test "
              "data set: \n",
              "H2O with enum/real values, "
              "lamba search and missing values"
              " accuracy from training "
              "data set: ",
              "H2O with enum/real values,"
              " lamba search and missing values"
              " accuracy from test data set: ",
          ],
          template_att_str=[
              "Sklearn with enum/real values,"
              " missing values"
              " intercept and weights: \n",
              "Sklearn with enum/real values,"
              " missing values"
              " logloss from training "
              "data set: ",
              "Sklearn with enum/real values, "
              "missing values"
              " logloss from test data set: ",
              "Sklearn with enum/real values, "
              "missing values"
              " confusion matrix from "
              "training data set: \n",
              "Sklearn with enum/real values, "
              "missing values"
              " confusion matrix from test"
              " data set: \n",
              "Sklearn with enum/real values, "
              "missing values"
              " accuracy from training"
              " data set: ",
              "Sklearn with enum/real values, "
              "missing values"
              " accuracy from test data set: ",
          ],
          att_str_fail=[
              "Intercept and weights are not" " equal!",
              "Logloss from training data " "set differ too much!",
              "Logloss from test data set " "differ too much!",
              "",
              "",
              "Accuracies from training data" " set differ too much!",
              "Accuracies from test data set " "differ too much!",
          ],
          att_str_success=[
              "Intercept and weights are close" " enough!",
              "Logloss from training data set" " are close enough!",
              "Logloss from test data set are" " close enough!",
              "",
              "",
              "Accuracies from training data" " set are close enough!",
              "Accuracies from test data set " "are close enough!",
          ],
          can_be_better_than_template=[
              True,
              True,
              True,
              True,
              True,
              True,
              True,
          ],
          just_print=[True, True, True, True, True, True, False],
          failed_test_number=self.test_failed,
          template_params=[
              p_weights,
              p_logloss_train,
              p_cm_train,
              p_accuracy_training,
              p_logloss_test,
              p_cm_test,
              p_accuracy_test,
          ],
          ignored_eps=self.ignored_eps,
          allowed_diff=self.allowed_diff,
      )
  
      # print out test results and update test_failed_array status to reflect if tests have failed
      self.test_failed_array[
          self.test_num
      ] += pyunit_utils.show_test_results(
          "test7_missing_enum_values_lambda_search",
          num_test_failed,
          self.test_failed,
      )
      self.test_num += 1
  
imports: import sys
  import os
  import random
  import math
  import numpy
  import time
  import h2o
  from builtins import range
  from tests import pyunit_utils
  from h2o.estimators.glm import H2OGeneralizedLinearEstimator
  from h2o.grid.grid_search import H2OGridSearch
  from scipy import stats
  from sklearn.linear_model import LogisticRegression
  from sklearn import metrics
label_desc: Use image node_id 9 for calling the TestGLMMultinomial obj's underlying member method code with example usage: obj.test7_missing_enum_values_lambda_search() without return types