class_docstr: This class is created to test the GLM algo with Gaussian family.  In this case, the relationship
  between the response Y and the predictor vector X is assumed to be Y = W^T * X + E where E is
  unknown Gaussian noise.  We generate random data set using the exact formula.  Since we know
  what the W is and there are theoretical solutions to calculating W, p-values, we know the solution
   to W/p-values/MSE for test and training data set for each randomly generated
  data set.  Hence, we are able to evaluate the H2O GLM Model generated using the same random data
  sets.  When regularization and other parameters are enabled, theoretical solutions are no longer
  available.  However, we can still evaluate H2O GLM model performance by comparing the MSE from H2O model
  and to the theoretical limits since they all are using the same data sets.  As long as they do not
  deviate too much, we consider the H2O model performance satisfactory.  In particular, I have
  written 8 tests in the hope to exercise as many parameters settings of the GLM algo with Gaussian
  distribution as possible.  Tomas has requested 2 tests to be added to test his new feature of
  missing_values_handling for predictors with both categorical/real columns.  Here is a list of
  all tests:
  
  test1_glm_and_theory(): theoretical values for weights, p-values and MSE are calculated.
      H2O GLM model is built Gaussian family with the same random data set.  We compare
      the weights, p-values, MSEs generated from H2O with theory.
  test2_glm_lambda_search(): test lambda search with alpha set to 0.5 per Tomas's
      suggestion.  Make sure MSEs generated here is comparable in value to H2O
      GLM model with no regularization.
  test3_glm_grid_search_over_params(): test grid search with over
      various alpha values while lambda is set to be the best value obtained
      from test 2.  The best model performance hopefully will generate MSEs
      close to H2O with no regularization in test 1.
  test4_glm_remove_collinear_columns(): test parameter remove_collinear_columns=True
      with lambda set to best lambda from test 2, alpha set to best alpha from Gridsearch
      and solver set to the one which generate the smallest test MSEs.  The same data set
      is used here except that we randomly choose predictor columns to repeat and scale.
      Make sure MSEs generated here is comparable in value to H2O GLM with no
      regularization.
  test5_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      only real value predictors.  The same data sets as before is used.  However, we
      go into the predictor matrix and randomly decide to change a value to be
      nan and create missing values.  Since no regularization is enabled in this case,
      we are able to calculate a theoretical weight/p-values/MSEs where we can
      compare our H2O models with.
  test6_enum_missing_values(): Test parameter missing_values_handling="MeanImputation" with
      mixed predictors (categorical/real value columns).  We first generate a data set that
      contains a random number of columns of categorical and real value columns.  Next, we
      encode the categorical columns.  Then, we generate the random data set using the formula
      Y = W^T * X+ E as before.  Next, we go into the predictor matrix (before encoding) and randomly
      decide to change a value to be nan and create missing values.  Since no regularization
      is enabled in this case, we are able to calculate a theoretical weight/p-values/MSEs
      where we can compare our H2O models with.
  test7_missing_enum_values_lambda_search(): Test parameter
      missing_values_handling="MeanImputation" with mixed predictors (categorical/real value columns).
      We first generate a data set that contains a random number of columns of categorical and real
      value columns.  Next, we encode the categorical columns using true one hot encoding.  Then,
      we generate the random data set using the formula Y = W^T * X+ E as before.  Next, we go into
      the predictor matrix (before encoding) and randomly  decide to change a value to be nan and
      create missing values.  Lambda-search will be enabled with alpha set to 0.5.  Since the
      encoding is different in this case than in test6, we will compute a theoretical weights/MSEs
      and compare the best H2O model MSEs with theoretical calculations and hope that they are close.
class_docstr_tok: ["This","class","is","created","to","test","the","GLM","algo","with","Gaussian","family",".","In","this","case",",","the","relationship","between","the","response","Y","and","the","predictor","vector","X","is","assumed","to","be","Y","=","W^T","*","X","+","E","where","E","is","unknown","Gaussian","noise",".","We","generate","random","data","set","using","the","exact","formula",".","Since","we","know","what","the","W","is","and","there","are","theoretical","solutions","to","calculating","W",",","p-values",",","we","know","the","solution","to","W\/p-values\/MSE","for","test","and","training","data","set","for","each","randomly","generated","data","set",".","Hence",",","we","are","able","to","evaluate","the","H2O","GLM","Model","generated","using","the","same","random","data","sets",".","When","regularization","and","other","parameters","are","enabled",",","theoretical","solutions","are","no","longer","available",".","However",",","we","can","still","evaluate","H2O","GLM","model","performance","by","comparing","the","MSE","from","H2O","model","and","to","the","theoretical","limits","since","they","all","are","using","the","same","data","sets",".","As","long","as","they","do","not","deviate","too","much",",","we","consider","the","H2O","model","performance","satisfactory",".","In","particular",",","I","have","written","8","tests","in","the","hope","to","exercise","as","many","parameters","settings","of","the","GLM","algo","with","Gaussian","distribution","as","possible",".","Tomas","has","requested","2","tests","to","be","added","to","test","his","new","feature","of","missing_values_handling","for","predictors","with","both","categorical\/real","columns",".","Here","is","a","list","of","all","tests",":","test1_glm_and_theory","(",")",":","theoretical","values","for","weights",",","p-values","and","MSE","are","calculated",".","H2O","GLM","model","is","built","Gaussian","family","with","the","same","random","data","set",".","We","compare","the","weights",",","p-values",",","MSEs","generated","from","H2O","with","theory",".","test2_glm_lambda_search","(",")",":","test","lambda","search","with","alpha","set","to","0.5","per","Tomas's","suggestion",".","Make","sure","MSEs","generated","here","is","comparable","in","value","to","H2O","GLM","model","with","no","regularization",".","test3_glm_grid_search_over_params","(",")",":","test","grid","search","with","over","various","alpha","values","while","lambda","is","set","to","be","the","best","value","obtained","from","test","2",".","The","best","model","performance","hopefully","will","generate","MSEs","close","to","H2O","with","no","regularization","in","test","1.","test4_glm_remove_collinear_columns","(",")",":","test","parameter","remove_collinear_columns=True","with","lambda","set","to","best","lambda","from","test","2",",","alpha","set","to","best","alpha","from","Gridsearch","and","solver","set","to","the","one","which","generate","the","smallest","test","MSEs",".","The","same","data","set","is","used","here","except","that","we","randomly","choose","predictor","columns","to","repeat","and","scale",".","Make","sure","MSEs","generated","here","is","comparable","in","value","to","H2O","GLM","with","no","regularization",".","test5_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","only","real","value","predictors",".","The","same","data","sets","as","before","is","used",".","However",",","we","go","into","the","predictor","matrix","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Since","no","regularization","is","enabled","in","this","case",",","we","are","able","to","calculate","a","theoretical","weight\/p-values\/MSEs","where","we","can","compare","our","H2O","models","with",".","test6_enum_missing_values","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","We","first","generate","a","data","set","that","contains","a","random","number","of","columns","of","categorical","and","real","value","columns",".","Next",",","we","encode","the","categorical","columns",".","Then",",","we","generate","the","random","data","set","using","the","formula","Y","=","W^T","*","X+","E","as","before",".","Next",",","we","go","into","the","predictor","matrix","(","before","encoding",")","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Since","no","regularization","is","enabled","in","this","case",",","we","are","able","to","calculate","a","theoretical","weight\/p-values\/MSEs","where","we","can","compare","our","H2O","models","with",".","test7_missing_enum_values_lambda_search","(",")",":","Test","parameter","missing_values_handling=","''","MeanImputation","''","with","mixed","predictors","(","categorical\/real","value","columns",")",".","We","first","generate","a","data","set","that","contains","a","random","number","of","columns","of","categorical","and","real","value","columns",".","Next",",","we","encode","the","categorical","columns","using","true","one","hot","encoding",".","Then",",","we","generate","the","random","data","set","using","the","formula","Y","=","W^T","*","X+","E","as","before",".","Next",",","we","go","into","the","predictor","matrix","(","before","encoding",")","and","randomly","decide","to","change","a","value","to","be","nan","and","create","missing","values",".","Lambda-search","will","be","enabled","with","alpha","set","to","0.5",".","Since","the","encoding","is","different","in","this","case","than","in","test6",",","we","will","compute","a","theoretical","weights\/MSEs","and","compare","the","best","H2O","model","MSEs","with","theoretical","calculations","and","hope","that","they","are","close","."]
docstr: This function is written to load in a training/test data sets with predictors followed by the response
  as the last column.  We then calculate the weights/bias and the p-values using derived formulae
  off the web.
  
  :param training_data_file: string representing the training data set filename
  :param test_data_file:  string representing the test data set filename
  :param has_categorical: bool indicating if the data set contains mixed predictors (both enum and real)
  :param true_one_hot:  bool True: true one hot encoding is used.  False: reference level plus one hot
  encoding is used
  :param validation_data_file: optional string, denoting validation file so that we can concatenate
   training and validation data sets into a big training set since H2O model is using a training
   and a validation data set.
  
  :return: a tuple containing weights, p-values, training data set MSE and test data set MSE
docstr_tok: ["This","function","is","written","to","load","in","a","training\/test","data","sets","with","predictors","followed","by","the","response","as","the","last","column",".","We","then","calculate","the","weights\/bias","and","the","p-values","using","derived","formulae","off","the","web",".",":","param","training_data_file",":","string","representing","the","training","data","set","filename",":","param","test_data_file",":","string","representing","the","test","data","set","filename",":","param","has_categorical",":","bool","indicating","if","the","data","set","contains","mixed","predictors","(","both","enum","and","real",")",":","param","true_one_hot",":","bool","True",":","true","one","hot","encoding","is","used",".","False",":","reference","level","plus","one","hot","encoding","is","used",":","param","validation_data_file",":","optional","string",",","denoting","validation","file","so","that","we","can","concatenate","training","and","validation","data","sets","into","a","big","training","set","since","H2O","model","is","using","a","training","and","a","validation","data","set",".",":","return",":","a","tuple","containing","weights",",","p-values",",","training","data","set","MSE","and","test","data","set","MSE"]
code: def theoretical_glm(
      self,
      training_data_file,
      test_data_file,
      has_categorical,
      true_one_hot,
      validation_data_file="",
  ):
      """
      This function is written to load in a training/test data sets with predictors followed by the response
      as the last column.  We then calculate the weights/bias and the p-values using derived formulae
      off the web.
  
      :param training_data_file: string representing the training data set filename
      :param test_data_file:  string representing the test data set filename
      :param has_categorical: bool indicating if the data set contains mixed predictors (both enum and real)
      :param true_one_hot:  bool True: true one hot encoding is used.  False: reference level plus one hot
      encoding is used
      :param validation_data_file: optional string, denoting validation file so that we can concatenate
       training and validation data sets into a big training set since H2O model is using a training
       and a validation data set.
  
      :return: a tuple containing weights, p-values, training data set MSE and test data set MSE
  
      """
      # read in the training data
      training_data_xy = np.asmatrix(
          np.genfromtxt(training_data_file, delimiter=",", dtype=None)
      )
      test_data_xy = np.asmatrix(
          np.genfromtxt(test_data_file, delimiter=",", dtype=None)
      )
  
      if (
          len(validation_data_file) > 0
      ):  # validation data set exist and add it to training_data
          temp_data_xy = np.asmatrix(
              np.genfromtxt(
                  validation_data_file, delimiter=",", dtype=None
              )
          )
          training_data_xy = np.concatenate(
              (training_data_xy, temp_data_xy), axis=0
          )
  
      # if predictor contains categorical data, perform imputation during encoding of enums to binary bits
      if has_categorical:
          training_data_xy = pyunit_utils.encode_enum_dataset(
              training_data_xy,
              self.enum_level_vec,
              self.enum_col,
              true_one_hot,
              np.any(training_data_xy),
          )
          test_data_xy = pyunit_utils.encode_enum_dataset(
              test_data_xy,
              self.enum_level_vec,
              self.enum_col,
              true_one_hot,
              np.any(training_data_xy),
          )
  
      # replace missing values with column mean before proceeding for training/test data sets
      if np.isnan(training_data_xy).any():
          inds = np.where(np.isnan(training_data_xy))
          col_means = np.asarray(np.nanmean(training_data_xy, axis=0))[
              0
          ]
          training_data_xy[inds] = np.take(col_means, inds[1])
  
          if np.isnan(test_data_xy).any():
              # replace the actual means with column means from training
              inds = np.where(np.isnan(test_data_xy))
              test_data_xy = pyunit_utils.replace_nan_with_mean(
                  test_data_xy, inds, col_means
              )
  
      (num_row, num_col) = training_data_xy.shape
  
      dof = num_row - num_col  # degree of freedom in t-distribution
  
      response_y = training_data_xy[:, num_col - 1]
      training_data = training_data_xy[:, range(0, num_col - 1)]
  
      # generate weight vector W = (X^T*X)^(-1)*X^T*Y
      # form the X matrix here
      temp_ones = np.asmatrix(np.ones(num_row)).transpose()
      x_mat = np.concatenate((temp_ones, training_data), axis=1)
  
      mat_inv = np.linalg.pinv(x_mat.transpose() * x_mat)
      t_weights = mat_inv * x_mat.transpose() * response_y
  
      # calculate training data MSE here
      t_predict_y = x_mat * t_weights
      delta = t_predict_y - response_y
      mse_train = delta.transpose() * delta
  
      # calculate 2-sided p-values here
      mysd = mse_train / dof
      se = np.sqrt(mysd * np.diag(mat_inv))
      tval = abs(
          t_weights.transpose() / se
      )  # ensure floating point division here
      p_values = scipy.stats.t.sf(tval, dof) * 2
  
      # calculate test data MSE
      test_response_y = test_data_xy[:, num_col - 1]
      test_data = test_data_xy[:, range(0, num_col - 1)]
  
      t_predict = pyunit_utils.generate_response_glm(
          t_weights, test_data, 0, self.family
      )
      (num_row_t, num_col_t) = test_data.shape
  
      temp = t_predict - test_response_y
      mse_test = temp.transpose() * temp / num_row_t  # test data MSE
  
      return (
          np.array(t_weights.transpose())[0].tolist(),
          np.array(p_values)[0].tolist(),
          mse_train[0, 0] / num_row,
          mse_test[0, 0],
      )
  
imports: import sys
  import random
  import os
  import numpy
  import scipy
  import math
  from scipy import stats
  from builtins import range
  import time
  import h2o
  from tests import pyunit_utils
  from h2o.estimators.glm import H2OGeneralizedLinearEstimator
  from h2o.grid.grid_search import H2OGridSearch
label_desc: Use image node_id 11 for calling the TestGLMGaussian obj's underlying member method code with example usage: obj.theoretical_glm(training_data_file, test_data_file, has_categorical, true_one_hot, validation_data_file) and returns: np,  np,  unknown,  mse_test