Random Machines Regression Approach: an ensemble support vector regression model with free kernel choice
Anderson Ara Mateus Maia Samuel Macêdo Francisco Louzada
stat.ML cs.LG stat.AP






Department of Statistics

       Federal University of Bahia

       Salvador, Bahia, Brazil
       

Department of Statistics

       Federal University of Bahia

       Salvador, Bahia, Brazil
       

Department of Natural Sciences and Mathemathics

       Federal Institute of Pernambuco

       Recife, Pernambuco, Brazil
                     

Institute of Mathematical and Computer Sciences 

       University of São Paulo 

       São Carlos, São Paulo, Brazil


Machine learning techniques always aim to reduce the generalized prediction error. In order to reduce it, ensemble methods present a good approach combining several models that results in a greater forecasting capacity. The Random Machines already have been demonstrated as strong technique, i.e: high predictive power, to classification tasks, in this article we propose an procedure to use the bagged-weighted support vector model to regression problems. Simulation studies were realized over artificial datasets, and over real data benchmarks. The results exhibited a good performance of Regression Random Machines through lower generalization error without needing to choose the best kernel function during tuning process.  



Support Vector Regression Bagging Kernel Functions 

















§ INTRODUCTION

The prediction of new observations or events through statistical models is one of the main objectives of supervised statistical learning methods. Currently, machine learning models have several applications in regression tasks in a wide range of science fields, for instance, economy - predicting bitcoin's price <cit.>, biology - predicting biological proprieties from plants <cit.> or classifying gene functions <cit.>, and physics - predicting electrical proprieties from materials. Inside this type of regression models, there is the support vector regression model that was proposed by <cit.> and has been used extensively as a optimal solution when compared with other traditional base-line methods <cit.>.  

Besides SVR, the ensemble method is a statistical learning approach that combines models in order to achieve greater predictive capacity. The combination of singular models can enhance predictive power and increase its generalization power <cit.>. Even novel approaches, as deep learning models, can also benefit from ensemble procedures <cit.>. There are two main types of ensemble algorithms: bagging <cit.> that uses independent bootstrap samples to create multiple models and built a final classifier by average mean or majority vote, reducing the variance, and boosting algorithms <cit.> that built sequential models in order to assign different weights based on their errors.

Bagging method does not require a specific type of base classifier, and can be be used to improve predictions in regression tasks <cit.>. This method can be used to enhance the a single support vector regression model and others kind of algorithms. The bagging approach using support vector regression models is already reported in literature through diverse applications. As examples, can be cited the works: for predict protein retention time <cit.>, for predict time series <cit.>, electric load forecasting <cit.>, for forecast building occupation <cit.> and to predict blood pressure measures <cit.>. 

Nonetheless the different number of works that presents the bagging based on support vector regression models, there is no proposal of standard framework to choose which kernel function will be used in ensemble that use SVR as base-learners. In support vector models the kernel function and its hyperparameters have a compelling impact on the efficiency of the algorithm <cit.>. Generally, this selection is made by a grid-search, which choose those parameters that produces the lower test error inside a grid of possible combinations, by random search <cit.>, or by bayesian optimization algorithms <cit.>. All of them are computationally expensive and and can consume too much time. This work introduces a novel model that gives a solution for the kernel function's choice using the bagged supported vector regression with efficient computational time and robust predictive power named as the Regression Random Machines (RRM). The method received this name because it uses random kernel choice for each model that composes the bagged support vector regression method, besides propose weights to these regressors, increasing the predictive power of the final model. The result was validated over simulation studies, beyond using the algorithm on diverse benchmarking datasets.

The following chapter is organized on the ensuing outline. Section 3.2 presents a theoretical description about the support vector machine method, proposed by Vapnik <cit.>, the challenges on the selection of hyperparameters and standard kernel functions; Section 3.3 presents a overview of the bagging algorithm and bagged SVR; Section 3.4 introduce how the proposed Regression Random Machines (RRM) approach works in detail, followed by the simulations studies in Section 3.5, as well as the applications in real data in Section 3.6. In the last section, is proposed a discussion about correlation and strength of bagging base models, and how they affect the general method's performance .




§ SUPPORT VECTOR MACHINE

Support vector regression machines <cit.> have been proposed as a generalized version of support vector machines <cit.> to regression tasks. Instead of creating a hyperplane with maximum margin in feature space, as done in support vector machines, the support vector regression build a optimal hyper-tube, which have all observations inside it, that maximizes the distance between the observations and the center of hypertube (Figure <ref>). 



Supposing a database given by {𝐱_𝐢,𝐲_𝐢}, i=1,…, n, y_i∈ℝ, where n is the number of observations. The proposed objective function which will be minimized its given by 

    min_𝐰1/2𝐰·𝐰



    s.t=  
    𝐲_i-f̂(𝐱_i) ≤ϵ
    f̂(𝐱_i) - 𝐲_i≤ϵ


are satisfied for i=1,…,n; and where f̂(𝐱_i)=𝐰·𝐱_i - b. Vapnik <cit.> showed that there are cases where the hypertube cannot be created with all observations inside it, so they introduced the slacks variables ℰ, ℰ' and the new optimization of convex objective function was given by


    min_𝐰,ℰ,ℰ'1/2𝐰·𝐰 + C∑_i=1^n(ℰ+ℰ')



    s.t=  
    𝐲_i-f̂(𝐱_i) ≤ℰ + ϵ
    f̂(𝐱_i) - 𝐲_i≤ℰ^' + ϵ
    
      0 ≥ℰ, ℰ^'


for i=1,…,n and hold f̂(𝐱_i)=𝐰·𝐱_i - b.

The hypertube also can be derived the dual to maximum-margin margin optimization given by the following Equation <ref> and the constraints <cit.>


    max_α,α^'( 1/2∑_i=1^n∑_j=1^n (α_i-α^'_i)(α_j-α^'_j)𝐱_i·𝐱_j + ∑_i=1^n (α_i-α^'_i) y_i - ϵ∑_i=1^n  (α_i-α^'_i) )



    s.t=  
        ∑_i=1^n  (α_i-α^'_i)=0 
    
          C ≥α_i,α^'_i≥ 0

    

for, i=1,…,n where

    
    w=(α_i-α^'_i)y_i
        b=1/n∑_i=1^n∑_j=1^n 
        (α_i-α^'_i)𝐱_i·𝐱_j-y_i

    

This approach of SVR works well with linear regression problems. However, there are cases where exist the non-linearity between the explaining and predictor variables. In these cases it may be used the kernel trick, based in Mercer's Theorem to deal with non-linearity. Using kernel methods, rather than consider the input space, it is considered higher feature spaces, where the observations could be linearly separable through the following function K(𝐱_𝐢,𝐱_𝐣)=ϕ(𝐱_𝐢)·ϕ(𝐱_𝐣)  that replaces the inner product in Equation <ref>.

The functions K(x,y)=ϕ(𝐱)·ϕ(𝐲) are defined as the semidefinite kernel functions <cit.>. Various types of kernel functions are used in distinct regression examples. The choice of particular kernels functions provide unique nonlinear mappings and the performance of the resulting SVR often depends on the appropriate choice of the kernel <cit.>.  There are several kernel functions in the general framework for SVR, which some of the most commons were used in this paper. They are presented in Table <ref>, and have hyperparameters γ and d, which γ∈ℝ^+, d ∈ℕ. The polynomial kernel, for instance, represent an transformation of the feature space to a determined degree. On the other hand, the radial basis kernel function as gaussian and laplacian, produces a feature space of infinite dimension.




 
 

 
 Determining which the best kernel by grid search, or other search method, can be an expensive and harrowing computational problem <cit.>.  To solve it, many works have tried to develop a methodology which can improve the selection of the best kernel function <cit.>. Regression Random Machines method proposes an efficient alternative to work through a framework where it is avoidable this exhaustive search.



§ BAGGING


Bagging is an acronym of Bootstrapping Aggregation, which was proposed by Breiman <cit.>. Bagging is an ensemble method that can be used for different prediction tasks. In general, the Bootstrapping Aggregating generates data sets by random sampling with replacement from the training set with the same size n, also known as bootstrap samples. Then, each model h_j(x_i) is trained independently for each bootstrapping sample j, ∀ j ∈{1,…,B}. The final bagging model, for regression tasks, is given by the following equation,

    H(𝐱)=1/n∑_i=1^Bh_i(𝐱),


where h_i(𝐱) is the model generated to each bootstrap sample from i=1,,…,B, and B is the number of total bootstrap samples. 

Breiman <cit.> also reported that approximately 1/3 of the observations from a database were not selected at each bootstrap sampling process. These observations were named as Out of Bag samples. Therefore, they could be used as test samples since they were not used to train the bootstrap models.





 §.§ Bagging SVR


Considering bagging procedure, the function h_i(𝐱) from (<ref>) can be any model and it can improve the predictive power of non-parametric regression methods <cit.>. One possibility is to use the SVR as the base model in order to lower the generalization error. Besides the applications already shown, the use of the bagged SVR for regression tasks can be listed: content-based image retrieval <cit.>, solar power forecasting <cit.>, quantifying urban land cover <cit.>, wind power prediction <cit.> and a trimmed bagging approach <cit.>. 

Despite the some works applied bagged SVR, none of them present a general framework to deal with the choice of the best kernel function. Often they choose it by trial evaluation, by a grid search or random search. As this proceeding is computationally expensive <cit.>, this paper proposed a novel bagging approach that can overcome the difficult to choose the best kernel function, besides showing an improvement in the prediction capacity by combining several different SVR models and varying the kernel functions: the Regression Random Machines.  



§ REGRESSION RANDOM MACHINES

Given a training set {(x_i,y_i)}_i=1^N with 𝐱_𝐢∈ℝ^p and y_i∈ℝ, ∀ i=1,…,n; the kernel bagging method initialize by training single models h_r(𝐱), where r=1,…,R, and R is the  total number of different kernel functions that could be used in support vector regression models. For example, if R=4 a possible choice is define h_1 as SVR with Linear kernel , h_2 as SVR with Polinomial kernel, h_3 as SVR with Gaussian kernel and h_4 as SVR with Laplacian kernel. 

Each model is validated for the test set {(x_k,y_k)}_k=1^V, and the root mean square error (RMSE_r), which we will refer as δ_i, is calculated for each model, ∀ r=1,…,R, where R is the number of kernel functions that will be used. As the range of the dependent variable in regression (y) is broad, the vector of root means squares δ is divided by its deviation in order to standardize the error. Afterwards, sample probabilities, λ_r, are calculated by the Equation (<ref>) for each kernel function 


    λ_r=e^-βδ_r/∑_i=1^Re^-βδ_i,

with ∀ r=1,…,R.


Subsequently, B bootstrap samples are sampled from the training set. A support vector regression model g_k is trained for each bootstrap sample, k=i,…,B and the kernel function that will be used for g_k will be determined by a random choice with probability λ_r, ∀ r=1,…,R.
The probabilities λ_r are higher if determined kernel function used in h_r(𝐱) has lower generalization error measured from the calculated RMSE over the test set. Consequently, the models with lower RMSE will frequently appear when the random kernel selection for each bootstrap model is done. 

The parameter β, named as correlation parameter, will tune the penalty of the generalization error of each model. Figure <ref> shows that small values of β create heavy-tail penalty functions while greater beta's values represent light-tail penalty. The parameter gets its name because it can determine the diversity between the chosen kernel functions, since high values further penalize the performance differences between each SVR model types. For instance, considering a value of β=0, the result of vector of probabilities is given by Equation <ref> its λ={0.25,0.25,0.25,0.25}, which means that all kernels have the same chance to be sampled in each bootstrap model, i.e: maximum diversity between kernel functions.  On other hand, a large value for β would quickly scale difference between models, consequently, the vector of probabilities λ would accumulate in a single kernel, and just one would be sampled, i.e: minimum diversity since its the same kernel function for all bootstrap samples returning to the traditional bagging approach.




After, a weight w_i is assigned to each bootstrap model calculated for g_i ∀ i=1,…,B. The weight is given by the Equation (<ref>).


    w_i=e^-βΛ_i/∑_j=1^Be^-βΛ_j,  i=1,…,B ,




where Λ_i is the Root Mean Square Error of model's prediction g_i using the Out of Bag Samples (OOBG_i), obtained from i bootstrap sample ∀ i=1,…,B, as test set. 














    G(𝐱_𝐢)= ∑_j^Bw_jg_j(𝐱_𝐢),  i=1,…,N.


All the modeling process is summed up in the pseudo-code exposed in Algorithm <ref>.




The entire Regression Random Machines is schematically presented in Figure <ref>, where it is designed all the steps used in all cases presented in this article. 






§ ARTIFICIAL DATA APPLICATION


Different scenarios were used to study the Regression Random Machines on simulated data. The objective was to evaluate the performance and behavior of the model when we have a controlled experiment. Eight different data sets generation scenarios were tested. The Models 1-5 are toy examples and can be found in <cit.>, the  Model 6 in <cit.>, the Model 7 in <cit.> and the Model 8 is presented in <cit.>. The simulations from 1-7 has the vector of independent predictions X=(X_1,…,X_p) and X follow a uniform distribution [0,1]^p. In the Model 8 each predictor X_i follow a standard normal distribution. Also, we define the transformation X_i=2(X-0.5),  i=1,…,p. For each case the sample size changed among the values of n ={30,100,1000}.  All the scenarios are described below:


    
  * Model 1: p=2, Y=X^2_1+e^-X^2_2 + 𝒩(0,0.25)
    
  * Model 2: p=8, Y=X_1X_2+X^2_3-X_4X_7+X_5X_8-X^2_6+𝒩(0,0.5)
    
  * Model 3: p=4, Y= -sin(X_1)+X^2_2+X_3-e^-X^2_4+𝒩(0,0.5)
    
  * Model 4: p=4, Y= X_1+(2X_2-1)^2+2sin(2πX_3)/(2-sin(2πX_3))+sin(2πX_4)+2cos(2πX_4)+3sin^2(2πX_4)+4cos^2(2πX_4)+𝒩(0,0.5)
     
  * Model 5: p=8,  Y=1_X_1 > 0+ X^3_2+1_X_3+X_4-X_6-X_5  > 1+X_7+e^-X^2_8+𝒩(0,0.5)
    
  * Model 6: p=6, Y=X^2_1+X^2_2X_3e^-|X_4|+X_6-X_5+𝒩(0,0.5)
    
  * Model 7: p=4, Y=X_1+3X^2_2-2e^-X_3+X_4
    
  * Model 8: p=6, Y=X_1+0.707 X^2_2 + 21_X_3>0+0.873 log (X_1)|X_3|+0.894 X_2 X_4+21_X_5>0+0.464e^X_6+𝒩(0,1)
    

An illustration about how the regression hyperplane created by the RRM can be seen in Figure <ref> which the Model 1 of data generation is used as example.

The repeated holdout with 30 repetitions was used as validation technique with a split ratio of training-test of 70%-30%. The Table <ref> summarizes the result, and from it is possible to realize that the Regression Random Machine outperformed the others methods in the majority simulations setups that were presented. The bold RMSE mean values indicates that this result was the lower among all methods. The default parameters of models were γ=1, B=100, C=1 , β=2, d=2 and ϵ=0.1.






















§ REAL DATA APPLICATION


The methodology was applied on 26 real-world datasets from the UCI Repository <cit.> to evaluate its performance. The datasets present a wide variety in the number of observations, dimensionality, and type of data. In addition, all of them represent a regression task. Table <ref> summarizes all datasets considered. The continuous features were scaled to zero mean and unit variance, in the exception of the discrete features which were went through a one-hot-encoding process. The validation technique used was the repeated holdout with 30 repetitions and a split ratio of training-test of 70%-30%.




The Regression Random Machines was compared with the bagged SVR approach using each one of single kernel functions showed in  Table <ref>, and with the standard SVR procedure with the same kernel functions. The chosen parameters were: the parameter ϵ=1, the cost parameter C=1, the number of bootstrap samples B=100, the degree of polynomial kernel d=2, and the hyperparameter γ from the Table <ref>, γ=1. The result is resumed in the Figure <ref> considering the Root Mean Squared Error.

As demonstrated in Figure <ref>, the RRM shows lower generalization error than the other bagged support vectors using unique kernel functions. Comparing the RRM with the traditional bagged SVR, it outperforms almost 90.9% of times the Kernel Linear Bagging, 96.9% for the Kernel Polynomial Bagging, 97.2% for the Gaussian Bagging, and 94.7% for the Laplacian Kernel Bagging. This results shows off that the random weighted choice of the kernels functions reduced,  mostly, the error from the predicted values. The difference is also present when the Regression Random Machines is put on comparison with the singular SVR, where the RM is more accurate 91.2% of times considering the Kernel Linear, 96.4% for the Kernel Polynomial, 94.6% for the Gaussian Bagging, and 84.5% for the Laplacian Kernel.





Another way to evaluate the method's performance is through the Error Score (ES) vector. This index formulation it is given by

    ES_i=ε_i - ε_min/ε_max-ε_min,


where the ε it just the Root Mean Squared Error vector over test observations for all methods, and where ε_i is the individual value for that technique ∀ i=1,…,R. For instance, suppose three algorithms: Regression Random Machines, SVR.Lin Regression and BSVR.Lin, then, after calculating the RMSE for each of them over a test set, the vector ε={0.1,0.5,0.3} is obtained, with the coordinates for each kernel function respectively. Thus, the ES vector is given by ES={0,1,0.5} which means that ES_1=0 was the Error Score for RRM, ES_2=1 for SVR.Lin and ES_2=0.5 for BSVR.Lin. The Figure <ref> shows boxplots for the mean values for the Error Score over all the 30 holdout repetitions for all 26 benchmarking data sets.



Analyzing the results it is clear to see that the RRM have, generally, a good performance when compared with the traditional methods. This approach also deals with the problem of the choice of best kernel function, since is not necessary to perform a grid-search among all the different functions and define which one has lower test error. For this reason, the RRM algorithm can be considered efficient, as it can reduce the prediction error and the computational cost.

As the hyperparameter tuning is an important procedure in of the support vector machine regression algorithm <cit.>, the value of γ was changed in order to study how its variation affects the behavior of RRM. The setting of the parameters was the set of values γ={2^-3,2^-2,2^-1,2^0,2^1,2^2,2^3} over the same 25 data sets (removed the Airbnb). The result is shown in Figure <ref>, where it is possible to see that the RRM surpassed the other bagging and single models. As said before the selection of these hyparameters, as the kernel function, has a direct impact on the model performance, and the results fortify the supposition that RRM gives a good and consistent result for a wide range γ values. 







§ STRENGTH AND CORRELATION EVALUATION


In this section we explain the reason why the Regression Random Machines approach is an ensemble approach that can reduce the generalization error. The random selection of kernel functions works to diversify different functions that belong to a Reproducing Kernel Hilbert Space (RHKS). The goal of this procedure is to diminish the correlation between regression models that constitute the RRM and increase strength of them since these both components result in greater results to bagged classifiers <cit.>. 

 The correlation concept can be defined as a measure of how much models are similar, while the strength of a model relies on how well it correctly predicts an observation. The estimation of a correlation measure can have different approaches. Considering classification models, for instance, a method to estimate the correlation between models is to calculate the area from decision boundaries that overlaps among them <cit.>. Other estimation method, still considering classification context, is used by <cit.>, who defines the similarity, through the agreement measure, as the number of observations that are equally labeled with the same class by different models.
 
 In the regression approach the correlation/similarity estimation between models can be calculated as the mean of the upper triangle from the correlation matrix given in Equation <ref>.
 
 
    Σ_corr = 
    [     ρ_ŷ_1,ŷ_1     ρ_ŷ_1,ŷ_2             …   ρ_ŷ_1,ŷ_B-1     ρ_ŷ_1,ŷ_B;                   ρ_ŷ_2,ŷ_2             …   ρ_ŷ_2,ŷ_B-1     ρ_ŷ_2,ŷ_B;                                         ⋱             ⋮             ⋮;                                           ρ_ŷ_B-1,ŷ_B-1   ρ_ŷ_B-1,ŷ_B;                                                             ρ_ŷ_B,ŷ_B ] _B× B

 
 The values of ρ_ŷ_i,ŷ_j are calculated by
 
 
    ρ_i,j=∑ _k=1^T(ŷ_i_k-ŷ_̂î)(ŷ_j_k-ŷ_̂ĵ)/√(∑_k=1^T(ŷ_i_k-ŷ_̂î))√(∑ _k=1^T(ŷ_j_k-ŷ_̂ĵ)).



for all i≠ j=1,…,B, and ŷ_̂î it is the vector of predictions from observations that belongs to the test set. The strength in this article will be estimated using the Error Score presented in the previous Section 3.6, since it capture well the prediction performance and has the same range of the correlation measure. As the ES is directly proportional to the Root Mean Square Error, it also can be considered a strength measure. Smalls values of RMSE produced by a regression model implies in a stronger model. 

In order to assess the correlation and strength of the RRM in comparison with the traditional bagged version of SVR, the algorithm was applied over all models of simulated data presented at the Section 3.5. We look for the model which has the lowest RMSE and Error Score (i.e: greater strength) and the lowest correlation measure. A model with small agreement can benefit more from the bagging procedure <cit.>. However, just small values of correlation are not enough, since this lower value can represent a weak model, i.e, a model which is not capable to predict new observations well. The result is summarized in Table <ref>. Both, RMSE and Agreement were calculated using a 30 Repeated Holdout validation set with split ratio of 70-30% training-test. The parameters of the methods were: B=100, γ=1, C=1, β=2 and ϵ=0.1.

The strength of the models is affected by the agreement and vice-versa, so optimize both measures at same time is a difficult effort. The relation between them can be analyzed in the Table <ref>. Observing simultaneously the RMSE and the Agreement measure from traditional bagging approaches exists a traded-off between them. Considering a data set as example Table <ref>, if the RMSE is the lower among them, its agreement is the highest. This trade-off is minimized in RRM case, which, despite presents the lower RMSE in most of the cases, this is not reflected in the highest agreement measure among all methods. Therefore,  Regression Random Machines has the low correlation and great strength, desirable features to produce a good bagging approach.




The idea of how the random selection of kernel functions can increase the diversity of Regression Random Machines models, when compared with the traditional SVR bagging algorithm, can not be so clear at the first moment. In order to observe clearly how it works, Model 1 will be used, with n=1000, as study case and see graphically the modelling process. Figure <ref> shows the level curves from the true data generation surface, the predicted hyperplane for RRM, and each single bootstrap model of a single kernel function. It can be seen that for each different kernel function the regression surface is distinct. which promotes diversity and subsequent reduction of correlation between models. Moreover, it is possible to notice that the Regression Random Machines surface, built through the combination of these single models, is the one that is closer to the real data generation hyperplane, reinforcing that it works as the best model in that case.







The same behavior was observed on the 26 real data sets, where the agreement is also calculated and compared with the strength (Error Score) of each model (Figure <ref>). Although the low values of generalization error from BSVR.Lin and BSVR.Lap they present large agreement values. On other hand, despite the low values of agreement from BSVR.Pol and BSVR.Gau they produce great values of Error Score. The unique method that can perform the optimal values for both it is the Regression Random Machines.





 §.§ The Correlation Coefficient β


Another way to study the correlation-strength trade-off in the RRM procedure is through the coefficient β presented in the Equations <ref>-<ref>. As mentioned before, the β cofficient can calibrate the diversity of kernel functions used during the bagging procedure. If we consider β=0 the RRM will hold that all kernel functions will be sampled and weighted equally. On other hand, if we use greater values of β the RRM's behavior will be close to traditional SVR' bagging, since just a single kernel function will be chosen.

To demonstrate this performance, we evaluate the standardized RMSE and Agreement on  benchmarks from Section 3.6 changing the values of β in a grid that range from 0 to 5, with the length of 21 intervals.
Both measures were calculated in a holdout validation with split ratio of 70-30%, and setting the parameters B=100, γ=1 and d=2. The results are summarised in three Data Sets: Taiwan, Boston Housing and Friedman #1 presented in Figure <ref>.



Though different values of RMSE and Agreement, all of them present the same behavior: for small values of β, i.e: minimum correlation between models, we have an weaker (represented through the large RMSE values) model from RRM algorithm. As the beta value increases the weighting on the kernel functions and bootstrap models predictions are applied increasing the agreement and reducing the RMSE. However, at some inflection point the RMSE starts to increase and the agreement continues to grow. The reason for this result can be explained by <cit.>, where it is defined that stable base-models, SVR models in this case, will not benefit from the aggregation procedure, and may even depreciate the model. Large values of β starts to penalize the RRM in way that just one, or few different,  kernel functions will be chosen, and as SVR <cit.> is defined as a stable model (i.e: bootstrap replications produces small changes in the model) this may lead to worst results. Therefore, the key of the improvement from Regression Random Machines is to add the instability in SVR, necessary for bagging procedures <cit.>, through the random sample of kernel functions and the weights associated to the predictions.

The β also can be defined as a hyperparameter of the model and can be tuned in order to achieve the lowest generalization error. From some empirical results, in this article the default choice of β was 2, since in the most of cases the best choice have been around this value and there wasn't much improvement from the grid search procedure for this parameter.





§ FINAL COMMENTS


The main contribution of this paper is to propose a novel learning approach to do ensemble using Support Vector Regression models that can enhance the improve the traditional bagging SVR, and give an alternative to deal with the problem choosing the best kernel function that should be used. Through the Regression Random Machines, the combination of different SVR  models by the different kernel functions propose an algorithm that avoids the expensive computational cost of doing a grid or random search between the kernel functions, besides reduce the general prediction error. 

In order to quantify this reduction, suppose an number of B models calculated in a traditional bagging procedure and R as the number of kernels functions that will be evaluated and used in support vector models. In traditional bagging algorithm using SVR as base-models the number of total models that will be calculated to obtain the bests results is given by B × K, while using the Regression Random Machines approach this number reduces to B + K. Using an example of B=100 and K=400, we have that the traditional bagging algorithm would take approximately four times the computational cost than the proposed Random Machines since the ratio of calculated models is 400/104 (i.e: four time faster).

Furthermore, the results from RRM explored the strength and correlation characteristics in the bagging procedure, obtaining simultaneously lower generalization error and agreement, instead of traditional ensemble procedures using SVR as base models which cannot obtain them at the same time. 




Despite the success of Regression Random Machines there are some open problems. For instance, although the RRM avoids the kernel function choice, the tuning for some hyperparameters of SVR still necessary for achieve the best model, and considering the number of models in bagging, the tuning can be computationally expensive. Additionally, this methodology can be explored in other contexts, as the computational cost, and can be applied to any practical statistical learning problem. Future theoretical studies may be done with respect to the use of other and more kernel functions in the bagging step, besides other weighting functions approaches. 















































§ ACKNOWLEDGEMENTS


I would like to thank CAPES-CNPq, for financing this project and providing the necessary financial resources for the complete this research.






