"reg_alpha" parameter in XGBoost regressor. Is it bad to use high values?
reg_alpha penalizes the features which increase cost function. Meaning it finds the features that doesn't increase accuracy. But this makes prediction line smoother.
On some problems I also increase reg_alpha > 30 because it reduces both overfitting and test error.
But if it is a regression problem it's prediction will be close to mean on test set and it will maybe not catch anomalies good.
So i may say you can increase it as long as your test accuracy doesn't begin to fall down.
Lastly when increasing reg_alpha , keeping max_depth small might be a good practise.
I'm performing hyperparameter tuning with grid search and I realized that I was getting overfitting... I tried a lot of ways to reduce it, changing the "gamma", "subsample", "max_depth" parameters to reduce it, but I was still overfitting...
Then, I increased the "reg_alpha" parameters value to > 30....and them my model reduced overfitting drastically. I know that this parameter refers to L1 regularization term on weights, and maybe that's why solved my problem.
I just want to know if it has any problem using high values for reg_alpha like this?
I would appreciate your help :D