I have a data.set that contain around 3000 observations. Every observation falls in one of the five categories (these are pre-defined). I am using randomForest and gbm from the h2o package for classification. Before I run the algorithms i optimize the hyperparameters of each algorithm using the h2o.grid. For model validation I am having a look at the confussion matrix for both algorithms, for both in-sample and out-of-sample.The issue is that in sample the algorithms perform relatively well for all 5 categories but out of sample they they only perform well for one category. I guess that is very common. I have 2 explanations: the 5 categories are not equally divided, this means that for a couple categories I have a few observations.Second, is that the algorithms are overfitting. The 1m dollar question is how can someone avoid overfitting ? I thought of running a preliminary analysis on my data set, to see which variables (in my case most of them dummy variables) in my data set are the most important for the classification, and then use those, instead of the whole data set. This could be tested by taking the means for the variables per category and which variable has the highest difference across the categories. Could this be a good idea ?I understand that the subject is too broad, but I think some ideas from experienced people on this topic could be useful for everyone !
accuracy prediction out of sample gone bad ! over fitting ?
machine learning;classification;r;random forest;overfitting