How to handle Overfitting
This is a very general question, however, there are many different solutions as explained below. For your case, probably, item 2 is not the case because you can not gather a large number of data points. I would recommend using solutions 1, 3, 5, and 6 (I see you used this method but try to combine it with other solutions such as cross-validation, regularization, and feature selection).

Cross-validation: Use the initial training dataset to produce multiple mini train-test splits. Use these splits to tune your model.

For example, in k-fold cross-validation, partition the data into k subsets. Then, train the model on k-1 folds iteratively while using the remaining fold as the test set. In this way, you can use the cross-validation to tune hyperparameters with only the original training set.

Train with more data: Try to use more data points if possible.

Perform feature selection: There are many algorithms that you can use to perform feature selection and prevent from overfitting

Early stopping: When you’re training a learning algorithm iteratively, you can measure how well each iteration of the model performs.


Up until a certain number of iterations, new iterations improve the model. After that point, however, the model’s ability to generalize can weaken as it begins to overfit the training data.

Use regularization. As also will be discussed in item 7, the higher the complexity of the model the higher the chance of overfitting. For example, in the case of logistic regression, when the weights are large, the model gets complicated and it probably won't work on the unseen test dataset. Regularization helps to decrease the weights and so the complexity of the model.

Use ensembling methods such as Random Forest and Gradient Boosting. One of the main issues with decision trees is that they are prone to overfitting; i.e., high variance, it means that they work very well on training data but not on the unseen test dataset. One solution to prevent overfitting in the decision tree is to use ensembling methods such as Random Forest, which uses the majority votes for a large number of decision trees trained on different random subsets of the data.

Simplifying the model: very complex models are prone to overfitting. Decrease the complexity of the model to avoid overfitting. For example, in deep neural networks, the chance of overfitting is very high when the data is not large. Therefore, decreasing the complexity of the neural networks (e.g., reducing the number of hidden layers) could help to prevent overfitting.

Drop out method. In deep neural networks, randomly dropping some of the connections between layers by multiplying noise sampled from a Bernoulli distribution could help to prevent overfitting.