Why not evaluate over test fit results in RandomizedSearchCV?
when we are training a model we usually divide data into train, validation and test sets. Lets look at what are the propose of each set
Train Set: It is used by model to learn its parameters. Usually model reduces its cost on the train set and selects parameters that gives minimum cost.
Validation Set: By name, validation set is used to validate that model will also performs well on the data that it haven't seen yet. That gives us confidence that model is not memorizing the training data and performing very well on training data but not performing good on new data. If the model is complex enough then there is risk of model memorizing training data to improve its performance on training set but not well on validation data.
Usually we use cross validation in which we divide training set into n equal parts, then for each iteration we select one part as validation and remaining parts as training set.
Test Set: Test Set is set aside and only used at last when we are satisfied with our model to estimate how well your final model will perform on new data in wild. One main difference with validation set is that it is not used in any way to improve or change model or improve model but validation set help us select us final model. We do this because we don't want a model that is biased towards test data but will not perform well with data in wild.
I'm trying optimize hiperparameters for classifiers and regression methods in sklearn. And I have a question. Why when you evaluate the results, you choose for example the best train accuracy, instead of evaluate this result over the test, and iterate others values with others train accuracys to obtain the best test accuracy? Because clearly the parameters for the best train accuracy are not the same that the parameters for the best test accuracy.
Thanks!