In what stage should you plot validation and learning curve So lets assume I have a binary classification problem and I started out with a logistic regression model. I quickly evaluate the models accuracy (lets assume we don't have a skewed dataset). After this I start to implement other models to compare, I begin with checking the accuracy score and plot a ROC curve to evaluate AOC score.
Lets say that I carry on with my logistic regression model. I wan't to fine-tune it and use a grid search algorithm and from there carry on choosing the best hyperparamters w.r.t the accuracy score.
When do I actually plot the learning and validation curve? I pressume that I should plot it next to see how a larger set of paramters affect the score and then to see if my model is underfitting or overfitting.
But shouldn't I plot the learning curve much earlier for example in the first step where I choose to start with a logistic reg. model?
To sum it up: In what stage is it appropiate to plot a learning curve?
I understand that with &quot;learning curve&quot; you are referring to the plot of the loss function over the training data (or subsets of it) when optimizing with iterative methods, like gradient descent. This is sometimes referred to as the training loss plot.
You use a training loss plot during training, to evaluate the convergence of your training algorithm and model. Usually, together with the training loss plot, you visualize the validation loss, that is, the loss function computed on a held-out dataset, to evaluate whether your model is overfitting your training data.
Normally, the training loss gets a new point after each optimization step. For instance, in stochastic gradient descent, the training loss points are the loss evaluated at each minibatch. On the other hand, the validation loss curve gets a new point after each epoch, or at least after a few optimization steps have been performed.
This is different from a ROC curve, which is evaluated once the model is fully trained, to understand the behavior of the trained model.