Which model is better given their training and validation errors? Below you have the plots of the training and validation errors for two different models. Both plots show the RMSE values for the validation dataset versus the number of training epochs. It is observed that models get lower RMSE value as training progresses.

The model associated with the first plot is performing quite well. The gap is quite narrowed.



I think the model associated with this second plot is doing pretty good, but not as well as the other. The gap is much broader. 



The model of the first plot was trained using a data set containing 1 million of ratings, while the second one used only 100K. I'm implementing the collaborative filtering (CF) algorithm. I am optimising it using SGD.

 

Are any of these models overfitting or underfitting?
I would say that your intuition is correct: the model associated with the first plot is likely to generalise more than the one associated with the second plot.

In both cases, it doesn't seem that your model has overfitted the training data. Overfitting often occurs when the training error keeps decreasing but the validation error starts to increase. In both your plots, both the training and validation errors keep decreasing (even if slowly, after a while).

Underfitting occurs when your model hasn't learned enough even about your training data. The smaller the training and validation error, the more likely your model has not underfitted, but the value of RMSE depends on the range of your inputs. See e.g. What are good RMSE values? for more info.

See also this article Overfitting and Underfitting With Machine Learning Algorithms for a general overview of the concepts of overfitting and underfitting.