How to train a model when there is too much training data?
Option 1 is far superior, supposing the data has the same quality throughout.
The reason being is that it helps avoid overfitting. One of the biggest issues in ML is that you have a limited amount of data to train on, and the networks tend to overfit that is learn the specific of the set of data instead of generalizing to the problem intended.
Never repeating the data is actually the ideal situation for training.
The only caveat being is if you have the labels very accurately set in certain parts and more sloppily done in others - in that case sticking with the better quality data might be better.
I have an RNN and way too much training data. Iterating through the data in its entirety will take years. Right now I have two options for training:
1. Do one pass over as much data as possible
2. Find a select portion of the data and train for multiple epochs on it
Which one is better and why?