Will my validation loss eventually go down? I'm currently training a binary classifier that takes in 2 inputs, and outputs which object it thinks is &quot;better.&quot;
I have an absolutely massive dataset, about 2 trillion records, and I'm feeding these records into my network about 300k records at a time. Overfitting isn't really a concern as I'm only running one epoch, so the network is only really seeing new data every cycle.
So far, the training loss is decreasing steadily, and my training accuracy is rising.
My validation loss is decreasing slightly, but fluctuating a lot. My validation accuracy is rising at about the same rate as my training accuracy is.
Will I eventually see a large drop in validation loss as the network gets more &quot;confident?&quot; I know that's a very hard question to answer without knowing more details, but have you seen this type of behavior before in your models? I'm just a little spooked since these experiments take up so much time to run.
When training neural network, you need to train many epochs in order for the model to learn enough to generalize. What is happening now the model is changing weights to fit the data that it observes (lower training loss) but does not generalize (as evidenced by high variance on validation data).
There are a couple of options:

Downsampling your data. Take only the most meaningful data to train.
Scale up training. Either use graphics processing units (GPUs) or distributed training.
Run training for longer.