How to test dev set on Time Series data via forecasting
I feel your frustration. I am not sure if this is a complete answer, but I’ll try to add some value here.

Disclaimer: I am not familiar with the neural nets you mentioned but have used other times series nns. So my answers will be general.

First, there is nothing specifically wrong about retraining every day unless you are doing so at the expense of the number of training epochs. For example, you could conceivably need to train for 20k epochs to get the best predictive power. If this takes 20 hours, it may not feasible. So you may be tempted to train for 5000 epochs. This is probably a reason not to retrain every day. My recommendation would be to invest in more epochs and retrain less frequently. Hopefully the resulting model is more generalized and powerful in the long run.

If you DO still want to train every day, I would load the weights and biases from the previous training session and continue training for a smaller number number of epochs. Start with as many epochs as you can to produce the first model. Then add the additional row and continue training for a manageable number of epochs. TBH, one extra row should not really change your model weights and biases significantly unless the new y is pretty extreme. In fact, I doubt it would even be obvious.

Now, why does your accuracy drop with the new data? Several ideas:


Just stating the obvious. In case it’s not. You will need many training epochs. Each application is different, but sometimes people only train for a few epochs and don’t understand why the model sucks. You might need 200 epochs, you might need  2000000.
You are overfitting and the addition of the new data is generalizing the model a little more each day. Doubtful, but possible.
You are reinitializing your model with random weights and biases and do not have a seed set. So your model could be finding different local minimums each time you train. I would be surprised if it was always in the negative, but it may be that you have just flipped heads 25 times in a row. I would set a seed when initialize to take this out of the picture.


Hopefully there’s something of use in here.