Can RNN learn for each `t` in time from a whole new dataset (many entries)
So, from what I understand you are trying to predict the target column given all your feature columns and the information from the past. This feels like a time-series forecasting problem to me too (You can check variations online).

Vanilla RNN tends to vanish or explode the gradients especially given that you have such a large number of timestamps. I'd suggest you to try an LSTM in order to avoid it as much as you can. Ideally yes, it is supposed to remember what happened earlier and learn from that.