How to train a model using a daunting huge training dataset
Quite often with massive datasets the model doesn't actually need the whole data. So I think the first step is to check whether using the whole data is useful: run an ablation study where you use say 1%, then 2%, 3%, .., up to say 10% of the data (adapt the levels to your case of course). Each run consists in training on the x % subset and evaluating on a validation set (be sure to separate your real final test set before anything, this study should only use a validation set).
The goal is to estimate how much gain in performance is obtained by adding training data. Plotting the performance as a function of the amount of data should give a decent idea of the trend, even if it doesn't reach the point of maximal performance, i.e. when more data doesn't improve performance anymore. With this information you can make a better decision about how to proceed with the real training.