With gradient descent w/MSE on a regression, must/should every Epoch use the exact same training samples?
Your goal in regression should be to obtain the factors which result in the best fit model without over-fitting.  The more data you have in the training set, the better your regression will be.  Thus you would want to train on the most data, but you also want to have some data held out to validate that your model is not over-fit.  So this is where you should have your data split into say a 80/20 training and validation set.  And if data is scarce or you want that 20% to contribute to the model then you could do a 5 fold cross validation.

In the spirit of research perhaps you should try both of these routes, and report your findings.