Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps
The second batch should be [t=25,t=49] because a Stateful LSTM must be sequential, because as you rightly say, the state of the network is at the state of t=24, if you then feed t=2 (or anything other than t=25) then the sequencing is messed up.

With stateless LSTM you can create sequences from any starting point, but with stateless LSTM the sequence must be observed.

In subsequent epochs you could change your start index so that different sequences are inputted. Nothing wrong with that. I personally generate a random number between 0 and the sequence length.