How is batching normally performed for sequence data for an RNN/LSTM
This Udacity course notebook batches data in a way that is not intuitive to me.
For a long sequence of data, they first truncates the data so it can be evenly divided by batch_size. Next, they .reshape() the data to be (batch_size, -1). Then they creates a sliding window over these batches of sub-sequences. When the sliding window goes out of bounds, they add fake data (via wrap-around to) to the end.
This provided graphic might explain better than I can:

I'm just wondering if this practice is normal, or if there is a different way. It seems strange that the batches are non-consecutive sub-sequences. Wouldn't that make it hard to interpret the output of a single batch?
Is there a better approach? The woman in the video literally said something to the affect &quot;I don't know why it's done this way, but I've seen it before and the network trains fine&quot;.

You should check the documentation on padded sequences from pytorch. (If I had more experience with it I would give you a more detailed explanation, but truth if that I never really understood them!)
Packed Sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.PackedSequence.html#torch.nn.utils.rnn.PackedSequence
Pack padded sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_padded_sequence.html#torch.nn.utils.rnn.pack_padded_sequence
Pad packed sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_packed_sequence.html#torch.nn.utils.rnn.pad_packed_sequence
Pad sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pad_sequence.html#torch.nn.utils.rnn.pad_sequence
Pack sequence:
https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_sequence.html#torch.nn.utils.rnn.pack_sequence
The names are a bit confusing. But the idea is that you create a tensor with the size of your largest sequence in the batch. The other sequences will be padded to have the same size as the longest sequence in the bach. This packed padded sequence is given to the recurrent model (RNN, LTMS, GRU, your favorite). With that you can back arbitrary sequences with minor memory limitations.
