In PyTorch, why does the sequence length need to be provided as the first dimension of the input tensor for an RNN? I am confused as to why the sequence length is the first dimension of the input tensor for an RNN, while the batch size is the first dimension for any other kind of network (linear, CNN, etc.).
This makes me think that I haven't fully grasped the concept of RNN batches. Is each independent batch a different sequence? And is the same hidden state across batches? Is the hidden state maintained between timesteps for a given sequence (for vanilla/truncated BPTT)?
As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN.
Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.
To answer the second part of your question, normally, each sequence in a batch is independent of the others (since they commonly get sampled at random). So, there is no direct dependence between any two inputs in a batch (except, of course, for the fact that they are commonly expected to stem from some underlying shared data generating process which you want to approximate by the RNN).
And a hidden state is commonly maintained per batch element, i.e. there is one hidden state per batch element (i.e. per sequence).