title: How to correctly give inputs to Embedding, LSTM and Linear layers in PyTorch?
body: <p>I need some clarity on how to correctly prepare inputs for batch-training using different components of the <code>torch.nn</code> module. Specifically, I'm looking to create an encoder-decoder network for a seq2seq model.</p>
  
  <p>Suppose I have a module with these three layers, in order:</p>
  
  <ol>
  <li><code>nn.Embedding</code></li>
  <li><code>nn.LSTM</code></li>
  <li><code>nn.Linear</code></li>
  </ol>
  
  <h1><code>nn.Embedding</code></h1>
  
  <p><strong>Input:</strong> <code>batch_size * seq_length</code><br>
  <strong>Output:</strong> <code>batch_size * seq_length * embedding_dimension</code></p>
  
  <p>I don't have any problems here, I just want to be explicit about the expected shape of the input and output.</p>
  
  <h1><code>nn.LSTM</code></h1>
  
  <p><strong>Input:</strong> <code>seq_length * batch_size * input_size</code> (<code>embedding_dimension</code> in this case)<br>
  <strong>Output:</strong> <code>seq_length * batch_size * hidden_size</code><br>
  <strong>last_hidden_state:</strong> <code>batch_size * hidden_size</code><br>
  <strong>last_cell_state:</strong> <code>batch_size * hidden_size</code></p>
  
  <p>To use the output of the <code>Embedding</code> layer as input for the <code>LSTM</code> layer, I need to transpose axis 1 and 2.</p>
  
  <p>Many examples I've found online do something like <code>x = embeds.view(len(sentence), self.batch_size , -1)</code>, but that confuses me. How does this view ensure that elements of the same batch remain in the same batch? What happens when <code>len(sentence)</code> and <code>self.batch</code> size are of same size?</p>
  
  <h1><code>nn.Linear</code></h1>
  
  <p><strong>Input:</strong> <code>batch_size</code> x <code>input_size</code> (hidden_size of LSTM in this case or ??)<br>
  <strong>Output:</strong> <code>batch_size</code> x <code>output_size</code></p>
  
  <p>If I only need the <code>last_hidden_state</code> of <code>LSTM</code>, then I can give it as input to <code>nn.Linear</code>.</p>
  
  <p>But if I want to make use of Output (which contains all intermediate hidden states as well) then I need to change <code>nn.Linear</code>'s input size to <code>seq_length * hidden_size</code> and to use Output as input to <code>Linear</code> module I need to transpose axis 1 and 2 of output and then I can view with <code>Output_transposed(batch_size, -1)</code>.</p>
  
  <p>Is my understanding here correct? How do I carry out these transpose operations in tensors <code>(tensor.transpose(0, 1))</code>?</p>