title: Hidden size vs input size in RNN
body: <p><strong>Premise 1:</strong></p>
  
  <p>Regarding neurons in a RNN layer - it is my understanding that at "each time step, every neuron receives both the input vector x (t) and the output vector from the previous time step y (t –1)" <sup>[1]</sup>:</p>
  
  <p><img src="https://i.stack.imgur.com/dSxLU.png" alt="https://github.com/ebolotin6/ebolotin6.github.io/blob/master/images/rnn.png"></p>
  
  <p><strong>Premise 2:</strong></p>
  
  <p>It is also my understanding that in Pytorch's GRU layer, <em>input_size</em> and <em>hidden_size</em> mean the following:</p>
  
  <blockquote>
    <ul>
    <li><strong>input_size</strong> – The number of expected features in the input x</li>
    <li><strong>hidden_size</strong> – The number of features in the hidden state h</li>
    </ul>
  </blockquote>
  
  <p>So naturally, <em>hidden_size</em> should represent the number of neurons in a GRU layer.</p>
  
  <p><strong>My question:</strong></p>
  
  <p>Given the following GRU layer:</p>
  
  <pre><code># assume that hidden_size = 3
  
  class Encoder(nn.Module):
      def __init__(self, src_dictionary_size, hidden_size):
          super(Encoder, self).__init__()
          self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
          self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size)
  
  </code></pre>
  
  <p>Assuming a hidden_size of 3, my understanding is that the GRU layer above would have 3 neurons, each which accepts an input vector of size 3 simultaneously for every timestep. </p>
  
  <p><strong>My question is</strong>: why do the arguments to <em>hidden_size</em> and <em>input_size</em> have to be equal? I.e. why can't each of the 3 neurons accept say, an input vector of size 5?</p>
  
  <p>Case in point: both of the following produce size mismatch:</p>
  
  <pre><code>self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)
  self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size+1)
  </code></pre>
  
  <p>[1] Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn and TensorFlow (p. 388). O'Reilly Media. Kindle Edition. </p>
  
  <p>[3] <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.GRU" rel="noreferrer">https://pytorch.org/docs/stable/nn.html#torch.nn.GRU</a></p>
  
  <hr>
  
  <p>Adding full code for reproducibility:</p>
  
  <pre class="lang-py prettyprint-override"><code>import torch
  import torch.nn as nn
  
  class Encoder(nn.Module):
      def __init__(self, src_dictionary_size, hidden_size):
          super(Encoder, self).__init__()
          self.hidden_size = hidden_size
          self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
          self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)
  
      def forward(self, pad_seqs, seq_lengths, hidden):
          """
          Args:
            pad_seqs of shape (max_seq_length, batch_size, 1): Padded source sequences.
            seq_lengths: List of sequence lengths.
            hidden of shape (1, batch_size, hidden_size): Initial states of the GRU.
  
          Returns:
            outputs of shape (max_seq_length, batch_size, hidden_size): Padded outputs of GRU at every step.
            hidden of shape (1, batch_size, hidden_size): Updated states of the GRU.
          """
          embedded_sqs = self.embedding(pad_seqs).squeeze(2)
          packed_sqs = pack_padded_sequence(embedded_sqs, seq_lengths)
          packed_output, h_n = self.gru(packed_sqs, hidden)
          output, input_sizes = pad_packed_sequence(packed_output)
  
          return output, h_n
  
      def init_hidden(self, batch_size=1):
          return torch.zeros(1, batch_size, self.hidden_size)
  
  def test_Encoder_shapes():
      hidden_size = 5
      encoder = Encoder(src_dictionary_size=5, hidden_size=hidden_size)
  
      # maximum word count
      max_seq_length = 4
  
      # num sentences
      batch_size = 2
      hidden = encoder.init_hidden(batch_size=batch_size)
  
      # these are padded sequences (sentences of words). There are 2 sentences (i.e. 2 batches) with a maximum of 4 words.
      pad_seqs = torch.tensor([
          [1, 2],
          [2, 3],
          [3, 0],
          [4, 0]
      ]).view(max_seq_length, batch_size, 1)
  
      outputs, new_hidden = encoder.forward(pad_seqs=pad_seqs, seq_lengths=[4, 2], hidden=hidden)
      assert outputs.shape == torch.Size([4, batch_size, hidden_size]), f"Bad outputs.shape: {outputs.shape}"
      assert new_hidden.shape == torch.Size([1, batch_size, hidden_size]), f"Bad new_hidden.shape: {new_hidden.shape}"
      print('Success')
  
  test_Encoder_shapes()
  </code></pre>