title: Torch / Lua, which neural network structure for mini-batch training?
body: <p>I'm still working on implementing the mini-batch gradient update on my siamese neural network. Previously I had an implementation problem, that was <a href="https://stackoverflow.com/questions/31324633/torch-lua-how-to-correctly-implement-minibatch-training-in-a-siamese-neural-n">correctly solved here</a>. </p>
  
  <p>Now I realized that there was also a mistake in the architecture of my neural network, that is related to my incomplete understanding of the correct implementation.</p>
  
  <p>So far, I've always used a non-minibatch gradient descent approach, in which I was passing the training elements one by one to the gradient update. Now, I want to implement a gradient update through mini-batch, starting say with minibatches made of N=2 elements.</p>
  
  <p><strong>My question is: how should I change the architecture of my siamese neural network to make it able to handle a mini-batch of N=2 elements instead of a single element?</strong></p>
  
  <p>This is the (simplified) architecture of my siamese neural network:</p>
  
  <pre><code>nn.Sequential {
    [input -&gt; (1) -&gt; (2) -&gt; output]
    (1): nn.ParallelTable {
      input
        |`-&gt; (1): nn.Sequential {
        |      [input -&gt; (1) -&gt; (2) -&gt; output]
        |      (1): nn.Linear(6 -&gt; 3)
        |      (2): nn.Linear(3 -&gt; 2)
        |    }
        |`-&gt; (2): nn.Sequential {
        |      [input -&gt; (1) -&gt; (2) -&gt; output]
        |      (1): nn.Linear(6 -&gt; 3)
        |      (2): nn.Linear(3 -&gt; 2)
        |    }
         ... -&gt; output
    }
    (2): nn.CosineDistance
  }
  </code></pre>
  
  <p>I have:</p>
  
  <ul>
  <li>2 identical siamese neural networks (upper and lower)</li>
  <li>6 input units</li>
  <li>3 hidden units</li>
  <li>2 output units</li>
  <li>cosine distance function that compares the output of the two parallel neural networks</li>
  </ul>
  
  <p>Here's my code:</p>
  
  <pre><code>perceptronUpper= nn.Sequential()
  perceptronUpper:add(nn.Linear(input_number, hiddenUnits))
  perceptronUpper:add(nn.Linear(hiddenUnits,output_number))
  perceptronLower= perceptronUpper:clone('weight', 'gradWeights', 'gradBias', 
  'bias')
  
  parallel_table = nn.ParallelTable()
  parallel_table:add(perceptronUpper)
  parallel_table:add(perceptronLower)
  
  perceptron = nn.Sequential()
  perceptron:add(parallel_table)
  perceptron:add(nn.CosineDistance())
  </code></pre>
  
  <p>This architecture works very well if I have a gradient update function that takes 1 element; how should modify it to let it manage a <strong>minibatch</strong>?</p>
  
  <p><strong>EDIT:</strong> I probably should use the <a href="https://github.com/Element-Research/rnn/blob/master/Sequencer.lua" rel="nofollow noreferrer">nn.Sequencer() class</a>, by modifying the last two lines of my code in: </p>
  
  <pre><code>perceptron:add(nn.Sequencer(parallel_table))
  perceptron:add(nn.Sequencer(nn.CosineDistance())).
  </code></pre>
  
  <p>What do you guys think?</p>