title: mini batch backpropagation clarification
body: <p>Read through a lot of articles and now I am in my braindrain end, and need a fresh perspective on the concept of mini batch. I am new to machine learning and would appreciate any advice on whether my process is correct. Here is my premise:</p>
  <p>I have a dataset of 355 features and 8 class output. A total of 12200 data. This is a rough visualization of my neural network:
  <a href="https://i.stack.imgur.com/LsbRM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LsbRM.png" alt="Neural Network Sketch" /></a></p>
  <p>I decided on 181 neurons for Hidden layer 1, 96 neurons for Hidden Layer 2. I used ReLu activation for the hidden layers and Logistic for the output layer.</p>
  <p>To do the mini-batch, I set my batch size to 8. So I have a total of 1525 batch with 8 dataset per batch. Here is my step:</p>
  <ol>
  <li>Get 1st Batch of data (8 sets of 355 inputs and 8 outputs).</li>
  <li>Forward Propagation of the Batch.</li>
  <li>Get the Errors and calculate the sum of squares of error.
  For the sum of squares, I averaged the errors of the batch first, used the formula
  SumError = (1/8)*sum(error^2)</li>
  <li>Back Propagation of the Batch</li>
  <li>Get the average of the weights value after back propagation.</li>
  <li>Use the new weights as the weights for the next batch.</li>
  <li>Get next Batch of Data (8 sets of 355 inputs and 8 outputs).</li>
  <li>Repeat 2-7 using the new set of weights.</li>
  <li>When all batch is done, get the average of the SumError to get the Sum of Squares per epoch.</li>
  <li>Repeat 1-9 until SumError per epoch is small.</li>
  <li>Get the final weights to be used for validation</li>
  </ol>
  <p>That is the process of my mini batch. Is this correct? I mean, for the weights, do I use the weights calculated after each batch as the input weights for the next batch, or do I collect all the weights first (the starting weights will be used for all the batch), and then average the weights for all the batches? Then use the average weights as input to the next epoch?</p>