title: mini-batch gradient descent implementation in tensorflow
body: <p>When reading an tensorflow implementation for a deep learning model, I am trying to understand the following code segment included in the training process. </p>
  
  <pre><code>self.net.gradients_node = tf.gradients(loss, self.variables)
  for epoch in range(epochs):
              total_loss = 0
              for step in range((epoch*training_iters), ((epoch+1)*training_iters)):
                  batch_x, batch_y = data_provider(self.batch_size)
  
                  # Run optimization op (backprop)
                  _, loss, lr, gradients = sess.run((self.optimizer, self.net.cost, self.learning_rate_node, self.net.gradients_node), 
                                                    feed_dict={self.net.x: batch_x,
                                                               self.net.y: util.crop_to_shape(batch_y, pred_shape),
                                                               self.net.keep_prob: dropout})
  
                  if avg_gradients is None:
                      avg_gradients = [np.zeros_like(gradient) for gradient in gradients]
                  for i in range(len(gradients)):
                      avg_gradients[i] = (avg_gradients[i] * (1.0 - (1.0 / (step+1)))) + (gradients[i] / (step+1))
  
                  norm_gradients = [np.linalg.norm(gradient) for gradient in avg_gradients]
                  self.norm_gradients_node.assign(norm_gradients).eval()
  
  
  
                  total_loss += loss
  </code></pre>
  
  <p>I think it is related to mini-batch gradient descent, but I cannot understand how does it work, or I have some difficulties to connect it to the algorithm shown as follows</p>
  
  <p><a href="https://i.stack.imgur.com/h407Z.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/h407Z.jpg" alt="enter image description here"></a></p>