title: When does Tensorflow update weights and biases?
body: <p>When does tensorflow update weights and biases in the for loop?</p>
  
  <p>Below is the code from tf's github. <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py" rel="noreferrer">mnist_softmax.py</a></p>
  
  <pre><code>for _ in range(1000):
      batch_xs, batch_ys = mnist.train.next_batch(100)
      sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
  </code></pre>
  
  <ol>
  <li>When does tensorflow update weights and biases?</li>
  <li>Does it update them when running <code>sess.run()</code>? If so, Does it mean, in this program, tf update weights and biases 1000 times?</li>
  <li>Or does it update them after finishing the whole for loop?</li>
  <li>If 2. is correct, my next question is, does tf update the model using different training data every time (since it uses next_batch(100). There are 1000*100 training data points in total. But all data points are considered only once individually. Am I correct or did I misunderstand something?</li>
  <li>If 3. is correct, is it weird that after just one update step the model had been trained?
  I think I must be misunderstanding something, It would be really great  if anyone can give me a hint or refer some material.</li>
  </ol>