title: Tensorflow: how it trains the model?
body: <p>Working on Tensorflow, the first step is build a data graph and use session to run it. While, during my practice, such as the <a href="https://www.tensorflow.org/versions/r0.9/tutorials/mnist/beginners/index.html#Training" rel="nofollow">MNIST tutorial</a>. It firstly defines <em>loss</em> function and the <em>optimizer</em>, with the following codes (and the MLP model is defined before that):</p>
  
  <pre><code>cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) #define cross entropy error function
  
  loss = tf.reduce_mean(cross_entropy, name='xentropy_mean') #define loss
  
  optimizer = tf.train.GradientDescentOptimizer(learning_rate) #define optimizer
  
  global_step = tf.Variable(0, name='global_step', trainable=False) #learning rate
  
  train_op = optimizer.minimize(loss, global_step=global_step) #train operation in the graph
  </code></pre>
  
  <p><strong>The training process:</strong></p>
  
  <pre><code>train_step =tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
  
  
  for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
  </code></pre>
  
  <p>That's how Tensorflow did training in this case. But my question is, how did Tensorflow know which weight it needs to train and update? I mean, in the training codes, we only pass output <code>y</code> to <code>cross_entropy</code>, but for <code>optimizer</code> or <code>loss</code>, we didn't pass any information about the structure <strong>directly</strong>. In addition, we use dictionary to feed batch data to <code>train_step</code>, but <code>train_step</code> didn't directly use the data. <strong>How did Tensorflow know where to use these data as input?</strong></p>
  
  <p>To my question, I thought it might be all those variables or constants are stored in <strong>Tensor</strong>. Operations such as <code>tf.matmul()</code> should a "subclass" of Tensorflow's operation class(I haven't check the code yet). There might be some mechanism for Tensorflow to recognise relations among tensors (<code>tf.Variable()</code>, <code>tf.constant()</code>) and operations (<code>tf.mul()</code>, <code>tf.div()</code>...). I guess, it could check the <code>tf.xxxx()</code>'s super class to find out whether it is a tensor or operation. This assumption raises <strong>my second question</strong>: should I use Tensorflow's 'tf.xxx' function as possible to ensure tensorflow could build correct data flow graph, even sometimes it is more complicated than normal Python methods or some functions are supported better in Numpy than Tensorflow? </p>
  
  <p><strong>My last question</strong> is: Is there any link between Tensorflow and C++? I heard someone said Tensorflow is faster than normal Python since it uses C or C++ as backend. Is there any transform mechanism to transfer Tensorflow Python codes to C/C++?</p>
  
  <p>I'd also be graceful if someone could share some debugging habits in coding with Tensorflow, since currently I just set up some terminals (Ubuntu) to test each part/functions of my codes.</p>