title: Using two tensorflow models where one is for inference and another is for training
body: <p>I'm new to tensorflow and I'm trying to combine two models in one graph because I need one model's inference result to modify the loss function of the other model. I wrote the code and it runs without errors but I'm not sure whether I wrote it correctly so I'm writing this thread.</p>
  
  <p>In the code, I loaded two graphs like this</p>
  
  <pre><code>with tf.variable_scope("modelA"):
    new_saver = tf.train.import_meta_graph('modelA-1000.meta')
    new_saver.restore(sess, tf.train.latest_checkpoint('./'))
  
  with tf.variable_scope("modelB"):
    new_saver = tf.train.import_meta_graph('modelB-1000.meta')
    new_saver.restore(sess, tf.train.latest_checkpoint('./'))
  </code></pre>
  
  <p>and I used modelA's result to modify modelB's loss funcion as follows</p>
  
  <pre><code>output_A = tf.get_default_graph().get_tensor_by_name("modelA_output:0")
  output_B = tf.get_default_graph().get_tensor_by_name("modelB_output:0")
  loss = tf.reduce_mean(-tf.reduce_sum(output_A * tf.log(output_B ), reduction_indices=[1])) 
  </code></pre>
  
  <p>Then for training, I included only modelB variables to train since I want to make Model A for inference only.</p>
  
  <pre><code>model_vars = tf.trainable_variables()
  var_B = [var for var in model_vars if 'modelB' in var.name]
  gradient = tf.gradients(loss,var_B)
  trainer = tf.train.GradientDescentOptimizer(0.1)
  train_step = trainer.apply_gradients(zip(gradient ,var_B))
  
  ... declare session and prepare batch for training ...
  
  for i in range(10000):
      loss_ = train_step.run(loss, feed_dict={x: batch[0]})
  </code></pre>
  
  <p>I ran it and the code runs but the loss does not decrease. What did I do wrong? Thanks for reading!</p>