title: Fine-Tuning the Inception model in TensorFlow
body: <p>I want to use the pre-trained Inception model on my own data-set AND I also want to fine-tune the variables of the Inception model itself.</p>
  
  <p>I have downloaded the pre-trained Inception model for TensorFlow from the following link:</p>
  
  <p><a href="http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz" rel="nofollow">http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz</a></p>
  
  <p>I load the Inception model as follows:</p>
  
  <pre><code>graph = tf.Graph()
  with graph.as_default():
      with tf.gfile.FastGFile('classify_image_graph_def.pb', 'rb') as file:
          graph_def = tf.GraphDef()
          graph_def.ParseFromString(file.read())
          tf.import_graph_def(graph_def, name='')
  </code></pre>
  
  <p>(Side note on the API: It would be nice if I could just write <code>graph = tf.load_graph('inception.pb')</code> instead of these six nested and complicated lines.)</p>
  
  <p>Then I get a reference to the tensor for the last layer before the softmax-classifier in the Inception model:</p>
  
  <pre><code>last_layer = graph.get_tensor_by_name('pool_3:0')
  </code></pre>
  
  <p>Now I want to append a new softmax-classifier to the graph so I can train the new softmax-classifier AND train some or all of the variables in the Inception model. This is what I understand to be fine-tuning, as opposed to transfer learning where only the new softmax-classifier is trained on my own data-set.</p>
  
  <p>I then use PrettyTensor to append the new softmax-classifier (note that <code>y_true</code> is a placeholder variable):</p>
  
  <pre><code>with pt.defaults_scope(activation_fn=tf.nn.relu):
      y_pred, loss = pt.wrap(last_layer).\
          flatten().\
          softmax_classifier(class_count=10, labels=y_true)
  </code></pre>
  
  <p>But this gives a long error-message where the last part reads:</p>
  
  <pre><code>ValueError: Tensor("flatten/reshape/Const:0", shape=(2,), dtype=int32) must be from the same graph as Tensor("pool_3:0", shape=(1, 1, 1, 2048), dtype=float32).
  </code></pre>
  
  <p>So I am apparently not allowed to combine two graphs like this.</p>
  
  <p>I have also tried using a <code>reshape()</code> instead of <code>flatten()</code> as follows (note the last layer of the Inception model has 2048 features):</p>
  
  <pre><code>with pt.defaults_scope(activation_fn=tf.nn.relu):
      y_pred, loss = pt.wrap(last_layer).\
          reshape([-1, 2048]).\
          softmax_classifier(class_count=10, labels=y_true)
  </code></pre>
  
  <p>But this gives almost the same error:</p>
  
  <pre><code>ValueError: Tensor("reshape/Const:0", shape=(2,), dtype=int32) must be from the same graph as Tensor("pool_3:0", shape=(1, 1, 1, 2048), dtype=float32).
  </code></pre>
  
  <p>I have also tried wrapping it in a <code>graph.as_default()</code> like so:</p>
  
  <pre><code>with graph.as_default():
      with pt.defaults_scope(activation_fn=tf.nn.relu):
          y_pred, loss = pt.wrap(last_layer).\
              reshape([-1, 2048]).\
              softmax_classifier(class_count=10, labels=y_true)
  </code></pre>
  
  <p>But this gives a similar error:</p>
  
  <pre><code>ValueError: Tensor("ArgMax_1:0", shape=(?,), dtype=int64) must be from the same graph as Tensor("cross_entropy/ArgMax:0", shape=(1,), dtype=int64).
  </code></pre>
  
  <p>How would I do fine-tuning of the Inception model? I want to add a new softmax-classifier AND I want to fine-tune some or all of the variables in the Inception model itself.</p>
  
  <p>Thanks!</p>
  
  <hr>
  
  <p>EDIT:</p>
  
  <p>I have a partial solution to the problem.</p>
  
  <p>The error messages were because I didn't put all the code inside the <code>with graph.as_default():</code> block. Putting all the code inside that block fixes the error-messages and I can now append a new softmax-layer to the Inception model using PrettyTensor, as described above.</p>
  
  <p>However, the Inception model is apparently a 'frozen' graph which means that all variables have been converted to constants before it was saved.</p>
  
  <p>So my question is now, whether I can somehow 'unfreeze' the graph for the Inception model, so I can continue training some or all of the variables of its graph? How would I do this?</p>
  
  <p>Or should I instead use the new MetaGraph functionality?</p>
  
  <p><a href="https://www.tensorflow.org/versions/r0.11/how_tos/meta_graph/index.html" rel="nofollow">https://www.tensorflow.org/versions/r0.11/how_tos/meta_graph/index.html</a></p>
  
  <p>Where can I download a pre-trained MetaGraph for the Inception model?</p>