title: TensorFlow: Restoring variables from two checkpoints after combining two graphs
body: <p>I got an issue with TensorFlow, hope you can help :D</p>
  
  <p>My goal: say I got two graphs, Graph A and Graph B, each one composed of three convolutional layers. What I want is to link the second layer of Graph A to the third layer of Graph B.</p>
  
  <p>I've successfully combined two graphs using the information given here: <a href="https://stackoverflow.com/questions/33748552/tensorflow-how-to-replace-a-node-in-a-calculation-graph">Tensorflow: How to replace a node in a calculation graph?</a></p>
  
  <p>This is what I got:
  <a href="https://i.stack.imgur.com/VDjdM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VDjdM.png" alt="enter image description here"></a></p>
  
  <p>The problem comes when I try to load variable weights. I've followed this solution, but didn't work: <a href="https://stackoverflow.com/questions/35733917/tensorflow-restoring-variables-from-from-multiple-checkpoints">TensorFlow: Restoring variables from from multiple checkpoints</a></p>
  
  <p>There is no problem when loading the weights of graph A, but this error arises <strong>when loading the weights of graph B</strong>:</p>
  
  <pre><code>INFO:tensorflow:Restoring parameters from ./models/merge_graph_A.ckpt
  INFO:tensorflow:Restoring parameters from ./models/merge_graph_B.ckpt
  ---------------------------------------------------------------------------
  NotFoundError                             Traceback (most recent call last)
  &lt;ipython-input-27-1e08224fde39&gt; in &lt;module&gt;()
        1 with tf.Session(graph=graphs_merged) as sess:
        2     saver_A.restore(sess, './models/merge_graph_A.ckpt')
  ----&gt; 3     saver_B.restore(sess, './models/merge_graph_B.ckpt')
        4 
        5     outcome = sess.run(output, feed_dict={
  
  /home/solano/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.pyc in restore(self, sess, save_path)
     1753     if context.in_graph_mode():
     1754       sess.run(self.saver_def.restore_op_name,
  -&gt; 1755                {self.saver_def.filename_tensor_name: save_path})
     1756     else:
     1757       self._build_eager(save_path, build_save=False, build_restore=True)
  
  /home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
      903     try:
      904       result = self._run(None, fetches, feed_dict, options_ptr,
  --&gt; 905                          run_metadata_ptr)
      906       if run_metadata:
      907         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
  
  /home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
     1135     if final_fetches or final_targets or (handle and feed_dict_tensor):
     1136       results = self._do_run(handle, final_targets, final_fetches,
  -&gt; 1137                              feed_dict_tensor, options, run_metadata)
     1138     else:
     1139       results = []
  
  /home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
     1353     if handle is None:
     1354       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
  -&gt; 1355                            options, run_metadata)
     1356     else:
     1357       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)
  
  /home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
     1372         except KeyError:
     1373           pass
  -&gt; 1374       raise type(e)(node_def, op, message)
     1375 
     1376   def _extend_graph(self):
  
  NotFoundError: Key Variables_A/W_c1 not found in checkpoint
       [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
  
  Caused by op u'save/RestoreV2', defined at:
    File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
      "__main__", fname, loader, pkg_name)
    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
      exec code in run_globals
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in &lt;module&gt;
      app.launch_new_instance()
    File "/home/solano/.local/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
      app.start()
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 486, in start
      self.io_loop.start()
    File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
      super(ZMQIOLoop, self).start()
    File "/home/solano/.local/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
      handler_func(fd_obj, events)
    File "/home/solano/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
      return fn(*args, **kwargs)
    File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
      self._handle_recv()
    File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
      self._run_callback(callback, msg)
    File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
      callback(*args, **kwargs)
    File "/home/solano/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
      return fn(*args, **kwargs)
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
      return self.dispatch_shell(stream, msg)
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
      handler(stream, idents, msg)
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
      user_expressions, allow_stdin)
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
      interactivity=interactivity, compiler=compiler, result=result)
    File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
      if self.run_code(code, result):
    File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "&lt;ipython-input-22-4397b861df66&gt;", line 5, in &lt;module&gt;
      return_elements=["conv_A_2/output:0"], name="")
    File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
      return func(*args, **kwargs)
    File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 553, in import_graph_def
      op_def=op_def)
    File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
      op_def=op_def)
    File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
      self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
  
  NotFoundError (see above for traceback): Key Variables_A/W_c1 not found in checkpoint
       [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
  </code></pre>
  
  <p>Something really interesting is that, <strong>if I changed the order of merging</strong>, taking the first two layers from B and the third layer from A, <strong>the error comes when loading the weights of graph A, just the opposite</strong>.</p>
  
  <hr>
  
  <p><strong>CODE</strong></p>
  
  <p>Loading graph A.</p>
  
  <pre><code>tf.reset_default_graph()
  graph_A = tf.Graph()
  
  with graph_A.as_default():
      saver_A = tf.train.import_meta_graph('./models/merge_graph_A.ckpt.meta')
  
  graph_A_def = graph_A.as_graph_def()
  </code></pre>
  
  <p>Loading Graph B.</p>
  
  <pre><code>tf.reset_default_graph()
  graph_B = tf.Graph()
  
  with graph_B.as_default():
      saver_B = tf.train.import_meta_graph('./models/merge_graph_B.ckpt.meta')
  
  graph_B_def = graph_B.as_graph_def()
  </code></pre>
  
  <p>Merging the two graphs:</p>
  
  <pre><code>tf.reset_default_graph() # not necessary
  with tf.Graph().as_default() as graphs_merged:
      input_image = tf.placeholder(tf.float32, (None, None, None, 3), name='input')
  
      bottleneck, = tf.import_graph_def(graph_A_def, input_map={"input_A:0": input_image},
                             return_elements=["conv_A_2/output:0"], name="")
  
      # Notice how the input is the bottleneck.
      output, = tf.import_graph_def(graph_B_def, input_map={"conv_B_2/output:0": bottleneck}, # HERE'S THE PROBLEM
                             return_elements=["conv_B_3/output:0"], name="")
  </code></pre>
  
  <p>Loading weights:</p>
  
  <pre><code>with tf.Session(graph=graphs_merged) as sess:
      saver_A.restore(sess, './models/merge_graph_A.ckpt')
      saver_B.restore(sess, './models/merge_graph_B.ckpt') # HERE'S THE PROBLEM
  
      outcome = sess.run(output, feed_dict={
          input_image: image_test
      })
  </code></pre>
  
  <p>Thanks everybody!!!</p>
  
  <hr>
  
  <p><em>EDIT: It's a problem of the saver. All checkpoints have been saved with the same prefix 'save'. This causes that, when merging, TensorFlow changes the prefix of the second graph to 'save_1', making impossible to the saver to find the variables it has to restore, trying instead to restore the variables using the operations prefixed with 'save'.</em></p>
  
  <p><strong>So the question changes</strong>: is there a way to change the name of this ops after importing the meta graph here: <code>saver_A = tf.train.import_meta_graph('./models/merge_graph_A.ckpt.meta')</code> ?</p>
  
  <pre><code>save/Const
  save/SaveV2/tensor_names
  save/SaveV2/shape_and_slices
  save/SaveV2
  save/control_dependency
  save/RestoreV2/tensor_names
  save/RestoreV2/shape_and_slices
  save/RestoreV2
  save/Assign
  save/Assign_1
  save/Assign_2
  save/Assign_3
  save/Assign_4
  save/Assign_5
  save/restore_all
  
  save_1/Const
  save_1/SaveV2/tensor_names
  save_1/SaveV2/shape_and_slices
  save_1/SaveV2
  save_1/control_dependency
  save_1/RestoreV2/tensor_names
  save_1/RestoreV2/shape_and_slices
  save_1/RestoreV2
  save_1/Assign
  save_1/Assign_1
  save_1/Assign_2
  save_1/Assign_3
  save_1/Assign_4
  save_1/Assign_5
  save_1/restore_all
  </code></pre>