title: Tensorflow restore `tf.Session` saved checkpoint using `tf.train.MonitoredTrainingSession`
body: <p>I have code for training a CNN using <code>tf.train.MonitoredTrainingSession</code>.</p>
  
  <p>When I create a new <code>tf.train.MonitoredTrainingSession</code> I can pass the <code>checkpoint</code> directory as an input parameter to the session and it will automatically restore the latest saved <code>checkpoint</code> it could find. And I can set up the <code>hooks</code> to train until some step. For example, if the <code>checkpoint</code>'s step is <code>150,000</code> and I would like to train until <code>200,000</code> I will put the <code>last_step</code> to <code>200,000</code>. </p>
  
  <p>The above process works perfectly as long as the latest <code>checkpoint</code> was saved using a <code>tf.train.MonitoredTrainingSession</code>. However, if I try to restore a <code>checkpoint</code> that was saved using a normal <code>tf.Session</code> then all hell breaks loose. It can't find some keys in the graph and all.</p>
  
  <p>The training is done with this:</p>
  
  <pre><code>with tf.train.MonitoredTrainingSession(
      checkpoint_dir=FLAGS.retrain_dir,
      hooks=[tf.train.StopAtStepHook(last_step=FLAGS.max_training_steps),
             tf.train.NanTensorHook(loss),
             _LoggerHook()],
      config=tf.ConfigProto(
          log_device_placement=FLAGS.log_device_placement)) as mon_sess:
    while not mon_sess.should_stop():
      mon_sess.run(train_op)
  </code></pre>
  
  <p>If the <code>checkpoint_dir</code> attribute had a folder with no checkpoints, this will start all over. If it had a <code>checkpoint</code> that was saved from a previous training session, it will restore the latest <code>checkpoint</code> and will continue training.</p>
  
  <p>Now, I am restoring the latest <code>checkpoint</code> and modifying some variables and saving them:</p>
  
  <pre><code>saver = tf.train.Saver(variables_to_restore)
  
  ckpt = tf.train.get_checkpoint_state(FLAGS.train_dir)
  
  with tf.Session() as sess:
    if ckpt and ckpt.model_checkpoint_path:
      # Restores from checkpoint
      saver.restore(sess, ckpt.model_checkpoint_path)
      print(ckpt.model_checkpoint_path)
      restored_step = ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1]
      FLAGS.max_training_steps = int(restored_step) + FLAGS.max_training_steps
    else:
      print('No checkpoint file found')
      return
  
    prune_convs(sess)
    saver.save(sess, FLAGS.retrain_dir+"model.ckpt-"+restored_step)
  </code></pre>
  
  <p>As you can see, just before <code>saver.save...</code> I am pruning all convolutional layers in the network. No need to describe how and why is that done. The point is that the network is in fact modified. Then I save the network to a <code>checkpoint</code>.</p>
  
  <p>Now, if I deploy test on the saved modified network, the test works just fine. However, when I try to run the <code>tf.train.MonitoredTrainingSession</code> on the <code>checkpoint</code> that was saved, it says:</p>
  
  <blockquote>
    <p>Key conv1/weight_loss/avg not found in checkpoint</p>
  </blockquote>
  
  <p>Also, I have noticed that the <code>checkpoint</code> that was saved with <code>tf.Session</code> has half of the size of the <code>checkpoint</code> that was saved with <code>tf.train.MonitoredTrainingSession</code></p>
  
  <p>I know I'm doing it wrong, any suggestions how to make this work?</p>