How to resume training of a model?
With tensorflow, currently the most straightforward and easy way to get persistence for your model is to use a tf.train.MonitoredTrainingSession. You just need to use it instead the normal tf.Session() that is frequently used. This an illustrative Python snippet:



with tf.train.MonitoredTrainingSession(checkpoint_dir='/tmp/mymodel',
                                       save_summaries_secs=600) as sess:
   _ = sess.run(train_op, feed_dict={...})


With this, your model is automagically saved every 600 secs in /tmp/mymodel and restored the next time you restart the program.