TensorFlow - Resume training in middle of an epoch?
The network doesn't store its training progress with respect to training data - this is not part of its state, because at any point you could decide to change what data set to feed it. You could maybe modify it so that it knew about the training data and progress, stored in some tensor somewhere, but that would be unusual. So, in order to do this, you will need to save and make use of additional data outside of the TensorFlow framework.

Probably the simplest thing to do is add the epoch number to the filename. You are already adding the current step within the epoch, so just add in the epoch multiplied:

saver.save(sess, 'my-model', global_step=epoch*1000000+step)


When you load the file, you can parse the filename to discover what epoch and step you were on and use those as the start point for the xrange functions. To make this easier to re-start from any given checkpoint, you could use argparse to allow your script to take the name of the checkpoint file you want to use.

In brief, it might look like this:

# Near top of script
import argparse
import re

# Before main logic
parser = argparse.ArgumentParser()
parser.add_argument('checkpoint')
args = parser.parse_args()

start_epoch = 0
start_step = 0
if args.checkpoint:
    saver.restore(sess, tf.train.latest_checkpoint(args.checkpoint))
    found_num = re.search(r'\d+', args.checkpoint)
    if found_num:
        checkpoint_id = int(found_num.group(0))
        start_epoch = checkpoint_id // 1000000
        start_step = checkpoint_id % 1000000

# Change to xrange:
for epoch in xrange(start_epoch, 25):
    for step in xrange(start_step, 1000000):
        sess.run(..training_op..) # etc

    # At end of epoch loop, you need to re-set steps:
    start_step = 0


You may want to reduce the number of checkpoints you are creating - as it stands you would have 25,000 checkpoint files generated by your code.

Another option would be use a single checkpoint file, and  to save and restore a Python pickle of a simple dict containing the state at the time you made the checkpoint, with a similar name.