How to take and restore snapshots of model training on another VM in Google Colab?
There is a 12 hour time limit for training DL models on GPU, according to google colab. Other people have had similar questions in the past, but there has been no clear answer on how to save and load models halfway through training when the 12 hour limits get exceeded, including saving the number of epochs that has been completed/saving other parameters. Is there an automated script for me to save the relevant parameters and resume operations on another VM? I am a complete noob; clear cut answers will be much appreciated.

As far as I know, there is no way to automatically reconnect to another VM whenever you reach the 12 hours limit. So in any case, you have to manually reconnect when the time is up.

As Bob Smith points out, you can mount Google Drive in Colab VM so that you can save and load data from there. In particular, you can periodically save model checkpoints so that you can load the most recent one whenever you connect to a new Colab VM.


Mount Drive in your Colab VM:

from google.colab import drive
drive.mount('/content/gdrive')

Create a saver in your graph:

saver = tf.train.Saver()

Periodically (e.g. every epoch) save a checkpoint in Drive:

saver.save(session, CHECKPOINT_PATH)



When you connect to a new Colab VM (because of the timeout), mount Drive again in your VM and restore the most recent checkpoint before the training phase:

saver.restore(session, CHECKPOINT_PATH)
...
# Start training with the restored model.


Take a look at the documentation to read more about tf.train.Saver.
