title: Reloading from checkpoing during AWS Sagemaker Training
body: <p>Sagemaker is a great tool to train your models, and we save some money by using AWS spot instances. However, training jobs sometimes get stopped in the middle. We are using some mechanisms to continue from the latest checkpoint after a restart. See also the <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html" rel="nofollow noreferrer">docs</a>.</p>
  <p>Still, how do you efficiently test such a mechanism? Can you trigger it yourself? Otherwise you have to wait until the spot instance actually ís restarted.</p>
  <p>Also, are you expected to use the linked <code>checkpoint_s3_uri</code> argument or the <code>model_dir</code> for this? E.g. the <code>TensorFlow</code> estimator <a href="https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator" rel="nofollow noreferrer">docs</a> seem to suggest something <code>model_dir</code>for checkpoints.</p>