title: Save, Recover, and Continue Updating Learning Curves while Training a CNN if Server Crashes Suddenly
body: <p>I am training a deep learning model with TensorFlow on a remote server. The problem is that I am only allocated 2 hours of training at a time and the server may crash at any points for various reason.</p>
  <p>I know the training of my model will take me at least 48 hours to complete. I would like to be able after the model is fully trained (48+ hours) to display the training curves from start to finish with no breaks in between.</p>
  <p>I am able to pick wherever the training was when it last crashes using callbacks (save best weights) but I am unsure how to achieve this with the training curves (loss + accuracy).</p>
  <p>Thank you very much for your help.</p>