The strange loss fluctuation when loading previous trained model
I am using PyTorch for deep learning now. 

I trained a model before and had the parameters saved. Loss values before the end of the training were about 0.003~0.006.

However, when I load the same model with the same training data, loss values at first fluctuate to around 0.5. 

The loss values then decrease very quickly to around 0.01 in ~10 iterations and now decreasing slowly.

Does anyone know why this situation keep happens? Since I am loading the same model/ training data. I was expecting the loss values would start at a similar level as the end of last training.

When resuming a training, you should not only load the network's weights but also the optimizer state. For that, you can use torch.save:

torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)


Then, for resuming the training:

model = TheModelClass(*args, **kwargs)
model.train()
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']


If you don't save the optimizer state, you loss important information such as the current learning rate, momentum etc. This is probably the cause of your problem.

Reference:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training
