Should I prefer the model with the lowest validation loss or the highest validation accuracy to deploy?
Okay, I think it's better if we distinguish loss and accuracy first via Jeremy's answer, and I agree with him with the sentence &quot;low or huge loss is a subjective metric&quot;.
The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand correctly what you need from your model.
For example, with Cifar 10, and you need the more correct label the better, you can believe in accuracy. However, if you want your model to make sure its result is the correct, area under receiver operating characteristic curve (AUROC) maybe the better choice.
For example, classification problem with 3 class, correct label y = 1:

Good accuracy, bad AUROC: the output probability from softmax [0.3,0.4,0.3]
Good accuracy, good AUROC: [0.1,0.8,0.1]

And with imbalanced dataset, Precision, Recall and F1-score will be more suitable.