Evaluating pytorch models: `with torch.no_grad` vs `model.eval()`
If you're reading this post because you've been encountering RuntimeError: CUDA out of memory, then with torch.no grad(): will likely to help save the memory. Using only model.eval() is unlikely to help with the OOM error.
The reason for this is that torch.no grad() disables autograd completely (you can no longer backpropagate), reducing memory consumption and speeding up computations.
However, you will still be able to call the gardients when using model.eval(). Personally, I find this design decision intriguing. So, what is the purpose of .eval()? It seems its main functionality is to deactivate the Dropout during the evaluation time.
To summarize, if you use torch.no grad(), no intermediate tensors are saved, and you can possibly increase the batch size in your inference.
When I want to evaluate the performance of my model on the validation set, is it preferred to use with torch.no_grad: or model.eval()?