How to solve RuntimeError: CUDA out of memory?
I try to run an inference using a cli to get the predictions from a detection and recognition model. With cuda10.2 it takes 15 mins for the inference to complete but I have cuda11.3 which takes 3 hours, I want to reduce this time.
Note : My hardware does not support cuda10.2.
hence I have following packages installed,

cudatoolkit  11.3.1  h2bc3f7f_2
pytorch  1.10.0  py3.7_cuda11.3_cudnn8.2.0_0  pytorch
torchvision  0.11.0  py37_cu113 pytorch

I get this error while I run the inference cli,
RuntimeError: CUDA out of memory. Tried to allocate 2.05 GiB (GPU 0; 5.81 GiB total capacity; 2.36 GiB already allocated; 1.61 GiB free; 2.38 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Tried :

To change the batch_size both for detection and recognition

Kindly help!
Thank you.

I use the following solutions whenever I encounter &quot;CUDA out of memory&quot; error. Here are the solutions, from simple to hard:
1- Try to reduce the batch size. First, train the model on each datum (batch_size=1) to save time. If it works without error, you can try a higher batch size but if it does not work, you should look to find another solution.
2- Try to use a different optimizer since some optimizers require less memory than others. For instance, SGD requires less memory than Adam.
3- Try to use a simpler model with fewer parameters.
4- Try to divide the model into two (or more than two) separate parts. Then update each part's parameters separately in each epoch. Note that whenever you want to compute gradients and update parameters of one part, the parameters of the other part of the model should be frozen. This leads to a lower amount of RAM required in each step which may solve the problem.
5- Lastly, if none of the above solutions work, GPU computation cannot be used.
Note that you can use a combination of these solutions which may lead to preventing the error. For instance, perhaps using a smaller batch size and simpler optimizer may work in some situations.
