PULSE on github (link provided) RuntimeError: CUDA out of memory.... preventing the program "run.py" from executing
(As a student I am kind of new to this but did quite a bit of research and I got pretty far, I'm super into learning something new through this!)
This issue is for the project pulse -&gt; https://github.com/adamian98/pulse
the readme if you scroll down a bit on the page, gives a much better explanation than I could. It will also give a direct &quot;correct&quot; path to judge my actions against and make solving the problem a lot easier.
Objective: run program using the run.py file
Issue: I got a &quot;RuntimeError: CUDA out of memory&quot; despite having a compatible gpu and enough vram
Knowledge: when it comes to coding i just started a few days ago and have a dozen hours with anaconda now, comfterable creating environments.
What I did was... (the list below is a summary and the specific details are after it)

install anaconda

use this .yml file -&gt; https://github.com/leihuayi/pulse/blob/feature/docker/pulse.yml (it changes dependencies to work for windows which is why I needed to grab a different one than the one supplied on the master github page) to create a new environment and install the required packages. It worked fantastically! I only got an error trying to install dlib, it didn't seem compatible with A LOT of the packages and my python version.

I installed the cuda toolkit 10.2 , cmake 3.17.2, and tried to install dlib into the environment directly. the errors spat out in a blaze of glory. The dlib package seems to be only needed for a different .py file and not run.py though so I think it may be unrelated to this error


logs are below and I explain my process in more detail
START DETAILS AND LOGS: from here until the &quot;DETAILS 2&quot; section should be enough information to solve, the rest past there is in case
error log for runing out of memory--&gt; (after executing the &quot;run.py&quot; file)
Loading Synthesis Network
Loading Mapping Network
Running Mapping Network
Traceback (most recent call last):
File &quot;C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\run.py&quot;, line 58, in
model = PULSE(cache_dir=kwargs[&quot;cache_dir&quot;])
File &quot;C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\PULSE.py&quot;, line 44, in init
latent_out = torch.nn.LeakyReLU(5)(mapping(latent))
File &quot;C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py&quot;, line 550, in call
result = self.forward(*input, **kwargs)
File &quot;C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py&quot;, line 233, in forward
x = super().forward(x)
File &quot;C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\container.py&quot;, line 100, in forward
input = module(input)
File &quot;C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py&quot;, line 550, in call
result = self.forward(*input, **kwargs)
File &quot;C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py&quot;, line 38, in forward
return F.linear(x, self.weight * self.w_mul, bias)
File &quot;C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\functional.py&quot;, line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)

End of error log.
NVIDIA-SMI LOG WHILE RUNNING (checking free memory)
C:\Users\micha&gt;nvidia-smi --query-gpu=memory.free --format=csv --loop=1
memory.free [MiB]


5991 MiB
5991 MiB
5991 MiB
5991 MiB
5897 MiB
5781 MiB
5685 MiB
1643 MiB
5991 MiB
5991 MiB

the program stopped at the 1643MiB
DETAILS PART 1:
I have an nvidia gpu with 6gb of memory which, (according to other logs posted by the creator of the project-&gt; should have enough memory for this to work &quot;We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well&quot; -adamian98 ). I'm trying to fix that error and get the run.py to work as intended.
Here is the system info using the numba -s command in anaconda (includes hardware info: gpu, my windows version, memory, python version etc.)
START CUDA INFO

Hardware Information
Machine : AMD64
CPU Name : znver1
CPU Count : 16
Number of accessible CPUs : 16
List of accessible CPUs cores : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
CFS Restrictions (CPUs worth of runtime) : None

CPU Features : 64bit adx aes avx avx2 bmi bmi2
clflushopt clwb clzero cmov cx16
cx8 f16c fma fsgsbase fxsr lzcnt
mmx movbe mwaitx pclmul popcnt
prfchw rdpid rdrnd rdseed sahf sha
sse sse2 sse3 sse4.1 sse4.2 sse4a
ssse3 wbnoinvd xsave xsavec
xsaveopt xsaves

Memory Total (MB) : 15789
Memory Available (MB) : 8421

OS Information
Platform Name : Windows-10-10.0.18362-SP0
Platform Release : 10
OS Name : Windows
OS Version : 10.0.18362
OS Specific Version : 10 10.0.18362 SP0 Multiprocessor Free
Libc Version : ?

Python Information
Python Compiler : MSC v.1916 64 bit (AMD64)
Python Implementation : CPython
Python Version : 3.8.5
Python Locale : en_CA.cp1252

LLVM Information
LLVM Version : 10.0.1

CUDA Information
CUDA Device Initialized : True
CUDA Driver Version : 10020
CUDA Detect Output:
Found 1 CUDA devices
id 0 b'GeForce GTX 1660 Ti with Max-Q Design' [SUPPORTED]
compute capability: 7.5
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported

CUDA Librairies Test Output:
Finding cublas from
named cublas.dll
trying to open library... ERROR: failed to open cublas:
Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cusparse from
named cusparse.dll
trying to open library... ERROR: failed to open cusparse:
Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cufft from
named cufft.dll
trying to open library... ERROR: failed to open cufft:
Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding curand from
named curand.dll
trying to open library... ERROR: failed to open curand:
Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding nvvm from
named nvvm.dll
trying to open library... ERROR: failed to open nvvm:
Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cudart from
named cudart.dll
trying to open library... ERROR: failed to open cudart:
Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding libdevice from
searching for compute_20... ERROR: can't open libdevice for compute_20
searching for compute_30... ERROR: can't open libdevice for compute_30
searching for compute_35... ERROR: can't open libdevice for compute_35
searching for compute_50... ERROR: can't open libdevice for compute_50

ROC information
ROC Available : False
ROC Toolchains : None
HSA Agents Count : 0
HSA Agents:
None
HSA Discrete GPUs Count : 0
HSA Discrete GPUs : None

SVML Information
SVML State, config.USING_SVML : True
SVML Library Loaded : True
llvmlite Using SVML Patched LLVM : True
SVML Operational : True

Threading Layer Information
TBB Threading Layer Available : False
+--&gt; Disabled due to Unknown import problem.
OpenMP Threading Layer Available : True
+--&gt;Vendor: MS
Workqueue Threading Layer Available : True
+--&gt;Workqueue imported successfully.

Numba Environment Variable Information
None found.

Conda Information
Conda Build : 3.20.5
Conda Env : 4.9.2
Conda Platform : win-64
Conda Python Version : 3.8.5.final.0
Conda Root Writable : True

Installed Packages
blas 1.0 mkl
ca-certificates 2020.12.8 haa95532_0
certifi 2020.12.5 py38haa95532_0
cffi 1.14.0 py38h7a1dbc1_0
chardet 3.0.4 py38haa95532_1003
cryptography 2.9.2 py38h7a1dbc1_0
cudatoolkit 10.2.89 h74a9793_1 anaconda
cycler 0.10.0 py38_0
freetype 2.9.1 ha9979f8_1
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.9 py_1
intel-openmp 2019.4 245
jpeg 9b hb83a4c4_2
kiwisolver 1.2.0 py38h74a9793_0
libcxx 7.0.0 h1ad3211_1002 conda-forge
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_0
llvm-meta 7.0.0 0 conda-forge
m2-bash 4.3.042 5
m2-gcc-libs 5.3.0 4
m2-libedit 3.1 20150326
m2-libffi 3.2.1 2
m2-libreadline 6.3.008 8
m2-msys2-runtime 2.5.0.17080.65c939c 3
m2-ncurses 6.0.20160220 2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
matplotlib 3.1.3 py38_0
matplotlib-base 3.1.3 py38h64f37c6_0
mkl 2019.4 245
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.0.15 py38h14836fe_0
mkl_random 1.1.0 py38hf9181ef_0
msys2-conda-epoch 20160418 1
ninja 1.9.0 py38h74a9793_0
numpy 1.18.1 py38h93ca92e_0
numpy-base 1.18.1 py38hc3f5095_1
olefile 0.46 py_0
openssl 1.1.1i h2bbff1b_0
pandas 1.0.3 py38h47e9c7a_0
pillow 7.1.2 py38hcc1f983_0
pip 20.0.2 py38_3
powershell_shortcut 0.0.1 3
pycparser 2.20 py_2
pyopenssl 19.1.0 pyhd3eb1b0_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
pysocks 1.7.1 py38haa95532_0
python 3.8.2 he1778fa_13
python-dateutil 2.8.1 py_0
pytorch 1.5.0 py3.8_cuda102_cudnn7_0 pytorch
pytz 2020.1 py_0
qt 5.9.7 vc14h73c81de_0
requests 2.23.0 py38_0
scipy 1.4.1 py38h9439919_0
setuptools 46.2.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.14.0 py38haa95532_0
sqlite 3.31.1 h2a8f88b_1
tk 8.6.8 hfa6e2cd_0
torchvision 0.6.0 py38_cu102 pytorch
tornado 6.0.4 py38he774522_1
urllib3 1.25.8 py38_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.34.2 py38_0
win_inet_pton 1.1.0 py38haa95532_0
wincertstore 0.2 py38_0
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.3.7 h508b16e_0

No errors reported.

Warning log
Warning (roc): Error initialising ROC: No ROC toolchains found.
Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init:

HSA is not currently supported on this platform (win32).`


END CUDA INFO
The section below is most likely not needed for this question, here in case:
DETAILS PART 2 (It sounded like dlib (and cmake) were only required to align the faces before therefore theoretically it isn't needed in run.py, but I'm not 100 percent sure. I omitted the section including the dlib specific errors):
I got the .yml installed successfully into the anaconda environment I named &quot;pulse3&quot; with no errors as well as the CUDA toolkit 10.2 and cmake 3.17.2. Dlib is the only thing that gave me some trouble spiting out a bunch of incompatibility errors with the other packages on windows with my python version 3.8.2.

based on new log evidence using this script simultaneously alongside the run.py file

 &quot;C:\Users\micha&gt;nvidia-smi --query-gpu=memory.free --format=csv --loop=1&quot; 


alongside using gpuz to monitor,
as well as the error produced from executing run.py in an anaconda environment with all required packages except dlib installed correctly:

RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)


resulted in a logged usage of up to but not exceeding 4.502GB of VRAM on a gtx 1660ti
Therefore, I can  strongly infer that the project PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (https://github.com/adamian98/pulse) does need somewhere between 6.01GB and 8.00GB of VRAM to execute the &quot;run.py&quot; file, therefore a gpu with 6gb is insufficient
so the error:

RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)


is most likely due to hardware limitations.
Notes To Help Improve the Pulse Project:
This result, though inconclusive, contradicts this theory from the creator of the project:
&quot;Unfortunately 2GB is not enough memory to store all of the gradients necessary during optimization. We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well.&quot; -adamian98
Possible factors of error (see logs in question for more info if needed):
1.needing to execute the .py file from outside the anaconda environment (despite having the file inside the environment as well)
2.lack of successful dlib package installation
3.the input image not being the correct size (it was 75x77 pixels, and was yoinked from the project samples)
4.the input image not being the correct format (it was.png)

the image not being in the correct place \anaconda3\envs\pulse3\Library\qml\Qt3D
(the anaconda environment was named pulse3 that is why the path uses pulse3)

