title: what(): Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices()
body: <p>I am working on Ubuntu 20.04.</p>
  <p>I am trying to test a Neural Network and I have some scripts which are written in python that try to use the GPU. My scripts work fine and I can train and test the model I am trying to build.</p>
  <p>However, I noticed that if for some reason my script does not arrive at its end (e.g., I have an error at the end of the <code>main</code> function) I cannot start again another script to try again to test the model.</p>
  <p>What I get is the following error:</p>
  <pre><code>(software1) myusername@mylaptopname:~/folder1$ bash experiments/test.sh 
  Using tensorboardX
  [log - coco.py] in class COCO
  [log - coco.py] in class COCO1
  importing1
  terminate called after throwing an instance of 'c10::Error'
    what():  Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW
  Exception raised from device_count_impl at ../c10/cuda/CUDAFunctions.cpp:97 (most recent call first):
  frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7febf4b41a22 in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libc10.so)
  frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x5b (0x7febf4b3e3db in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libc10.so)
  frame #2: &lt;unknown function&gt; + 0xd513 (0x7febf4d9f513 in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
  frame #3: c10::cuda::device_count_ensure_non_zero() + 0xe (0x7febf4d9f54e in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
  frame #4: THCudaInit(THCState*) + 0x21 (0x7feb7984a541 in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
  frame #5: &lt;unknown function&gt; + 0x2b05964 (0x7feb796ca964 in /home/myusername/.virtualenvs/software1/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
  frame #6: at::Context::lazyInitCUDA()::{lambda()#1}::operator()() const + 0x3a (0x7feb427dc592 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #7: void std::__invoke_impl&lt;void, at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::__invoke_other, at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;) + 0x24 (0x7feb427dd179 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #8: std::__invoke_result&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;::type std::__invoke&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::__invoke_result&amp;&amp;, (at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;)...) + 0x33 (0x7feb427dce70 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #9: std::call_once&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::once_flag&amp;, at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;)::{lambda()#1}::operator()() const + 0x27 (0x7feb427dc893 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #10: std::call_once&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::once_flag&amp;, at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;)::{lambda()#2}::operator()() const + 0x2b (0x7feb427dc8c1 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #11: std::call_once&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::once_flag&amp;, at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;)::{lambda()#2}::_FUN() + 0x12 (0x7feb427dc8d6 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #12: &lt;unknown function&gt; + 0x1247f (0x7fec01f6347f in /lib/x86_64-linux-gnu/libpthread.so.0)
  frame #13: &lt;unknown function&gt; + 0x444c8 (0x7feb427dc4c8 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #14: void std::call_once&lt;at::Context::lazyInitCUDA()::{lambda()#1}&gt;(std::once_flag&amp;, at::Context::lazyInitCUDA()::{lambda()#1}&amp;&amp;) + 0x86 (0x7feb427dc973 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #15: at::Context::lazyInitCUDA() + 0x3a (0x7feb427dc60a in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #16: &lt;unknown function&gt; + 0x44458 (0x7feb427dc458 in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #17: &lt;unknown function&gt; + 0x4447f (0x7feb427dc47f in /home/myusername/folder1/src/lib/model/networks/DCNv2/_ext.cpython-38-x86_64-linux-gnu.so)
  frame #18: &lt;unknown function&gt; + 0x11b8a (0x7fec021c0b8a in /lib64/ld-linux-x86-64.so.2)
  frame #19: &lt;unknown function&gt; + 0x11c91 (0x7fec021c0c91 in /lib64/ld-linux-x86-64.so.2)
  frame #20: _dl_catch_exception + 0xe5 (0x7fec020d7915 in /lib/x86_64-linux-gnu/libc.so.6)
  frame #21: &lt;unknown function&gt; + 0x160bf (0x7fec021c50bf in /lib64/ld-linux-x86-64.so.2)
  frame #22: _dl_catch_exception + 0x88 (0x7fec020d78b8 in /lib/x86_64-linux-gnu/libc.so.6)
  frame #23: &lt;unknown function&gt; + 0x155fa (0x7fec021c45fa in /lib64/ld-linux-x86-64.so.2)
  frame #24: &lt;unknown function&gt; + 0x134c (0x7fec01f4c34c in /lib/x86_64-linux-gnu/libdl.so.2)
  frame #25: _dl_catch_exception + 0x88 (0x7fec020d78b8 in /lib/x86_64-linux-gnu/libc.so.6)
  frame #26: _dl_catch_error + 0x33 (0x7fec020d7983 in /lib/x86_64-linux-gnu/libc.so.6)
  frame #27: &lt;unknown function&gt; + 0x1b59 (0x7fec01f4cb59 in /lib/x86_64-linux-gnu/libdl.so.2)
  frame #28: dlopen + 0x4a (0x7fec01f4c3da in /lib/x86_64-linux-gnu/libdl.so.2)
  &lt;omitting python frames&gt;
  frame #31: python() [0x685611]
  frame #32: python() [0x5c5530]
  frame #47: python() [0x5f38f1]
  frame #53: python() [0x600f54]
  frame #54: python() [0x5c5530]
  
  experiments/test.sh: line 20: 33055 Aborted                 (core dumped) python test.py ddd --exp_id software1 --dataset nuscenes --val_split mini_val --run_dataset_eval --num_workers 8 --nuscenes_att --velocity --gpus 0 --pointcloud --radar_sweeps 6 --max_pc_dist 60.0 --pc_z_offset -0.0 --load_model ../exp/ddd/software1/model_last.pth --flip_test
  
  </code></pre>
  <p>The only way I have to solve this problem is to reboot my laptop and this is annoying of course.</p>
  <p>I know that it is quite generic what I am asking but I was wondering if there is a way to <em>reset</em> the use of the GPU since to me this looks like a problem in which the allocation of GPU has not been &quot;closed&quot; correctly. Indeed, every time I reboot the problem disappears.</p>
  <p>Could you help me solve this problem?</p>