YOLOv7 RuntimeError: CUDA error: unknown error
Getting the following error with my current setup after following this guide when trying to train a custom model for YOLOv7: https://docs.nvidia.com/cuda/wsl-user-guide/index.html.
OS: Ubuntu (Windows 10 WSL)
Hardware: 16gb RAM, RTX 3070
Python Version: 3.8.10
Driver Version: 517.48
PyTorch Version:
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
1.10.0a0+3fd9dcf
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; print(torchvision.__version__)
0.11.0a0

Cuda Version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jul_14_19:41:19_PDT_2021
Cuda compilation tools, release 11.4, V11.4.100
Build cuda_11.4.r11.4/compiler.30188945_0

Some common issues that have solved this issue for other people that I've attempted:

Restarting the computer
apt-get install nvidia-modprobe (https://github.com/pytorch/pytorch/issues/49081)

Docker Container:
sudo docker run --name yolov7 --gpus all -it -v &quot;/mnt/c/coco/&quot;:&quot;/coco/&quot; -v &quot;/mnt/c/yolov7/&quot;:&quot;/yolov7/&quot; --shm-size=16gb nvcr.io/nvidia/pytorch:21.08-py3
The part that I don't understand is that torch appears to be operating correctly:
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.cuda.current_device())
0
&gt;&gt;&gt; torch.rand(1)
tensor([0.3052])

nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76.02    Driver Version: 517.48       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:2B:00.0  On |                  N/A |
|  0%   46C    P8    27W / 240W |    423MiB /  8192MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

From Within Docker Container:
python train.py --workers 1 --device 0 --batch-size 2 --data data/coco.yaml --img-size 1920 --cfg cfg/training/yolov7.yaml --weights 'yolov7_training.pt' --name yolov7 --hyp data/hyp.scratch.custom.yaml
The above call produces the following error:
Traceback (most recent call last):
  File &quot;train.py&quot;, line 616, in &lt;module&gt;
    train(hyp, opt, device, tb_writer)
  File &quot;train.py&quot;, line 361, in train
    pred = model(imgs)  # forward
  File &quot;/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/yolov7/models/yolo.py&quot;, line 599, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File &quot;/yolov7/models/yolo.py&quot;, line 625, in forward_once
    x = m(x)  # run
  File &quot;/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/yolov7/models/common.py&quot;, line 108, in forward
    return self.act(self.bn(self.conv(x)))
  File &quot;/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py&quot;, line 395, in forward
    return F.silu(input, inplace=self.inplace)
  File &quot;/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py&quot;, line 1901, in silu
    return torch._C._nn.silu(input)
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.


Even on WSL, your Host is Windows and will have the same limitations.
On windows there is a watchdog that kills too long kernels (WDDM).
If your NVidia GPU is a secondary GPU, try to disable WDDM TDR.
If it's your primary GPU, you can experiment screen freezes when you use pyTorch.
there is some informations on how to disable WDDM in this post : https://stackoverflow.com/a/13185441/4866974
