title: yolov5-6.0 RuntimeError: CUDA error: device-side assert triggered
body: <p>I'm sorry, but I don't speak English, so I use a translator.
  Until a few days ago, there were no problems with learning. But now I get the following error:</p>
  <pre><code>:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [15,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [103,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [104,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [105,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [106,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [107,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [108,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [86,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
  C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [63,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; &quot;index out of bounds&quot;` failed.
                 Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:   0%|    | 0/5 [00:01&lt;?, ?it/s]
  Traceback (most recent call last):
    File &quot;D:\Jupyter\NationalPark\yolov5-6.0\train.py&quot;, line 621, in &lt;module&gt;
      main(opt)
    File &quot;D:\Jupyter\NationalPark\yolov5-6.0\train.py&quot;, line 518, in main
      train(opt.hyp, opt, device, callbacks)
    File &quot;D:\Jupyter\NationalPark\yolov5-6.0\train.py&quot;, line 354, in train
      results, maps, _ = val.run(data_dict,
    File &quot;C:\Users\Administrator\anaconda3\envs\torch\lib\site-packages\torch\autograd\grad_mode.py&quot;, line 28, in decorate_context
      return func(*args, **kwargs)
    File &quot;D:\Jupyter\NationalPark\yolov5-6.0\val.py&quot;, line 179, in run
      loss += compute_loss([x.float() for x in train_out], targets)[1]  # box, obj, cls
    File &quot;D:\Jupyter\NationalPark\yolov5-6.0\utils\loss.py&quot;, line 148, in __call__
      t[range(n), tcls[i]] = self.cp
  RuntimeError: CUDA error: device-side assert triggered
  </code></pre>
  <p><code>os.environ['CUDA_LAUNCH_BLOCKING'] = &quot;1&quot;</code>is added to the top of train.py.</p>
  <p><strong>the code</strong></p>
  <pre><code>conda activate torch
  d:
  cd D:\Jupyter\NationalPark\yolov5-6.0
  python train.py --img 640 --batch 100 --epochs 600 --data data/custom2.yaml --cfg models/yolov5n.yaml --weights yolov5n.pt --workers 16 --patience 300
  </code></pre>
  <p>And detect.py runs fine.</p>
  <p>Is there any more information I should write down?
  if letting me know Thanks.</p>