title: Loss becomes NaN in training
body: <p>I am working on a Faster-RCNN model with below codes.
  I am using <a href="https://public.roboflow.com/object-detection/chess-full" rel="nofollow noreferrer">roboflow chess pieces dataset</a></p>
  <pre class="lang-py prettyprint-override"><code>def get_model(n_classes):
      model = models.detection.fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)
      in_features = model.roi_heads.box_predictor.cls_score.in_features
      model.roi_heads.box_predictor = models.detection.faster_rcnn.FastRCNNPredictor(in_features, n_classes)
      return model
  </code></pre>
  <p>Dataset class, _getitem_ part</p>
  <pre class="lang-py prettyprint-override"><code>
      def __getitem__(self, index):
          id = self.ids[index]
          image = self._load_image(id)
          # target = self._load_target(id)
          target = copy.deepcopy(self._load_target(id))
          boxes = torch.tensor([t[&quot;bbox&quot;] for t in target])
          new_boxes = torch.add(boxes[:,:2],boxes[:,2:])
          boxes = torch.cat((boxes[:,:2],new_boxes),1)
          labels = torch.tensor([t[&quot;category_id&quot;] for t in target], dtype=torch.int64)
          
          image = torch.from_numpy(image).permute(2,0,1)
  
          targ = {} 
          targ['boxes'] = boxes
          targ['labels'] = labels
          targ['image_id'] = torch.tensor(index)
          targ['area'] = (boxes[:,2]-boxes[:,0]) * (boxes[:,3]-boxes[:,1]) # we have a different area
          targ['iscrowd'] = torch.tensor([t[&quot;iscrowd&quot;] for t in target], dtype=torch.int64)
  
          return image, targ
  </code></pre>
  <p>The pipeline with above codes works fine without transforms. Predicted bbox seems good and mAPs are between 0.4 and 0.8 after 10 epochs.</p>
  <p>However, when I try to implement augmentation like below on above code pieces</p>
  <pre class="lang-py prettyprint-override"><code>def get_transforms(train=False):
      if train:
          transform = A.Compose([
              ToTensorV2()
          ], bbox_params=A.BboxParams(format='pascal_voc',label_fields=[&quot;labels&quot;]))
      else:
          transform = A.Compose([
              ToTensorV2()
          ], bbox_params=A.BboxParams(format='pascal_voc',label_fields=[&quot;labels&quot;]))
      return transform
  
  </code></pre>
  <p>Dataset class, _getitem_ part</p>
  <pre class="lang-py prettyprint-override"><code>    def __getitem__(self, index):
          id = self.ids[index]
          image = self._load_image(id)
          # target = self._load_target(id)
          target = copy.deepcopy(self._load_target(id))
          boxes = torch.tensor([t[&quot;bbox&quot;] for t in target])
          new_boxes = torch.add(boxes[:,:2],boxes[:,2:])
          boxes = torch.cat((boxes[:,:2],new_boxes),1)
          labels = torch.tensor([t[&quot;category_id&quot;] for t in target], dtype=torch.int64)
          
          if self.transforms is not None:
              transformed = self.transforms(image=image, bboxes=boxes, labels=labels)
              image = transformed['image']
              boxes = torch.tensor(transformed['bboxes']).view(len(transformed[&quot;bboxes&quot;]),4)
              labels = torch.tensor(transformed[&quot;labels&quot;],dtype=torch.int64)
  
          else:
              image = torch.from_numpy(image).permute(2,0,1)
  
          targ = {}
          targ['boxes'] = boxes
          targ['labels'] = labels
          targ['image_id'] = torch.tensor(index)
          targ['area'] = (boxes[:,2]-boxes[:,0]) * (boxes[:,3]-boxes[:,1]) # we have a different area
          targ['iscrowd'] = torch.tensor([t[&quot;iscrowd&quot;] for t in target], dtype=torch.int64)
  
          return image, targ
  </code></pre>
  <p>I end up with NaN loss.</p>
  <p>This is the last output I get with batch_size 10</p>
  <pre class="lang-py prettyprint-override"><code>Epoch: [0]  [10/18]  eta: 0:02:41  lr: 0.003237  loss: 2.3237 (2.6498)  loss_classifier: 1.4347 (1.8002)  loss_box_reg: 0.7538 (0.7682)  loss_objectness: 0.0441 (0.0595)  loss_rpn_box_reg: 0.0221 (0.0220)  time: 20.2499  data: 0.1298
  Loss is nan, stopping training
  {'loss_classifier': tensor(nan, grad_fn=&lt;NllLossBackward0&gt;), 'loss_box_reg': tensor(nan, grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(nan, grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward0&gt;), 'loss_rpn_box_reg': tensor(nan, dtype=torch.float64, grad_fn=&lt;DivBackward0&gt;)}
  </code></pre>
  <ul>
  <li>Why does the loss become NaN?</li>
  <li>How to find the problem cause it?</li>
  </ul>
  <p><strong>Edit</strong>: I am using patches and some of my training examples are empty(no object). Meanwhile the model train on the these patches I noticed the values next to loss values in parenthesis increase.I couldn't find what these parenthesis refer to, but I think it connected with last image or batch.(I was using batch_size 1).</p>
  <p>There is a few lines of output while it was process on empty images. I tried with Adam and SGD, results are same.</p>
  <pre class="lang-py prettyprint-override"><code>Epoch: [0]  [17/26]  eta: 0:00:14  lr: 0.003601  loss: 2.4854 (3.9266)  loss_classifier: 1.1224 (2.2893)  loss_box_reg: 0.7182 (1.2226)  loss_objectness: 0.0497 (0.3413)  loss_rpn_box_reg: 0.0116 (0.0735)  time: 1.6587  data: 0.0102 # before empty image
  Epoch: [0]  [18/26]  eta: 0:00:12  lr: 0.003801  loss: 2.8132 (61.1689)  loss_classifier: 1.5675 (28.8652)  loss_box_reg: 0.7563 (29.8348)  loss_objectness: 0.1070 (2.2412)  loss_rpn_box_reg: 0.0145 (0.2278)  time: 1.6240  data: 0.0098 # after empty image
  </code></pre>