title: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0(when checking arugment mat1 in method wrapper_addmm
body: <p>I trained a faster r cnn in order to detect tools. I already define my model and every thing worked. But to have a cleaner code without gloabal variables I tried to write a class MyModel who will automatically define every objet and train the model. So on this class I defined a class called self.dataset = ToolDataset.</p>
  <p>On this first class I have defined my input (an image) and my output (a target which is a dictionnary with bboxes, labels, area …).
  Then I built a data loader (so I have a self.data_loader), and I used the function train_one_epoch of the engine librarie. On this function, I gave in input my model (a faster r cnn), my data loader, and the device who is cuda:0 (I printed it). This function iterate on my data loader. This function defines a list of images and a list of targets, and converts the values of the lists into the good device.
  And then it calls : <code>model(images, targets)</code>. And on this step I got the error with the two devices founded (I pasted the error at the end of the message).</p>
  <p>I got the error even if every tensor (my images, and every values of my target dictionary) returned True for the command tensor.is_cuda. So I really don’t understand why does the error say that I have also a cpu device. I show you my function train , train_one_epoch, and my variables images and targets :</p>
  <p>train method :</p>
  <pre><code>def train(self, num_epoch = 10, gpu = True):
          
          if gpu : 
              CUDA_LAUNCH_BLOCKING=&quot;1&quot;
  
              #torch.set_default_tensor_type(torch.FloatTensor) 
              model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
              use_cuda = torch.cuda.is_available()
              device = torch.device(&quot;cuda:0&quot; if use_cuda else &quot;cpu&quot;)
              model.to(device)
              if self.multi_object_detection == False : 
                  num_classes = 2 # ['Tool', 'background']
              else : 
                  print(&quot;need to set a multi object detection code&quot;)
  
              in_features = torch.tensor(model.roi_heads.box_predictor.cls_score.in_features, dtype = torch.int64).to(device)
              print(&quot;in_features = {}&quot;.format(in_features))
              model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
              print( &quot;model.roi_heads.box_predictor {}&quot;.format( model.roi_heads.box_predictor))
              
              model_parameters = filter(lambda p: p.requires_grad, model.parameters())
              #params = sum([np.prod(p.size()) for p in model_parameters])
              params = [p for p in model.parameters() if p.requires_grad]
  
              
              optimizer = torch.optim.SGD(params, lr=0.001, momentum=0.9, weight_decay=0.0005)
              lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)
              gc.collect()
              num_epochs = 5
              FILE_model_dict_gpu = &quot;model_state_dict__gpu_lab2_and_lab7_5epoch.pth&quot;
              list_of_list_losses = []
              print(&quot;device = &quot;, device)
              
              if (self.data_loader.dataset) == None :
                  self.build_dataloader(device)
              
              for epoch in tqdm(range(num_epochs)):
  
                  # Train for one epoch, printing every 10 iterations
                  train_his_, list_losses, list_losses_dict = train_one_epoch(model, optimizer, self.data_loader, device, epoch, print_freq=10)
                  list_of_list_losses.append(list_losses)
                  # Compute losses over the validation set
                  #val_his_ = validate_one_epoch(model, val_data_loader, device, print_freq=10)
  
                  # Update the learning rate
                  print(&quot;lr before update : &quot;, lr_scheduler)
                  lr_scheduler.step()
                  print(&quot;lr after update : &quot;, lr_scheduler)
                  # Store loss values to plot learning curves afterwork.
                  if epoch == 0: 
                      train_history = {k: [v] for k, v in train_his_.items()}
                      #val_history = {k: [v] for k, v in val_his_.items()}
                  else: 
                      for k, v in train_his_.items():train_history[k] += [v]
                  #   for k, v in val_his_.items():val_history[k] += [v]
  
                  # On peut save le modèle dans la boucle en ajoutant un critère : si la validation decroit
                  # torch.save(model, save_path)
  
                  torch.cuda.empty_cache()
                  gc.collect()
  </code></pre>
  <p>train_one_epoch function (I print some information that will be shown on the output at the end of the message)</p>
  <pre><code>
  def train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq):
  
      model.train()
      metric_logger = utilss.MetricLogger(delimiter=&quot;  &quot;)
      metric_logger.add_meter('lr', utilss.SmoothedValue(window_size=1, fmt='{value:.6f}'))
      header = 'Epoch: [{}]'.format(epoch)
      list_losses = []
      list_losses_dict = []
      for i, values in tqdm(enumerate(metric_logger.log_every(data_loader, print_freq, header))):
          images, targets = values
          for image in images : 
              print(&quot;before the to(device) operation, image.is_cuda = {}&quot;.format(image.is_cuda))
          images = list(image.to(device, dtype=torch.float) for image in images)
          targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
          #images = [image.cuda() for image in images]
          for image in images : 
              print(image)
              print(&quot;after the to(device) operation, image.is_cuda = {}&quot;.format(image.is_cuda))
          for target in targets :
              for t, dict_value in target.items():
                  print(&quot;after the to(device) operation, dict_value.is_cuda = {}&quot;.format(dict_value.is_cuda))
  
          print(&quot;images = {}&quot;.format(images))
          print(&quot;targets = {}&quot;.format(targets))
  
          # Feed the training samples to the model and compute the losses
          loss_dict = model(images, targets)
          losses = sum(loss for loss in loss_dict.values())
  
          # reduce losses over all GPUs for logging purposes
          loss_dict_reduced = utilss.reduce_dict(loss_dict)
          losses_reduced = sum(loss for loss in loss_dict_reduced.values())
          loss_value = losses_reduced.item()
          print(&quot;Loss is {}, stopping training&quot;.format(loss_value))
          if not math.isfinite(loss_value):
              print(&quot;Loss is {}, stopping training&quot;.format(loss_value))
              print(loss_dict_reduced)
              sys.exit(1)
          list_losses.append(loss_value)
  
          # Pytorch function to initialize optimizer
          optimizer.zero_grad()
          # Compute gradients or the backpropagation
          losses.backward()
          # Update current gradient
          optimizer.step()
  
  </code></pre>
  <p>And I show you my output with the error (with my images and target, and the error) :</p>
  <pre><code>in_features = 1024
  model.roi_heads.box_predictor FastRCNNPredictor(
    (cls_score): Linear(in_features=1024, out_features=2, bias=True)
    (bbox_pred): Linear(in_features=1024, out_features=8, bias=True)
  )
  device =  cuda:0
  
  100%|██████████| 515/515 [00:00&lt;00:00, 112118.06it/s]
  100%|██████████| 761/761 [00:00&lt;00:00, 111005.96it/s]
    0%|          | 0/5 [00:00&lt;?, ?it/s]
  0it [00:00, ?it/s]
  
  before the to(device) operation, image.is_cuda = True
  tensor([[[0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0235, 0.0235, 0.0235,  ..., 0.0235, 0.0235, 0.0235],
           [0.0353, 0.0353, 0.0353,  ..., 0.0314, 0.0314, 0.0314]],
  
          [[0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0039, 0.0039, 0.0039],
           [0.0235, 0.0235, 0.0235,  ..., 0.0157, 0.0157, 0.0157],
           [0.0353, 0.0353, 0.0353,  ..., 0.0235, 0.0235, 0.0235]],
  
          [[0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0078, 0.0078, 0.0078],
           [0.0235, 0.0235, 0.0235,  ..., 0.0196, 0.0196, 0.0196],
           [0.0353, 0.0353, 0.0353,  ..., 0.0275, 0.0275, 0.0275]]],
         device='cuda:0')
  after the to(device) operation, image.is_cuda = True
  after the to(device) operation, dict_value.is_cuda = True
  after the to(device) operation, dict_value.is_cuda = True
  after the to(device) operation, dict_value.is_cuda = True
  after the to(device) operation, dict_value.is_cuda = True
  after the to(device) operation, dict_value.is_cuda = True
  images = [tensor([[[0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0235, 0.0235, 0.0235,  ..., 0.0235, 0.0235, 0.0235],
           [0.0353, 0.0353, 0.0353,  ..., 0.0314, 0.0314, 0.0314]],
  
          [[0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           [0.0078, 0.0078, 0.0078,  ..., 0.0000, 0.0000, 0.0000],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0039, 0.0039, 0.0039],
           [0.0235, 0.0235, 0.0235,  ..., 0.0157, 0.0157, 0.0157],
           [0.0353, 0.0353, 0.0353,  ..., 0.0235, 0.0235, 0.0235]],
  
          [[0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           [0.0078, 0.0078, 0.0078,  ..., 0.0118, 0.0118, 0.0118],
           ...,
           [0.0078, 0.0078, 0.0078,  ..., 0.0078, 0.0078, 0.0078],
           [0.0235, 0.0235, 0.0235,  ..., 0.0196, 0.0196, 0.0196],
           [0.0353, 0.0353, 0.0353,  ..., 0.0275, 0.0275, 0.0275]]],
         device='cuda:0')]
  targets = [{'boxes': tensor([[1118.8964,    0.0000, 1368.9186,  399.3243],
          [1043.0958,  111.4863, 1332.4319,  426.1295]], device='cuda:0',
         dtype=torch.float64), 'labels': tensor([1, 1], device='cuda:0'), 'index': tensor([311], device='cuda:0'), 'area': tensor([99839.9404, 91037.6485], device='cuda:0', dtype=torch.float64), 'iscrowd': tensor([0], device='cuda:0')}]
  
  /home/nathaneberrebi/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/c10/core/TensorImpl.h:1156.)
    return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
  0it [00:02, ?it/s]
    0%|          | 0/5 [00:02&lt;?, ?it/s]
  
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  &lt;ipython-input-15-51a35da5b1fe&gt; in &lt;module&gt;
  ----&gt; 1 class_model.train()
  
  &lt;ipython-input-7-d44d099a7743&gt; in train(self, num_epoch, gpu)
      144 
      145                 # Train for one epoch, printing every 10 iterations
  --&gt; 146                 train_his_, list_losses, list_losses_dict = train_one_epoch(model, optimizer, self.data_loader, device, epoch, print_freq=10)
      147                 list_of_list_losses.append(list_losses)
      148                 # Compute losses over the validation set
  
  &lt;ipython-input-6-347c12a81a2f&gt; in train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq)
      519 
      520         # Feed the training samples to the model and compute the losses
  --&gt; 521         loss_dict = model(images, targets)
      522         losses = sum(loss for loss in loss_dict.values())
      523 
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
     1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
     1050                 or _global_forward_hooks or _global_forward_pre_hooks):
  -&gt; 1051             return forward_call(*input, **kwargs)
     1052         # Do not call functions when jit is used
     1053         full_backward_hooks, non_full_backward_hooks = [], []
  
  ~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
       95             features = OrderedDict([('0', features)])
       96         proposals, proposal_losses = self.rpn(images, features, targets)
  ---&gt; 97         detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
       98         detections = self.transform.postprocess(detections, images.image_sizes, original_image_sizes)
       99 
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
     1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
     1050                 or _global_forward_hooks or _global_forward_pre_hooks):
  -&gt; 1051             return forward_call(*input, **kwargs)
     1052         # Do not call functions when jit is used
     1053         full_backward_hooks, non_full_backward_hooks = [], []
  
  ~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/roi_heads.py in forward(self, features, proposals, image_shapes, targets)
      752         box_features = self.box_roi_pool(features, proposals, image_shapes)
      753         box_features = self.box_head(box_features)
  --&gt; 754         class_logits, box_regression = self.box_predictor(box_features)
      755 
      756         result: List[Dict[str, torch.Tensor]] = []
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
     1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
     1050                 or _global_forward_hooks or _global_forward_pre_hooks):
  -&gt; 1051             return forward_call(*input, **kwargs)
     1052         # Do not call functions when jit is used
     1053         full_backward_hooks, non_full_backward_hooks = [], []
  
  ~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/faster_rcnn.py in forward(self, x)
      280             assert list(x.shape[2:]) == [1, 1]
      281         x = x.flatten(start_dim=1)
  --&gt; 282         scores = self.cls_score(x)
      283         bbox_deltas = self.bbox_pred(x)
      284 
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
     1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
     1050                 or _global_forward_hooks or _global_forward_pre_hooks):
  -&gt; 1051             return forward_call(*input, **kwargs)
     1052         # Do not call functions when jit is used
     1053         full_backward_hooks, non_full_backward_hooks = [], []
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
       94 
       95     def forward(self, input: Tensor) -&gt; Tensor:
  ---&gt; 96         return F.linear(input, self.weight, self.bias)
       97 
       98     def extra_repr(self) -&gt; str:
  
  ~/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
     1845     if has_torch_function_variadic(input, weight):
     1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
  -&gt; 1847     return torch._C._nn.linear(input, weight, bias)
     1848 
     1849 
  
  RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument mat1 in method wrapper_addmm)
  </code></pre>
  <p>Thank you very much for your help, I'm having this issue since while. And I cannot torch.jit.trace my last model (before trying to clean my code using a class to build automatically every object with just one function train) because of the same error. And I need to fix it to use this model in a c++ code.
  Let me know if you need further informations.</p>
  <p>Here is my toch env :</p>
  <pre><code>PyTorch version: 1.9.0
  Is debug build: False
  CUDA used to build PyTorch: 11.1
  ROCM used to build PyTorch: N/A
  
  OS: Ubuntu 20.04.2 LTS (x86_64)
  GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
  Clang version: Could not collect
  CMake version: Could not collect
  Libc version: glibc-2.31
  
  Python version: 3.8 (64-bit runtime)
  Python platform: Linux-5.8.0-59-generic-x86_64-with-glibc2.10
  Is CUDA available: True
  CUDA runtime version: Could not collect
  GPU models and configuration: GPU 0: GeForce RTX 3060 Laptop GPU
  Nvidia driver version: 460.80
  cuDNN version: Could not collect
  HIP runtime version: N/A
  MIOpen runtime version: N/A
  
  Versions of relevant libraries:
  [pip3] numpy==1.20.2
  [pip3] numpydoc==1.1.0
  [pip3] torch==1.9.0
  [pip3] torchaudio==0.9.0a0+33b2469
  [pip3] torchvision==0.10.0
  [conda] blas                      1.0                         mkl  
  [conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
  [conda] mkl                       2021.2.0           h06a4308_296  
  [conda] mkl-service               2.4.0            py38h497a2fe_0    conda-forge
  [conda] mkl_fft                   1.3.0            py38h42c9631_2  
  [conda] mkl_random                1.2.2            py38h1abd341_0    conda-forge
  [conda] numpy                     1.18.5                   pypi_0    pypi
  [conda] numpy-base                1.20.2           py38hfae3a4d_0  
  [conda] numpydoc                  1.1.0                      py_1    conda-forge
  [conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
  [conda] torch                     1.9.0                    pypi_0    pypi
  [conda] torchaudio                0.9.0                      py38    pytorch
  [conda] torchvision               0.10.0               py38_cu111    pytorch
  </code></pre>