Weird behavior when calling cuda() on different tensors in pytorch
I am trying to train a pytorch neural network on a GPU device. In order to do so, I load my inputs and network onto the default cuda enabled GPU decive. However, when I load my inputs, the model's weights do not stay cuda tensors. Here is my train function

def train(network: nn.Module, name: str, learning_cycles: dict, num_epochs):
    # check we have a working gpu to train on
    assert(torch.cuda.is_available())
    # load model onto gpu
    network = network.cuda()
    # load train and test data with a transform
    transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    train_set = torchvision.datasets.CIFAR10(root='./data', train=True, 
 download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=128, 
shuffle=True, num_workers=2)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(network.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(num_epochs):
        for i, data in enumerate(train_loader, 0):
            inputs, labels = data
            # load inputs and labels onto gpu
            inputs, labels = inputs.cuda(), labels.cuda()
            optimizer.zero_grad()
            outputs = network(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()


When calling train, I get the following error.

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same


Interestingly, when I delete the line inputs, labels = inputs.cuda(), labels.cuda() I get the error RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

I would very much like to train my network, and I have searched the internet to no avail. Any good ideas?

Given that a device mismatch crops up regardless of the device the inputs are on, it's likely that some of your model's parameters are not being moved over to the GPU when you call network = network.cuda(). You have model parameters on both the CPU and the GPU.

Post your model code. It's likely you have a Pytorch module in an incorrect container.

Lists of modules should be in a nn. ModuleList. Modules in a Python list will not transfer over. Compare

layers1 = [nn.Linear(256, 256), nn.Linear(256, 256), nn.Linear(256, 256)]
layers2 = nn.ModuleList([nn.Linear(256, 256), nn.Linear(256, 256), nn.Linear(256, 256)])


If you called model.cuda() on a model with the above two lines, the layers in layer1 would remain on the CPU, while the layers in layer2 would be moved to the GPU.

Similarly, a list of nn.Parameter objects should be contained in an nn.ParameterList object.

There's also nn. ModuleDict and nn.ParameterDict for dictionary containers.
