Instruction: Facing this error while classifying Images, containing 10 classes in pytorch, in ResNet50. My code is:

Input:
This is the code I am implementing: I am using a subset of the CalTech256 dataset to classify images of 10 different kinds of animals. We will go over the dataset preparation, data augmentation and then steps to build the classifier.

def train_and_validate(model, loss_criterion, optimizer, epochs=25):
    '''
    Function to train and validate
    Parameters
        :param model: Model to train and validate
        :param loss_criterion: Loss Criterion to minimize
        :param optimizer: Optimizer for computing gradients
        :param epochs: Number of epochs (default=25)

    Returns
        model: Trained Model with best validation accuracy
        history: (dict object): Having training loss, accuracy and validation loss, accuracy
    '''

    start = time.time()
    history = []
    best_acc = 0.0

    for epoch in range(epochs):
        epoch_start = time.time()
        print("Epoch: {}/{}".format(epoch+1, epochs))

        # Set to training mode
        model.train()

        # Loss and Accuracy within the epoch
        train_loss = 0.0
        train_acc = 0.0

        valid_loss = 0.0
        valid_acc = 0.0

        for i, (inputs, labels) in enumerate(train_data_loader):

            inputs = inputs.to(device)
            labels = labels.to(device)

            # Clean existing gradients
            optimizer.zero_grad()

            # Forward pass - compute outputs on input data using the model
            outputs = model(inputs)

            # Compute loss
            loss = loss_criterion(outputs, labels)

            # Backpropagate the gradients
            loss.backward()

            # Update the parameters
            optimizer.step()

            # Compute the total loss for the batch and add it to train_loss
            train_loss += loss.item() * inputs.size(0)

            # Compute the accuracy
            ret, predictions = torch.max(outputs.data, 1)
            correct_counts = predictions.eq(labels.data.view_as(predictions))

            # Convert correct_counts to float and then compute the mean
            acc = torch.mean(correct_counts.type(torch.FloatTensor))

            # Compute total accuracy in the whole batch and add to train_acc
            train_acc += acc.item() * inputs.size(0)

            #print("Batch number: {:03d}, Training: Loss: {:.4f}, Accuracy: {:.4f}".format(i, loss.item(), acc.item()))

        # Validation - No gradient tracking needed
        with torch.no_grad():

            # Set to evaluation mode
            model.eval()

            # Validation loop
            for j, (inputs, labels) in enumerate(valid_data_loader):
                inputs = inputs.to(device)
                labels = labels.to(device)

                # Forward pass - compute outputs on input data using the model
                outputs = model(inputs)

                # Compute loss
                loss = loss_criterion(outputs, labels)

                # Compute the total loss for the batch and add it to valid_loss
                valid_loss += loss.item() * inputs.size(0)

                # Calculate validation accuracy
                ret, predictions = torch.max(outputs.data, 1)
                correct_counts = predictions.eq(labels.data.view_as(predictions))

                # Convert correct_counts to float and then compute the mean
                acc = torch.mean(correct_counts.type(torch.FloatTensor))

                # Compute total accuracy in the whole batch and add to valid_acc
                valid_acc += acc.item() * inputs.size(0)

                #print("Validation Batch number: {:03d}, Validation: Loss: {:.4f}, Accuracy: {:.4f}".format(j, loss.item(), acc.item()))

        # Find average training loss and training accuracy
        avg_train_loss = train_loss/train_data_size
        avg_train_acc = train_acc/train_data_size

        # Find average training loss and training accuracy
        avg_valid_loss = valid_loss/valid_data_size
        avg_valid_acc = valid_acc/valid_data_size

        history.append([avg_train_loss, avg_valid_loss, avg_train_acc, avg_valid_acc])

        epoch_end = time.time()

        print("Epoch : {:03d}, Training: Loss: {:.4f}, Accuracy: {:.4f}%, \n\t\tValidation : Loss : {:.4f}, Accuracy: {:.4f}%, Time: {:.4f}s".format(epoch, avg_train_loss, avg_train_acc*100, avg_valid_loss, avg_valid_acc*100, epoch_end-epoch_start))

        # Save if the model has best accuracy till now
        torch.save(model, dataset+'_model_'+str(epoch)+'.pt')

    return model, history

# Load pretrained ResNet50 Model
resnet50 = models.resnet50(pretrained=True)
#resnet50 = resnet50.to('cuda:0')

# Freeze model parameters
for param in resnet50.parameters():
    param.requires_grad = False
# Change the final layer of ResNet50 Model for Transfer Learning
fc_inputs = resnet50.fc.in_features

resnet50.fc = nn.Sequential(
    nn.Linear(fc_inputs, 256),
    nn.ReLU(),
    nn.Dropout(0.4),
    nn.Linear(256, num_classes), # Since 10 possible outputs
    nn.LogSoftmax(dim=1) # For using NLLLoss()
)

# Convert model to be used on GPU
# resnet50 = resnet50.to('cuda:0')

# Change the final layer of ResNet50 Model for Transfer Learning
fc_inputs = resnet50.fc.in_features

resnet50.fc = nn.Sequential(
    nn.Linear(fc_inputs, 256),
    nn.ReLU(),
    nn.Dropout(0.4),
    nn.Linear(256, num_classes), # Since 10 possible outputs
    nn.LogSoftmax(dienter code herem=1) # For using NLLLoss()
)

# Convert model to be used on GPU
# resnet50 = resnet50.to('cuda:0')`enter code here`

Error is this:

  RuntimeError                              Traceback (most recent call
  last)  in ()
        6 # Train the model for 25 epochs
        7 num_epochs = 30
  ----> 8 trained_model, history = train_and_validate(resnet50, loss_func, optimizer, num_epochs)
        9
       10 torch.save(history, dataset+'_history.pt')

   in train_and_validate(model,
  loss_criterion, optimizer, epochs)
       43
       44             # Compute loss
  ---> 45             loss = loss_criterion(outputs, labels)
       46
       47             # Backpropagate the gradients

  ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
  call(self, *input, **kwargs)
      539             result = self._slow_forward(*input, **kwargs)
      540         else:
  --> 541             result = self.forward(*input, **kwargs)
      542         for hook in self._forward_hooks.values():
      543             hook_result = hook(self, input, result)

  ~\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in
  forward(self, input, target)
      202
      203     def forward(self, input, target):
  --> 204         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
      205
      206

  ~\Anaconda3\lib\site-packages\torch\nn\functional.py in
  nll_loss(input, target, weight, size_average, ignore_index, reduce,
  reduction)    1836                          .format(input.size(0),
  target.size(0)))    1837     if dim == 2:
  -> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)    1839     elif dim == 4:    1840         ret = torch._C._nn.nll_loss2d(input, target,
  weight, _Reduction.get_enum(reduction), ignore_index)

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes'
  failed.  at
  C:\Users\builder\AppData\Local\Temp\pip-req-build-0i480kur\aten\src\THNN/generic/ClassNLLCriterion.c:97

Output:
This happens when there are either incorrect labels in your dataset, or the labels are 1-indexed (instead of 0-indexed). As from the error message, cur_target must be smaller than the total number of classes (10). To verify the issue, check the maximum and minimum label in your dataset. If the data is indeed 1-indexed, just minus one from all annotations and you should be fine.

Note, another possible reason is that there exists some -1 labels in the data. Some (esp older) datasets use -1 as indication of a wrong/dubious label. If you find such labels, just discard them.


Instruction: slicing each raw of a tensor

Input:
I have a tensor of size (1124823 x 13). I want to go from the center of each raw of the tensor take five elements from the left and five from the right. I do such this simple code. It gives me

TypeError: iteration over a 0-d tensor

i = 0
j = 6
base = feat.size(0)//2
for i in feat[i, j]:
    x = feat[base - 5: base:+5]
    i += 1

Output:
What about this:

x = torch.rand(100,13)
center = x.size(1) // 2
x1 = x[:, center:center+5] # torch.Size([100, 5]) (right)
x2 = x[:, center-5:center] # torch.Size([100, 5]) (left)

Is that what you want?


Instruction: RuntimeError: Could not run 'aten::xxxx', Can't load Pytorch model trained with TPU

Input:
Training for MLM was added based on the Japanese model of BERT.
At that time, we used TPU on Google Colab.
I get the following error when loading the created model.
Is there a way to load the model?

code

from transformers import BertJapaneseTokenizer, BertForMaskedLM

# Load pre-trained model
model = BertForMaskedLM.from_pretrained('/content/drive/My Drive/Bert/models/sample/')
model.eval()

output

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    469             try:
--> 470                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    471             except Exception:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    528                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 529         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    530

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    701     unpickler.persistent_load = persistent_load
--> 702     result = unpickler.load()
    703

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in _rebuild_xla_tensor(data, dtype, device, requires_grad)
    151 def _rebuild_xla_tensor(data, dtype, device, requires_grad):
--> 152     tensor = torch.from_numpy(data).to(dtype=dtype, device=device)
    153     tensor.requires_grad = requires_grad

RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'XLATensorId' backend. 'aten::empty.memory_format' is only available for these backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId, CPUTensorId, MkldnnCPUTensorId, SparseCUDATensorId].

Output:
I ran into the same error while using transformers, this is how I solved it.

After training on Colab, I had to send the model to the CPU. Basically, run:

model.to('cpu')

Then save the model, which allowed me to import the weights in another instance.

As implied by the error,

RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'XLATensorId' backend. 'aten::empty.memory_format' is only available for these backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId, CPUTensorId, MkldnnCPUTensorId, SparseCUDATensorId]


Instruction: Can my PyTorch forward function do additional operations?

Input:
Typically a forward function strings together a bunch of layers and returns the output of the last one. Can I do some additional processing after that last layer before returning? For example, some scalar multiplication and reshaping via .view?

I know that the autograd somehow figures out gradients. So I don’t know if my additional processing will somehow screw that up. Thanks.

Output:
pytorch tracks the gradients via the computational graph of the tensors, not through the functions. As long as your tensors has requires_grad=True property and their grad is not None you can do (almost) whatever you like and still be able to backprop.
As long as you are using pytorch's operations (e.g., those listed in here and here) you should be okay.

For more info see this.

For example (taken from torchvision's VGG implementation):

class VGG(nn.Module):

    def __init__(self, features, num_classes=1000, init_weights=True):
        super(VGG, self).__init__()
        #  ...

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)  # <-- what you were asking about
        x = self.classifier(x)
        return x

A more complex example can be seen in torchvision's implementation of ResNet:

class Bottleneck(nn.Module):
    def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
                 base_width=64, dilation=1, norm_layer=None):
        super(Bottleneck, self).__init__()
        # ...

    def forward(self, x):
        identity = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)

        if self.downsample is not None:    # <-- conditional execution!
            identity = self.downsample(x)

        out += identity  # <-- inplace operations
        out = self.relu(out)

        return out


Instruction: How to deploy an existing pytorch model previously trained with Amazon Sagemaker and stored in S3 bucket

Input:
I have trained a Pytorch model using SageMaker and the model is now stored in an S3 bucket.
I am trying to retrieve that model and deploying it.

This is the code I am using:

estimator = sagemaker.model.FrameworkModel(
    model_data= #link to  model location in s3
    image=  # image
    role=role,
    entry_point='train.py',
    source_dir='pytorch_source',
    sagemaker_session = sagemaker_session
)

predictor = estimator.deploy(initial_instance_count=1, instance_type="ml.p2.xlarge")

But after the deployment process (that seems to run smoothly), the predictor is just a NoneType.
I haven't found any weird message in the logs...

I have also made another attempt with the following code:

estimator = PyTorchModel(model_data= #link to model location in s3
                             role=role,
                             image= #image
                             entry_point='pytorch_source/train.py',
                            predictor_cls = 'pytorch_source/train.py',
                           framework_version = '1.1.0')

predictor = estimator.deploy(initial_instance_count=1, instance_type="ml.p2.xlarge")

But it doesn't even complete the deployment.

Can anyone help with this?

Output:
I actually solved using PyTorchModel with the following settings:

estimator = PyTorchModel(model_data='#path to model,
                             role=role,
                             source_dir='pytorch_source',
                             entry_point='deploy.py',
                            predictor_cls = ImgPredictor,
                           framework_version = '1.1.0')

where ImgPredictor is

from sagemaker.predictor import RealTimePredictor, json_deserializer

class ImgPredictor(RealTimePredictor):
    def __init__(self, endpoint_name, sagemaker_session):
        super(ImgPredictor, self).__init__(endpoint_name, sagemaker_session, content_type='application/x-image',
                                           deserializer = json_deserializer ,accept='application/json')

and deploy.py contains the required functions input_fn, output_fn, model_fn and predict_fn.
Also, a requirements.txt file was missing in the source directory.


Instruction: Torchscripting a module with _ConvNd in forward

Input:
I am using PyTorch 1.4 and need to export a model with convolutions inside a loop in forward:

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()

    def forward(self, x):
        for i in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
            x = torch.nn.Relu()(conv(x))
        return x

torch.jit.script(MyCell())

This gives the following error:

RuntimeError:
Arguments for call are not valid.
The following variants are available:

  _single(float[1] x) -> (float[]):
  Expected a value of type 'List[float]' for argument 'x' but instead found type 'Tensor'.

  _single(int[1] x) -> (int[]):
  Expected a value of type 'List[int]' for argument 'x' but instead found type 'Tensor'.

The original call is:
  File "***/torch/nn/modules/conv.py", line 187
                 padding=0, dilation=1, groups=1,
                 bias=True, padding_mode='zeros'):
        kernel_size = _single(kernel_size)
                      ~~~~~~~ <--- HERE
        stride = _single(stride)
        padding = _single(padding)
'Conv1d.__init__' is being compiled since it was called from 'Conv1d'
  File "***", line ***
    def forward(self, x):
        for _ in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
                   ~~~~~~~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(conv(x))
        return x
'Conv1d' is being compiled since it was called from 'MyCell.forward'
  File "***", line ***
    def forward(self, x, h):
        for _ in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(conv(x))
        return x

I have also tried pre-defining the conv's then putting them in a list inside __init__, but such a type is not allowed by TorchScript:

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.conv = [torch.nn.Conv1d(1, 1, 2*i+3) for i in range(5)]

    def forward(self, x):
        for i in range(len(self.conv)):
            x = torch.nn.Relu()(self.conv[i](x))
        return x

torch.jit.script(MyCell())

This instead gives:

RuntimeError:
Module 'MyCell' has no attribute 'conv' (This attribute exists on the Python module, but we failed to convert Python type: 'list' to a TorchScript type.):
  File "***", line ***
    def forward(self, x):
        for i in range(len(self.conv)):
                           ~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(self.conv[i](x))
        return x

So how to export this module? Background: I am exporting Mixed-scale Dense Networks (source) to TorchScript; while nn.Sequential may work for this simplified case, practically I need to convolve with all the historical convolution outputs in each iteration, which is more than chaining the layers.

Output:
You can use nn.ModuleList() in the following way.

Also, note that you can't subscript nn.ModuleList currently probably due to a bug as mentioned in issue#16123, but use the workaround as mentioned below.

class MyCell(nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.conv = nn.ModuleList([torch.nn.Conv1d(1, 1, 2*i+3) for i in range(5)])
        self.relu = nn.ReLU()

    def forward(self, x):
        for mod in self.conv:
            x = self.relu(mod(x))
        return x

>>> torch.jit.script(MyCell())
RecursiveScriptModule(
  original_name=MyCell
  (conv): RecursiveScriptModule(
    original_name=ModuleList
    (0): RecursiveScriptModule(original_name=Conv1d)
    (1): RecursiveScriptModule(original_name=Conv1d)
    (2): RecursiveScriptModule(original_name=Conv1d)
    (3): RecursiveScriptModule(original_name=Conv1d)
    (4): RecursiveScriptModule(original_name=Conv1d)
  )
  (relu): RecursiveScriptModule(original_name=ReLU)
)


Instruction: Adding a Linear layer to my LSTM made the validation loss skyrocket in PyTorch

Input:
My model was:

    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])

        out = torch.nn.SELU()(lstm_out)
        return out, hidden

It now is:

    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)
        batch_size = lstm_out.size(0)

        flattened_out = lstm_out.view(-1, self.hidden_size * 2)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])
        out = self.linear(flattened_out)

        out = torch.nn.functional.relu(out)

        view_out = out.view(batch_size, self.seq_length, -1)

        return view_out, hidden

I used to get validation loss (with MSELoss) under 1000 after 2-3 epochs. Now with the Linear layer, it is skyrocketing up to 15000 even after 10 epochs. Why would this be?

Output: You can try lowering the learning rate :)


Instruction: installing CuPy with Pytorch

Input:
I want to pass tensors from Pytorch to CuPy and back to do some ops not available in Pytorch. What's the recommended way to install them together?

I already installed Pytorch using conda, and it has installed cudatoolkit package. I'd prefer to use conda to install cupy as well. However CuPy installation instructions recommend uninstalling cudatoolkit (at the very bottom).

So, how do I make sure both Pytorch and CuPy are using the same cuda version, as well as CuDNN, NVCC, etc? Or should I not use conda to install cupy?

Output: Just do conda install -c conda-forge cupy and it should work. That part of documentation is outdated and to be updated.


Instruction: Pytorch cut 2d array by lengths on first dimension

Input:
I have a 2d array, let's say of size torch.tensor(batch_size, 1000).  The 1000 array from the second dimension is actually variable length.  I have a second array of size [batch_size] containing the length for each rows...

Here is an example code snippet:

# preds is the 2d array of size [batch_size, 1000]
# lengths is a 1d array containing the lengths of each row of preds
res_pred = []
for i in range(len(preds)):
    length = lengths[i].item()
    res_pred += [preds[i][:length]]

result = torch.cat(res_pred).flatten()

I do the same thing for my targets and then I can apply a loss function to both.

I was wondering if there was a single vectorized operation I could do to extract all batch_size vectors of variable lengths and torch.cat them together. Right now I am looping on the first dimension, but this feel slow.

Thanks,

Output:
You can create a 2D mask tensor with the number of True's in i-th row given by lengths[i]. Here's one example:

batch_size = 6
n = 5

preds = torch.arange(batch_size * n).reshape(batch_size, n)
# tensor([[ 0,  1,  2,  3,  4],
#         [ 5,  6,  7,  8,  9],
#         [10, 11, 12, 13, 14],
#         [15, 16, 17, 18, 19],
#         [20, 21, 22, 23, 24],
#         [25, 26, 27, 28, 29]])

#lengths = np.random.randint(0, n+1, batch_size)
lengths = torch.randint(0, n+1, (batch_size, ))
# tensor([2, 0, 5, 3, 3, 2])

Let's create the mask and get our result (probably there is a better way to create such a mask, but that's what I came up with):

#mask = np.tile(range(n), (batch_size,1)) < lengths[:,None]
mask = torch.arange(n).repeat((batch_size,1)) < lengths[:, None]
# tensor([[ True,  True, False, False, False],
#        [False, False, False, False, False],
#        [ True,  True,  True,  True,  True],
#        [ True,  True,  True, False, False],
#        [ True,  True,  True, False, False],
#        [ True,  True, False, False, False]])

#result = preds[mask]
result = torch.masked_select(preds, mask)
# tensor([0, 1, 10, 11, 12, 13, 14, 15, 16, 17, 20, 21, 22, 25, 26])

This produces the same result as your code:

res_pred = []
for i in range(len(preds)):
    length = lengths[i].item()
    res_pred += [preds[i][:length]]

result = torch.cat(res_pred).flatten()


Instruction: Gaussian filter in PyTorch

Input:
I am looking for a way to apply a Gaussian filter to an image (tensor) only using PyTorch functions. Using numpy, the equivalent code is

import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

# Define 2D Gaussian kernel
def gkern(kernlen=256, std=128):
    """Returns a 2D Gaussian kernel array."""
    gkern1d = signal.gaussian(kernlen, std=std).reshape(kernlen, 1)
    gkern2d = np.outer(gkern1d, gkern1d)
    return gkern2d

# Generate random matrix and multiply the kernel by it
A = np.random.rand(256*256).reshape([256,256])

# Test plot
plt.figure()
plt.imshow(A*gkern(256, std=32))
plt.show()

The closest suggestion I found is based on this post:

import torch.nn as nn

conv = nn.Conv2d(in_channels = 1, out_channels = 1, kernel_size=264, bias=False)
with torch.no_grad():
    conv.weight = gaussian_weights

But it gives me the error NameError: name 'gaussian_weights' is not defined. How can I make it work?

Output:
There is a Pytorch class to apply Gaussian Blur to your image:
torchvision.transforms.GaussianBlur(kernel_size, sigma=(0.1, 2.0))

Check the documentation for more info


Instruction: HTTPError: HTTP Error 403: Forbidden on Google Colab

Input:
I am trying to download MNIST data in PyTorch using the following code:

train_loader = torch.utils.data.DataLoader(
      datasets.MNIST('data',
                      train=True,
                      download=True,
                      transform=transforms.Compose([
                         transforms.ToTensor(),
                         transforms.Normalize((0.1307,), (0.3081,))
                     ])),
      batch_size=128, shuffle=True)

and it gives the following error.

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz
0it [00:00, ?it/s]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-2-2fee284dabb8> in <module>()
      5                       transform=transforms.Compose([
      6                          transforms.ToTensor(),
----> 7                          transforms.Normalize((0.1307,), (0.3081,))
      8                      ])),
      9       batch_size=128, shuffle=True)

11 frames
/usr/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    648 class HTTPDefaultErrorHandler(BaseHandler):
    649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    651
    652 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

How do I solve this? The notebook was working before, I'm trying to rerun it but I got this error.

Output:
This is a new bug, reported here: https://github.com/pytorch/vision/issues/1938

See that thread for some potential workarounds until the issue is fixed in pytorch itself.


Instruction: Questions regarding image processing for pre-trained image classifier in PyTorch

Input:
I am trying to use a popular pre-trained VGG model for image classification in PyTorch but noticed that the image is resized to 256 and crop it to 224 for pre-processing images in most of the programs. I am curious why we do resize it to 256 first and crop it instead of resizing it to 224 directly.

transforms = transforms.Compose([transforms.Resize(256),
                                 transforms.CenterCrop(224),
                                 transforms.ToTensor(),
                                 transforms.Normalize([0.485, 0.456, 0.406],
                                                     [0.229, 0.224, 0.225])])

Output:
For image classification tasks, typically the object of interest is located in the center of the image. It is thus common practice (for inference) to take a central crop of the image cutting away some border (this does not apply in general, however, as exact preprocessing depends strongly on how the network was trained).

As per the "why cropping and not resizing directly", this is a byproduct of data augmentation during training: taking a random crop of the image is a very common data augmentation technique. At inference time, resizing the whole image to the input size instead of applying a crop influences the scale of the objects in the image, which negatively affects the network's performance (because you're evaluating on data that has a "format" different from the one you trained on and CNNs are not scale-invariant).


Instruction: Strange Move Assignment Operator Signature

Input:
I came across an unfamiliar move assignment operator signature in Pytorch' tensor backend (ATen, source).
Just out of curiosity, what does the && operator do at the end of

Tensor & Tensor::operator=(Tensor && rhs) &&

While I'm familiar with move semantics and the usual copy/move constructor and assignment operator signatures, I could not find any documentation online about the syntax above.

I would be grateful if someone could explain what this operator does, how it differs from the usual move assignment operation, and when it should be used.

Output:
Objects of a class used as  expressions can be rvalues or lvalues. The move assignment operator is a member function of a class.

This declaration

Tensor & Tensor::operator=(Tensor && rhs) &&

means that this move assignment operator is called for rvalue object of the class.

Here is a demonstrative program.

#include <iostream>

struct A
{
    A & operator =( A && ) &
    {
        std::cout << "Calling the move assignment operator for an lvalue object\n";
        return *this;
    }

    A & operator =( A && ) &&
    {
        std::cout << "Calling the move assignment operator for an rvalue object\n";
        return *this;
    }
};

int main()
{
    A a;

    a = A();

    A() = A();

    return 0;
}

The program output is

Calling the move assignment operator for an lvalue object
Calling the move assignment operator for an rvalue object

That is in this statement

    a = A();

the left hand operand of the assignment is an lvalue.

In this statement

    A() = A();

the left hand operand of the assignment is rvalue (a temporary object).


Instruction: Verify convolution theorem using pytorch

Input:
Basically this theorem is formulated as below:

F(f*g) = F(f)xF(g)

I know this theorem but I just simply cannot reproduce the result by using pytorch.

Below is a reproducable code:

import torch
import torch.nn.functional as F

# calculate f*g
f = torch.ones((1,1,5,5))
g = torch.tensor(list(range(9))).view(1,1,3,3).float()
conv = F.conv2d(f, g, bias=None, padding=2)

# calculate F(f*g)
F_fg = torch.rfft(conv, signal_ndim=2, onesided=False)

# calculate F x G
f = f.squeeze()
g = g.squeeze()

# need to pad into at least [w1+w2-1, h1+h2-1], which is 7 in our case.
size = f.size(0) + g.size(0) - 1

f_new = torch.zeros((7,7))
g_new = torch.zeros((7,7))

f_new[1:6,1:6] = f
g_new[2:5,2:5] = g

F_f = torch.rfft(f_new, signal_ndim=2, onesided=False)
F_g = torch.rfft(g_new, signal_ndim=2, onesided=False)
FxG = torch.mul(F_f, F_g)

print(FxG - F_fg)

here is the result for print(FxG - F_fg)

tensor([[[[[ 0.0000e+00,  0.0000e+00],
       [ 4.1426e+02,  1.7270e+02],
       [-3.6546e+01,  4.7600e+01],
       [-1.0216e+01, -4.1198e+01],
       [-1.0216e+01, -2.0223e+00],
       [-3.6546e+01, -6.2804e+01],
       [ 4.1426e+02, -1.1427e+02]],

      ...

      [[ 4.1063e+02, -2.2347e+02],
       [-7.6294e-06,  2.2817e+01],
       [-1.9024e+01, -9.0105e+00],
       [ 7.1708e+00, -4.1027e+00],
       [-2.6739e+00, -1.1121e+01],
       [ 8.8471e+00,  7.1710e+00],
       [ 4.2528e+01,  9.7559e+01]]]]])

and you can see that the difference is not always 0.

can someone tell me why and how to do this properly?

Thanks

Output:
So I took a closer look at what you've done so far. I've identified three sources of error in your code. I'll try to sufficiently address each of them here.
1. Complex arithmetic
PyTorch doesn't currently support multiplication of complex numbers (AFAIK). The FFT operation simply returns a tensor with a real and imaginary dimension. Instead of using torch.mul or the * operator we need to explicitly code complex multiplication.

(a + ib) * (c + id) = (a*c - b*d) + i(a*d + b*c)

2. The definition of convolution
The definition of "convolution" often used in CNN literature is actually different from the definition used when discussing the convolution theorem. I won't go into detail, but the theoretical definition flips the kernel before sliding and multiplying. Instead, the convolution operation in pytorch, tensorflow, caffe, etc... doesn't do this flipping.
To account for this we can simply flip g (both horizontally and vertically) before applying the FFT.
3. Anchor position
The anchor-point when using the convolution theorem is assumed to be the upper left corner of the padded g. Again, I won't go into detail about this but it's how the math works out.

The second and third point may be easier to understand with an example. Suppose you used the following g
[1 2 3]
[4 5 6]
[7 8 9]

instead of g_new being
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 1 2 3 0 0]
[0 0 4 5 6 0 0]
[0 0 7 8 9 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]

it should actually be
[5 4 0 0 0 0 6]
[2 1 0 0 0 0 3]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[8 7 0 0 0 0 9]

where we flip the kernel vertically and horizontally, then apply circular shift so that the center of the kernel is in the upper left corner.

I ended up rewriting most of your code and generalizing it a bit. The most complex operation is defining g_new properly. I decided to use a meshgrid and modulo arithmetic to simultaneously flip and shift the indices. If something here doesn't make sense to you please leave a comment and I'll try to clarify.
import torch
import torch.nn.functional as F

def conv2d_pyt(f, g):
    assert len(f.size()) == 2
    assert len(g.size()) == 2

    f_new = f.unsqueeze(0).unsqueeze(0)
    g_new = g.unsqueeze(0).unsqueeze(0)

    pad_y = (g.size(0) - 1) // 2
    pad_x = (g.size(1) - 1) // 2

    fcg = F.conv2d(f_new, g_new, bias=None, padding=(pad_y, pad_x))
    return fcg[0, 0, :, :]

def conv2d_fft(f, g):
    assert len(f.size()) == 2
    assert len(g.size()) == 2

    # in general not necessary that inputs are odd shaped but makes life easier
    assert f.size(0) % 2 == 1
    assert f.size(1) % 2 == 1
    assert g.size(0) % 2 == 1
    assert g.size(1) % 2 == 1

    size_y = f.size(0) + g.size(0) - 1
    size_x = f.size(1) + g.size(1) - 1

    f_new = torch.zeros((size_y, size_x))
    g_new = torch.zeros((size_y, size_x))

    # copy f to center
    f_pad_y = (f_new.size(0) - f.size(0)) // 2
    f_pad_x = (f_new.size(1) - f.size(1)) // 2
    f_new[f_pad_y:-f_pad_y, f_pad_x:-f_pad_x] = f

    # anchor of g is 0,0 (flip g and wrap circular)
    g_center_y = g.size(0) // 2
    g_center_x = g.size(1) // 2
    g_y, g_x = torch.meshgrid(torch.arange(g.size(0)), torch.arange(g.size(1)))
    g_new_y = (g_y.flip(0) - g_center_y) % g_new.size(0)
    g_new_x = (g_x.flip(1) - g_center_x) % g_new.size(1)
    g_new[g_new_y, g_new_x] = g[g_y, g_x]

    # take fft of both f and g
    F_f = torch.rfft(f_new, signal_ndim=2, onesided=False)
    F_g = torch.rfft(g_new, signal_ndim=2, onesided=False)

    # complex multiply
    FxG_real = F_f[:, :, 0] * F_g[:, :, 0] - F_f[:, :, 1] * F_g[:, :, 1]
    FxG_imag = F_f[:, :, 0] * F_g[:, :, 1] + F_f[:, :, 1] * F_g[:, :, 0]
    FxG = torch.stack([FxG_real, FxG_imag], dim=2)

    # inverse fft
    fcg = torch.irfft(FxG, signal_ndim=2, onesided=False)

    # crop center before returning
    return fcg[f_pad_y:-f_pad_y, f_pad_x:-f_pad_x]

# calculate f*g
f = torch.randn(11, 7)
g = torch.randn(5, 3)

fcg_pyt = conv2d_pyt(f, g)
fcg_fft = conv2d_fft(f, g)

avg_diff = torch.mean(torch.abs(fcg_pyt - fcg_fft)).item()

print('Average difference:', avg_diff)

Which gives me
Average difference: 4.6866085767760524e-07

This is very close to zero. The reason we don't get exactly zero is simply due to floating point errors.


Instruction: Pytorch error: Could not run 'aten::slow_conv3d_forward' with arguments from the 'CUDATensorId' backend

Input:
I am training a CNN on CUDA GPU which takes 3D medical images as input and outputs a classifier. I suspect there may be a bug in pytorch. I am running pytorch 1.4.0.  The GPU is 'Tesla P100-PCIE-16GB'. When I run the model on CUDA I get the error

Traceback (most recent call last):
  File "/home/ub/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-55-cc0dd3d9cbb7>", line 1, in <module>
    net(cc)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "<ipython-input-2-19e11966d1cd>", line 181, in forward
    out = self.layer1(x)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 480, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Could not run 'aten::slow_conv3d_forward' with arguments from the 'CUDATensorId' backend. 'aten::slow_conv3d_forward' is only available for these backends: [CPUTensorId, VariableTensorId].

To replicate the issue:

#input is a 64,64,64 3d image batch with 2 channels
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv3d(2, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv3d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(16 * 16*16 * 64, 1000)
        self.fc2 = nn.Linear(1000, 2)
        # self.softmax =  nn.LogSoftmax(dim=1)

    def forward(self, x):
        # print(out.shape)
        out = self.layer1(x)
        # print(out.shape)
        out = self.layer2(out)
        # print(out.shape)
        out = out.reshape(out.size(0), -1)
        # print(out.shape)
        out = self.drop_out(out)
        # print(out.shape)
        out = self.fc1(out)
        # print(out.shape)
        out = self.fc2(out)
        # out = self.softmax(out)
        # print(out.shape)
        return out

net = Convnet()
input = torch.randn(16, 2, 64, 64, 64)
net(input)

Output:
Initially, I was thinking the error message indicates that 'aten::slow_conv3d_forward' is not implemented with GPU (CUDA). But after looked at your network, it does not make sense to me, since Conv3D is a very basic op, and Pytorch team should implement this in CUDA.

Then I dived a bit about the source code, finding that the input is not a CUDA tensor, which causes the problem.

Here is a working sample:

import torch
from torch import nn

#input is a 64,64,64 3d image batch with 2 channels
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv3d(2, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv3d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(16 * 16*16 * 64, 1000)
        self.fc2 = nn.Linear(1000, 2)
        # self.softmax =  nn.LogSoftmax(dim=1)

    def forward(self, x):
        # print(out.shape)
        out = self.layer1(x)
        # print(out.shape)
        out = self.layer2(out)
        # print(out.shape)
        out = out.reshape(out.size(0), -1)
        # print(out.shape)
        out = self.drop_out(out)
        # print(out.shape)
        out = self.fc1(out)
        # print(out.shape)
        out = self.fc2(out)
        # out = self.softmax(out)
        # print(out.shape)
        return out

net = ConvNet()
input = torch.randn(16, 2, 64, 64, 64)
net.cuda()
input = input.cuda() # IMPORTANT to reassign your tensor
net(input)

Remember when you put a model from CPU to GPU, you can directly call .cuda(), but if you put a tensor from CPU to GPU, you will need to reassign it, such as tensor = tensor.cuda(), instead of only calling tensor.cuda(). Hope that helps.

Output:

tensor([[-0.1588,  0.0680],
        [ 0.1514,  0.2078],
        [-0.2272, -0.2835],
        [-0.1105,  0.0585],
        [-0.2300,  0.2517],
        [-0.2497, -0.1019],
        [ 0.1357, -0.0475],
        [-0.0341, -0.3267],
        [-0.0207, -0.0451],
        [-0.4821, -0.0107],
        [-0.1779,  0.1247],
        [ 0.1281,  0.1830],
        [-0.0595, -0.1259],
        [-0.0545,  0.1838],
        [-0.0033, -0.1353],
        [ 0.0098, -0.0957]], device='cuda:0', grad_fn=<AddmmBackward>)


Instruction: Pytorch Quantization RuntimeError: Trying to create tensor with negative dimension

Input:
I am trying out the pytorch quantization module. When doing static post training quantization I follow the
next procedure detailed in the documentation:

adding QuantStub and DeQuantStub modules
Fuse operations
Specify qauntization config
torch.quantization.prepare()
Calibrate the model by running inference against a calibration dataset
torch.quantization.convert()

However, when calibrating the model after preparing it the program breaks.

The error appears at the last fully connected layers. It seems that the observers introduced in the graph are trying to create an histogram of negative dimension.

Here is the error:

    x = self.fc(x)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 552, in __call__
    hook_result = hook(self, input, result)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/quantize.py", line 74, in _observer_forward_hook
    return self.activation_post_process(output)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/observer.py", line 805, in forward
    self.bins)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/observer.py", line 761, in _combine_histograms
    histogram_with_output_range = torch.zeros((Nbins * downsample_rate))
RuntimeError: Trying to create tensor with negative dimension -4398046511104: [-4398046511104]

The fully connected are built as

class LinearReLU(nn.Sequential):
    def __init__(self, in_neurons, out_neurons):
        super(LinearReLU, self).__init__(
            nn.Linear(in_neurons, out_neurons),
            nn.ReLU(inplace=False)
        )

They are appended in the fc(x) as fc = nn.Sequential(*([LinearReLU, LinearReLU, ...]).

However, I suspect that it has something to do with the reshape between the convolutions and the fully connected layers.

x = x.reshape(-1, size)

Until now I have not been able to solve this error.

Thanks in advance

Output:
For anybody that has the same problem.

The solution is in this line in pytorch quantization documentation:

  View-based operations like view(), as_strided(), expand(), flatten(), select(), python-style indexing, etc - work as on regular tensor (if quantization is not per-channel)

The problem was using reshape and doing per channel quantization. If I do meanof the last two channels there is no problem.


Instruction: Pytorch model stuck at 0.5 though loss decreases consistently

Input:
This is using PyTorch

I have been trying to implement UNet model on my images, however, my model accuracy is always exact 0.5. Loss does decrease.

I have also checked for class imbalance. I have also tried playing with learning rate. Learning rate affects loss but not the accuracy.

My architecture below ( from here )

""" `UNet` class is based on https://arxiv.org/abs/1505.04597

The U-Net is a convolutional encoder-decoder neural network.
Contextual spatial information (from the decoding,
expansive pathway) about an input tensor is merged with
information representing the localization of details
(from the encoding, compressive pathway).

Modifications to the original paper:
(1) padding is used in 3x3 convolutions to prevent loss
    of border pixels
(2) merging outputs does not require cropping due to (1)
(3) residual connections can be used by specifying
    UNet(merge_mode='add')
(4) if non-parametric upsampling is used in the decoder
    pathway (specified by upmode='upsample'), then an
    additional 1x1 2d convolution occurs after upsampling
    to reduce channel dimensionality by a factor of 2.
    This channel halving happens with the convolution in
    the tranpose convolution (specified by upmode='transpose')

    Arguments:
        in_channels: int, number of channels in the input tensor.
                     Default is 3 for RGB images. Our SPARCS dataset is 13 channel.
              depth: int, number of MaxPools in the U-Net. During training, input size needs to be
                     (depth-1) times divisible by 2
        start_filts: int, number of convolutional filters for the first conv.
            up_mode: string, type of upconvolution. Choices: 'transpose' for transpose convolution

"""

class UNet(nn.Module):

    def __init__(self, num_classes, depth, in_channels, start_filts=16, up_mode='transpose', merge_mode='concat'):

        super(UNet, self).__init__()

        if up_mode in ('transpose', 'upsample'):
            self.up_mode = up_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for upsampling. Only \"transpose\" and \"upsample\" are allowed.".format(up_mode))

        if merge_mode in ('concat', 'add'):
            self.merge_mode = merge_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for merging up and down paths.Only \"concat\" and \"add\" are allowed.".format(up_mode))

        # NOTE: up_mode 'upsample' is incompatible with merge_mode 'add'
        if self.up_mode == 'upsample' and self.merge_mode == 'add':
            raise ValueError("up_mode \"upsample\" is incompatible with merge_mode \"add\" at the moment "
                             "because it doesn't make sense to use nearest neighbour to reduce depth channels (by half).")

        self.num_classes = num_classes
        self.in_channels = in_channels
        self.start_filts = start_filts
        self.depth = depth

        self.down_convs = []
        self.up_convs = []

        # create the encoder pathway and add to a list
        for i in range(depth):
            ins = self.in_channels if i == 0 else outs
            outs = self.start_filts*(2**i)
            pooling = True if i < depth-1 else False

            down_conv = DownConv(ins, outs, pooling=pooling)
            self.down_convs.append(down_conv)

        # create the decoder pathway and add to a list
        # - careful! decoding only requires depth-1 blocks
        for i in range(depth-1):
            ins = outs
            outs = ins // 2
            up_conv = UpConv(ins, outs, up_mode=up_mode, merge_mode=merge_mode)
            self.up_convs.append(up_conv)

        self.conv_final = conv1x1(outs, self.num_classes)

        # add the list of modules to current module
        self.down_convs = nn.ModuleList(self.down_convs)
        self.up_convs = nn.ModuleList(self.up_convs)

        self.reset_params()

    @staticmethod
    def weight_init(m):
        if isinstance(m, nn.Conv2d):

            #https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/
            ##Doc: https://pytorch.org/docs/stable/nn.init.html?highlight=xavier#torch.nn.init.xavier_normal_
            init.xavier_normal_(m.weight)
            init.constant_(m.bias, 0)

    def reset_params(self):
        for i, m in enumerate(self.modules()):
            self.weight_init(m)

    def forward(self, x):
        encoder_outs = []

        # encoder pathway, save outputs for merging
        for i, module in enumerate(self.down_convs):
            x, before_pool = module(x)
            encoder_outs.append(before_pool)

        for i, module in enumerate(self.up_convs):
            before_pool = encoder_outs[-(i+2)]
            x = module(before_pool, x)

        # No softmax is used. This means we need to use
        # nn.CrossEntropyLoss is your training script,
        # as this module includes a softmax already.
        x = self.conv_final(x)
        return x

Parameters are :

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x,y = train_sequence[0] ; batch_size = x.shape[0]
model = UNet(num_classes = 2, depth=5, in_channels=5, merge_mode='concat').to(device)
optim = torch.optim.Adam(model.parameters(),lr=0.01, weight_decay=1e-3)
criterion = nn.BCEWithLogitsLoss() #has sigmoid internally
epochs = 1000

The function for training is :

import torch.nn.functional as f

def train_model(epoch,train_sequence):
    """Train the model and report validation error with training error
    Args:
        model: the model to be trained
        criterion: loss function
        data_train (DataLoader): training dataset
    """
    model.train()
    for idx in range(len(train_sequence)):
        X, y = train_sequence[idx]
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)

        outputs = model(images)
        print(masks.shape, outputs.shape)
        loss = criterion(outputs, masks)
        optim.zero_grad()
        loss.backward()
        # Update weights
        optim.step()
    # total_loss = get_loss_train(model, data_train, criterion)

My function for calculating loss and accuracy is below:

def get_loss_train(model, train_sequence):
    """
        Calculate loss over train set
    """
    model.eval()
    total_acc = 0
    total_loss = 0
    for idx in range(len(train_sequence)):
        with torch.no_grad():
            X, y = train_sequence[idx]
            images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
            masks = Variable(torch.from_numpy(y)).to(device)

            outputs = model(images)
            loss = criterion(outputs, masks)
            preds = torch.argmax(outputs, dim=1).float()
            acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
            total_acc = total_acc + acc
            total_loss = total_loss + loss.cpu().item()
    return total_acc/(len(train_sequence)), total_loss/(len(train_sequence))

Edit : Code which runs (calls) the functions:

for epoch in range(epochs):
    train_model(epoch, train_sequence)
    train_acc, train_loss = get_loss_train(model,train_sequence)
    print("Train Acc:", train_acc)
    print("Train loss:", train_loss)

Can someone help me identify as why is accuracy always exact 0.5?

Edit-2:

As asked accuracy_check_for_batch function is here:

def accuracy_check_for_batch(masks, predictions, batch_size):
    total_acc = 0
    for index in range(batch_size):
        total_acc += accuracy_check(masks[index], predictions[index])
    return total_acc/batch_size

and

def accuracy_check(mask, prediction):
    ims = [mask, prediction]
    np_ims = []
    for item in ims:
        if 'str' in str(type(item)):
            item = np.array(Image.open(item))
        elif 'PIL' in str(type(item)):
            item = np.array(item)
        elif 'torch' in str(type(item)):
            item = item.numpy()
        np_ims.append(item)

    compare = np.equal(np_ims[0], np_ims[1])
    accuracy = np.sum(compare)

    return accuracy/len(np_ims[0].flatten())

Output:
I found the mistake.

model = UNet(num_classes = 2, depth=5, in_channels=5, merge_mode='concat').to(device)

should be

model = UNet(num_classes = 1, depth=5, in_channels=5, merge_mode='concat').to(device)

because I am using BCELosswithLogits.


Instruction: How to create a train-val split in custom image datasets using PyTorch?

Input:
I want to create a train+val set from my original trainset. The directory is split into train and test. I load the original train set and want to split it into train and val sets so I can evaulate validation loss during training using the train_loader and val_loader.

There's not a lot of documentation about this which explains things clearly.

Output:
Check out the answer here.

I've posted it below as well.

======================================================

The data is read using ImageFolder. Task is binary image classification with 498 images in the dataset which are equally distributed among both classes (249 images each).

img_dataset = ImageFolder(..., transforms=t)

1. SubsetRandomSampler

dataset_size = len(img_dataset)
dataset_indices = list(range(dataset_size))

np.random.shuffle(dataset_indices)

val_split_index = int(np.floor(0.2 * dataset_size))

train_idx, val_idx = dataset_indices[val_split_index:], dataset_indices[:val_split_index]

train_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(val_idx)

train_loader = DataLoader(dataset=img_dataset, shuffle=False, batch_size=8, sampler=train_sampler)
validation_loader = DataLoader(dataset=img_dataset, shuffle=False, batch_size=1, sampler=val_sampler)

2. random_split

Here, out of the 498 total images, 400 get randomly assigned to train and the rest 98 to validation.

dataset_train, dataset_valid = random_split(img_dataset, (400, 98))

train_loader = DataLoader(dataset=dataset_train, shuffle=True, batch_size=8)
val_loader = DataLoader(dataset=dataset_valid, shuffle=False, batch_size=1)

3. WeightedRandomSampler

  if someone stumbled here searching for WeightedRandomSampler, check @ptrblck's answer here for a good explanation of what is happening below.

Now, how does WeightedRandomSampler fit in creating train+val set? Because unlike SubsetRandomSampler or random_split(),  we're not splitting for train and val here. We're simply ensuring that each batch gets equal number of classes during training.

So, my guess is we need to use WeightedRandomSampler after random_split() or SubsetRandomSampler. But this wouldn't ensure that train and val have similar ratio between classes.

target_list = []

for _, t in imgdataset:
    target_list.append(t)

target_list = torch.tensor(target_list)
target_list = target_list[torch.randperm(len(target_list))]

# get_class_distribution() is a function that takes in a dataset and
# returns a dictionary with class count. In this case, the
# get_class_distribution(img_dataset)  returns the following -
# {'class_0': 249, 'class_0': 249}
class_count = [i for i in get_class_distribution(img_dataset).values()]
class_weights = 1./torch.tensor(class_count, dtype=torch.float)

class_weights_all = class_weights[target_list]

weighted_sampler = WeightedRandomSampler(
    weights=class_weights_all,
    num_samples=len(class_weights_all),
    replacement=True
)


Instruction: TypeError: unsupported operand type(s) for +: 'Tensor' and 'dict'

Input:
I am new to the world of neural networks and I am trying to implement a CNN generator from this model  and these equations (N=32) in order to make motion generation. I wrote the following code, where H_txt is a dictionary containing, as keys, the name of my clips and as values, a vector representing the action shown in the clip, and z is a white gaussian noise of dimension (1, 256).

N=32

class CNNGenerator(nn.Module):

    def __init__(self, htxt = H_txt):
        super(CNNGenerator, self).__init__()

        self.htxt = htxt
        self.conv1 = nn.Conv1d(1, 1, 3)
        self.conv2 = nn.Conv1d(1, 1, 3)
        self.conv3 = nn.Conv1d(1, 1, 3)
        self.conv4 = nn.Conv1d(4, 4, 3)
        self.conv5 = nn.Conv1d(2, 2, 3)
        self.conv6 = nn.Conv1d(8, 8, 3)
        self.conv7 = nn.Conv1d(4, 4, 3)
        self.conv8 = nn.Conv1d(16, 16, 3)
        self.conv9 = nn.Conv1d(8, 8, 3)
        self.conv10 = nn.Conv1d(32, 32, 3)
        self.conv11 = nn.Conv1d(16, 16, 3)
        self.conv12 = nn.Conv1d(32, 32, 3)
        self.conv13 = nn.Conv1d(1, 1, 3)
        self.conv14 = nn.Conv1d(2, 2, 3)
        self.conv15 = nn.Conv1d(2, 2, 3)
        self.conv16 = nn.Conv1d(4, 4, 3)
        self.conv17 = nn.Conv1d(4, 4, 3)
        self.conv18 = nn.Conv1d(8, 8, 3)
        self.conv19 = nn.Conv1d(8, 8, 3)
        self.conv20 = nn.Conv1d(16, 16, 3)
        self.conv21 = nn.Conv1d(16, 16, 3)
        self.conv22 = nn.Conv1d(32, 32, 3)
        self.conv23 = nn.Conv1d(32, 32, 3)

    def forward(self, x):
        x[0] = self.conv1(F.relu(self.conv2(z) + self.htxt))

        x[1] = self.conv3(F.relu(self.conv4(z) + self.htxt))
        x[2] = self.conv5(F.relu(self.conv6(z) + self.htxt))
        x[3] = self.conv7(F.relu(self.conv8(z) + self.htxt))
        x[4] = self.conv9(F.relu(self.conv10(z) + self.htxt))
        x[5] = self.conv11(F.relu(self.conv12(z) + self.htxt))
        h = np.zeros(np.log2(N))

        h[0] = x[0]

        h[1] = nn.AdaptiveAvgPool1d(2*h[0]) + self.conv13(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv14(h[0])+x[1]))))
        h[2] = nn.AdaptiveAvgPool1d(2*h[1]) + self.conv15(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv16(h[1])+x[2]))))
        h[3] = nn.AdaptiveAvgPool1d(2*h[2]) + self.conv17(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv18(h[2])+x[3]))))
        h[4] = nn.AdaptiveAvgPool1d(2*h[3]) + self.conv19(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv20(h[3])+x[4]))))
        h[5] = nn.AdaptiveAvgPool1d(2*h[4]) + self.conv21(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv22(h[4])+x[5]))))
        A = self.conv23(h[np.log2(N)])
        return A

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

    net = CNNGenerator()
    z= torch.randn(1, 1, 256)       #k=256
    out = net(z)
    print(out)

When I run my code, I get the following error message, coming from my  forward function :

TypeError: unsupported operand type(s) for +: 'Tensor' and 'dict'

My code doesn't like the fact that I try to sum a tensor with a dictionary. I haven't found any solution online, so I was wondering if it was possible to sum my tensor with a dictionary? Is there a function that can convert my dictionary into a tensor?
I tried to convert the dictionary into an array with np.asarray() but I got an error message saying I can't use a numpy function for this.

Thanks a lot for reading my message

Output:
Your question is missing details about your variables but based on the error i am giving you my answer.You are adding a dictionary and a tensor which is giving you the error.If you want to add the values of dictionary to the tensor then you must convert the dictionary into a tensor.And also why are you adding the dictionary directly because here there is no need for keys.If you want to concatenate the values in the dictionary after converting it into tensor along the desired axis then you need to use torch.cat function.
To add the values in the dictionary to the tensor get the values from dict and convert to tensor by something like this

torch.Tensor(list(htxt.values()))


Instruction: Can autograd in pytorch handle a repeated use of a layer within the same module?

Input:
I have a layer layer in an nn.Module and use it two or more times during a single forward step. The output of this layer is later inputted to the same layer. Can pytorch's autograd compute the grad of the weights of this layer correctly?
def forward(x):
    x = self.layer(x)
    x = self.layer(x)
    return x

Complete example:
import torch
import torch.nn as nn
import torch.nn.functional as F

class net(nn.Module):
    def __init__(self,in_dim,out_dim):
        super(net,self).__init__()
        self.layer = nn.Linear(in_dim,out_dim,bias=False)

    def forward(self,x):
        x = self.layer(x)
        x = self.layer(x)
        return x

input_x = torch.tensor([10.])
label = torch.tensor([5.])
n = net(1,1)
loss_fn = nn.MSELoss()

out = n(input_x)
loss = loss_fn(out,label)
n.zero_grad()
loss.backward()

for param in n.parameters():
    w = param.item()
    g = param.grad

print('Input = %.4f; label = %.4f'%(input_x,label))
print('Weight = %.4f; output = %.4f'%(w,out))
print('Gradient w.r.t. the weight is %.4f'%(g))
print('And it should be %.4f'%(4*(w**2*input_x-label)*w*input_x))

Output:
Input = 10.0000; label = 5.0000
Weight = 0.9472; output = 8.9717
Gradient w.r.t. the weight is 150.4767
And it should be 150.4766

In this example, I have defined a module with only one linear layer (in_dim=out_dim=1 and no bias). w is the weight of this layer; input_x is the input value; label is the desired value. Since the loss is chosen as MSE, the formula for the loss is
((w^2)*input_x-label)^2
Computing by hand, we have
dw/dx = 2*((w^2)*input_x-label)*(2*w*input_x)
The output of my example above shows that autograd gives the same result as computed by hand, giving me a reason to believe that it can work in this case. But in a real application, the layer may have inputs and outputs of higher dimensions, a nonlinear activation function after it, and the neural network could have multiple layers.
What I want to ask is: can I trust autograd to handle such situation, but a lot more complicated than that in my example? How does it work when a layer is called iteratively?

Output:
This will work just fine. From the perspective of the autograd engine this isn't a cyclic application since the resulting computation graph will unwrap the repeated computation as a linear sequence. To illustrate this, for a single layer you might have:

x -----> layer --------+
           ^           |
           |  2 times  |
           +-----------+

From the autograd perspective this looks like:

x ---> layer ---> layer ---> layer

Here layer is the same layer copied 3 times over the graph. This means when computing the gradient for the layer's weights they will be accumulated from all the three stages. So when using backward:

x ---> layer ---> layer ---> layer ---> loss_func
                                            |
       lback <--- lback <--- lback <--------+
         |          |          |
         |          v          |
         +------> weights <----+
                   _grad

Here lback represents the local derivative of the layer forward transformation which uses the upstream gradient as an input. Each one adds to the layer's weights_grad.

Recurrent Neural Networks use this repeated application of layers (cells) at their basis. See for example this tutorial about Classifying Names with a Character-Level RNN.


Instruction: Parallel analog to torch.nn.Sequential container

Input: Just wondering, why I can't find subj in torch.nn? nn.Sequential is pretty convinient, it allows to define networks in one place, clear and visual, but restricted to very simple ones! With parallel analog (and litle help of "identity" nodes for residual connections) it forms a complete method to construct any feedforward net combinatorial way. Am I missing something?

Output:
Well, maybe it shouldn't be in standard module collection, just because it can be defined really simple:

class ParallelModule(nn.Sequential):
    def __init__(self, *args):
        super(ParallelModule, self).__init__( *args )

    def forward(self, input):
        output = []
        for module in self:
            output.append( module(input) )
        return torch.cat( output, dim=1 )

Inheriting "Parallel" from "Sequential" is ideologically bad, but works well.
Now one can define networks like pictured, with following code:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.net = nn.Sequential(
            nn.Conv2d(  1, 32, 3, padding=1 ),        nn.ReLU(),
            nn.Conv2d( 32, 64, 3, padding=1 ),        nn.ReLU(),
            nn.MaxPool2d( 3, stride=2 ),              nn.Dropout2d( 0.25 ),

            ParallelModule(
                nn.Conv2d(  64, 64, 1 ),
                nn.Sequential(
                    nn.Conv2d(  64, 64, 1 ),          nn.ReLU(),
                    ParallelModule(
                        nn.Conv2d(  64, 32, (3,1), padding=(1,0) ),
                        nn.Conv2d(  64, 32, (1,3), padding=(0,1) ),
                    ),
                ),
                nn.Sequential(
                    nn.Conv2d(  64, 64, 1 ),          nn.ReLU(),
                    nn.Conv2d(  64, 64, 3, padding=1 ), nn.ReLU(),
                    ParallelModule(
                        nn.Conv2d(  64, 32, (3,1), padding=(1,0) ),
                        nn.Conv2d(  64, 32, (1,3), padding=(0,1) ),
                    ),
                ),
                nn.Sequential(
                    #PrinterModule(),
                    nn.AvgPool2d( 3, stride=1, padding=1 ),
                    nn.Conv2d(  64, 64, 1 ),
                ),
            ),
            nn.ReLU(),
            nn.Conv2d( 256, 64, 1 ),                  nn.ReLU(),

            nn.Conv2d( 64, 128, 3, padding=1 ),       nn.ReLU(),
            nn.MaxPool2d( 3, stride=2 ),              nn.Dropout2d( 0.5 ),
            nn.Flatten(),
            nn.Linear( 4608, 128 ),                   nn.ReLU(),
            nn.Linear(  128,  10 ),                   nn.LogSoftmax( dim=1 ),
        )

    def forward(self, x):
        return self.net.forward( x )


Instruction: Appending missing values according to index

Input:
Say if i have a tensor

values = torch.tensor([5, 3, 2, 8])

and a corresponding index to values

index = torch.tensor([0, 2, 4, 5])

and assuming that I want to insert in the missing index (1 and 3), with a fixed value (100), such that I get

values = torch.tensor([5, 100, 3, 100, 2, 8])

Is there a vectorized way to do it in PyTorch (or numpy)?

Output:
You can fill it with 100 first and then fill it with the original values.

in pytorch

import torch

result = torch.empty(6, dtype = torch.int32).fill_(100)
values = torch.tensor([5, 3, 2, 8], dtype = torch.int32)
index = torch.tensor([0, 2, 4, 5])
result[index] = values
print(result)

in numpy

import numpy as np

result = np.full((6,), 100)
index = np.array([0, 2, 4, 5])
values = np.array([5, 3, 2, 8])
result[index] = values
print(result)


Instruction: Getting the index of successive minimum comparisons

Input:
Say if i have a tensor

values = torch.tensor([5., 4., 8., 3.])

and i want to take the minimum over every 2 successive values, meaning

min(5., 4.) = 4.
min(8., 3.) = 3.

is there a vectorized way of doing it and still obtaining back the relative index of the minimum? Meaning what I want as output is:

min_index = [1, 1]
#min_index[0] == 1 as 4. is the minimum of (5., 4.) and is in index 1 of (5., 4.)
#min_index[1] == 1 as 3. is the minimum of (8., 3.) and is in index 1 of (8., 3.)

Output:
I think reshaping your tensor will make it a lot easier.
After that the torch.min automatically returns the minimum values and indexes.

import torch

values = torch.tensor([5., 4., 8., 3.])
values_reshaped = values.reshape(-1,2) # works for any length
minimums, index = torch.min(values_reshaped, axis = -1)
print(minimums) # tensor of the minimum values
print(index) # tensor of indexes


Instruction: How to perform advanced indexing in PyTorch?

Input:
Is there a way of doing the following without looping?

S, N, H = 9, 7, 4

a = torch.randn(S, N, H)

# tensor with integer values between 1, S of shape (N,)
lens = torch.randint(1, S + 1, (N,))

res = torch.zeros(N, H)

for i in range(N):
    res[i] = a[lens[i] - 1, i, :]

Output:
Yes, I believe this works.

import torch

S, N, H = 9, 7, 4

a = torch.randn(S, N, H)

# tensor with integer values between 1, S of shape (N,)
lens = torch.randint(0, S, (N,))
i = torch.tensor(range(0,7))
res = torch.zeros(N, H)

res = a[lens, i, :]
print(res)

And why did you make lens 1 from S+1 and then do lens[i]-1 ? I just changed it so lens is 0 from S for convenience. However if you need lens to be 1 from S+1, you can change
res = a[lens, i, :]
to
res = a[lens-1, i, :]


Instruction: Any guarantees that Torch won't mess up with an already allocated CUDA array?

Input:
Assume we allocated some array on our GPU through other means than PyTorch, for example by creating a GPU array using numba.cuda.device_array. Will PyTorch, when allocating later GPU memory for some tensors, accidentally overwrite the memory space that is being used for our first CUDA array? In general, since PyTorch and Numba use the same CUDA runtime and thus I assume the same mechanism for memory management, are they automatically aware of memory regions used by other CUDA programs or does each one of them see the entire GPU memory as his own? If it's the latter, is there a way to make them aware of allocations by other CUDA programs?

EDIT: figured this would be an important assumption: assume that all allocations are done by the same process.

Output:
  Will PyTorch, when allocating later GPU memory for some tensors, accidentally overwrite the memory space that is being used for our first CUDA array?

No.

  are they automatically aware of memory regions used by other CUDA programs ...

They are not "aware", but each process gets its own separate context ...

  ... or does each one of them see the entire GPU memory as his own?

.... and contexts have their own address spaces and isolation. So neither, but there is no risk of memory corruption.

  If it's the latter, is there a way to make them aware of allocations by other CUDA programs?

If by "aware" you mean "safe", then that happens automatically. If by "aware" you imply some sort of interoperability, then that is possible on some platforms, but it is not automatic.

  ... assume that all allocations are done by the same process.

That is a different situation. In general, the same process implies a shared context, and shared contexts share a memory space, but all the normal address space protection rules and facilities apply, so there is not a risk of loss of safety.


Instruction: Loading & Freezing a Pretrained Model to Combine with a New Network

Input:
I have a pretrained model and would like to build a classifier on top of it. I’m trying to load and freeze the weights of the pretrained model, and pass its outputs to the new classifier, which I’d like to optimise. Here is what I have so far, I’m a little stuck on a TypeError: forward() missing 1 required positional argument: 'x' error from the nn.Sequential line:

import model #model.py contains the architecture of the pretrained model

class Classifier(nn.Module):
    def __init__(self):
        ...
    def forward(self, x):
        ...

net = model.Model()
net.load_state_dict(checkpoint["net"])

for c in net.children():
    for param in child.parameters():
        params.requires_grad = False

model = nn.Sequential(nn.ModuleList(net()), Classifier())

Output: I finally solved this issue after a discussion with @ptrblck from the PyTorch Forums. The solution is similar to Shai's answer, only that because net contains an instance of the model.Model class, one should do model = nn.Sequential(net, Classifier()) instead, without calling nn.ModuleList().


Instruction: Get value of variable index in particular dimension

Input:
Say if i have a tensor that is

value = torch.tensor([

    [[0, 0, 0], [1, 1, 1]],
    [[2, 2, 2], [3, 3, 3]],
])

essentially with shape (2,2,3).

Now say if i have an index = [1, 0], which means I want to take:

# row 1 of [[0, 0, 0], [1, 1, 1]], giving me: [1, 1, 1]
# row 0 of [[2, 2, 2], [3, 3, 3]], giving me: [2, 2, 2]

So that the final output:

output = torch.tensor([[1, 1, 1], [2, 2, 2]])

is there a vectorized way to achieve this?

Output:
You can use advanced indexing.
I can't find a good pytorch document about this, but I believe it works as same as numpy, so here's the numpy's document about indexing.

import torch

value = torch.tensor([

    [[0, 0, 0], [1, 1, 1]],
    [[2, 2, 2], [3, 3, 3]],
])

index = [1, 0]
i = range(0,2)

result = value[i, index]
# same as result = value[i, index, :]

print(result)


Instruction: Is there any way to use both a GPU accelerator and Torch in google cloud AI platform for model deployment?

Input:
I already have a torch model (BERT), and I'd like to use the ai-platform service to get online predictions using a GPU, but I can't figure out how to do it.

The following command, without an accelerator, works:

gcloud alpha ai-platform versions create {VERSION} --model {MODEL_NAME} --origin=gs://{BUCKET}/models/ --python-version=3.5 --runtime-version=1.14 --package-uris=gs://{BUCKET}/packages/my-torch-package-0.1.tar.gz,gs://cloud-ai-pytorch/torch-1.0.0-cp35-cp35m-linux_x86_64.whl --machine-type=mls1-c4-m4 --prediction-class=predictor.CustomModelPrediction

However, if I try to add the accelerator parameter:

--accelerator=^:^count=1:type=nvidia-tesla-k80

I get the following error message:

ERROR: (gcloud.alpha.ai-platform.versions.create) INVALID_ARGUMENT: Field: version.machine_type Error: GPU accelerators are not supported on the requested machine type: mls1-c4-m4
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: 'GPU accelerators are not supported on the requested machine type:
      mls1-c4-m4'
    field: version.machine_type

But if I use a different machine type, that I know I can use with an accelerator, I get the following error:

ERROR: (gcloud.alpha.ai-platform.versions.create) FAILED_PRECONDITION: Field: framework Error: Machine type n1-highcpu-4 does not support CUSTOM_CLASS.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: Machine type n1-highcpu-4 does not support CUSTOM_CLASS.
    field: framework

It's like any machine that supports GPU accelerators doesn't support custom classes (required AFAIK to use Torch), and any machine that supports custom classes doesn't support GPU accelerators.

Any way to make it work?

There are a bunch of tutorials on how to use ai-platform with Torch, but I can't see the point of using gcloud to train and predict if you have to do everything on the CPU so that feels very odd to me.

Output:
As for now, using Custom Prediction Routines is in Beta. In addition, using other machine types than mls1-c1-m2 is also in Beta.

Nevertheless, as you can see in the previously referenced link, GPU's are not available for mls1-like machines. At the same time, these are the only machine types that allow models outside TensorFlow.

In summary, probably deploying your prediction model in Torch and using a GPU might not be a feasible option right now.


Instruction: How to implement differentiable hamming loss in pytorch?

Input:
How to implement a differentiable loss function that counts the number of wrong predictions?

output = [1,0,4,10]
target = [1,2,4,15]
loss = np.count_nonzero(output != target) / len(output) # [0,1,0,1] -> 2 / 4 -> 0.5

I have tried a few implementations but they are not differentiable. RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

def hamming_loss(output, target):
  #loss = torch.tensor(torch.nonzero(output != target).size(0)).double() / target.size(0)
  #loss = torch.sum((output != target), dim=0).double() / target.size(0)
  loss = torch.mean((output != target).double())
  return loss

Maybe there is some similar but differential loss function?

Output:
Why don't you convert your discrete predictions (e.g., [1, 0, 4, 10]) with "soft" predictions, i.e. probability of each label (e.g., output becomes a 4x(num labels) probability vectors).
Once you have "soft" predictions, you can compute the cross entropy loss between the predicted output probabilities and the desired targets.


Instruction: How to collect the predictions for each observation in pytorch?

Input:
Im trying to make a RNN myself using a loop rather than the nn.rnn module. But I would still like to use pytorch's backward for backpropogation.

Each observation makes 1 prediction which is then used for subsequent predictions. But how should each prediction be stored such that we can calculate the loss and get back propagation?

can we just create a normal list and append to it with each prediction? Or will that not allow back propogation?

Output:
Yes. That would work. PyTorch creates a computational graph with nodes for all the operations that you perform during the forward pass. So, for each operation in the for loop, no matter how you store the outputs (or even if you discard them), you should still be able to call backward on the loss as the information required for computing gradients is already there in the graph.

Here's a toy example:

import torch
from torch import nn
import torch.nn.functional as F

class RNN(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim):
        super().__init__()
        self.wh = nn.Linear(hidden_dim, hidden_dim)
        self.wx = nn.Linear(in_dim, hidden_dim)
        self.wo = nn.Linear(hidden_dim, out_dim)
        self.hidden_dim = hidden_dim

    def forward(self, xs, hidden):
        outputs = []
        hiddens = []
        hiddens.append(hidden)
        for i, x in enumerate(xs):
            hiddens.append(torch.tanh(self.wx(x) + self.wh(hiddens[i])))
            outputs.append(F.log_softmax(self.wo(hiddens[i+1]), dim=0))
        return outputs, hiddens

    def init_hidden(self):
        return torch.zeros(self.hidden_dim)

# Initialize the input and output
x = torch.tensor([[1., 0., 0.], [0., 1., 0.], [1., 0., 0.]])
y = torch.tensor([1])

# Initialize the network, hidden state and loss function
rnn = RNN(3, 10, 2)
hidden = rnn.init_hidden()
nll = nn.NLLLoss()

# Forward pass
outputs, hidden_states = rnn(x, hidden)

# Compute loss
loss = nll(outputs[-1].unsqueeze(0), y)

# Call Backward on the loss
loss.backward()

# Inspect the computed gradients
print(rnn.wh.weight.grad)   # Gives a tensor of shape (10, 10) in this case


Instruction: Pytorch ImageNet dataset

Input:
I am unable to download the original ImageNet dataset from their official website. However, I found out that pytorch has ImageNet as one of it’s torch vision datasets.

Q1. Is that the original ImageNet dataset?

Q2. How do I get the classes for the dataset like it’s being done in Cifar-10

classes = [‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’]

Output:
The torchvision.datasets.ImageNet is just a class which allows you to work with the ImageNet dataset. You have to download the dataset yourself (e.g. from http://image-net.org/download-images) and pass the path to it as the root argument to the ImageNet class object.
Note that the option to download it directly by passing the flag download=True is no longer possible:
if download is True:
    msg = ("The dataset is no longer publicly accessible. You need to "
           "download the archives externally and place them in the root "
           "directory.")
    raise RuntimeError(msg)
elif download is False:
    msg = ("The use of the download flag is deprecated, since the dataset "
           "is no longer publicly accessible.")
    warnings.warn(msg, RuntimeWarning)

(source)
If you just need to get the class names and the corresponding indices without downloading the whole dataset (e.g. if you are using a pretrained model and want to map the predictions to labels), then you can download them e.g. from here or from this github gist.


Instruction: BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification

Input:
I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes.
I just started using the Huggingface Transformer package and BERT with PyTorch. What I need is a classifier with a softmax layer on top so that I can do 5-way classification. Confusingly, there seem to be two relevant options in the Transformer package: BertForSequenceClassification and BertForMultipleChoice.
Which one should I use for my 5-way classification task? What are the appropriate use cases for them?
The documentation for BertForSequenceClassification doesn't mention softmax at all, although it does mention cross-entropy. I am not sure if this class is only for 2-class classification (i.e. logistic regression).

Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

The documentation for BertForMultipleChoice mentions softmax, but the way the labels are described, it sound like this class is for multi-label classification (that is, a binary classification for multiple labels).

Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices] where num_choices is the size of the second dimension of the input tensors.

Thank you for any help.

Output:
The answer to this lies in the (admittedly very brief) description of what the tasks are about:

  [BertForMultipleChoice] [...], e.g. for RocStories/SWAG tasks.

When looking at the paper for SWAG, it seems that the task is actually learning to choose from varying options. This is in contrast to your "classical" classification task, in which the "choices" (i.e., classes) do not vary across your samples, which is exactly what BertForSequenceClassification is for.

Both variants can in fact be for an arbitrary number of classes (in the case of BertForSequenceClassification), respectively choices (for BertForMultipleChoice), via changing the labels parameter in the config. But, since it seems like you are dealing with a case of "classical classification", I suggest using the BertForSequenceClassification model.

Shortly addressing the missing Softmax in BertForSequenceClassification: Since classification tasks can compute loss across classes indipendent of the sample (unlike multiple choice, where your distribution is changing), this allows you to use Cross-Entropy Loss, which factors in Softmax in the backpropagation step for increased numerical stability.


Instruction: Freeze only some lines of a torch.nn.Embedding object

Input:
I am quite a newbie to Pytorch, and I am trying to implement a sort of "post-training" procedure on embeddings.

I have a vocabulary with a set of items, and I have learned one vector for each of them.
I keep the learned vectors in a nn.Embedding object.
What I'd like to do now is to add a new item to the vocabulary without updating the already learned vectors. The embedding for the new item would be initialized randomly, and then trained while keeping all the other embeddings frozen.

I know that in order to prevent a nn.Embedding to be trained, I need to set to False its requires_grad variable. I have also found this other question that is similar to mine. The best answer proposes to

either store the frozen vectors and the vector to train in different nn.Embedding objects, the former with requires_grad = False and the latter with requires_grad = True
or store the frozen vectors and the new one in the same nn.Embedding object, computing the gradient on all vectors, but descending it is only on the dimensions of the vector of of the new item. This, however, leads to a relevant degradation in performances (which I want to avoid, of course).

My problem is that I really need to store the vector for the new item in the same nn.Embedding object as the frozen vectors of the old items. The reason for this constraint is the following: when building my loss function with the embeddings of the items (old and new), I need to lookup the vectors based on the ids of the items, and for performances reasons I need to use Python slicing. In other words, given a list of item ids item_ids, I need to do something like vecs = embedding[item_ids]. If I used two different nn.Embedding items for the old items and the and new one I would need to use an explicit for-loop with if-else conditions, which would lead to worse performances.

Is there any way I can do this?

Output:
If you look at the implementation of nn.Embedding it uses the functional form of embedding in the forward pass. Therefore, I think you could implement a custom module that does something like this:

import torch
from torch.nn.parameter import Parameter
import torch.nn.functional as F

weights_freeze = torch.rand(10, 5)  # Don't make parameter
weights_train = Parameter(torch.rand(2, 5))
weights = torch.cat((weights_freeze, weights_train), 0)

idx = torch.tensor([[11, 1, 3]])
lookup = F.embedding(idx, weights)

# Desired result
print(lookup)
lookup.sum().backward()
# 11 corresponds to idx 1 in weights_train so this has grad
print(weights_train.grad)


Instruction: Pytorch Autograd gives different gradients when using .clamp instead of torch.relu

Input:
I'm still working on my understanding of the PyTorch autograd system. One thing I'm struggling at is to understand why .clamp(min=0) and nn.functional.relu() seem to have different backward passes.

It's especially confusing as .clamp is used equivalently to relu in PyTorch tutorials, such as https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn.

I found this when analysing the gradients of a simple fully connected net with one hidden layer and a relu activation (linear in the outputlayer).

to my understanding the output of the following code should be just zeros. I hope someone can show me what I am missing.

import torch
dtype = torch.float

x = torch.tensor([[3,2,1],
                  [1,0,2],
                  [4,1,2],
                  [0,0,1]], dtype=dtype)

y = torch.ones(4,4)

w1_a = torch.tensor([[1,2],
                     [0,1],
                     [4,0]], dtype=dtype, requires_grad=True)
w1_b = w1_a.clone().detach()
w1_b.requires_grad = True

w2_a = torch.tensor([[-1, 1],
                     [-2, 3]], dtype=dtype, requires_grad=True)
w2_b = w2_a.clone().detach()
w2_b.requires_grad = True

y_hat_a = torch.nn.functional.relu(x.mm(w1_a)).mm(w2_a)
y_a = torch.ones_like(y_hat_a)
y_hat_b = x.mm(w1_b).clamp(min=0).mm(w2_b)
y_b = torch.ones_like(y_hat_b)

loss_a = (y_hat_a - y_a).pow(2).sum()
loss_b = (y_hat_b - y_b).pow(2).sum()

loss_a.backward()
loss_b.backward()

print(w1_a.grad - w1_b.grad)
print(w2_a.grad - w2_b.grad)

# OUT:
# tensor([[  0.,   0.],
#         [  0.,   0.],
#         [  0., -38.]])
# tensor([[0., 0.],
#         [0., 0.]])
#

Output: The reason is that clamp and relu produce different gradients at 0. Checking with a scalar tensor x = 0 the two versions: (x.clamp(min=0) - 1.0).pow(2).backward() versus (relu(x) - 1.0).pow(2).backward(). The resulting x.grad is 0 for the relu version but it is -2 for the clamp version. That means relu chooses x == 0 --> grad = 0 while clamp chooses x == 0 --> grad = 1.


Instruction: Torch squeeze and the batch dimension

Input:
does anyone here know if the torch.squeeze function respects the batch (e.g. first) dimension? From some inline code it seems it does not.. but maybe someone else knows the inner workings better than I do.
Btw, the underlying problem is that I have tensor of shape (n_batch, channel, x, y, 1). I want to remove the last dimension with a simple function, so that I end up with a shape of (n_batch, channel, x, y).
A reshape is of course possible, or even selecting the last axis. But I want to embed this functionality in a layer so that I can easily add it to a ModuleList or Sequence object.
EDIT: just found out that for Tensorflow (2.5.0) the function tf.linalg.diag DOES respect batch dimension. Just a FYI that it might differ per function you are using

Output:
No! squeeze doesn't respect the batch dimension. It's a potential source of error if you use squeeze when the batch dimension may be 1. Rule of thumb is that only classes and functions in torch.nn respect batch dimensions by default.

This has caused me headaches in the past. I recommend using reshape or only using squeeze with the optional input dimension argument. In your case you could use .squeeze(4) to only remove the last dimension. That way nothing unexpected happens. Squeeze without the input dimension has led me to unexpected results, specifically when

the input shape to the model may vary
batch size may vary
nn.DataParallel is being used (in which case batch size for a particular instance may be reduced to 1)


Instruction: PyTorch Inequality Gradient

Input:
I wrote a PyTorch model roughly as follows:

import torch.nn as nn
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = nn.Sequential(nn.Linear(64 * 64, 16), nn.LeakyReLU(0.2))
        self.layer2 = nn.Sequential(nn.Linear(16, 32), nn.LeakyReLU(0.2))
        self.layer3 = nn.Sequential(nn.Linear(32, 64), nn.LeakyReLU(0.2))
        self.layer4 = nn.Sequential(nn.Linear(64, 15), nn.Tanh())
    def forward(self, x):
        return (self.layer4(self.layer3(self.layer2(self.layer1(x)))) < 0).float()

Notice what I want to do: I want forward to return a tensor of 0s and 1s. However, this does not train, probably because the derivative of the inequality is zero.

How can I make a model like this one train, for example, if I want to do image segmentation?

Output:
As you said, you can't train something like x<0.
You should be fine even if you get rid of the <0 part and use

    return self.layer4(self.layer3(self.layer2(self.layer1(x))))

as long as you are using the appropriate loss. I think what you would  want to use is nn.BCEWithLogitsLoss. In that case you should get the Tanh out of the last layer since nn.BCEWithLogitsLoss internally computes with sigmoid.
(There are options of using nn.BCEloss() with sigmoid at the last layer, or even stick with Tanh, but I don't think there's a reason to take the long way.)

So in the training phase, the neural network tries as hard as it could to fit the output to 0s and 1s. After that, it is the testing phase that you should take the output of the layer, and give it some kind of threshold to change the values to precisely 1s and 0s.(like you did (output<0).float())

You will find useful sources if you search for multilabel classification.


Instruction: Getting 'tensor is not a torch image' for data type

Input:
As the question suggests, I'm trying to convert images to tensor.

X, y = train_sequence[idx]

images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
masks = Variable(torch.from_numpy(y)).to(device)
print(type(images)) ## Output: <class 'torch.Tensor'>

images = transforms.Normalize((0.5, 0.5, 0.5, 0.5, 0.5), (0.5, 0.5, 0.5,0.5, 0.5))(images)
masks =  transforms.Normalize((0.5), (0.5))(masks)

But I get the error at
 ---> 19         images = transforms.Normalize((0.5, 0.5, 0.5, 0.5, 0.5), (0.5, 0.5, 0.5,0.5, 0.5))(images)

TypeError: tensor is not a torch image.

Output:
This is because as of now, torchvison.transforms.Normalize only supports images with 2 or 3 channels, without the batch dimension, ie. (C, H, W). So instead of passing in a 4D tensor, something like this would work:

image = torch.ones(3, 64, 64)
image = transforms.Noramalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))(image)

Also, because the 0.5 values represent the mean and standard deviation of the image channels, there should normally only be 3 channels (you don't "normalise" the batch dimension, only the spatial ones), so instead of using a tuple of length 5, do (0.5, 0.5, 0.5).