Instruction: Slicing torch tensors with list of indeces

Input:
I'm doing a reinforcement learning project, and I'm trying to get a tensor that represents the expected reward of all the given actions. I have a long tensor of chosen actions of size batch with values of either zero or one (the two potential actions). I have a tensor of expected rewards for each action of size batch * action_size, and I want a tensor of size batch.

For example, if batch size was 4, then I have

action = tensor([1,0,0,1])
expectedReward = tensor([[3,7],[5,9],[-1,12],[0,1]])

and what I want is

rewardForActions = tensor([7,5,-1,1])

I thought this would answer my question, but it's not the same at all, because if I went with that solution, it would end up with a 4*4 tensor, selecting from each row 4 times, instead of once.

Any ideas?

Output:
You could do

rewardForActions = expectedReward.index_select(1, action).diagonal()
# tensor([ 7,  5, -1,  1])


Instruction: Why do we need clone the grad_output and assign it to grad_input when defining a ReLU autograd function?

Input:
I'm walking through the autograd part of pytorch tutorials. I have two questions:

Why do we need clone the grad_output and assign it to grad_input other than simple assignment during backpropagation?
What's the purpose of grad_input[input < 0] = 0? Does it mean we don't update the gradient when input less than zero?

Here's the code:

class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

Link here:
https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-defining-new-autograd-functions

Thanks a lot in advance.

Output:
  Why do we need clone the grad_output and assign it to grad_input other than simple assignment during backpropagation?

tensor.clone() creates a copy of tensor that imitates the original tensor's requires_grad field. clone is a way to copy the tensor while still keeping the copy as a part of the computation graph it came from.

So, grad_input is part of the same computation graph as grad_output and if we compute the gradient for grad_output, then the same will be done for grad_input.

Since we make changes in grad_input, we clone it first.

  What's the purpose of 'grad_input[input < 0] = 0'? Does it mean we don't update the gradient when input less than zero?

This is done as per ReLU function's definition. The ReLU function is f(x)=max(0,x). It means if x<=0 then f(x)=0, else f(x)=x. In the first case, when x<0, the derivative of f(x) with respect to x is f'(x)=0. So, we perform grad_input[input < 0] = 0. In the second case, it is f'(x)=1, so we simply pass the grad_output to grad_input (works like an open gate).


Instruction: Install PyTorch from requirements.txt

Input:
Torch documentation says use
pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

to install the latest version of PyTorch. This works when I do it manually but when I add it to req.txt and do pip install -r req.txt, it fails and says  ERROR: No matching distribution.
Edit: adding the whole line from req.txt and error here.
torch==1.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.htmltorch==1.4.0+cpu

ERROR: Could not find a version that satisfies the requirement torch==1.4.0+cpu (from -r requirements.txt (line 1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0)
ERROR: No matching distribution found for torch==1.4.0+cpu (from -r requirements.txt (line 1))

Output:
Add --find-links in requirements.txt before torch
--find-links https://download.pytorch.org/whl/torch_stable.html

torch==1.2.0+cpu

Source: https://github.com/pytorch/pytorch/issues/29745#issuecomment-553588171


Instruction: How to apply linear layer to 2D layer only in one dimension (by row or by column) - partially connected layers

Input:
I'm trying to apply a linear layer to a 2D matrix of tensors connecting it only by column as in the picture below.

The input shape is (batch_size, 3, 50). I first tried with 2D convolution, adding a 1 channel dimension, so input shape is (batch_size, 1, 3, 50)

import torch.nn as nn
import torch

class ColumnConv(nn.Module):
    def __init__(self):
        self.layers = nn.Sequential(
            nn.Conv2d(
                in_channels=1,
                out_channels=1,
                kernel_size=(3, 1),
                stride=1,
            ),  # shape is B, 1, 1, 50
            nn.ReLU(),
            nn.Flatten() #shape is B, 50
        )

    def forward(self, x):
        return self.layers(x)

But it doesn't seem to work.
I'm planning to use a list of 50 nn.Linear layers and apply them to column slices of input, but it seems much more like a workaround not optimized for performance.

Is there a more "pytorchic" way of doing this?

Output:
The PyTorch nn.Linear module can be applied to multidimensional input, the linear will be applied to the last dimension so to apply by column the solution is to swap rows and columns.

linear_3_to_1 = nn.Linear(3, 1)

x = torch.randn(1, 1, 3, 50)
x = x.transpose(2, 3) #swap 3 and 50
out = linear_3_to_1(x).flatten()


Instruction: Get indices of elements in tensor a that are present in tensor b

Input:
For example, I want to get the indices of elements valued 0 and 2 in tensor a. These values, (0 and 2) are stored in tensor b. I have devised a pythonic way to do so (shown below), but I don't think list comprehensions are optimized to run on GPU, or maybe there is a more PyTorchy way to do it that I am unaware of.

import torch
a = torch.tensor([0, 1, 0, 1, 1, 0, 2])
b = torch.tensor([0, 2])
torch.tensor([x in b for x in a]).nonzero()

>>>> tensor([[0],
             [2],
             [5],
             [6]])

Any other suggestions or is this an acceptable way?

Output:
Here's a more efficient way to do it (as suggested in the link posted by jodag in comments...):

(a[..., None] == b).any(-1).nonzero()


Instruction: How can I count the number of 1's and 0's in the second dimension of a PyTorch Tensor?

Input:
I have a tensor with size: torch.Size([64, 2941]), which is 64 batches of 2941 elements.

Across all 64 batches, I want to count the total number of 1s and 0s in the second dimension of the tensor, all the way to the 2941th so that I will have those counts as a tensor of size torch.Size([2941])

How do I do that?

Output:
You can sum them:

import torch
torch.manual_seed(2020)

# x is a fake data, it should be your tensor with 64x2941
x = (torch.rand((3,4)) > 0.5).to(torch.int32)
print(x)
# tensor([[0, 0, 1, 0],
#         [0, 0, 1, 1],
#         [0, 1, 1, 1]], dtype=torch.int32)

ones = (x == 1.).sum(dim=0)
print(ones)
# tensor([0, 1, 3, 2])

And if x is binary, you can have the number of zeros by a simple subtraction:

zeros = x.shape[1] - ones


Instruction: I fine tuned a pre-trained BERT for sentence classification, but i cant get it to predict for new sentences

Input:
below is the result of my fine-tuning.

Training Loss   Valid. Loss Valid. Accur.   Training Time   Validation Time
epoch
1   0.16    0.11    0.96    0:02:11 0:00:05
2   0.07    0.13    0.96    0:02:19 0:00:05
3   0.03    0.14    0.97    0:02:22 0:00:05
4   0.02    0.16    0.96    0:02:21 0:00:05

next i tried to use the model to predict labels from a csv file. i created a label column, set the type to int64 and run the prediction.

print('Predicting labels for {:,} test sentences...'.format(len(input_ids)))
model.eval()
# Tracking variables
predictions , true_labels = [], []
# Predict
for batch in prediction_dataloader:
  # Add batch to GPU
  batch = tuple(t.to(device) for t in batch)

  # Unpack the inputs from our dataloader
  b_input_ids, b_input_mask, b_labels = batch

  # Telling the model not to compute or store gradients, saving memory and
  # speeding up prediction
  with torch.no_grad():
      # Forward pass, calculate logit predictions
      outputs = model(b_input_ids, token_type_ids=None,
                      attention_mask=b_input_mask)

  logits = outputs[0]

  # Move logits and labels to CPU
  logits = logits.detach().cpu().numpy()
  label_ids = b_labels.to('cpu').numpy()

  # Store predictions and true labels
  predictions.append(logits)
  true_labels.append(label_ids)

however, while i am able to print out the predictions[4.235, -4.805] etc, and the true_labels[NaN,NaN.....], i am unable to actually get the predicted labels{0 or 1}. Am i missing something here?

Output:
The output of the models are logits, i.e., the probability distribution before normalization using softmax.

If you take your output: [4.235, -4.805] and run softmax over it

In [1]: import torch
In [2]: import torch.nn.functional as F
In [3]: F.softmax(torch.tensor([4.235, -4.805]))
Out[3]: tensor([9.9988e-01, 1.1856e-04])

You get get 99% probability score for label 0. When you have the logits as a 2D tensor, you can easily get the classes by calling

logits.argmax(0)

The NaNs values in your true_labels are probably a bug in how you load the data, it has nothing to do with the BERT model.


Instruction: Why override Dataset instead of directly pass in input and labels, pytorch

Input:
Sorry if what I say here is wrong -- new to pytorch.

From what I can tell there are two main ways of getting training data and passing through a network. One is to override Dataset and the other is to just prepare your data correctly and then iterate over it, like shown in this example: pytorch classification example

which does something like

rnn(input, hidden, output)
for i in range(input.size()[0]):
    output, hidden = rnn(input[i], hidden)

The other way would be to do something like

for epoch in range(epochs):
    for data, target in trainloader:
        computer model etc

where in this method, trainloader is from doing something like

trainloader = DataLoader(my_data)

after overriding getitem and len

My question here, is what are the differences between these methods, and why would you use one over the other? Also, it seems to me that overriding Dataset doesn't work for something that has lets say an input layer of size 100 nodes with an output of 10 nodes, since when you return getitem it needs a pair of (data, label). This seems like a case where I probably don't understand how to use Dataset very well, but that is why I'm asking in the first place. I think I read something about a collate function which might help in this scenario?

Output:
Dataset class and the Dataloader class in PyTorch help us to feed our own training data into the network. Dataset class is used to provide an interface for accessing all the training or testing samples in your dataset. In order to achieve this, you have to implement at least two methods, __getitem__ and __len__ so that each training sample can be accessed by its index. In the initialization part of the class, we load the dataset (as float type) and convert them into Float torch tensors. __getitem__ will return the features and target value.

What are the differences between these methods?

In PyTorch either you can prepare your data such that the PyTorch DataLoader can consume it and you get an iterable object or you can overload the default DataLoader to perform some custom operations like if you want to do some preprocessing of text/images, stack frames from videos clips, etc.

Our DataLoader behaves like an iterator, so we can loop over it and fetch a different mini-batch every time.

Basic Sample

from torch.utils.data import DataLoader

train_loader = DataLoader(dataset=train_data, batch_size=16, shuffle=True)
valid_loader = DataLoader(dataset=valid_data, batch_size=16, shuffle=True)

# To retrieve a sample mini-batch, one can simply run the command below —
# it will return a list containing two tensors:
# one for the features, another one for the labels.

next(iter(train_loader))
next(iter(valid_loader))

Custom Sample

import torch
from torch.utils.data import Dataset, Dataloader

class SampleData(Dataset):
def __init__(self, data):
        self.data = torch.FloatTensor(data.values.astype('float'))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        target = self.data[index][-1]
        data_val = self.data[index] [:-1]
        return data_val,target

train_dataset = SampleData(train_data)
valid_dataset = SampleData(valid_data)

device = "cuda" if torch.cuda.is_available() else "cpu"
kwargs = {'num_workers': 1, 'pin_memory': True} if device=='cuda' else {}
train_loader = DataLoader(train_dataset, batch_size=train_batch_size, shuffle=True, **kwargs)
test_loader = DataLoader(valid_dataset, batch_size=test_batch_size, shuffle=False, **kwargs)

Why would you use one over the other?

It solely depends on your use-case and the amount of control you want. PyTorch has given you all the power and it is you who is going to decide how much you want to. Suppose you are solving a simple image classification problem, then,

You can simply put all the images in a root folder with each subfolder containing the samples belonging to a particular class and label the folder with the class name. When training we just need to specify the path to the root folder and the PyTorch DataLoader will automatically pick images from each folder and training the model.

But on the other hand, if you have classifying video clips or video sequences generally known as video tagging in a large video file then you need to write your custom DataLoader to load the frames from the video, stack it and give input to the DataLoader.

Use can find some useful links below for further reference:

https://pytorch.org/docs/stable/data.html

https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel

https://pytorch.org/tutorials/beginner/data_loading_tutorial.html


Instruction: Extracting indexes from a max pool over uniform data

Input:
I'm trying to find max points in a 2D tensor for a given kernel size, but I'm having issues with a special case where all the values are uniform. For example, given the following example, I would like to mark each point as a max point:

+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+

If I run torch.nn.functional.max_pool2d with a kernel size=3, stride=1, and padding=1, I get the following indicies:

+---+---+---+----+
| 0 | 0 | 1 |  2 |
+---+---+---+----+
| 0 | 0 | 1 |  2 |
+---+---+---+----+
| 4 | 4 | 5 |  6 |
+---+---+---+----+
| 8 | 8 | 9 | 10 |
+---+---+---+----+

What changes do I need to account for to instead obtain the following indicies?

+----+----+----+----+
| 1  | 2  | 3  |  4 |
+----+----+----+----+
|  5 |  6 |  7 |  8 |
+----+----+----+----+
|  9 | 10 | 11 | 12 |
+----+----+----+----+
| 13 | 14 | 15 | 16 |
+----+----+----+----+

Output:
You can do the following:

a = torch.ones(4,4)
indices = (a == torch.max(a).item()).nonzero()

What this does is return a [16,2] sized tensor with the 2D coordinates of the max value(s), i.e. [0,0], [0,1], .., [3,3]. The torch.max part should be easy to understand, nonzero() considers the boolean tensor given by (a == torch.max(a).item()), takes False to be 0, and returns the non-zero indices. Hope this helps!


Instruction: torch assign not in place by tensor slicing in pytorch

Input:
I am trying to convert my current code, which assigns tensors in place, to an outer operation.
Meaning currently the code is

self.X[:, nc:] = D

Where D is in the same shape as self.X[:, nc:]
But I would like to convert it to

sliced_index = ~ somehow create an indexed tensor from self.X[:, nc:]
self.X = self.X.scatter(1,sliced_index,mm(S_, Z[:, :n - nc]))

And don't know how to create that index mask tensor that represents only the entries in the sliced tensor

Minimal example:

a = [[0,1,2],[3,4,5]]
D = [[6],[7]]
Not_in_place = [[0,1,6],[3,4,7]]

Output:
A masked scatter is a little easier. The mask itself can be computed as an in-place operation after which you can use masked_scatter

mask = torch.zeros(self.X.shape, device=self.X.device, dtype=torch.bool)
mask[:, nc:] = True
self.X = self.X.masked_scatter(mask, D)

A more specialized version which relies on broadcasting but should be more efficient would be

mask = torch.zeros([1, self.X.size(1)], device=self.X.device, dtype=torch.bool)
mask[0, nc:] = True
self.X = self.X.masked_scatter(mask, D)


Instruction: Create slice mask in pytorch?

Input:
Is there a way to specify a mask based on a slice operation?
For example

A = torch.arange(6).view((2,3))
# A = [[0,1,2], [3,4,5]]

mask_slice = torch.mask_slice(A[:,1:])
# mask_slice = [[0,1,1],[0,1,1]]

Output:
You can do something like this (if I got your question right):

mask_slice = torch.zeros(A.shape, dtype=bool)
mask_slice[:, 1:] = 1

# tensor([[False,  True,  True],
#         [False,  True,  True]])


Instruction: Multi class classifcation with Pytorch

Input:
I'm new with Pytorch and I need a clarification on multiclass classification.

I'm fine-tuning the DenseNet neural network, so it can recognize 3 different classes.

Because it's a multiclass problem, I have to replace the classification layer in this way:

kernelCount = self.densenet121.classifier.in_features
self.densenet121.classifier = nn.Sequential(nn.Linear(kernelCount, 3), nn.Softmax(dim=1))

And use CrossEntropyLoss as the loss function:

loss = torch.nn.CrossEntropyLoss(reduction='mean')

By reading on Pytorch forum, I found that CrossEntropyLoss applys the softmax function on the output of the neural network. Is this true? Should I remove the Softmax activation function from the structure of the network?

And what about the test phase? If it's included, I have to call the softmax function on the output of the model?

Thanks in advance for your help.

Output:
Yes, CrossEntropyLoss applies softmax implicitly. You should remove the softmax layer at the end of the network since softmax is not idempotent, therefore applying it twice would be a semantic error.

As far as evaluation/testing goes. Remember that softmax is a monotonically increasing operation (meaning the relative order of outputs doesn't change when you apply it). Therefore the result of argmax before and after softmax will give the same result.

The only time you may want to perform softmax explicitly during evaluation would be if you need the actual confidence value for some reason. If needed you can apply softmax explicitly using torch.softmax on the network output during evaluation.


Instruction: RuntimeError: size mismatch, m1: [4 x 784], m2: [4 x 784] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:136

Input:
I have executed the following code

   import matplotlib.pyplot as plt
   import torch
   import torch.nn as nn
   import torch.optim as optim
   from torch.autograd import Variable
   from torch.utils import data as t_data
   import torchvision.datasets as datasets
   from torchvision import transforms
   data_transforms = transforms.Compose([transforms.ToTensor()])
  mnist_trainset = datasets.MNIST(root='./data', train=True,
                           download=True, transform=data_transforms)
batch_size=4
dataloader_mnist_train = t_data.DataLoader(mnist_trainset,
                                           batch_size=batch_size,
                                           shuffle=True
                                           )

def make_some_noise():
    return torch.rand(batch_size,100)

class generator(nn.Module):

    def __init__(self, inp, out):

        super(generator, self).__init__()

        self.net = nn.Sequential(
                                 nn.Linear(inp,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,1000),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(1000,800),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(800,out)
                                    )

    def forward(self, x):
        x = self.net(x)
        return x

class discriminator(nn.Module):

    def __init__(self, inp, out):

        super(discriminator, self).__init__()

        self.net = nn.Sequential(
                                 nn.Linear(inp,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,784),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(784,200),
                                 nn.ReLU(inplace=True),
                                 nn.Linear(200,out),
                                 nn.Sigmoid()
                                    )

    def forward(self, x):
        x = self.net(x)
        return x

def plot_img(array,number=None):
    array = array.detach()
    array = array.reshape(28,28)

    plt.imshow(array,cmap='binary')
    plt.xticks([])
    plt.yticks([])
    if number:
        plt.xlabel(number,fontsize='x-large')
    plt.show()

d_steps = 100
g_steps = 100

gen=generator(4,4)
dis=discriminator(4,4)

criteriond1 = nn.BCELoss()
optimizerd1 = optim.SGD(dis.parameters(), lr=0.001, momentum=0.9)

criteriond2 = nn.BCELoss()
optimizerd2 = optim.SGD(gen.parameters(), lr=0.001, momentum=0.9)

printing_steps = 20

epochs = 5

for epoch in range(epochs):

    print (epoch)

    # training discriminator
    for d_step in range(d_steps):
        dis.zero_grad()

        # training discriminator on real data
        for inp_real,_ in dataloader_mnist_train:
            inp_real_x = inp_real
            break

        inp_real_x = inp_real_x.reshape(batch_size,784)
        dis_real_out = dis(inp_real_x)
        dis_real_loss = criteriond1(dis_real_out,
                              Variable(torch.ones(batch_size,1)))
        dis_real_loss.backward()

        # training discriminator on data produced by generator
        inp_fake_x_gen = make_some_noise()
        #output from generator is generated
        dis_inp_fake_x = gen(inp_fake_x_gen).detach()
        dis_fake_out = dis(dis_inp_fake_x)
        dis_fake_loss = criteriond1(dis_fake_out,
                                Variable(torch.zeros(batch_size,1)))
        dis_fake_loss.backward()

        optimizerd1.step()

    # training generator
    for g_step in range(g_steps):
        gen.zero_grad()

        #generating data for input for generator
        gen_inp = make_some_noise()

        gen_out = gen(gen_inp)
        dis_out_gen_training = dis(gen_out)
        gen_loss = criteriond2(dis_out_gen_training,
                               Variable(torch.ones(batch_size,1)))
        gen_loss.backward()

        optimizerd2.step()

    if epoch%printing_steps==0:
        plot_img(gen_out[0])
        plot_img(gen_out[1])
        plot_img(gen_out[2])
        plot_img(gen_out[3])
        print("\n\n")

On running the code,following error is shown

 File "mygan.py", line 105, in <module>
    dis_real_out = dis(inp_real_x)
    RuntimeError: size mismatch, m1: [4 x 784], m2: [4 x 784] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:136

How can I resolve this?

I got the code from https://blog.usejournal.com/train-your-first-gan-model-from-scratch-using-pytorch-9b72987fd2c0

Output:
The error hints that the tensor you fed into the discriminator has incorrect shape. Now let's try to find out what the shape of the tensor is, and what shape is expected.

The tensor itself has a shape of [batch_size x 784] because of the reshape operation above. The discriminator network, on the other hand, expects a tensor with a last dimension of 4. This is because the first layer in the discriminator network is nn.Linear(inp, 784), where inp = 4.

A linear layer nn.Linear(input_size, output_size), expects the final dimension of the input tensor to be equal to input_size, and generates output with the final dimension projected to output_size. In this case, it expects an input tensor of shape [batch_size x 4], and outputs a tensor of shape [batch_size x 784].

And now to the real issue: the generator and discriminator that you defined has incorrect size. You seem to have changed the 300 dimension size from the blog post to 784, which I assume is the size of your image (28 x 28 for MNIST). However, the 300 is not the input size, but rather a "hidden state size" -- the model uses a 300-dimensional vector to encode your input image.

What you should do here is to set the input size to 784, and the output size to 1, because the discriminator makes a binary judgment of fake (0) or real (1). For the generator, the input size should be equal to the "input noise" that you randomly generate, in this case 100. The output size should also be 784, because its output is the generated image, which should be the same size as the real data.

So, you only need to make the following changes to your code, and it should run smoothly:

gen = generator(100, 784)
dis = discriminator(784, 1)


Instruction: How to break PyTorch autograd with in-place ops

Input:
I'm trying to understand better the role of in-place operations in PyTorch autograd.
My understanding is that they are likely to cause problems since they may overwrite values needed during the backward step.

I'm trying to build an example where an in-place operation breaks the auto differentiation, my idea is to overwrite some value needed during the backpropagation after it has been used to compute some other tensor.

I'm using the assignment as the in-place operation (I tried += with the same result), I double-checked it is an in-place op in this way:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
y = x
y[3] = -1
print(x)

prints:

tensor([ 0.,  1.,  2., -1.,  4.], grad_fn=<CopySlices>)

This is my attempt to break autograd:

Without the in-place op:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
out1 = x ** 2
out2 = out1 / 10
# out1[3] += 100
out2.sum().backward()
print(x.grad)

This prints

tensor([0.0000, 0.2000, 0.4000, 0.6000, 0.8000])

With the in-place op:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
out1 = x ** 2
out2 = out1 / 10
out1[3] = 0
out2.sum().backward()
print(x.grad)

This prints:

tensor([0.0000, 0.2000, 0.4000, 0.6000, 0.8000])

I was expecting to obtain differents grads.

What is the item assignment doing? I don't get the grad_fn=<CopySlices>.
Why does it return the same grads?
Is there a working example of in-place operations that break autograd?
Is there a list of non-backwards compatible PyTorch ops?

Output:
A working example of an in-place operations that breaks autograd:
  x = torch.ones(5, requires_grad=True)
  x2 = (x + 1).sqrt()
  z = (x2 - 10)
  x2[0] = -1
  z.sum().backward()

Raises:
RuntimeError: one of the variables needed for gradient computation has been modified by an in-place operation: [torch.FloatTensor [5]], which is output 0 of SqrtBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


Instruction: Why bilinear scaling of images with PIL and pytorch produces different results?

Input:
In order to feed an image to the pytorch network I first need to downscale it to some fixed size. At first I've done it using PIL.Image.resize() method, with interpolation mode set to BILINEAR. Then I though it would be more convenient to first convert a batch of images to pytorch tensor and then use torch.nn.functional.interpolate() function to scale the whole tensor at once on a GPU ('bilinear' interpolation mode as well). This lead to a decrease of the model accuracy because now during inference a type of scaling (torch) was different from the one used during training (PIL). After that, I compared two methods of downscaling visually and found out that they produce different results. Pillow downscaling seems more smooth. Do these methods perform different operations under the hood though both being bilinear? If so, I am also curious if there is a way to achieve the same result as Pillow image scaling with torch tensor scaling?

Original image (the well-known Lenna image)

Pillow scaled image:

Torch scaled image:

Mean channel absolute difference map:

Demo code:

import numpy as np
from PIL import Image
import torch
import torch.nn.functional as F
from torchvision import transforms
import matplotlib.pyplot as plt

pil_to_torch = transforms.ToTensor()
res_shape = (128, 128)

pil_img = Image.open('Lenna.png')
torch_img = pil_to_torch(pil_img)

pil_image_scaled = pil_img.resize(res_shape, Image.BILINEAR)
torch_img_scaled = F.interpolate(torch_img.unsqueeze(0), res_shape, mode='bilinear').squeeze(0)

pil_image_scaled_on_torch = pil_to_torch(pil_image_scaled)
relative_diff = torch.abs((pil_image_scaled_on_torch - torch_img_scaled) / pil_image_scaled_on_torch).mean().item()
print('relative pixel diff:', relative_diff)

pil_image_scaled_numpy = pil_image_scaled_on_torch.cpu().numpy().transpose([1, 2, 0])
torch_img_scaled_numpy = torch_img_scaled.cpu().numpy().transpose([1, 2, 0])
plt.imsave('pil_scaled.png', pil_image_scaled_numpy)
plt.imsave('torch_scaled.png', torch_img_scaled_numpy)
plt.imsave('mean_diff.png', np.abs(pil_image_scaled_numpy - torch_img_scaled_numpy).mean(-1))

Python 3.6.6, requirements:

cycler==0.10.0
kiwisolver==1.1.0
matplotlib==3.2.1
numpy==1.18.2
Pillow==7.0.0
pyparsing==2.4.6
python-dateutil==2.8.1
six==1.14.0
torch==1.4.0
torchvision==0.5.0

Output:
"Bilinear interpolation" is an interpolation method.
But downscaling an image is not necessarily only accomplished using interpolation.
It is possible to simply resample the image as a lower sampling rate, using an interpolation method to compute new samples that don't coincide with old samples. But this leads to aliasing (which is what you get when higher frequency components in the image cannot be represented at the lower sampling density, "aliasing" the energy of these higher frequencies onto lower frequency components; that is, new low frequency components appear in the image after the resampling).
To avoid aliasing, some libraries apply a low-pass filter (remove high frequencies that cannot be represented at the lower sampling frequency) before resampling. The subsampling algorithm in these libraries do much more than just interpolating.
The difference you see is because these two libraries take different approaches, one tries to avoid aliasing by low-pass filtering, the other doesn't.
To obtain the same results in Torch as in Pillow, you need to explicitly low-pass filter the image yourself. To get identical results you will have to figure out exactly how Pillow filters the image, there are different methods and different possible parameter settings. Looking at the source code is the best way to find out exactly what they do.


Instruction: Building a deep reinforcement learning with a cnn q - approximation

Input: I'm new in DRL. Starting from this code https://github.com/jaromiru/cwcf, I would like to substitute the MLP used for the q function approximation with a CNN, but I don't know how to do. Can anybody help me? Thanks

Output:
Try going through this it has a detailed explanation on how to build DQN to solve the CartPole problem. You can also have a look at this which has implementations of many DRL algorithms

Then you can replace the code in agent.py present in repo with DQN agent code


Instruction: How can i convert an ndarray shape (32 by xxx) into a single column dataframe, while discarding the last ten objects of this array

Input:
After running a binary classification NLP model over a dataset with batch size of 32, i have an nparray of predictions of size 32 by 300 and the last batch is of size 24. i am trying to rearrange these values in a dataframe.

predictions.append(logits.argmax(1))

[array([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0,
       0, 0, 0, 1, 0, 0, 0, 0, 0, 0]), array([0, 1, 0, 0, 0, 0, 0, 0............

however, when i try to convert it to a dataframe,

df.labels = pd.DataFrame(predictions)
df.labels.head()

this is what i get

0    0.0
1    0.0
2    0.0
3    0.0
4    0.0
Name: labels, dtype: float64

i want to have the values as 1 or 0 integers, not floats
the final data batch of 32 has NaN from value 24 to 32, instead of being blank

Output:
If your column contains NaN it will always be of float type. The presence of NaN values explains therefore why the column is float.

You must have a dimension problem, your predictions variable is two dimensional, 32 x 10, you should flatten it first and then add your 24 elements, see documentions here :

predictions = predictions.flatten()

to_append = logits.argmax(1) # this is your array with 24 elements
predictions = predictions.append(to_append)

What happens here is that whan you append a row of 24 elements to a DataFrame with 32 columns, the last columns from 23 to 32 will exist and will automatically be filled with NaN. See some examples here and here.

Why do you want to transform NaN to blank? If blank means an empty string, you shouldn't do that, as you will mix float and string in your column.

If you want Integer values. You should make an imputation of the NaN values with a constant Integer value (let us say 888):

df = df.fillna(888)

Then you can transform all to int using the function astype :

df = df.astype('int16')


Instruction: weird problem with Pytorch's mse_loss function

Input:
Traceback (most recent call last):
  File "c:/Users/levin/Desktop/programming/nn.py", line 208, in <module>
    agent.train(BATCHSIZE)
  File "c:/Users/levin/Desktop/programming/nn.py", line 147, in train
    output = F.mse_loss(prediction, target)
  File "C:\Users\levin\Anaconda3\lib\site-packages\torch\nn\functional.py", line 2203, in mse_loss
    if not (target.size() == input.size()):
AttributeError: 'NoneType' object has no attribute 'size'

This above is the Error that I'm constantly getting and I really don't know how to fix it.

This some code that might be important

    def train(self, BATCHSIZE):
        trainsample = random.sample(self.memory, BATCHSIZE)

        for state, action, reward, new_state, gameovertemp in trainsample:
            if gameovertemp:
                target = torch.tensor(reward).grad_fn
            else:
                target = reward + self.gamma * torch.max(self.dqn.forward(new_state))

            self.dqn.zero_grad()
            prediction = torch.max(self.dqn.forward(state))
            #print(prediction, "prediction")
            #print(target, "target")
            output = F.mse_loss(prediction, target)
            output.backward()
            self.optimizer.step()

Output:
As stated in a comment the error due to either target of input to be None and is not related to the size() attribute.

The problem is probably at this line:

target = torch.tensor(reward).grad_fn

Here you convert reward to a new Tensor. However, a Tensor created by the user always has a grad_fn equal to None (as explained in Pytorch Autograd).

To have a grad_fn a Tensor must be the result of some computation, not a static value.

The thing is that mse_loss does not expect target to be differentiable, as the name suggest it is just the value to be compared.

Try to remove the grad_fn from this line the raw Tensor should be sufficient.


Instruction: PyTorch TensorBoard write frequency

Input:
I'm trying to write my training and validation losses to tensorboard using torch (torch.utils.tensorboard) and it looks like it only writes up to 1000 data points, no matter what the actual number of iterations are. For example, running the following code,

writer1 = SummaryWriter('runs/1')
writer2 = SummaryWriter('runs/2')

for i in range(2000):
    writer1.add_scalar('tag', 1, i)

for i in range(20000):
    writer2.add_scalar('tag', 1, i)

both yield 1000 points exactly when examining and downloaded csv, and even on the tensorboard dashboard, the first points start at steps 5 and 18 and increment such that the total number of steps are 1000, rather than 2,000 and 20,000.

I don't know if this is tensorboard's default behaviour or if its PyTorch's decision, but either way, is there a way to write every single step?

Output:
Actually I found the answer here. So the SummaryWriter is saving at every epoch, but to load everything, tensorboard has to be started with the flag --samples_per_plugin scalars=0. 0 tells tensorboard to load all points, while 100 would mean a total of 100 points for example

To sum up, I started tensorboard with the command tensorboard --logdir=logs --samples_per_plugin scalars=0


Instruction: kth-value per row in pytorch?

Input:
Given

import torch
A = torch.rand(9).view((3,3)) # tensor([[0.7455, 0.7736, 0.1772],\n[0.6646, 0.4191, 0.6602],\n[0.0818, 0.8079, 0.6424]])
k = torch.tensor([0,1,0])
A.kthvalue_vectoriezed(k) -> [0.1772,0.6602,0.0818]

Meaning I would like to operate on each column with a different k.

Not kthvalue nor topk offers such API.
Is there a vectorized way around that?
Remark - kth value is not the value in the kth index, but the kth smallest element. Pytorch docs

  torch.kthvalue(input, k, dim=None, keepdim=False, out=None) -> (Tensor, LongTensor)
  Returns a namedtuple (values, indices) where values is the k th smallest element of each row of the input tensor in the given dimension dim. And indices is the index location of each element found.

Output:
Assuming you don't need indices into original matrix (if you do, just use fancy indexing for the second return value as well) you could simply sort the values (by last index by default) and return appropriate values like so:

def kth_smallest(tensor, indices):
    tensor_sorted, _ = torch.sort(tensor)
    return tensor_sorted[torch.arange(len(indices)), indices]

And this test case gives you your desired values:

tensor = torch.tensor(
    [[0.7455, 0.7736, 0.1772], [0.6646, 0.4191, 0.6602], [0.0818, 0.8079, 0.6424]]
)

print(kth_smallest(tensor, [0, 1, 0])) # -> [0.1772,0.6602,0.0818]


Instruction: Create a torch::Tensor in C++ to change the shape

Input:
I have a tensor array, and I want to change the shape of tensor. I tried to use torch.view, but it raise an exception that "shape[1] is invalid for input of size 10000". Anyone can give me a tips for the error information?

int shape[] = {1,1,100,100};
torch::Tensor img = torch::zeros((100,100),torch::KF32);
torch::Tensor tmg = img.view(*shape);

Output:
C++ is not python so constructs like unpacking with * obviously will not work. Same goes for (, ), you should use object which can be "auto-casted" to IntArrayRef.

Creating objects basics

ArrayRef is a template class which means it can hold different C++ types and IntArrayRef is an alias for ArrayRef<int>. This class has a few constructors (e.g. from standard C-style array, std::vector, std::array or std::initializer_list).

Both torch::zeros and view method of torch::Tensor require this exact object.

What you can do is:

/* auto to feel more "Pythonic" */
auto img = torch::zeros({100, 100}, torch::kF32);
auto tmg = img.view({1, 1, 100, 100});

{1, 1, 100, 100} is std::initializer_list<int> type so ArrayRef<int> (a.k.a. IntArrayRef) can be constructed from it (probably moved as this object is an rvalue).

Same thing happens for torch::zeros.

Easier way for this case

What you have here could be accomplished easier though with unsqueeze like this:

auto img = torch::zeros({100, 100}, torch::kF32);
auto unsqueezed = img.unsqueeze(0).unsqueeze(0);

Where 0 in the dimension.

About libtorch

All in all read the reference and check types at least if you want to work with C++. I agree docs could use some work but if you know something about C++ it shouldn't be too hard to follow even into source code which might be needed sometimes.


Instruction: Reset Random Initialization on Pytorch

Input:
I want to do some experiments on feedforward neural networks. To make a fair comparison, I need them to have the exact same random initialisation. How can I do it?

Is there a way to save the same initial weights such that I can train a network and then reinitialize it exactly as it was before?

I have been trying to save the initial parameters on a list, called 'init' , and then reassign the parameters  but it did not work:

i = 0
for name, param in model.named_parameters():
    param = init[i]
    i += 1

Any suggestion?

Output:
You can try seeding random via:

torch.manual_seed(seed)
torch.manual_seed_all(seed)

Note you have to seed random before each model initialisation. If this doesn't work, try the following:

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False


Instruction: How can I get the MSE of a tensor across a specific dimension?

Input:
I have 2 tensors with .size of torch.Size([2272, 161]). I want to get mean-squared-error between them. However, I want it along each of the 161 channels, so that my error tensor has a .size of torch.Size([161]). How can I accomplish this?

It seems that torch.nn.MSELoss doesn't let me specify a dimension.

Output:
For the nn.MSELoss you can specify the option reduction='none'. This then gives you back the squared error for each entry position of both of your tensors. Then you can apply torch.sum/torch.mean.

a = torch.randn(2272,161)
b = torch.randn(2272,161)
loss = nn.MSELoss(reduction='none')
loss_result = torch.sum(loss(a,b),dim=0)

I don't think there is a direct way to specify at the initialisation of the loss to which dimension to apply mean/sum. Hope that helps!


Instruction: Modifying validation function for single image instead of Tencrop

Input:
I have a PublicTest function that runs every epoch for validation and there is a transform test variable that transforms the validation data as above:

    transform_test = transforms.Compose([
    transforms.TenCrop(cut_size),
    transforms.Lambda(lambda crops: torch.stack([transforms.ToTensor()(crop) for crop in crops])),
])

def PublicTest(epoch):
    global PublicTest_acc
    global best_PublicTest_acc
    global best_PublicTest_acc_epoch
    net.eval()
    PublicTest_loss = 0
    correct = 0
    total = 0
    for batch_idx, (inputs, targets) in enumerate(PublicTestloader):
        bs, ncrops, c, h, w = np.shape(inputs)
        inputs = inputs.view(-1, c, h, w)
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        outputs_avg = outputs.view(bs, ncrops, -1).mean(1)  # avg over crops
        loss = criterion(outputs_avg, targets)
        PublicTest_loss += loss.item()
        _, predicted = torch.max(outputs_avg.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(PublicTestloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                           % (PublicTest_loss / (batch_idx + 1), 100. * correct / total, correct, total))

I'd like to modify the code so instead of Tencrop each image would be validated once. I changed the transform_test like this

 transform_test = transforms.Compose([transforms.ToTensor()])

I run the code but had the realized that PublicTest have ncrops as a parameter and received the error:

 File "mainpro_FER.py", line 147, in PublicTest
    bs, ncrops, c, h, w = np.shape(inputs)
ValueError: not enough values to unpack (expected 5, got 4)

removed ncrops parameter in the function, tried again had this error:

  IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

So I need a way to make this function work for single image for every images in batch unlike TenCrop.
Thanks.

Output:
Nevermind guys, I figured out just like this:

def PublicTest(epoch):
    global PublicTest_acc
    global best_PublicTest_acc
    global best_PublicTest_acc_epoch
    net.eval()
    PublicTest_loss = 0
    correct = 0
    total = 0
    for batch_idx, (inputs, targets) in enumerate(PublicTestloader):
        bs, c, h, w = np.shape(inputs)
        inputs = inputs.view(-1, c, h, w)
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        # outputs_avg = outputs.view(bs, ncrops, -1).mean(1)  # avg over crops
        loss = criterion(outputs, targets)
        PublicTest_loss += loss.item()
        _, predicted = torch.max(outputs.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(PublicTestloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                           % (PublicTest_loss / (batch_idx + 1), 100. * correct / total, correct, total))


Instruction: How can I get one array to return only the masked values define by another array with Numpy / PyTorch?

Input:
I have a mask, which has a shape of: [64, 2895] and an array pred which has a shape of [64, 2895, 161].

mask is binary with only 0s and 1s. What I want to do is reduce pred so that it maintains 64 batches, and along the 2895, wherever there is a 1 in the mask for each batch, return the related pred.

So as a simplified example, if:

mask = [[1, 0, 0],
        [1, 1, 0],
        [0, 0, 1]]
pred = [[[0.12, 0.23, 0.45, 0.56, 0.57],
         [0.91, 0.98, 0.97, 0.96, 0.95],
         [0.24, 0.46, 0.68, 0.80, 0.15]],

        [[1.12, 1.23, 1.45, 1.56, 1.57],
         [1.91, 1.98, 1.97, 1.96, 1.95],
         [1.24, 1.46, 1.68, 1.80, 1.15]],

        [[2.12, 2.23, 2.45, 2.56, 2.57],
         [2.91, 2.98, 2.97, 2.96, 2.95],
         [2.24, 2.46, 2.68, 2.80, 2.15]]]

What I want is:

[[[0.12, 0.23, 0.45, 0.56, 0.57]],

 [[1.12, 1.23, 1.45, 1.56, 1.57],
  [1.91, 1.98, 1.97, 1.96, 1.95]],

 [[2.24, 2.46, 2.68, 2.80, 2.15]]]

I realize that there are different dimensions, I hope that that's possible. If not, then fill in the missing dimensions with 0. Either numpy or pytorch would be helpful. Thank you.

Output:
If you want a vectorized computation then different dimension seems not possible, but this would give you the one with masked entry filled with 0:

# pred: torch.size([64, 2895, 161])
# mask: torch.size([64, 2895])

result = pred * mask[:, :, None]
# extend mask with another dimension so now it can do entry-wise multiplication

and result is exactly what you want


Instruction: What does "*" operator do before the tensor size in PyTorch?

Input:
I am now learning building neural networks in PyTorch. Here are the codes cut from the .py file:

x = torch.unsqueeze(torch.linspace(-1, 1, 1000), dim=1)
y = x.pow(2) + 0.1*torch.normal(torch.zeros(*x.size()))

I am quite comfused about the utility of the * operator before  x.size(). I tried to delete it and plot the scatter graph, which was proved the same as the one with * not removed.

I also checked the official documentation of size in https://pytorch.org/docs/stable/tensors.html but I couldn't figure it out.

Image of torch.size item in documentation

I'd appreciate it very much if you may help me.

Output:
the reason that * makes no difference in the results here is because torch.zero except both a variable number of arguments and a collection like a list or tuple as mentioned here. It does not mean * itself is useless.

Then, since torch.Size() class is a subclass of python tuple, one can unpack it using *. (x.size() will return a torch.Size() object)

So to wrap up, x.size() would give you (1000, 1) and *x.size() in the argument would give you 1000, 1 and both are accepted by torch.zeros()


Instruction: how target[y == l] = label work is labeling make_blobs dataset in pytorch tutorial?

Input:
I am learning PyTorch with this blog: [PyTorch Introduction to Neural Network] (https://medium.com/biaslyai/pytorch-introduction-to-neural-network-feedforward-neural-network-model-e7231cff47cb)
 I found this piece of code confusing:

from sklearn.datasets import make_blobs
def blob_label(y, label, loc): # assign labels
    target = numpy.copy(y)
    for l in loc:
        target[y == l] = label
    return target
x_train, y_train = make_blobs(n_samples=40, n_features=2, cluster_std=1.5, shuffle=True)
x_train = torch.FloatTensor(x_train)
y_train = torch.FloatTensor(blob_label(y_train, 0, [0]))
y_train = torch.FloatTensor(blob_label(y_train, 1, [1,2,3]))
x_test, y_test = make_blobs(n_samples=10, n_features=2, cluster_std=1.5, shuffle=True)
x_test = torch.FloatTensor(x_test)
y_test = torch.FloatTensor(blob_label(y_test, 0, [0]))
y_test = torch.FloatTensor(blob_label(y_test, 1, [1,2,3]))

how exactly is target[y == l] = label working in here?
how this line of code assigns 1s and 0s to the data?
Thanks!

Output:
It's Boolean or “mask” index arrays see doc for more

>>> y = torch.arange(9).reshape(3,3)
>>> y
tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])
>>> target = np.copy(y)

so, target[y == l] = label
y == l give us boolean array like (if l = 0)

>>> y == 0
tensor([[ True, False, False],
        [False, False, False],
        [False, False, False]])

and we can access and assign value using boolean array y == 0

>>> target[y == 0]
array([0], dtype=int64)

>>> target[y == 0] = 999
>>> target
array([[999,   1,   2],
       [  3,   4,   5],
       [  6,   7,   8]], dtype=int64)


Instruction: Why the backpropagation process can still work when I included 'loss.backward()' in 'with torch.no_grad():'?

Input:
I'm working with a linear regression example in PyTorch. I know I did wrong including 'loss.backward()' in 'with torch.no_grad():', but why it worked well with my code?

According to pytorch docs, torch.autograd.no_grad is a context-manager that disabled gradient calculation. So I'm really confused.

Code here:

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt
# Toy dataset
x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],
                    [9.779], [6.182], [7.59], [2.167], [7.042],
                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)

y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],
                    [3.366], [2.596], [2.53], [1.221], [2.827],
                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)

input_size = 1
output_size = 1
epochs = 100
learning_rate = 0.05

model = nn.Linear(input_size, output_size)

criterion = nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# training
for epoch in range(epochs):
    # convert numpy to tensor
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)

    # forward
    out = model(inputs)
    loss = criterion(out, targets)

    # backward
    with torch.no_grad():
        model.zero_grad()
        loss.backward()
        optimizer.step()
        print('inputs grad : ', inputs.requires_grad)
    if epoch % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, epochs, loss.item()))

predicted = model(torch.from_numpy(x_train)).detach().numpy()
plt.plot(x_train, y_train, 'ro', label='Original data')
plt.plot(x_train, predicted, label='Fitted line')
plt.legend()
plt.show()

# Save the model checkpoint
torch.save(model.state_dict(), 'model\linear_model.ckpt')

Thanks in advance for answering my question.

Output:
This worked because the loss calculation has happened before the no_grad and you keep calculating the gradients according to that loss calculation (which calculation had gradient enabled).

Basically, you continue update the weights of your layers using the gradients calculated outside of the no_grad.

When you actually use the no_grad:

for epoch in range(epochs):
    # convert numpy to tensor
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)
    with torch.no_grad():  # no_grad used here
    # forward
        out = model(inputs)
        loss = criterion(out, targets)
        model.zero_grad()
        loss.backward()
        optimizer.step()
        print('inputs grad : ', inputs.requires_grad)
    if epoch % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, epochs, loss.item()))

Then you will get the proper error, saying:

element 0 of tensors does not require grad and does not have a grad_fn.

That is, you use no_grad where is not appropriate to use it.

If you print the .requires_grad of loss, then you will see that loss has requires_grad.

That is, when you do this:

for epoch in range(epochs):
    # convert numpy to tensor
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)

    # forward
    out = model(inputs)
    loss = criterion(out, targets)

    # backward
    with torch.no_grad():
        model.zero_grad()
        loss.backward()

        optimizer.step()
        print('inputs grad : ', inputs.requires_grad)
        print('loss grad : ', loss.requires_grad)  # Prints loss.require_rgad
    if epoch % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, epochs, loss.item()))

You will see:

inputs grad :  False
loss grad :  True

Additionally, the

print('inputs grad : ', inputs.requires_grad)

Will always print False. That is, if you do

for epoch in range(epochs):
    # convert numpy to tensor
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)

    print('inputs grad : ', inputs.requires_grad). # Print the inputs.requires_grad

    # forward
    out = model(inputs)
    loss = criterion(out, targets)

    # backward
    with torch.no_grad():
        model.zero_grad()
        loss.backward()

        optimizer.step()
        print('inputs grad : ', inputs.requires_grad)
        print('loss grad : ', loss.requires_grad)
    if epoch % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, epochs, loss.item()))

You will get:

inputs grad :  False
inputs grad :  False
loss grad :  True

That is, you are using wrong things to check what you did wrong. The best thing that you can do is to read again the docs of PyTorch on gradient mechanics.


Instruction: Why `torch.cuda.is_available()` returns False even after installing pytorch with cuda?

Input:
On a Windows 10 PC with an NVidia GeForce 820M
I installed CUDA 9.2 and cudnn 7.1 successfully,
and then installed PyTorch using the instructions at pytorch.org:
pip install torch==1.4.0+cu92 torchvision==0.5.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

But I get:
>>> import torch
>>> torch.cuda.is_available()
False

Output:
Your graphics card does not support CUDA 9.0.
Since I've seen a lot of questions that refer to issues like this I'm writing a broad answer on how to check if your system is compatible with CUDA, specifically targeted at using PyTorch with CUDA support. Various circumstance-dependent options for resolving issues are described in the last section of this answer.

The system requirements to use PyTorch with CUDA are as follows:

Your graphics card must support the required version of CUDA
Your graphics card driver must support the required version of CUDA
The PyTorch binaries must be built with support for the compute capability of your graphics card

Note: If you install pre-built binaries (using either pip or conda) then you do not need to install the CUDA toolkit or runtime on your system before installing PyTorch with CUDA support. This is because PyTorch, unless compiled from source, is always delivered with a copy of the CUDA library.

1. How to check if your GPU/graphics card supports a particular CUDA version
First, identify the model of your graphics card.
Before moving forward ensure that you've got an NVIDIA graphics card. AMD and Intel graphics cards do not support CUDA.
NVIDIA doesn't do a great job of providing CUDA compatibility information in a single location. The best resource is probably this section on the CUDA Wikipedia page. To determine which versions of CUDA are supported

Locate your graphics card model in the big table and take note of the compute capability version. For example, the GeForce 820M compute capability is 2.1.
In the bullet list preceding the table check to see if the required CUDA version is supported by the compute capability of your graphics card. For example, CUDA 9.2 is not supported for compute compatibility 2.1.

If your card doesn't support the required CUDA version then see the options in section 4 of this answer.
Note: Compute capability refers to the computational features supported by your graphics card. Newer versions of the CUDA library rely on newer hardware features, which is why we need to determine the compute capability in order to determine the supported versions of CUDA.

2. How to check if your GPU/graphics driver supports a particular CUDA version
The graphics driver is the software that allows your operating system to communicate with your graphics card. Since CUDA relies on low-level communication with the graphics card you need to have an up-to-date driver in order use the latest versions of CUDA.
First, make sure you have an NVIDIA graphics driver installed on your system. You can acquire the newest driver for your system from NVIDIA's website.
If you've installed the latest driver version then your graphics driver probably supports every CUDA version compatible with your graphics card (see section 1). To verify, you can check Table 3 in the CUDA release notes. In rare cases I've heard of the latest recommended graphics drivers not supporting the latest CUDA releases. You should be able to get around this by installing the CUDA toolkit for the required CUDA version and selecting the option to install compatible drivers, though this usually isn't required.
If you can't, or don't want to upgrade the graphics driver then you can check to see if your current driver supports the specific CUDA version as follows:
On Windows

Determine your current graphics driver version (Source https://www.nvidia.com/en-gb/drivers/drivers-faq/)

Right-click on your desktop and select NVIDIA Control Panel. From the
NVIDIA Control Panel menu, select Help > System Information. The
driver version is listed at the top of the Details window. For more
advanced users, you can also get the driver version number from the
Windows Device Manager. Right-click on your graphics device under
display adapters and then select Properties. Select the Driver tab and
read the Driver version. The last 5 digits are the NVIDIA driver
version number.

Visit the CUDA release notes and scroll down to Table 3. Use this table to verify your graphics driver is new enough to support the required version of CUDA.

On Linux/OS X
Run the following command in a terminal window
nvidia-smi

This should result in something like the following
Sat Apr  4 15:31:57 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 206...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   35C    P8    16W / 175W |    502MiB /  7974MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1138      G   /usr/lib/xorg/Xorg                           300MiB |
|    0      2550      G   /usr/bin/compiz                              189MiB |
|    0      5735      G   /usr/lib/firefox/firefox                       5MiB |
|    0      7073      G   /usr/lib/firefox/firefox                       5MiB |
+-----------------------------------------------------------------------------+

Driver Version: ###.## is your graphic driver version. In the example above the driver version is 435.21.
CUDA Version: ##.# is the latest version of CUDA supported by your graphics driver. In the example above the graphics driver supports CUDA 10.1 as well as all compatible CUDA versions before 10.1.
Note: The CUDA Version displayed in this table does not indicate that the CUDA toolkit or runtime are actually installed on your system. This just indicates the latest version of CUDA your graphics driver is compatible with.
To be extra sure that your driver supports the desired CUDA version you can visit Table 3 on the CUDA release notes page.

3. How to check if a particular version of PyTorch is compatible with your GPU/graphics card compute capability
Even if your graphics card supports the required version of CUDA then it's possible that the pre-compiled PyTorch binaries were not compiled with support for your compute capability. For example, in PyTorch 0.3.1 support for compute capability <= 5.0 was dropped.
First, verify that your graphics card and driver both support the required CUDA version (see Sections 1 and 2 above), the information in this section assumes that this is the case.
The easiest way to check if PyTorch supports your compute capability is to install the desired version of PyTorch with CUDA support and run the following from a python interpreter
>>> import torch
>>> torch.zeros(1).cuda()

If you get an error message that reads
Found GPU0 XXXXX which is of cuda capability #.#.
PyTorch no longer supports this GPU because it is too old.

then that means PyTorch was not compiled with support for your compute capability. If this runs without issue then you should be good to go.
Update If you're installing an old version of PyTorch on a system with a newer GPU then it's possible that the old PyTorch release wasn't compiled with support for your compute capability. Assuming your GPU supports the version of CUDA used by PyTorch, then you should be able to rebuild PyTorch from source with the desired CUDA version or upgrade to a more recent version of PyTorch that was compiled with support for the newer compute capabilities.

4. Conclusion
If your graphics card and driver support the required version of CUDA (section 1 and 2) but the PyTorch binaries don't support your compute capability (section 3) then your options are

Compile PyTorch from source with support for your compute capability (see here)
Install PyTorch without CUDA support (CPU-only)
Install an older version of the PyTorch binaries that support your compute capability (not recommended as PyTorch 0.3.1 is very outdated at this point). AFAIK compute capability older than 3.X has never been supported in the pre-built binaries
Upgrade your graphics card

If your graphics card doesn't support the required version of CUDA (section 1) then your options are

Install PyTorch without CUDA support (CPU-only)
Install an older version of PyTorch that supports a CUDA version supported by your graphics card (still may require compiling from source if the binaries don't support your compute capability)
Upgrade your graphics card


Instruction: How can I save PyTorch's DataLoader instance?

Input: I want to save PyTorch's torch.utils.data.dataloader.DataLoader instance, so that I can continue training where I left off (keeping shuffle seed, states and everything).

Output:
It's quite simple. One should design their own Sampler which takes the starting index and shuffles the data by itself:
import random
from torch.utils.data.dataloader import Sampler

random.seed(224)  # use a fixed number

class MySampler(Sampler):
    def __init__(self, data, i=0):
        random.shuffle(data)
        self.seq = list(range(len(data)))[i * batch_size:]

    def __iter__(self):
        return iter(self.seq)

    def __len__(self):
        return len(self.seq)

Now save the last index i somewhere and the next time instantiate the DataLoader using it:
train_dataset = MyDataset(train_data)
train_sampler = MySampler(train_dataset, last_i)
train_data_loader = DataLoader(dataset=train_dataset,
                               batch_size=batch_size,
                               sampler=train_sampler,
                               shuffle=False)  # don't forget to set DataLoader's shuffle to False

It's quite useful when training on Colab.


Instruction: How do I assign a numpy.int64 to a torch.cuda.FloatTensor?

Input:
Im not able to assign int64 to torch tensor. I have got the following tensor

tempScale = torch.zeros((total, len(scale))).cuda() if useGpu else torch.zeros((nbPatchTotal, len(scale)))

In my code, when I'm using the following line, it's throwing an error message

tmpScale[:, j] = scale

The error message is

TypeError: can't assign a numpy.int64 to a torch.cuda.FloatTensor

what am I missing?

Output:
You have to convert scale to a torch tensor of the same type and device as tmpScale before assignment.

tmpScale[:, j] = torch.from_numpy(scale).to(tmpScale)

Note that this is casting scale from an int64 to a float32 which will likely result in a loss of precision if values in scale have magnitude larger than 224 (about 16 million).


Instruction: Empty state_dict with vector or tuple of layers in nn.Module

Input:
I switched to using a Version with a parametrized number of layers of torch.nn.Module like Net_par below, only to find out all the saved state_dicts were empty after quite some optimizing.^^

This method is the recommended saving operation (https://pytorch.org/docs/stable/notes/serialization.html#recommend-saving-models), still layers stored in a vector (or tuple, for that matter) are discarded when constructing the state_dict.

torch.save works properly in contrast, but adds to data and limits robustness. This feels a little like a bug, can anybody help with a workaround?

Minimal example for comparison between parametrized and fixed layer count:

import torch
import torch.nn as nn

class Net_par(nn.Module):
    def __init__(self,layer_dofs):
        super(Net_par, self).__init__()
        self.layers=[]
        for i in range(len(layer_dofs)-1):
            self.layers.append(nn.Linear(layer_dofs[i],layer_dofs[i+1]))
    def forward(self, x):
        for i in range(len(self.layers)-1):
            x = torch.tanh(self.layers[i](x))
        return torch.tanh(self.layers[len(self.layers)-1](x))

class Net_sta(nn.Module):
    def __init__(self,dof1,dof2):
        super(Net_sta, self).__init__()
        self.layer=nn.Linear(dof1,dof2)
    def forward(self, x):
        return torch.tanh(self.layer1(x))

if __name__=="__main__":
    net_par=Net_par((3,4))
    net_sta=Net_sta(3,4)
    print(str(net_par.state_dict()))
    #OrderedDict()                 <------Why?!
    print(str(net_sta.state_dict()))
    #OrderedDict([('layer.weight', tensor([[...
    #   ...]])), ('layer.bias', tensor([...   ...]))])

Output:
You need to use nn.ModuleList() instead of simple python list.

class Net_par(nn.Module):
    ...
    self.layers = nn.ModuleList([])


Instruction: How to change DataLoader in PyTorch to read one image for prediction?

Input:
Currently, I have a pre-trained model that uses a DataLoader for reading a batch of images for training the model.
self.data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=False,
   num_workers=1, pin_memory=True)

...

model.eval()
for step, inputs in enumerate(test_loader.data_loader):
   outputs = model(torch.cat([inputs], 1))

...

I want to process (make predictions) on images, as they arrive from a queue. It should be similar to a code that reads a single image and runs the model to make predictions on it. Something along the following lines:
from PIL import Image

new_input = Image.open(image_path)
model.eval()
outputs = model(torch.cat([new_input ], 1))

I was wondering if you could guide me how to do this and apply the same transformations in the DataLoader.

Output:
You can use do it with IterableDataset :

from torch.utils.data import IterableDataset

class MyDataset(IterableDataset):
    def __init__(self, image_queue):
      self.queue = image_queue

    def read_next_image(self):
        while self.queue.qsize() > 0:
            # you can add transform here
            yield self.queue.get()
        return None

    def __iter__(self):
        return self.read_next_image()

and batch_size = 1 :

import queue
import torchvision.transforms.functional as TF

buffer = queue.Queue()
new_input = Image.open(image_path)
buffer.put(TF.to_tensor(new_input))
# ... Populate queue here

dataset = MyDataset(buffer)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=1)
for data in dataloader:
   model(data) # data is one-image batch of size [1,3,H,W] where 3 - number of color channels


Instruction: How to convert a list of different-sized tensors to a single tensor?

Input:
I want to convert a list of tensors with different sizes to a single tensor.

I tried torch.stack, but it shows an error.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-237-76c3ff6f157f> in <module>
----> 1 torch.stack(t)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 5 and 6 in dimension 1 at C:\w\1\s\tmp_conda_3.7_105232\conda\conda-bld\pytorch_1579085620499\work\aten\src\TH/generic/THTensor.cpp:612

My list of tensors:

[tensor([-0.1873, -0.6180, -0.3918, -0.5849, -0.3607]),
 tensor([-0.6873, -0.3918, -0.5849, -0.9768, -0.7590, -0.6707]),
 tensor([-0.6686, -0.7022, -0.7436, -0.8231, -0.6348, -0.4040, -0.6074, -0.6921])]

I have also tried this in a different way, instead of tensors, I used a lists of these individual tensors and tried to make a tensor out of it. That also showed an error.

list: [[-0.18729999661445618, -0.6179999709129333, -0.3917999863624573, -0.5849000215530396, -0.36070001125335693], [-0.6873000264167786, -0.3917999863624573, -0.5849000215530396, -0.9768000245094299, -0.7590000033378601, -0.6707000136375427], [-0.6686000227928162, -0.7021999955177307, -0.7436000108718872, -0.8230999708175659, -0.6348000168800354, -0.40400001406669617, -0.6074000000953674, -0.6920999884605408]]

The error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-245-489aea87f307> in <module>
----> 1 torch.FloatTensor(t)

ValueError: expected sequence of length 5 at dim 1 (got 6)

Apparently, it says, it is expecting the same length of lists if I am not wrong.

Can anyone help here?

Output:
I agree with @helloswift123, you cannot stack tensors of different lengths.

Also, @helloswift123's answer will work only when the total number of elements is divisible by the shape that you want. In this case, the total number of elements is 19 and in no case, it can be reshaped into something useful since it is a prime number.

torch.cat() as suggested,

data = [torch.tensor([-0.1873, -0.6180, -0.3918, -0.5849, -0.3607]),
                torch.tensor([-0.6873, -0.3918, -0.5849, -0.9768, -0.7590, -0.6707]),
                torch.tensor([-0.6686, -0.7022, -0.7436, -0.8231, -0.6348, -0.4040, -0.6074, -0.6921])]
dataTensor = torch.cat(data)
dataTensor.numel()

Output:

tensor([-0.1873, -0.6180, -0.3918, -0.5849, -0.3607, -0.6873, -0.3918, -0.5849,
        -0.9768, -0.7590, -0.6707, -0.6686, -0.7022, -0.7436, -0.8231, -0.6348,
        -0.4040, -0.6074, -0.6921])
19

Possible solution:

This is also not a perfect solution but might solve this problem.

# Have a list of tensors (which can be of different lengths)
data = [torch.tensor([-0.1873, -0.6180, -0.3918, -0.5849, -0.3607]),
        torch.tensor([-0.6873, -0.3918, -0.5849, -0.9768, -0.7590, -0.6707]),
        torch.tensor([-0.6686, -0.7022, -0.7436, -0.8231, -0.6348, -0.4040, -0.6074, -0.6921])]

# Determine maximum length
max_len = max([x.squeeze().numel() for x in data])

# pad all tensors to have same length
data = [torch.nn.functional.pad(x, pad=(0, max_len - x.numel()), mode='constant', value=0) for x in data]

# stack them
data = torch.stack(data)

print(data)
print(data.shape)

Output:

tensor([[-0.1873, -0.6180, -0.3918, -0.5849, -0.3607,  0.0000,  0.0000,  0.0000],
        [-0.6873, -0.3918, -0.5849, -0.9768, -0.7590, -0.6707,  0.0000,  0.0000],
        [-0.6686, -0.7022, -0.7436, -0.8231, -0.6348, -0.4040, -0.6074, -0.6921]])
torch.Size([3, 8])

This will append zeros to the end of any tensor which is having fewer elements and in this case you can use torch.stack() as usual.

I hope this helps!


Instruction: Python, Pytorch, having trouble using cuda

Input:
OS: Windows 10

Python 3.7.4 (Conda)

GPU: GTX 980

So I installed CUDA toolkit v.10.1, got the matching cuDNN files, installed the cuda 10.1 enabled pytorch version and in addition to that i updated my gpu drivers. but if i now try to check wether my gpu is available with torch.cuda.is_available() it still get a False. Any Ideas??

Output:
You need to make sure that your python is running on GPU instead of CPU.

You can do this by starting your IDE on GPU by selecting from right-click options. Just right click on your IDE and select Run with graphics processer. Like Shown in picture below.

Alternatively, you can go to Nvidia Control Panel > Manage 3D Settings > Program Settings and customize the default Graphics Processor of Python and your IDE to NVIDIA GPU. The control panel can be seen below.


Instruction: Sum over indices with value (1)

Input:
I have a tensor with a shape (2, 2, 3) like:

a= tensor ([[[2, 0, 2],[1, 0, 0]],[[1, 0, 1],[0, 1, 0]]])

I want to find the indices of the values (1), then make 2 to the power of those indices and finally add the results for the last dimension, so the final result should be like:

tensor ([[[],[2^0]],[[2^0+2^2],[2^1]]])

My actual tensor is much bigger than this example, so I don’t want to use "for" loop, and I have to use broadcasting…

I was thinking of something like torch.pow(2,(a == 1).nonzero()).sum(), but it doesn’t work. I have to find a way to apply (a== 1).nonzero() only for the last dimension, any suggestion? Thanks.

Output:
Change it to NumPy and apply this:

a_n = a.numpy()
a_n = np.apply_along_axis(func1d=lambda x: np.sum(np.power(2,np.where(x==1))[0]), axis=2, arr=a_n)

a = torch.Tensor(a_n)

Basically, it applies the function you want in axis=2, I'm supposing your larger array is something like (2,2,n)


Instruction: How to properly run my project using GPU?

Input:
I am really new to torch and machine learning. I am trying to run my project using GPU. I tried to have such modification to my code:

model = Challenge()
    model = model.to(torch.device('cuda'))

However, I am still having following error:

Traceback (most recent call last):
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 112, in <module>
    main()
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 91, in main
    stats)
  File "C:/Users/ruidong/Desktop/YZR temp/Project2/train_challenge.py", line 40, in _evaluate_epoch
    output = model(X)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ruidong\Desktop\YZR temp\Project2\model\challenge.py", line 48, in forward
    z = F.relu(self.conv1(x))
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "C:\Users\ruidong\Anaconda3\envs\EECS445\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward

Any suggestions? really appreciate.

Output: The model is correctly moved to GPU. However, for a model that is placed in GPU, you need to pass the tensors that are in GPU too. The error is because you are passing tensor that is placed in CPU in a model that is in GPU. Just do the same for inputs before passing them to model


Instruction: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True

Input:
I'm a student and a beginner in Python and PyTorch both. I have a very basic Neural Network for which I am encountering the mentioned RunTimeError. The code to reproduce the error is this:

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
import matplotlib.pyplot as plt

# Ensure Reproducibility
torch.manual_seed(0)

# Data Generation
x = torch.randn((100,1), requires_grad = True)
y = 1 + 2 * x + 0.3 * torch.randn(100,1)
# Shuffles the indices
idx = np.arange(100)
np.random.shuffle(idx)

# Uses first 80 random indices for train
train_idx = idx[:70]
# Uses the remaining indices for validation
val_idx = idx[70:]

# Generates train and validation sets
x_train, y_train = x[train_idx], y[train_idx]
x_val, y_val = x[val_idx], y[val_idx]

class OurFirstNeuralNetwork(nn.Module):
    def __init__(self):
        super(OurFirstNeuralNetwork, self).__init__()
        # Here we "define" our Neural Network Architecture
        self.fc1 = nn.Linear(1, 5)
        self.non_linearity_fc1 = nn.ReLU()
        self.fc2 = nn.Linear(5,1)
        #self.non_linearity_fc2 = nn.ReLU()

    def forward(self, x):
        # The forward pass
        # Here we define how activations "flow" between neurons. We've already discussed the "Sum" and "Transformation" steps of the forward pass.
        sum_fc1 = self.fc1(x)
        transformation_fc1 = self.non_linearity_fc1(sum_fc1)
        sum_fc2 = self.fc2(transformation_fc1)
        #transformation_fc2 = self.non_linearity_fc2(sum_fc2)
        # The transformation_fc2 is also the output of our model which symbolises the end of our forward pass.
        return sum_fc2

# Instantiate the model and train

model = OurFirstNeuralNetwork()
print(model)
print(model.state_dict())
n_epochs = 1000
loss_fn = nn.MSELoss(reduction='mean')
optimizer = optim.Adam(model.parameters())

for epoch in range(n_epochs):

    model.train()
    optimizer.zero_grad()
    prediction = model(x_train)
    loss = loss_fn(y_train, prediction)
    print(epoch, loss)
    loss.backward(retain_graph=True)
    optimizer.step()

print(model.state_dict())

Everything is basic and standard and this works fine.

However, when I take out the "retain_graph=True" argument, it throws the RunTimeError. From reading various forums, I understand that this is to do with the graph getting thrown away after the first iteration but I have seen many tutorials and blogs where loss.backward() is the way to go especially since it conserves memory. But I am not able to conceptually grasp why the same does not work for me.

Any help is appreciated and my apologies if the way in which I have asked my question is not in the expected format. I am open to feedback and will oblige to include more details or rephrase the question so that it is easier for everyone. Thank you in advance!

Output:
You need to add optimizer.zero_grad() after optimizer.step() to zero out the gradients.

Why you need to do this?

When you do loss.backward() torch will compute gradients for parameters and update the parameter's .grad property. When you do optimizer.step(), the parameters are updated using the .grad property as  i.e `parameter = parameter - lr*parameter.grad.

Since you do not clear the gradients and call backward the second time, it will compute dl/d(updated param) which will require to backpropagate through paramter.grad of the first pass. When doing backward, the computation graph of this gradients is not stored and hence you have to pass retain_graph= True to get rid of error. However, we don't want to do that for updating params. Rather we want to clear gradients, and restart with a new computation graph therefore, you need to zero the gradients with a .zero_grad call.

Also see Why do we need to call zero_grad() in PyTorch?


Instruction: torch.nn.conv2d does not give the same result as torch.nn.functional.conv2d

Input:
This is my code:

l1 = nn.Conv2d(3, 2, kernel_size=3, stride=2).double() #Layer
l1wt = l1.weight.data #filter
inputs = np.random.rand(3, 3, 5, 5) #input
it = torch.from_numpy(inputs) #input tensor
output1 = l1(it) #output
output2 = torch.nn.functional.conv2d(it, l1wt, stride=2) #output
print(output1)
print(output2)

I would expect to get the same result for output1 and output2, but they are not.
Am I doing something wrong are do nn and nn.functional work different?

Output:
I think you forgot the bias.

inp = torch.rand(3,3,5,5)
a = nn.Conv2d(3,2,3,stride=2)
a(inp)
nn.functional.conv2d(inp, a.weight.data, bias=a.bias.data)

Looks the same to me


Instruction: Pytorch cross entropy input dimensions

Input:
I'm trying to develop a binary classifier with Huggingface's BertModel and Pytorch.
The classifier module is something like this:

class SSTClassifierModel(nn.Module):

  def __init__(self, num_classes = 2, hidden_size = 768):
    super(SSTClassifierModel, self).__init__()
    self.number_of_classes = num_classes
    self.dropout = nn.Dropout(0.01)
    self.hidden_size = hidden_size
    self.bert = BertModel.from_pretrained('bert-base-uncased')
    self.classifier = nn.Linear(hidden_size, num_classes)

  def forward(self, input_ids, att_masks,token_type_ids,  labels):
    _, embedding = self.bert(input_ids, token_type_ids, att_masks)
    output = self.classifier(self.dropout(embedding))
    return output

The way I train the model is as follows:

loss_function = BCELoss()
model.train()
for epoch in range(NO_OF_EPOCHS):
  for step, batch in enumerate(train_dataloader):
        input_ids = batch[0].to(device)
        input_mask = batch[1].to(device)
        token_type_ids = batch[2].to(device)
        labels = batch[3].to(device)
        # assuming batch size = 3, labels is something like:
        # tensor([[0],[1],[1]])
        model.zero_grad()
        model_output = model(input_ids,
                             input_mask,
                             token_type_ids,
                             labels)
        # model output is something like: (with batch size = 3)
        # tensor([[ 0.3566, -0.0333],
                 #[ 0.1154,  0.2842],
                 #[-0.0016,  0.3767]], grad_fn=<AddmmBackward>)

        loss = loss_function(model_output.view(-1,2) , labels.view(-1))

I'm doing the .view()s because of the Huggingface's source code for BertForSequenceClassification here which uses the exact same way to compute the loss. But I get this error:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2068     if input.numel() != target.numel():
   2069         raise ValueError("Target and input must have the same number of elements. target nelement ({}) "
-> 2070                          "!= input nelement ({})".format(target.numel(), input.numel()))
   2071
   2072     if weight is not None:

ValueError: Target and input must have the same number of elements. target nelement (3) != input nelement (6)

Is there something wrong with my labels? or my model's output? I'm really stuck here. The documentation for Pytorch's BCELoss says:

Input: (N,∗) where ∗ means, any number of additional dimensions
Target: (N,∗), same shape as the input

How should I make my labels the same shape as the model output? I feel like there's something huge that I'm missing but I can't find it.

Output:
Few observations:

The code that you refer to uses CrossEntropyLoss but you are using BCELoss.
CrossEntropyLoss takes prediction logits (size: (N,D)) and target labels (size: (N,)) whereas BCELoss takes p(y=1|x) (size: (N,)) and target labels (size: (N,)) as p(y=0|x) can be computed from p(y=1|x)
CrossEntropyLoss expects logits i.e whereas BCELoss expects probability value

Solution:

Since you pass an (N,2) tensor, it gives an error. You only need to pass p(y=1|x), therefore you can do

loss = loss_function(model_output.view(-1,2)[:,1] , labels.view(-1))

above I assumed that the second value is p(y=1|x).

A cleaner way would be to make model output only one value i.e p(y=1|x) and pass it to the loss function. It seems from the code that you are passing logit values and not probability values, so you may also need to compute sigmoid (model_output) if you want to use BCELoss or alternatively you can use BCEWithLogitsLoss.

Another alternative is to change the loss to CrossEntropyLoss that should work too as it can work for binary labels too.


Instruction: Get local world size in torch distributed training

Input:
Suppose I have 2 machines with 4 GPUs each. Suppose that each instance of the training algorithm requires 2 GPUs.
I would like to run 4 processes, 2 for each machine, each process using 2 GPUs.

How can I make each process retrieve the number of local processes running on the same machine?
I can detect the world size with

torch.distributed.get_world_size()

and the global rank with

torch.distributed.get_rank()

But, given that I would like not to hard code parameters, is there a way to recover that on each node are running 2 processes? This will be usefull to me to assign GPUs to each process equally.

Example: Suppose I know that a machine has 4 GPUs and that there are 2 processes on it, I will assign GPUs [0, 1] to process with local rank 0 and GPUs [2, 3] to process with local rank 1. I know total number of processes but I cannot understand if they are on the same machine, so I cannot decide how many GPUs they are allowed to use.

I need a function that would be called torch.distributed.get_local_world_size()

Output:
torch.cuda.device_count() is essentially the local world size and could be useful in determining how many GPUs you have available on each device. If you can't do that for some reason, using plain MPI might help
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank() # device rank - [0,1]

torch.cuda.device(i)
ngpus = torch.cuda.device_count()
print(ngpus, " gpus on machine", i) # here's local world size for each process

but I think it would work just to call torch.cuda.device_count() in any case without adding this dependency. I am pretty new here so if you can, please let me know how this answer can be improved.


Instruction: Pytorch Dataloader shuffle with multiple dataset

Input:
I'm trying to make custom Dataloader with multiple datasets.

My question is that if I use (shuffle = True) in the Dataloader option, is it possible to shuffle the same order in multiple Dataloader?

For example:

dataloader1: label = [5 , 4, 15, 16]

dataloader2: label = [5 , 4, 15, 16]

Output:
Edit: Pytorch's dataloaders have an implemented solution for that already.

See here: https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler you can specify the sampler yourself. So you can create a generator and give it to all dataloaders.

Old (and a bit more hacky) answer:

If keeping the order is really important, instead of making a custom dataloader, it may be better to make a custom dataset.

Note that is only possible if all datasets have the same number of examples. Or not use part of the data of the larger datasets.

Something in those lines should work:

class ManyDatasetsInOne(Dataset):
    def __init__(self, **parameters):
        self.dataset1 = dataset1(**parameters_1)
        self.dataset2 = dataset2(**parameters_2)

    def __len__(self):
        return len(self.dataset1)

    def __getitem__(self, index):

        data1 = load_item(idx, self.dataset1)
        data2 = load_item(idx, self.dataset1)

        return data1, data2


Instruction: Pytorch Concatenate rows in alternate order

Input:
I am trying to code up the positional encoding in the transformers paper. In order to do so I need to do an operation similar to the following:

a = torch.arange(20).reshape(4,5)
b = a * 2
c = torch.cat([torch.stack([a_row,b_row]) for a_row, b_row in zip(a,b)])

I feel like there might be a faster way to do the above? perhaps by adding a dimension on to a and b?

Output:
So turns out simple concatenation and reshaping does the trick:

c = torch.cat([a, b], dim=-1).view(-1, a.shape[-1])

When I timed it with the following it was about 2.3x faster than @dennlinger's answer:

improved2 = timeit.timeit("c = torch.cat([a, b], dim=-1).view(-1, a.shape[-1])",
                          setup="import torch; from __main__ import a, b",
                          number=10000)
print(improved2/10000)
# 7.253780400003507e-06
print(improved / improved2)
# 2.3988091506044955


Instruction: Libtorch operator/syntax errors in Visual Studio

Input:
Hi recently I’ve installed Libtorch and I was able to use it in my new Visual Studio project without a problem. Currently I am trying to use Libtorch in an existing CUDA project. But I am having these strange errors when I include torch header and I couldn’t find any solution in the internet regarding to my problem. Does anyone have any idea what might be the cause of these errors?

Severity    Code    Description Project File    Line    Suppression State
Error   C2833   'operator {' is not a recognized operator or type   DepthSensing    e:\research\libtorch\include\c10\util\flat_hash_map.h   1433
Error   C2059   syntax error: 'newline' DepthSensing    e:\research\libtorch\include\c10\util\flat_hash_map.h   1433
Error   C2238   unexpected token(s) preceding ';'   DepthSensing    e:\research\libtorch\include\c10\util\flat_hash_map.h   1433
Error   C2143   syntax error: missing ';' before 'const'    DepthSensing    e:\research\libtorch\include\c10\util\flat_hash_map.h   1433
Error   C2833   'operator {' is not a recognized operator or type   DepthSensing    e:\research\libtorch\include\c10\util\order_preserving_flat_hash_map.h  1552
Error   C2059   syntax error: 'newline' DepthSensing    e:\research\libtorch\include\c10\util\order_preserving_flat_hash_map.h  1552
Error   C2238   unexpected token(s) preceding ';'   DepthSensing    e:\research\libtorch\include\c10\util\order_preserving_flat_hash_map.h  1552
Error   C2143   syntax error: missing ';' before 'const'    DepthSensing    e:\research\libtorch\include\c10\util\order_preserving_flat_hash_map.h  1552

Environment

Windows 10
CUDA 10.1
Visual Studio 2017
C++14

Output:
Thanks to @john I have realized that there was a macro in another library which has the same name as a typename in Libtorch library(which was a macro called V in my case), that’s why it was confused in compilation.  I am sticking to this solution for now.

warning C4003: not enough actual parameters for macro 'max' - Visual Studio 2010 C++


Instruction: PyTorch transferring index to 1-0

Input:
How to quickly set the elements in the given index list to 1 and others to 0?

For example,I have an ID pool like:
torch.arange(10),
for a given input index tensor([1,5,7,9,2]) wanna return tensor([0,1,1,0,0,1,0,1,0,1])

Output:
Easiest is to start with zeros and fill with ones using fancy indexing like this:

import torch

tensor = torch.zeros(10)
tensor[[1, 5, 7, 9, 2]] = 1

If your IDs are predefined (e.g. torch.arange(10)) and you want to get only those elements which are not zero you can do this:

import torch

ids = torch.arange(10)

mask = torch.zeros_like(ids).bool() # it has to be bool
mask[[1, 5, 7, 9, 2]] = True

torch.masked_select(ids, mask)

Which would give you:

tensor([1, 2, 5, 7, 9])


Instruction: Should I set model.eval for getting the current training loss in Pytorch?

Input: We set model.train() during training, but during my training iterations, I also want to do a forward pass of the training dataset to see what my new loss is. When doing this, should I temporarily set model.eval()?

Output:
If your network has layers which act different during inference (torch.nn.BatchNormNd and torch.nn.DropoutNd could be an example, for the second case all neurons will be used but scaled by inverted probability of keeping neurons, see here or here for example) and you want to test how your network performs currently (which is usually called a validation step) then it is mandatory to use module.eval().

It is a common (and very good!) practice to always switch to eval mode when doing inference-like things no matter if this changes your actual model.

EDIT:

You should also use with torch.no_grad(): block during inference, see official tutorial code as gradients are not needed during this phase and it's wasteful to compute them.


Instruction: Applying a 1D Convolution on a Tensor in Pytorch

Input:
I have a Tensor that represents a set of 1D signals, that are concatenated along the column axis. So say I have 300 1D signals that are of size 64. So [64x300]

I want to apply a smooth convolution / moving average kernel on it [0.2 0.2 0.2 0.2 0.2] on the GPU, but I am not sure exactly what is the API to do it. Can I be provided an example?

Output:
You can use regular torch.nn.Conv1d to do this.

Inputs

In your case you have 1 channel (1D) with 300 timesteps (please refer to documentation those values will be appropriately C_in and L_in).

So, for your input it would be (you need 1 there, it cannot be squeezed!):

import torch

inputs = torch.randn(64, 1, 300)

Convolution

You need torch.nn.Conv1d with kernel_size equal to 5 (as indicated by your elements: [0.2 0.2 0.2 0.2 0.2]) and no bias. I assume your output has to be of the same size (300) so 2 elements have to be padded at the beginning and end. All of this gives us this module:

module = torch.nn.Conv1d(
    in_channels=1, out_channels=1, kernel_size=5, padding=2, bias=False
)

Weights of this module (0.2 values) can be specified like this:

module.weight.data = torch.full_like(module.weight.data, 0.2)

torch.full_like will work for kernel of any size in case you want other size than 5.

Finally run it to average steps and you're done:

out = module(inputs)

GPU

If you want to use GPU just cast your module and inputs like this:

inputs = inputs.cuda()
module = module.cuda()

See CUDA documentation for more information.


Instruction: ValueError: sampler option is mutually exclusive with shuffle pytorch

Input:
i'm working on face recognition project using pytorch and mtcnn and after trained my training dataset , now i want to make prediction on test data set

this my trained code

optimizer = optim.Adam(resnet.parameters(), lr=0.001)
scheduler = MultiStepLR(optimizer, [5, 10])

trans = transforms.Compose([
   np.float32,
   transforms.ToTensor(),
   fixed_image_standardization
])
dataset = datasets.ImageFolder(data_dir, transform=trans)
img_inds = np.arange(len(dataset))
np.random.shuffle(img_inds)
train_inds = img_inds[:int(0.8 * len(img_inds))]
val_inds = img_inds[int(0.8 * len(img_inds)):]

train_loader = DataLoader(
   dataset,
   num_workers=workers,
   batch_size=batch_size,
   sampler=SubsetRandomSampler(train_inds)
)
val_loader = DataLoader(
   dataset,
   shuffle=True,
   num_workers=workers,
   batch_size=batch_size,
   sampler=SubsetRandomSampler(val_inds)
)

and if remove sampler=SubsetRandomSampler(val_inds) and put val_inds instead it will rise this error

  val_inds
      ^
  SyntaxError: positional argument follows keyword argument

i want to make prediction (select randomly from test data set) in pytorch?thats why i should use shuffle=True
i followed this repo facenet-pytorch

Output:
TLDR; Remove shuffle=True in this case as SubsetRandomSampler shuffles data already.

What torch.utils.data.SubsetRandomSampler does (please consult documentation when in doubt) is it will take a list of indices and return their permutation.

In your case you have indices corresponding to training (those are indices of elements in training Dataset) and validation.

Let's assume those look like that:

train_indices = [0, 2, 3, 4, 5, 6, 9, 10, 12, 13, 15]
val_indices = [1, 7, 8, 11, 14]

During each pass SubsetRandomSampler will return one number from those lists at random and those will be randomized again after all of them were returned (__iter__ will be called again).

So SubsetRandomSampler might return something like this for val_indices (analogously for train_indices):

val_indices = [1, 8, 11, 7, 14]  # Epoch 1
val_indices = [11, 7, 8, 14, 1]  # Epoch 2
val_indices = [7, 1, 14, 8, 11]  # Epoch 3

Now each of those numbers are an index to your original dataset. Please note validation is shuffled this way and so is train without using shuffle=True. Those indices do not overlap so data is splitted correctly.

Additional info

shuffle uses torch.utils.data.RandomSampler under the hood if shuffle=True is specified, see source code. This in turn is equivalent to using torch.utils.data.SubsetRandomSampler with all indices (np.arange(len(datatest))) specified.
you don't have to pre-shuffle np.random.shuffle(img_inds) as indices will be shuffled during each pass anyway
don't use numpy if torch provides the same functionality. There is torch.arange, mixing both libraries is almost never necessary.

Inference

Single image

Just pass it through your network an get output, e.g.:

module.eval()
with torch.no_grad():
    output = module(dataset[5380])

First line puts model in evaluation mode (changes behaviour of some layer), context manager turns off gradient (as it's not needed for predictions). Those are almost always used when "checking neural network output".

Checking validation dataset

Something along those lines, notice the same ideas applied as for single image:

module.eval()

total_batches = 0
batch_accuracy = 0
for images, labels in val_loader:
    total_batches += 1
    with torch.no_grad():
        output = module(images)
        # In case it outputs logits without activation
        # If it outputs activation you may have to use argmax or > 0.5 for binary case
        # Item gets float from torch.tensor
        batch_accuracy += torch.mean(labels == (output > 0.0)).item()

print("Overall accuracy: {}".format(batch_accuracy / total_batches))

Other cases

Please see some beginners guides or tutorials and understand those concepts as StackOverflow is not a place to re-do this work (rather concrete and small questions), thanks.