Instruction: Can anyone tell me how to checking if Pytorch model exists, and if it does, delete it and replace it with a new one?

Input: So I save a lot of torch models for training and with different batchsize and epochs, and the models are saves with strings of epoch and batchsize. Basically I sometimes change some layers hyperparamters and some augmentation to check the prediction results, but if the torch model is there, I want to delete it and replace it with the new one.

Output:
The simplest solution is simply saving a model with the same name, essentially overwriting the existing one. This is equivalent to checking if it exists, deleting and then saving.

If you want to explicitly check if it exists, you can do that easily with os.

import os
if os.path.exists('path/to/model.pth'):  # checking if there is a file with this name
    os.remove('path/to/model.pth')  # deleting the file
    torch.save(model, 'path/to/model.pth')  # saving a new model with the same name


Instruction: How to convert ndarray to autograd variable in GPU format?

Input:
I am trying to do something like this,

data = torch.autograd.Variable(torch.from_numpy(nd_array))

It comes under the type as Variable[torch.FloatTensor], But I need Variable[torch.cuda.FloatTensor] also I want to do this in pytorch version 0.3.0 which lacks few methods like to(device) or set_default_device

Output:
You can use cuda() method of your tensor.

If you'd like to use specific device you could go with context manager, e.g.

with torch.cuda.device(device_index):
    t = torch.FloatTensor(1.).cuda()

For more specific information check documentation for version 0.3.0.


Instruction: Get each sequence's last item from packed sequence

Input:
I am trying to put a packed and padded sequence through a GRU, and retrieve the output of the last item of each sequence. Of course I don't mean the -1 item, but the actual last, not-padded item. We know the lengths of the sequences in advance, so it should be as easy as to extract for each sequence the length-1 item.

I tried the following

import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

# Data
input = torch.Tensor([[[0., 0., 0.],
                       [1., 0., 1.],
                       [1., 1., 0.],
                       [1., 0., 1.],
                       [1., 0., 1.],
                       [1., 1., 0.]],

                      [[1., 1., 0.],
                       [0., 1., 0.],
                       [0., 0., 0.],
                       [0., 1., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.]],

                      [[0., 0., 0.],
                       [1., 0., 0.],
                       [1., 1., 1.],
                       [0., 0., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.]],

                      [[1., 1., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.],
                       [0., 0., 0.]]])

lengths = [6, 4, 3, 1]
p = pack_padded_sequence(input, lengths, batch_first=True)

# Forward
gru = torch.nn.GRU(3, 12, batch_first=True)
packed_output, gru_h = gru(p)

# Unpack
output, input_sizes = pad_packed_sequence(packed_output, batch_first=True)

last_seq_idxs = torch.LongTensor([x-1 for x in input_sizes])

last_seq_items = torch.index_select(output, 1, last_seq_idxs)

print(last_seq_items.size())
# torch.Size([4, 4, 12])

But the shape is not what I expect. I had expected to get 4x12, i.e. last item of each individual sequence x hidden.`

I could loop through the whole thing, and build a new tensor containing the items I need, but I was hoping for a built-in approach that took advantage of some smart math. I fear that manually looping and building, will result in very poor performance.

Output:
Instead of last two operations last_seq_idxs and last_seq_items you could just do last_seq_items=output[torch.arange(4), input_sizes-1].

I don't think index_select is doing the right thing. It will select the whole batch at the index you passed and therefore your output size is [4,4,12].


Instruction: What is the difference between view and view_as in PyTorch?

Input: I am building neural networks in Pytorch, I see view and view_as used interchangeably in various implementation what is the difference between them?

Output:
view and view_as are very similar with a slight difference. In view() the shape of the desired output tensor is to be passed in as the parameter, whereas in view_as() a tensor whose shape is to be mimicked is passed.
tensor.view_as(other) is equivalent to tensor.view(other.size())


Instruction: Is there a pytorch method to check the number of cpus?

Input: I can use this torch.cuda.device_count() to check the number of GPUs. I was wondering if there was something equivalent to check the number of CPUs.

Output:
just use this :

os.cpu_count()


Instruction: In a Kaggle kernel while having selected the GPU option when checking torch.cuda.is_available(), it says is not available

Input:
I created a kernel for a finished Kaggle competition in which i used pytorch. When checking if cuda is available, it returns False.

I checked the GPU option from settings and it says it is on in the bottom bar with resources info. I tried to restart the session without any changes.
What could be the problem? (cpu only pytorch installed maybe?)

Output: The problem was with the selected docker configuration from settings. Selecting the "Latest Available" fixed the problem.


Instruction: How to implement low-dimensional embedding layer in pytorch

Input:
I recently read a paper about embedding.

In Eq. (3), the f is a 4096X1 vector. the author try to compress the vector in to theta (a 20X1 vector) by using an embedding matrix E.

The equation is simple  theta = E*f

I was wondering if it can using pytorch to achieve this goal, then in the training, the E can be learned automatically.

How to finish the rest? thanks so much.

The demo code is follow:

import torch
from torch import nn

f = torch.randn(4096,1)

Output:
Assuming your input vectors are one-hot that is where "embedding layers" are used, you can directly use embedding layer from torch which does above as well as some more things. nn.Embeddings take non-zero index of one-hot vector as input as a long tensor. For ex: if feature vector is

f = [[0,0,1], [1,0,0]]

then input to nn.Embeddings will be

input = [2, 0]

However, what OP has asked in question is getting embeddings by matrix multiplication and below I will address that. You can define a module to do that as below. Since, param is an instance of nn.Parameter it will be registered as a parameter and will be optimized when you call Adam or any other optimizer.

class Embedding(nn.Module):
    def __init__(self, input_dim, embedding_dim):
        super().__init__()
        self.param = torch.nn.Parameter(torch.randn(input_dim, embedding_dim))

    def forward(self, x):
        return torch.mm(x, self.param)

If you notice carefully this is the same as a linear layer with no bias and slightly different initialization. Therefore, you can achieve the same by using a linear layer as below.

self.embedding = nn.Linear(4096, 20, bias=False)
# change initial weights to normal[0,1] or whatever is required
embedding.weight.data = torch.randn_like(embedding.weight)


Instruction: Training model in eval() mode gives better result in PyTorch?

Input:
I have a model with Dropout layers (with p=0.6). I ended up training the model in .eval() mode and again trained the model in .train() mode, I find that the training .eval() mode gave me better accuracy and quicker loss reduction on training data,

train(): Train loss : 0.832, Validation Loss : 0.821
eval(): Train loss : 0.323, Validation Loss : 0.251

Why is this so?

Output:
This seems like the model architecture is simple and when in train mode, is not able to capture the features in the data and hence undergoes underfitting.

eval() disables dropouts and Batch normalization, among other modules.

This means that the model trains better without dropout helping the model the learn better with more neurons, also increasing the layer size, increasing the number of layers, decreasing the dropout probability, helps.


Instruction: nn.Linear should be mismatch, but it works successfully

Input:
I'm confused about the in-feature of nn.linear. For out-feature of the model VGG-19's last nn.MaxPool2d, the tensor size is (512, 7, 7). The model below uses pooling function and resizes the tensor to (512, 49), then uses nn.linear(512, 7) directly. Why can't it work successfully without mismatch problem?

source

'''VGG11/13/16/19 in Pytorch.'''
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

cfg = {
    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 7)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)

Output:
Why is the assumption that this code works? I tested it, and got the following shapes, and the expected size mismatch error.

def forward(self, x):
    out = self.features(x) # torch.Size([1, 512, 7, 7])
    out = out.view(out.size(0), -1)  # torch.Size([1, 25088])
    out = F.dropout(out, p=0.5, training=self.training)  # torch.Size([1, 25088])
    out = self.classifier(out)  # RuntimeError: size mismatch, m1: [1 x 25088], m2: [512 x 7]
    return out

One mistake that you made with inferring the sizes, is that you omitted the batch dimension. That is why you may falsely conclude that the shape change with out.view(out.size(0), -1) is [512, 7, 7] -> [512, 49] instead the correct [b, 512, 7, 7] -> [b, 25088] where b is the batch size.

As expected when classifier is changed to

self.classifier = nn.Linear(25088, 7)

then the forward function works, no size mismatch error.


Instruction: Why does multi-class classification fails with sigmoid?

Input:
MNIST trained with Sigmoid fails while Softmax works fine

I am trying to investigate how different activation affects the final results, so I implemented a simple net for MNIST with PyTorch.

I am using NLLLoss (Negative log likelihood) as it implements Cross Entropy Loss when used with softmax.

When I have softmax as activation of the last layer, it works great.
But when I used sigmoid instead, I noticed that things fall apart

Here is my network code

def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 80)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.XXXX(x)

where XXXX is the activation function

both Sigmoid and Softmax output values between (0, 1).
Yes Softmax guarantees the sum of 1 but I am not sure if this answers why the training fails with Sigmoid.
Is there any detail I am not catching here?

Output:
Sigmoid + crossentropy can be used for multilabel classification (assume a picture with a dog and a cat, you want the model to return "dog and cat"). It works when the classes aren't mutually exclusive or the samples contain more than one object that you want to recognize.

In your case MNIST has mutually exclusive classes and in each image there is only one number, so it is better to use logsoftmax + negative loglikelihood, which assume that the classes are mutually exclusive and there is only one correct label associated to the image.

So, you can't really expect to have that behavior from sigmoid.


Instruction: Applying Kullback-Leibler (aka kl divergence) element-wise in Pytorch

Input:
I have two tensors named x_t, x_k with follwing shapes NxHxW and KxNxHxW respectively, where K, is the number of autoencoders used to reconstruct x_t (if you have no idea what is this, assume they're K different nets aiming to predict x_t, this probably has nothing to do with the question anyways) N is batch size, H matrix height, W matrix width.

I'm trying to apply Kullback-Leibler divergence algorithm to both tensors (after broadcasting x_t as x_k along the Kth dimension) using Pytorch's nn.functional.kl_div method.

However, it does not seem to be working as I expected. I'm looking to calcualte the kl_div between each observation in x_t and x_k resulting in a tensor of size KxN (i.e., kl_div of each observation for each K autoencoder).

The actual output is a single value if I use the reduction argument, and the same tensor size (i.e., KxNxHxW) if I do not use it.

Has anyone tried something similar?

Reproducible example:

import torch
import torch.nn.functional as F
#                  K   N   H  W
x_t = torch.randn(    10, 5, 5)
x_k = torch.randn( 3, 10, 5, 5)

x_broadcasted = x_t.expand_as(x_k)

loss = F.kl_div(x_t, x_k, reduction="none") # or "batchmean", or there are many options

Output:
It's unclear to me what exactly constitutes a probability distribution in your model. With reduction='none', kl_div, given log(x_n) and y_n, computes kl_div = y_n * (log(y_n) - log(x_n)), which is the "summed" part of the actual Kullback-Leibler divergence. Summation (or, in other words, taking the expectation) is up to you. If your point is that H, W are the two dimensions over which you want to take expectation, it's as simple as

loss = F.kl_div(x_t, x_k, reduction="none").sum(dim=(-1, -2))

Which is of shape [K, N]. If your network output is to be interpreted differently, you need to better specify which are the event dimensions and which are sample dimensions of your distribution.


Instruction: Pytorch: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead

Input:
Calling tensor.numpy() gives the error:

RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

tensor.cpu().detach().numpy() gives the same error.

Output:
 Error reproduced
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.numpy()

print(tensor1)
print(type(tensor1))

which leads to the exact same error for the line tensor1 = tensor1.numpy():
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
Traceback (most recent call last):
  File "/home/badScript.py", line 8, in <module>
    tensor1 = tensor1.numpy()
RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

Process finished with exit code 1

Generic solution
this was suggested to you in your error message, just replace var with your variable name
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.detach().numpy()

print(tensor1)
print(type(tensor1))

which returns as expected
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
[1. 2.]
<class 'numpy.ndarray'>

Process finished with exit code 0

Some explanation
You need to convert your tensor to another tensor that isn't requiring a gradient in addition to its actual value definition. This other tensor can be converted to a numpy array. Cf. this discuss.pytorch post. (I think, more precisely, that one needs to do that in order to get the actual tensor out of its pytorch Variable wrapper, cf. this other discuss.pytorch post).


Instruction: How can I reduce a tensor's last dimension in PyTorch?

Input:
I have tensor of shape (1, 3, 256, 256, 3). I need to reduce one of the dimensions to obtain the shape (1, 3, 256, 256). How can I do it?

Thanks!

Output:
If you intend to apply mean over the last dimension, then you can do so with:

In [18]: t = torch.randn((1, 3, 256, 256, 3))

In [19]: t.shape
Out[19]: torch.Size([1, 3, 256, 256, 3])

# apply mean over the last dimension
In [23]: t_reduced = torch.mean(t, -1)

In [24]: t_reduced.shape
Out[24]: torch.Size([1, 3, 256, 256])

# equivalently
In [32]: torch.mean(t, t.ndimension()-1).shape
Out[32]: torch.Size([1, 3, 256, 256])


Instruction: Code conversion from python 2 to python 3

Input:
I'm setting up a new algorithm which combines an object detector(bounding box detector) which is in python 3 and a mask generator which is in python 2. The problem here is I have several python 2 files which is required for the mask generation algorithm. So I tried 2to3 to convert all my python 2 files to python 3. The script seemed like working but as it was a deep learning algorithm(for mask generation when bounding box coordinates are given as input) which needs some pytorch weights to be loaded, while testing the model in python 3 the program was throwing out error like

  "RuntimeError: Expected object of type torch.FloatTensor but found
  type torch.cuda.FloatTensor for argument #2 ‘weight’"

I have searched in PyTorch forums but none of the posts were useful to me. Is it because my mask generation code is trained in python 2 ?

Does that means while loading the weights and testing the model I should use python 2 not python 3 ? It would be great if someone can shed some light on this. As a work around I can still use the object detector code downgraded to python 2. But still I want to know why it was throwing the error.

Output:
I just resolved the issue by re-installing the torch(0.4.0) and torchvision(0.2.1) for my conda environment. I had to downgrade the versions of both of them. Finally I was successful in converting my python 2.7 code to python 3. Thanks to 2to3 library. Actually this error was happening in the image normalize function of the PyTorch. That was an internal function which accepts the image array as tensors.

tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor


Instruction: Error libtorch_python.so: cannot open shared object file: No such file or directory

Input:
I'm trying to implement fastai pretrain language model and it requires torch to work. After run the code, I got some problem about the import torch._C

I run it on my linux, python 3.7.1, via pip: torch 1.0.1.post2, cuda V7.5.17. I'm getting this error:

Traceback (most recent call last):
  File "pretrain_lm.py", line 7, in <module>
    import fastai
  File "/home/andira/anaconda3/lib/python3.7/site-packages/fastai/__init__.py", line 1, in <module>
    from .basic_train import *
  File "/home/andira/anaconda3/lib/python3.7/site-packages/fastai/basic_train.py", line 2, in <module>
    from .torch_core import *
  File "/home/andira/anaconda3/lib/python3.7/site-packages/fastai/torch_core.py", line 2, in <module>
    from .imports.torch import *
  File "/home/andira/anaconda3/lib/python3.7/site-packages/fastai/imports/__init__.py", line 2, in <module>
    from .torch import *
  File "/home/andira/anaconda3/lib/python3.7/site-packages/fastai/imports/torch.py", line 1, in <module>
    import torch, torch.nn.functional as F
  File "/home/andira/anaconda3/lib/python3.7/site-packages/torch/__init__.py", line 84, in <module>
    from torch._C import *
ImportError: libtorch_python.so: cannot open shared object file: No such file or directory

So I tried to run this line:

from torch._C import *

and got same result

ImportError: libtorch_python.so: cannot open shared object file: No such file or directory

I checked /home/andira/anaconda3/lib/python3.7/site-packages/torch/lib and there are only libcaffe2_gpu.so and libshm.so file, and I can't find libtorch_python.so either. My question is, what is actually libtorch_python.so? I've read some of article and like most of it talked about undefined symbol, not cannot open shared object file: No such file or directory like mine. I'm new at python and torch so I really appreciate your answer.

Output:
My problem is solved. I'm uninstalling my torch twice
pip uninstall torch
pip uninstall torch

and then re-installing it back:
pip install torch==1.0.1.post2


Instruction: Matrix-vector multiplication for only one dimension in a tensor

Input:
Is it possible to multiply only one (last) dimension in a tensor alone with other vectors?

For example, assume a tensor T=[100, 20, 400] and a matrix M =[400, 400].
Is it possible to make the operation h_{transpose}*M*h, where h is the last dimension in the tensor T? In other words, is it possible to make use of (possibly pytorch) built-in functions to get the resulting tensor of size [100, 20, 1]?

Output:
I think the easiest (certainly the shortest) solution is with einsum.

import torch

T = torch.randn(100, 20, 400)
M = torch.randn(400, 400)

res = torch.einsum('abc,cd,abd->ab', (T, M, T)).unsqueeze(-1)

It basically says "for all (a, b, c, d) in bounds, multiply T[a, b, c] with M[c, d] and T[a, b, d] and accumulate it in res[a, b]".

Since einsum is implemented in terms of basic building blocks like mm, transpose etc, this could certainly be unrolled into a more "classical" solution, but right now my brain fails me at that.


Instruction: Regarding number of epochs for torchvision models

Input:
I was trying to find, how many epochs was the pretrained Alexnet model (available from torchvision) trained for on Imagenet and also what learning rate was used? I tried checking the checkpoint keys to see if any epoch info was stored.
Any suggestions on how to find it out?

Output:
According to this comment on GitHub by a PyTorch team member, most of the training was done with a variant of https://github.com/pytorch/examples/tree/master/imagenet. All the models were trained on Imagenet. According to the file:

The default learning rate schedule starts at 0.1 and decays by a factor of 10 every 30 epochs, though they recommend using 0.01 for Alexnet as initial learning rate.
The default value for epochs is 90.


Instruction: Pytorch - add rows of a 2D tensor element-wise

Input:
I have the following tensor :

ts = torch.tensor([[1,2,3],[4,6,7],[8,9,10]])
> tensor([[ 1,  2,  3],
    [ 4,  6,  7],
    [ 8,  9, 10]])

I am looking for a pytorch generic operation that adds all rows element-wise like that:

ts2 = ts[0]+ts[1]+ts[2]
print(ts2)
> tensor([13, 17, 20])

In reality, the number of rows corresponds to the batch size that vary.

Output:
You can sum over an axis/dimension like so:

torch.sum(ts, dim=0)


Instruction: Pytorch loss function error in the last batch

Input:
Assume that I have 77 samples to train my CNN, and my batch size is 10. Then the last batch has a batch size of 7 instead of 10. Somehow when I pass it to the loss function such as nn.MSELoss(), it gives me the error:

  RuntimeError: The size of tensor a (10) must match the size of tensor
  b (7) at non-singleton dimension 1

So pytorch doesn't support batches with different sizes?

My code in doubt:

import numpy as np
import torch
from torch import nn
import torchvision
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, (5,4))
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(64, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, x.shape[1] * x.shape[2] * x.shape[3])
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Net()

batch_size = 10

# Generating Artifical data
x_train = torch.randn((77,1,20,20))
y_train = torch.randint(0,10,size=(77,),dtype=torch.float)

trainset = torch.utils.data.TensorDataset(x_train,y_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=0)
# testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=0)

criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

for epoch in range(20):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i%10==0:
            print('epoch{}, step{}, loss: {}'.format(epoch + 1, i + 1, running_loss))
#             print("frac post = {}".format(frac_post))
            running_loss = 0.0

Output:
The problem is not due to the batch size, but to a failure to broadcast properly between the 10 outputs of your CNN and the single label provided in each example.
If you look at the model output and label tensor shapes during the batch where the error is thrown,

print(outputs.shape, labels.shape)
#out: torch.Size([7, 10]) torch.Size([7])

you'll see that the labels are stored in a singleton tensor. According to pytorch broadcasting rules, to be broadcastable two tensors have to be compatible in all trailing dimensions. In this case, the trailing dimension of the model output (10) is incompatible with that of the label (7).

To fix, either add a dummy dimension to the label (assuming you actually want to broadcast the labels to match your ten network outputs), or define a network with scalar outputs. For example:

y_train = torch.randint(0,10,size=(77,1),dtype=torch.float)

results in

print(outputs.shape, labels.shape)
#out: torch.Size([7, 10]) torch.Size([7,1])
# these are broadcastable


Instruction: how to load the gpu trained model into the cpu?

Input:
I am using PyTorch. I am going to use the already trained model on multiple GPUs with CPU. how to do this task?

I tried on Anaconda 3 and pytorch with cpu only i dont have gpu

model = models.get_pose_net(config, is_train=False)
gpus = [int(i) for i in config.GPUS.split(',')]
model = torch.nn.DataParallel(model, device_ids=gpus).cuda()

print('Created model...')
print(model)
checkpoint = torch.load(config.MODEL.RESUME)
model.load_state_dict(checkpoint)
model.eval()
print('Loaded pretrained weights...')

the error i got is

    AssertionError                            Traceback (most recent call  last)
<ipython-input-15-bbfcd201d332> in <module>()
      2 model = models.get_pose_net(config, is_train=False)
      3 gpus = [int(i) for i in config.GPUS.split(',')]
----> 4 model = torch.nn.DataParallel(model, device_ids=gpus).cuda()
      5 print('Created model...')
      6 print(model)

C:\Users\psl\Anaconda3\lib\site-packages\torch\nn\modules\module.py in cuda(self, device)
    258             Module: self
    259         """
--> 260         return self._apply(lambda t: t.cuda(device))
    261
    262     def cpu(self):

C:\Users\psl\Anaconda3\lib\site-packages\torch\nn\modules\module.py in

_apply(self, fn)
        185     def _apply(self, fn):
        186         for module in self.children():
    --> 187             module._apply(fn)
        188
        189         for param in self._parameters.values():

C:\Users\psl\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    185     def _apply(self, fn):
    186         for module in self.children():
--> 187             module._apply(fn)
    188
    189         for param in self._parameters.values():

C:\Users\psl\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    191                 # Tensors stored in modules are graph leaves, and we don't
    192                 # want to create copy nodes, so we have to unpack the data.
--> 193                 param.data = fn(param.data)
    194                 if param._grad is not None:
    195                     param._grad.data = fn(param._grad.data)

C:\Users\psl\Anaconda3\lib\site-packages\torch\nn\modules\module.py in <lambda>(t)
    258             Module: self
    259         """
--> 260         return self._apply(lambda t: t.cuda(device))
    261
    262     def cpu(self):

 C:\Users\psl\Anaconda3\lib\site-packages\torch\cuda\__init__.py in _lazy_init()
    159         raise RuntimeError(
    160             "Cannot re-initialize CUDA in forked subprocess. " + msg)
--> 161     _check_driver()
    162     torch._C._cuda_init()
    163     _cudart = _load_cudart()

C:\Users\psl\Anaconda3\lib\site-packages\torch\cuda\__init__.py in _check_driver()
     80 Found no NVIDIA driver on your system. Please check that you
     81 have an NVIDIA GPU and installed a driver from
---> 82 http://www.nvidia.com/Download/index.aspx""")
     83         else:
     84             # TODO: directly link to the alternative bin that needs install

AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

Output:
To force load the saved model onto cpu, use the following command.
torch.load('/path/to/saved/model', map_location='cpu')

In your case change it to
torch.load(config.MODEL.RESUME, map_location='cpu')


Instruction: How to dynamically index the tensor in pytorch?

Input:
For example, I got a tensor:

tensor = torch.rand(12, 512, 768)

And I got an index list, say it is:

[0,2,3,400,5,32,7,8,321,107,100,511]

I wish to select 1 element out of 512 elements on dimension 2 given the index list. And then the tensor's size would become (12, 1, 768).

Is there a way to do it?

Output:
There is also a way just using PyTorch and avoiding the loop using indexing and  torch.split:

tensor = torch.rand(12, 512, 768)

# create tensor with idx
idx_list = [0,2,3,400,5,32,7,8,321,107,100,511]
# convert list to tensor
idx_tensor = torch.tensor(idx_list)

# indexing and splitting
list_of_tensors = tensor[:, idx_tensor, :].split(1, dim=1)

When you call tensor[:, idx_tensor, :] you will get a tensor of shape: (12, len_of_idx_list, 768). Where the second dimension depends on your number of indices.

Using torch.split this tensor is split into a list of tensors of shape: (12, 1, 768).

So finally list_of_tensors contains tensors of the shape:

[torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768]),
 torch.Size([12, 1, 768])]


Instruction: Cannot Obtain Similar DL Prediction Result in Pytorch C++ API Compared to Python

Input:
I have trained a deep learning model using unet architecture in order to segment the nuclei in python and pytorch. I would like to load this pretrained model and make prediction in C++. For this reason, I obtained trace file(with pt extension). Then, I have run this code:

#include <iostream>

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

using namespace cv;

int main(int argc, const char* argv[]) {

    Mat image;
    image = imread("C:/Users/Sercan/PycharmProjects/samplepyTorch/test_2.png", CV_LOAD_IMAGE_COLOR);

    std::shared_ptr<torch::jit::script::Module> module = torch::jit::load("C:/Users/Sercan/PycharmProjects/samplepyTorch/epistroma_unet_best_model_trace.pt");
    module->to(torch::kCUDA);

    std::vector<int64_t> sizes = { 1, 3, image.rows, image.cols };
    torch::TensorOptions options(torch::ScalarType::Byte);
    torch::Tensor tensor_image = torch::from_blob(image.data, torch::IntList(sizes), options);
    tensor_image = tensor_image.toType(torch::kFloat);
    auto result = module->forward({ tensor_image.to(at::kCUDA) }).toTensor();

    result = result.squeeze().cpu();
    result = at::sigmoid(result);

    cv::Mat img_out(image.rows, image.cols, CV_32F, result.data<float>());

    cv::imwrite("img_out.png", img_out);

}

Image outputs ( First image: test image, Second image: Python prediction result, Third image: C++ prediction result):

As you see, C++ prediction output is not similar to python prediction output. Could you offer a solution to fix this problem?

Output:
Even though the question is old it might be useful to some. This answer is based on pytorch 1.5.0 release (and first stable version of C++ frontend), the case might be a little different in previous versions (though 1.4.0+ would work the same IIRC).

PyTorch C++ frontend code

no need to explicitly create torch::TensorOptions object if you only want to specify the type in torch::from_blob. Check Configuring Properties of Tensor in PyTorch notes, this will clear it up further. Basically, you can just use torch::ScalarType::Byte.
This type is equivalent to torch::kUInt8 which is easier to find in the docs IMO
No need to create std::vector object to keep shape as torch::from_blob has it's second argument of type IntArrayRef, which is a typedef for ArrayRef<int64_t> (see ArrayRef documentation). This class, in turn, has multiple overloaded constructors, one of which takes std::initializer_list (which is exactly yours { 1, 3, image.rows, image.cols })

With all that in mind you can create tensor_image in a single line like so (added auto as returned type is IMO obvious and const as it won't be modified further as the type is changed in the same line):

const auto tensor_image =
    torch::from_blob(image.data, {1, 3, image.rows, image.cols},
                     torch::kUInt8)
        .toType(torch::kFloat);

Actual error

OpenCV loads images in BGR (blue-green-red) format, while PyTorch usually uses RGB (say in torchvision in Python). Solution is to permute your image so the colors match.

Including above change, whole code becomes:

const auto tensor_image =
    torch::from_blob(image.data, {1, 3, image.rows, image.cols},
                     torch::kUInt8)
        .toType(torch::kFloat)
        .permute(0, 3, 2, 1);

And you should be fine now with your predictions. Maybe it would be beneficial to get tensor > 0 instead of sigmoid as it's probably binary classification and there is no need for this operation per se.

Other PyTorch related stuff

There is no need to use at (ATen - as described in docs, foundational tensor and mathematical operation library on which all else is built) namespace anymore as torch:: namespace redirects to it.

Clearer and less confusing options would be:

torch::kCUDA instead of at::kCUDA
torch::sigmoid instead of at::sigmoid

Also .data<T> is deprecated in favor of .data_ptr<T>

All in all you rarely need to use different namespace than torch:: and it's sub-namespaces.


Instruction: Understanding Gradient in Pytorch

Input:
I have some Pytorch code which demonstrates the gradient calculation within Pytorch, but I am thoroughly confused what got calculated and how it is used. This post here demonstrates the usage of it, but it does not make sense to me in terms of the back propagation algorithm. Looking at the gradient of in1 and in2 in the example below, I realized the gradient of in1 and in2 is the derivative of the loss function but my understanding is that the update needs to also account for the actual loss value as well? Where is the loss value getting used? Am I missing something here?

in1 = torch.randn(2,2,requires_grad=True)
in2 = torch.randn(2,2,requires_grad=True)
target = torch.randn(2,2)
l1 = torch.nn.L1Loss()
l2 = torch.nn.MSELoss()
out1 = l1(in1,target)
out2 = l2(in2,target)
out1.backward()
out2.backward()
in1.grad
in2.grad

Output:
Backpropagation is based on the chain-rule for calculating derivatives. This means the gradients are computed step-by-step from tail to head and always passed back to the previous step ("previous" w.r.t. to the preceding forward pass).

For scalar output the process is initiated by assuming a gradient of d (out1) / d (out1) = 1 to start the process. If you're calling backward on a (non-scalar) tensor though you need to provide the initial gradient since it is not unambiguous.

Let's look at an example that involves more steps to compute the output:

a = torch.tensor(1., requires_grad=True)
b = a**2
c = 5*b
c.backward()
print(a.grad)  # Prints: 10.

So what happens here?

The process is initiated by using d(c)/d(c) = 1.
Then the previous gradient is computed as d(c)/d(b) = 5 and multiplied with the downstream gradient (1 in this case), i.e. 5 * 1 = 5.
Again the previous gradient is computed as d(b)/d(a) = 2*a = 2 and multiplied again with the downstream gradient (5 in this case), i.e. 2 * 5 = 10.
Hence we arrive at a gradient value of 10 for the initial tensor a.

Now in effect this calculates d(c)/d(a) and that's all there is to it. It is the gradient of c with respect to a and hence no notion of a "target loss" is used (even if the loss was zero that doesn't mean the gradient has to be; it is up to the optimizer to step into the correct (downhill) direction and to stop once the loss got sufficiently small).


Instruction: How do I flatten a tensor in pytorch?

Input:
Given a tensor of multiple dimensions, how do I flatten it so that it has a single dimension?
torch.Size([2, 3, 5])    ⟶ flatten ⟶    torch.Size([30])

Output:
TL;DR: torch.flatten()

Use torch.flatten() which was introduced in v0.4.1 and documented in v1.0rc1:

>>> t = torch.tensor([[[1, 2],
                       [3, 4]],
                      [[5, 6],
                       [7, 8]]])
>>> torch.flatten(t)
tensor([1, 2, 3, 4, 5, 6, 7, 8])
>>> torch.flatten(t, start_dim=1)
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])

For v0.4.1 and earlier, use t.reshape(-1).

With t.reshape(-1):

If the requested view is contiguous in memory
this will equivalent to t.view(-1) and memory will not be copied.

Otherwise it will be equivalent to t.contiguous().view(-1).

Other non-options:

t.view(-1) won't copy memory, but may not work depending on original size and stride
t.resize(-1) gives RuntimeError (see below)
t.resize(t.numel()) warning about being a low-level method
(see discussion below)

(Note: pytorch's reshape() may change data but numpy's reshape() won't.)

t.resize(t.numel()) needs some discussion. The torch.Tensor.resize_ documentation says:

  The storage is reinterpreted as C-contiguous, ignoring the current strides (unless the target size equals the current size, in which case the tensor is left unchanged)

Given the current strides will be ignored with the new (1, numel()) size, the order of the elements may apppear in a different order than with reshape(-1). However, "size" may mean the memory size, rather than the tensor's size.

It would be nice if t.resize(-1) worked for both convenience and efficiency, but with torch 1.0.1.post2, t = torch.rand([2, 3, 5]);  t.resize(-1) gives:

RuntimeError: requested resize to -1 (-1 elements in total), but the given
tensor has a size of 2x2 (4 elements). autograd's resize can only change the
shape of a given tensor, while preserving the number of elements.

I raised a feature request for this here, but the consensus was that resize() was a low level method, and reshape() should be used in preference.


Instruction: 4d Input Tensor vs 1d Input Tensor (aka vector) to a neural network

Input:
Reading about machine learning, I keep seeing references to the "input vector" or "feature vector", a 1d tensor that holds the input to the neural network. So for example a 28x28 grayscale image would be a 784 dimensional vector.

Then I also keep seeing references to images being a 4 dimensional tensor with the dimensions being number in batch, color channel, height, and width. For example, this is how it's described in "Deep Learning with Python, by Francois Chollet".

I'm wondering, why is it described in the different ways? When would one be used vs the other?

Output:
There are two main considerations.

First is due to batching. Since we usually want to perform each optimization step based on gradient calculation for a number of training examples (and not just one), it is helpful to run the calculations for all of them at once. Therefore standard approach in many libraries is that the first dimension is the batch dimension, and all operations are applied independently for each subtensor along first dimension. Therefore most tensors in the actual code are at least 2-dimensional: [batch, any_other_dimensions...]. However, from the perspective of the neural network, batching is an implementation detail, so it is often skipped for clarity. Your link talks about 784 dimensional vectors, which are in practice almost undoubtedly processed in batches, so example tensors with batch size of 16 would be of size [batch, features] = [16, 784]. Summing up, we have the first dimension explained as batch, and then there are the any_other_dimensions... which in the above example happens to be a single features dimension of size 784.

Then come the 4 dimensional tensors, which arise when using convolutional neural networks, instead of fully connected ones. A fully connected network uses full matrices, which means that every neuron of the previous layer contributes to every neuron of the following layer. Convolutional neural networks can be seen as using a specially structured sparse matrix, where each neuron of the previous layer influences only some neurons of the following layer, namely those within some fixed distance of its location. Therefore, convolutions impose a spatial structure, which needs to be reflected in the intermediate tensors. Instead of [batch, features], we therefore need [batch, x, y] to reflect the spatial structure of the data. Finally, convolutional neural networks, in everyday practice, have a bit of admixture of fully-connected ones: they have the notion of multiple "features" which are localized spatially - giving raise to the so-called "feature maps" and the tensor raises to 4d: [batch, feature, x, y]. Each value tensor_new[b, f, x, x] is calculated based on all previous values tensor_previous[b', f', x', x'], subject to the following constraints:

b = b': we do not mix the batch elements
x' is at most some distance away from x and similarly for y': we only use the values in the spatial neighborhood
All f's are used: this is the "fully connected" part.

Convolutional neural networks are better suited to visual tasks than fully connected ones, which become infeasible for large enough images (imagine storing a fully connected matrix of size (1024 * 1024) ^ 2 for a 1024 x 1024px image). 4d tensors in CNNs are specific to 2d vision, you can encounter 3d tensors in 1d signal processing (for example sound): [batch, feature, time], 5d in 3d volume processing [batch, feature, x, y, z] and entirely different layouts in other kinds of networks which are neither fully-connected nor convolutional.

Summing up: if somebody tells you they are using 1d vectors, that's a simplification: almost surely the use at least two, for batching. Then, in the context of 2d computer vision, convolutional networks are the standard and they come with 4d tensors. In other scenarios, you may see even different layouts and dimensionalities. Keywords to google for more reading: fully connected neural networks, convolutional neural networks, minibatching or stochastic gradient descend (these two are closely related).


Instruction: Pytorch doesn't support one-hot vector?

Input:
I am very confused by how Pytorch deals with one-hot vectors. In this tutorial, the neural network will generate a one-hot vector as its output. As far as I understand, the schematic structure of the neural network in the tutorial should be like:

However, the labels are not in one-hot vector format. I get the following size

print(labels.size())
print(outputs.size())

output>>> torch.Size([4])
output>>> torch.Size([4, 10])

Miraculously, I they pass the outputs and labels to criterion=CrossEntropyLoss(), there's no error at all.

loss = criterion(outputs, labels) # How come it has no error?

My hypothesis:

Maybe pytorch automatically convert the labels to one-hot vector form. So, I try to convert labels to one-hot vector before passing it to the loss function.

def to_one_hot_vector(num_class, label):
    b = np.zeros((label.shape[0], num_class))
    b[np.arange(label.shape[0]), label] = 1

    return b

labels_one_hot = to_one_hot_vector(10,labels)
labels_one_hot = torch.Tensor(labels_one_hot)
labels_one_hot = labels_one_hot.type(torch.LongTensor)

loss = criterion(outputs, labels_one_hot) # Now it gives me error

However, I got the following error

  RuntimeError: multi-target not supported at
  /opt/pytorch/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

So, one-hot vectors are not supported in Pytorch? How does Pytorch calculates the cross entropy for the two tensor outputs = [1,0,0],[0,0,1] and labels = [0,2] ? It doesn't make sense to me at all at the moment.

Output:
PyTorch states in its documentation for CrossEntropyLoss that

This criterion expects a class index (0 to C-1) as the target for each value of a 1D tensor of size minibatch

In other words, it has your to_one_hot_vector function conceptually built in CEL and does not expose the one-hot API. Notice that one-hot vectors are memory inefficient compared to storing class labels.
If you are given one-hot vectors and need to go to class labels format (for instance  to be compatible with CEL), you can use argmax like below:
import torch

labels = torch.tensor([1, 2, 3, 5])
one_hot = torch.zeros(4, 6)
one_hot[torch.arange(4), labels] = 1

reverted = torch.argmax(one_hot, dim=1)
assert (labels == reverted).all().item()


Instruction: Dict[str, Any] or Dict[str, Field] in pytext

Input:
I'm reading the document of pytext (NLP modeling framework built on PyTorch) and this simple method from_config, a factory method to create a component from a config, has lines like Dict[str, Field] = {ExtraField.TOKEN_RANGE: RawField()}.

@classmethod
def from_config(cls, config: Config, model_input_config, target_config, **kwargs):
    model_input_fields: Dict[str, Field] = create_fields(
      model_input_config,
        {
            ModelInput.WORD_FEAT: TextFeatureField,
            ModelInput.DICT_FEAT: DictFeatureField,
            ModelInput.CHAR_FEAT: CharFeatureField,
        },
    )
    target_fields: Dict[str, Field] = {WordLabelConfig._name: WordLabelField.from_config(target_config)}
    extra_fields: Dict[str, Field] = {ExtraField.TOKEN_RANGE: RawField()}
    kwargs.update(config.items())
    return cls(
        raw_columns=config.columns_to_read,
        targets=target_fields,
        features=model_input_fields,
        extra_fields=extra_fields,
        **kwargs,
    )

and

    def preprocess(self, data: List[Dict[str, Any]]):
        tokens = []
        for row in data:
            tokens.extend(self.preprocess_row(row))
        return [{"text": tokens}]

How can a dictionary have keys with 2 items? What exactly is this?

I would appreciate any pointer!

Output:
What you're seeing are python type annotations. You can read about the syntax, design and rationale here and about the actual implementation (possible types, how to construct custom ones, etc) here. Note that here List and Dict are upper cased - Dict[str, Any] is meant to construct the type "a dictionary with string keys and Any values" and not to access an instance of that type.

Those are optional and by default are not used for anything (so you can just ignore them when reading your code, because python also does). However, there are tools like mypy which can interpret these type annotations and check whether they are consistent.

I don't know for sure how they are used in torchtext - I don't use it myself and I haven't found anything quickly searching the documentation - but they are likely helpful to the developers who use some special tooling. But they can also be helpful to you! From your perspective, they are best treated as comments rather than code. Reading the signature of preprocess you know that data should be a list of dicts with str keys and any value type. If you have bugs in your code and find that data is a str itself, you know for sure that it is a bug (perhaps not the only one).


Instruction: How to import Python package (Pytorch-neat) that is not installable from pip/conda repositories?

Input:
I am trying to used Pytorch-neat package https://github.com/uber-research/PyTorch-NEAT but I don't understand the workflow of using it. I already installed python-neat package and I can import it using import neat in my Jupyter notebook. But what should I do with Pytroch-neat code? There is no pytorch-neat package in Conda or pip repositories, so, I guess that this Pytroch-neat code is not compiled and distributed as the Python package for Jupyter notebook. But what should I do with this code? E.g. sample script contains the code:

import neat

from pytorch_neat.multi_env_eval import MultiEnvEvaluator

So - neat is package and I am importing it. But how to understand the from clause? Should I load Pytroch-neat scripts somehow in the previous cells of my notebook and then I can call this from clause? Or maybe I should compile Pytroch-neat package locally and install it from local repository and import it similarly to neat package. But if so, why the examples use from clause?

I am starting to use Python and I am greatly confused with all of this!

Output: To import from pytorch_neat you have to clone the repository and manually copy directory pytorch_neat into your site-packages (or any directory in sys.path).


Instruction: Pytorch. How does pin_memory work in Dataloader?

Input:
I want to understand how pin_memory in Dataloader works.
According to the documentation:
pin_memory (bool, optional) – If True, the data loader will copy tensors into CUDA pinned memory before returning them.

Below is a self-contained code example.
import torchvision
import torch

print('torch.cuda.is_available()', torch.cuda.is_available())
train_dataset = torchvision.datasets.CIFAR10(root='cifar10_pytorch', download=True, transform=torchvision.transforms.ToTensor())
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, pin_memory=True)
x, y = next(iter(train_dataloader))
print('x.device', x.device)
print('y.device', y.device)

Producing the following output:
torch.cuda.is_available() True
x.device cpu
y.device cpu

But I was expecting something like this, because I specified flag pin_memory=True in Dataloader.
torch.cuda.is_available() True
x.device cuda:0
y.device cuda:0

Also I run some benchmark:
import torchvision
import torch
import time
import numpy as np

pin_memory=True
train_dataset =torchvision.datasets.CIFAR10(root='cifar10_pytorch', download=True, transform=torchvision.transforms.ToTensor())
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, pin_memory=pin_memory)
print('pin_memory:', pin_memory)
times = []
n_runs = 10
for i in range(n_runs):
    st = time.time()
    for bx, by in train_dataloader:
        bx, by = bx.cuda(), by.cuda()
    times.append(time.time() - st)
print('average time:', np.mean(times))

I got the following results.
pin_memory: False
average time: 6.5701503753662

pin_memory: True
average time: 7.0254474401474

So pin_memory=True only makes things slower.
Can someone explain me this behaviour?

Output:
The documentation is perhaps overly laconic, given that the terms used are fairly niche. In CUDA terms, pinned memory does not mean GPU memory but non-paged CPU memory. The benefits and rationale are provided here, but the gist of it is that this flag allows the x.cuda() operation (which you still have to execute as usually) to avoid one implicit CPU-to-CPU copy, which makes it a bit more performant. Additionally, with pinned memory tensors you can use x.cuda(non_blocking=True) to perform the copy asynchronously with respect to host. This can lead to performance gains in certain scenarios, namely if your code is structured as

x.cuda(non_blocking=True)
perform some CPU operations
perform GPU operations using x.

Since the copy initiated in 1. is asynchronous, it does not block 2. from proceeding while the copy is underway and thus the two can happen side by side (which is the gain). Since step 3. requires x to be already copied over to GPU, it cannot be executed until 1. is complete - therefore only 1. and 2. can be overlapping, and 3. will definitely take place afterwards. The duration of 2. is therefore the maximum time you can expect to save with non_blocking=True. Without non_blocking=True your CPU would be waiting idle for the transfer to complete before proceeding with 2..
Note: perhaps step 2. could also comprise of GPU operations, as long as they do not require x - I am not sure if this is true and please don't quote me on that.
Edit: I believe you're missing the point with your benchmark. There are three issues with it

You're not using non_blocking=True in your .cuda() calls.
You're not using multiprocessing in your DataLoader, which means that most of the work is done synchronously on main thread anyway, trumping the memory transfer costs.
You're not performing any CPU work in your data loading loop (aside from .cuda() calls) so there is no work to be overlaid with memory transfers.

A benchmark closer to how pin_memory is meant to be used would be
import torchvision, torch, time
import numpy as np

pin_memory = True
batch_size = 1024 # bigger memory transfers to make their cost more noticable
n_workers = 6 # parallel workers to free up the main thread and reduce data decoding overhead
train_dataset =torchvision.datasets.CIFAR10(
    root='cifar10_pytorch',
    download=True,
    transform=torchvision.transforms.ToTensor()
)
train_dataloader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=batch_size,
    pin_memory=pin_memory,
    num_workers=n_workers
)
print('pin_memory:', pin_memory)
times = []
n_runs = 10

def work():
    # emulates the CPU work done
    time.sleep(0.1)

for i in range(n_runs):
    st = time.time()
    for bx, by in train_dataloader:
       bx, by = bx.cuda(non_blocking=pin_memory), by.cuda(non_blocking=pin_memory)
       work()
   times.append(time.time() - st)
print('average time:', np.mean(times))

which gives an average of 5.48s for my machine with memory pinning and 5.72s without.


Instruction: how to avoid split and sum of pieces in pytorch or numpy

Input:
I want to split a long vector into smaller unequal pieces, do a summation on each piece and gather the results into a new vector.
I need to do this in pytorch but I am also interested to see how this is done with numpy.
This can easily be accomplish by splitting the vector.
sizes = [3, 7, 5, 9]
X = torch.ones(sum(sizes))
Y = torch.tensor([s.sum() for s in torch.split(X, sizes)])

or with np.ones and np.split.
Is there a more efficient way to do this?
Edit:
Inspired by the first comment:
indices = np.cumsum([0]+sizes)[:-1]
Y = np.add.reduceat(X, indices.tolist())

solves it for numpy. I am still looking for a solution with pytorch.

Output:
index_add_ is your friend!

# inputs
sizes = torch.tensor([3, 7, 5, 9], dtype=torch.long)
x = torch.ones(sizes.sum())
# prepare an index vector for summation (what elements of x are summed to each element of y)
ind = torch.zeros(sizes.sum(), dtype=torch.long)
ind[torch.cumsum(sizes, dim=0)[:-1]] = 1
ind = torch.cumsum(ind, dim=0)
# prepare the output
y = torch.zeros(len(sizes))
# do the actual summation
y.index_add_(0, ind, x)


Instruction: 1D correlation between 2 matrices

Input:
I want to find 1D correlation between two matrices. These two matrices are the output of a convolution operation on two different images. Let's call the first matrix as matrix A and the other one as matrix B. Both these matrices have the shape 100 x 100 x 64 (say).

I've been following a research paper which basically computes 1D correlation between these two matrices (matrix A and matrix B) in one of the steps and the output of the correlation operation is also a matrix with the shape 100 x 100 x 64. The link to the paper can be found here. The network can be found on Page 4. The correlation part is in the bottom part of the network. A couple of lines have been mentioned about it in the 2nd paragraph of section 3.3 (on the same page, below the network).

I am not really sure what they mean by 1D correlation and more so how to implement it in Python. I am also confused as to how the shape of the output remains the same as the input after applying correlation. I am using the PyTorch library for implementing this network.

Any help will be appreciated. Thanks.

Output:
So they basically have 1 original image, which they treat as the left side view for the depth perception algorithm, but since you need stereo vision to calculate depth in a still image they use a neural structure to synthesise a right side view.

1 Dimensional Correlation takes 2 sequences and calculates the correlation at each point giving you another 1D sequence of the same length as the 2 inputs. So if you apply this correlation along a certain axis of a tensor the resultant tensor does not change shape.

Intuitively they thought it made sense to correlate the images along the horizontal axes a bit like reading the images like reading a book, but in this instance it should have an effect akin to identifying that things that are further away also appear to be points that are closer together in the left and right side views. The correlation is probably higher for left and right side data-points that are further away and this makes the depth classification for the neural network much easier.


Instruction: Conv1D with kernel_size=1 vs Linear layer

Input:
I'm working on very sparse vectors as input. I started working with simple Linear (dense/fully connected layers) and my network yielded pretty good results (let's take accuracy as my metric here, 95.8%).
I later tried to use a Conv1d with a kernel_size=1 and a MaxPool1d, and this network works slightly better (96.4% accuracy).
Question: How are these two implementation different ? Shouldn't a Conv1d with a unit kernel_size do the same as a Linear layer?
I've tried multiple runs, the CNN always yields slightly better results.

Output:
nn.Conv1d with a kernel size of 1 and nn.Linear give essentially the same results. The only differences are the initialization procedure and how the operations are applied (which has some effect on the speed). Note that using a linear layer should be faster as it is implemented as a simple matrix multiplication (+ adding a broadcasted bias vector)
@RobinFrcd your answers are either different due to MaxPool1d or due to the different initialization procedure.
Here are a few experiments to prove my claims:
def count_parameters(model):
    """Count the number of parameters in a model."""
    return sum([p.numel() for p in model.parameters()])

conv = torch.nn.Conv1d(8,32,1)
print(count_parameters(conv))
# 288

linear = torch.nn.Linear(8,32)
print(count_parameters(linear))
# 288

print(conv.weight.shape)
# torch.Size([32, 8, 1])
print(linear.weight.shape)
# torch.Size([32, 8])

# use same initialization
linear.weight = torch.nn.Parameter(conv.weight.squeeze(2))
linear.bias = torch.nn.Parameter(conv.bias)

tensor = torch.randn(128,256,8)
permuted_tensor = tensor.permute(0,2,1).clone().contiguous()

out_linear = linear(tensor)
print(out_linear.mean())
# tensor(0.0067, grad_fn=<MeanBackward0>)

out_conv = conv(permuted_tensor)
print(out_conv.mean())
# tensor(0.0067, grad_fn=<MeanBackward0>)

Speed test:
%%timeit
_ = linear(tensor)
# 151 µs ± 297 ns per loop

%%timeit
_ = conv(permuted_tensor)
# 1.43 ms ± 6.33 µs per loop

As Hanchen's answer show, the results can differ very slightly due to numerical precision.


Instruction: torch.nn.sequential vs. combination of multiple torch.nn.linear

Input:
I'm trying to create a multi layer neural net class in pytorch. I want to know if the following 2 pieces of code create the same network.

Model 1 with nn.Linear

class TestModel(nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim):
    super(TestModel, self).__init__()
    self.fc1 = nn.Linear(input_dim,hidden_dim)
    self.fc2 = nn.Linear(hidden_dim,output_dim)

def forward(self, x):
    x = nn.functional.relu(self.fc1(x))
    x = nn.functional.softmax(self.fc2(x))
    return x

Model 2 with nn.Sequential

class TestModel2(nn.Module):
def __init__(self, input, hidden, output):
    super(TestModel2, self).__init__()
    self.seq = nn.Sequential(
               nn.Linear(input_dim,hidden_dim),
               nn.ReLU(),
               nn.Linear(hidden_dim,output_dim),
               nn.Softmax()
               )

def forward(self, x):
    return self.seq(x)

Output:
Yes, these two pieces of code create the same network.
One way to convince yourself that this is true is to save both models to ONNX.

import torch.nn as nn

class TestModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(TestModel, self).__init__()
        self.fc1 = nn.Linear(input_dim,hidden_dim)
        self.fc2 = nn.Linear(hidden_dim,output_dim)

    def forward(self, x):
        x = nn.functional.relu(self.fc1(x))
        x = nn.functional.softmax(self.fc2(x))
        return x

class TestModel2(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(TestModel2, self).__init__()
        self.seq = nn.Sequential(
                nn.Linear(input_dim, hidden_dim),
                nn.ReLU(),
                nn.Linear(hidden_dim, output_dim),
                nn.Softmax()
                )

    def forward(self, x):
        return self.seq(x)

m = TestModel(1, 2, 3)
m2 = TestModel2(1, 2, 3)

torch.onnx.export(m, torch.Tensor([0]), "test.onnx", verbose=True)
/opt/anaconda3/envs/py36/bin/ipython:9: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
graph(%0 : Float(1)
      %1 : Float(2, 1)
      %2 : Float(2)
      %3 : Float(3, 2)
      %4 : Float(3)) {
  %5 : Float(1!, 2) = onnx::Transpose[perm=[1, 0]](%1), scope: TestModel/Linear[fc1]
  %6 : Float(2) = onnx::MatMul(%0, %5), scope: TestModel/Linear[fc1]
  %7 : Float(2) = onnx::Add(%6, %2), scope: TestModel/Linear[fc1]
  %8 : Float(2) = onnx::Relu(%7), scope: TestModel
  %9 : Float(2!, 3!) = onnx::Transpose[perm=[1, 0]](%3), scope: TestModel/Linear[fc2]
  %10 : Float(3) = onnx::MatMul(%8, %9), scope: TestModel/Linear[fc2]
  %11 : Float(3) = onnx::Add(%10, %4), scope: TestModel/Linear[fc2]
  %12 : Float(3) = onnx::Softmax[axis=0](%11), scope: TestModel
  return (%12);
}

torch.onnx.export(m2, torch.Tensor([0]), "test.onnx", verbose=True)
/opt/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py:475: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  result = self._slow_forward(*input, **kwargs)
graph(%0 : Float(1)
      %1 : Float(2, 1)
      %2 : Float(2)
      %3 : Float(3, 2)
      %4 : Float(3)) {
  %5 : Float(1!, 2) = onnx::Transpose[perm=[1, 0]](%1), scope: TestModel2/Sequential[seq]/Linear[0]
  %6 : Float(2) = onnx::MatMul(%0, %5), scope: TestModel2/Sequential[seq]/Linear[0]
  %7 : Float(2) = onnx::Add(%6, %2), scope: TestModel2/Sequential[seq]/Linear[0]
  %8 : Float(2) = onnx::Relu(%7), scope: TestModel2/Sequential[seq]/ReLU[1]
  %9 : Float(2!, 3!) = onnx::Transpose[perm=[1, 0]](%3), scope: TestModel2/Sequential[seq]/Linear[2]
  %10 : Float(3) = onnx::MatMul(%8, %9), scope: TestModel2/Sequential[seq]/Linear[2]
  %11 : Float(3) = onnx::Add(%10, %4), scope: TestModel2/Sequential[seq]/Linear[2]
  %12 : Float(3) = onnx::Softmax[axis=0](%11), scope: TestModel2/Sequential[seq]/Softmax[3]
  return (%12);
}

So both models result in the same ONNX graph with the same operations.


Instruction: How do I update a tensor in Pytorch after indexing twice?

Input:
I know how to update a tensor after indexing into part of it like this:

import torch

b = torch.tensor([0, 1, 0, 1], dtype=torch.uint8)
b[b] = 2
b
# tensor([0, 2, 0, 2], dtype=torch.uint8)

but is there a way I can update the original tensor after indexing into it twice? E.g.

i = 1
b = torch.tensor([0, 1, 0, 1], dtype=torch.uint8)
b[b][i] = 2
b
# tensor([0, 1, 0, 1], dtype=torch.uint8)

What I'd like is for b to be tensor([0, 1, 0, 2]) at the end. Is there a way to do this?

I know that I can do

masked = b[b]
masked[i] = 2
b[b] = masked
b
# tensor([0, 1, 0, 2], dtype=torch.uint8)

but is there any better way? It seems that this must be inefficient; if masked is very large, I'm updating many locations in b when I've really only changed one.

(In case a different approach than indexing twice would work better, the general problem I have is how to change the value in an original tensor at the ith location of a masked version of that tensor.)

Output:
I adopted another solution from here, and compared it to your solution:

Solution:

b[b.nonzero()[i]] = 2

Runtime comparison:

import torch as t
import numpy as np
import timeit

if __name__ == "__main__":

    np.random.seed(12345)
    b = t.tensor(np.random.randint(0,2, [1000]), dtype=t.uint8)
    # inconvenient way to think of a random index halfway that is 1.
    halfway = np.array(list(range(len(b))))[b == 1][len(b[b == 1]) //2]

    runs = 100000

    elapsed1 = timeit.timeit("mask=b[b]; mask[halfway] = 2; b[b] = mask",
                             "from __main__ import b, halfway", number=runs)

    print("Time taken (original): {:.6f} ms per call".format(elapsed1 / runs))

    elapsed2 = timeit.timeit("b[b.nonzero()[halfway]]=2",
                             "from __main__ import b, halfway", number=runs)

    print("Time taken (improved): {:.6f} ms per call".format(elapsed2 / runs))

Results:

Time taken (original): 0.000096 ms per call
Time taken (improved): 0.000047 ms per call

Results for vector of length 100000

Time taken: 0.010284 ms per call
Time taken: 0.003667 ms per call

So the solutions differ only by factor 2. I'm not sure if this is the optimal solution, but depending on your size (and how often you call the function) it should give you a rough idea of what you're looking at.


Instruction: PyTorch transforms on TensorDataset

Input:
I'm using TensorDataset to create dataset from numpy arrays.

# convert numpy arrays to pytorch tensors
X_train = torch.stack([torch.from_numpy(np.array(i)) for i in X_train])
y_train = torch.stack([torch.from_numpy(np.array(i)) for i in y_train])

# reshape into [C, H, W]
X_train = X_train.reshape((-1, 1, 28, 28)).float()

# create dataset and dataloaders
train_dataset = torch.utils.data.TensorDataset(X_train, y_train)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64)

How do I apply data augmentation (transforms) to TensorDataset?

For example, using ImageFolder, I can specify transforms as one of its parameters torchvision.datasets.ImageFolder(root, transform=...).

According to this reply by one of PyTorch's team members, it's not supported by default. Is there any alternative way to do so?

Feel free to ask if more code is needed to explain the problem.

Output:
By default transforms are not supported for TensorDataset. But we can create our custom class to add that option. But, as I already mentioned, most of transforms are developed for PIL.Image. But anyway here is very simple MNIST example with very dummy transforms. csv file with MNIST here.

Code:

import numpy as np
import torch
from torch.utils.data import Dataset, TensorDataset

import torchvision
import torchvision.transforms as transforms

import matplotlib.pyplot as plt

# Import mnist dataset from cvs file and convert it to torch tensor

with open('mnist_train.csv', 'r') as f:
    mnist_train = f.readlines()

# Images
X_train = np.array([[float(j) for j in i.strip().split(',')][1:] for i in mnist_train])
X_train = X_train.reshape((-1, 1, 28, 28))
X_train = torch.tensor(X_train)

# Labels
y_train = np.array([int(i[0]) for i in mnist_train])
y_train = y_train.reshape(y_train.shape[0], 1)
y_train = torch.tensor(y_train)

del mnist_train

class CustomTensorDataset(Dataset):
    """TensorDataset with support of transforms.
    """
    def __init__(self, tensors, transform=None):
        assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
        self.tensors = tensors
        self.transform = transform

    def __getitem__(self, index):
        x = self.tensors[0][index]

        if self.transform:
            x = self.transform(x)

        y = self.tensors[1][index]

        return x, y

    def __len__(self):
        return self.tensors[0].size(0)

def imshow(img, title=''):
    """Plot the image batch.
    """
    plt.figure(figsize=(10, 10))
    plt.title(title)
    plt.imshow(np.transpose( img.numpy(), (1, 2, 0)), cmap='gray')
    plt.show()

# Dataset w/o any tranformations
train_dataset_normal = CustomTensorDataset(tensors=(X_train, y_train), transform=None)
train_loader = torch.utils.data.DataLoader(train_dataset_normal, batch_size=16)

# iterate
for i, data in enumerate(train_loader):
    x, y = data
    imshow(torchvision.utils.make_grid(x, 4), title='Normal')
    break  # we need just one batch

# Let's add some transforms

# Dataset with flipping tranformations

def vflip(tensor):
    """Flips tensor vertically.
    """
    tensor = tensor.flip(1)
    return tensor

def hflip(tensor):
    """Flips tensor horizontally.
    """
    tensor = tensor.flip(2)
    return tensor

train_dataset_vf = CustomTensorDataset(tensors=(X_train, y_train), transform=vflip)
train_loader = torch.utils.data.DataLoader(train_dataset_vf, batch_size=16)

result = []

for i, data in enumerate(train_loader):
    x, y = data
    imshow(torchvision.utils.make_grid(x, 4), title='Vertical flip')
    break

train_dataset_hf = CustomTensorDataset(tensors=(X_train, y_train), transform=hflip)
train_loader = torch.utils.data.DataLoader(train_dataset_hf, batch_size=16)

result = []

for i, data in enumerate(train_loader):
    x, y = data
    imshow(torchvision.utils.make_grid(x, 4), title='Horizontal flip')
    break

Output:


Instruction: Fixed Gabor Filter Convolutional Neural Networks

Input:
I'm trying to build a CNN with some conv layers where half of the filters in the layer are fixed and the other half is learnable while training the model. But I didn't find anything about that.

what I'm trying to do is similar to what they did in this paper https://arxiv.org/pdf/1705.04748.pdf

Is there a way to do that in Keras, Pytorch...

Output:
Sure. In PyTorch you can use nn.Conv2d and

set its weight parameter manually to your desired filters
exclude these weights from learning

A simple example would be:

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.conv_learning = nn.Conv2d(1, 5, 3, bias=False)
        self.conv_gabor = nn.Conv2d(1, 5, 3, bias=False)
        # weights HAVE TO be wrapped in `nn.Parameter` even if they are not learning
        self.conv_gabor.weight = nn.Parameter(torch.randn(1, 5, 3, 3))

    def forward(self, x):
        y = self.conv_learning(x)
        y = torch.sigmoid(y)
        y = self.conv_gabor(y)

        return y.mean()

model = Model()
xs = torch.randn(10, 1, 30, 30)
ys = torch.randn(10)
loss_fn = nn.MSELoss()

# we can exclude parameters from being learned here, by filtering them
# out based on some criterion. For instance if all your fixed filters have
# "gabor" in name, the following will do
learning_parameters = (param for name, param in model.named_parameters()
                             if 'gabor' not in name)
optim = torch.optim.SGD(learning_parameters, lr=0.1)

epochs = 10
for e in range(epochs):
    y = model(xs)
    loss = loss_fn(y, ys)

    model.zero_grad()
    loss.backward()
    optim.step()


Instruction: nvcc and clang are not working well together when installing pytorch-gpu

Input:
I am trying to install pytorch with gpu support on my MacBook Pro following official instructions.

Things go smoothly until an error occurred:

[ 70%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/caffe2_gpu_generated_THCTensorMath.cu.o
nvcc fatal   : The version ('90000') of the host compiler ('Apple clang') is not supported
nvcc fatal   : The version ('90000') of the host compiler ('Apple clang') is not supported
CMake Error at caffe2_gpu_generated_THCBlas.cu.o.Release.cmake:219 (message):
  Error generating
  /Users/username/Dev/pytorch-gpu/pytorch/build/cCaMfafkee2 /ECrMraokre Faitle sc/acfaffef2e_2g_pgup_ug.ednierr/a_t_e/da_tTeHnC/Sslrece/pT.HcCu/..o/.cRaeflfeea2s_eg.pcum_agkeen:e2r1a9t e(dm_eTsHsCaBglea)s:.

It seems that CUDA and clang are not working well together.

I searched over internet and found these posts, but they did not solve my problem:

Revert Apple Clang Version For NVCC
https://github.com/pytorch/pytorch/issues/3047

Here's my environment:

macOS Sierra 10.12.6 (16G1618)
NVIDIA GeForce GT 750M
CUDA Driver Version: 387.178
GPU Driver Version: 378.05.05.25f11
Cuda compilation tools, release 8.0, V8.0.61
(Previous)Apple LLVM version 9.0.0 (clang-900.0.39.2)
(After downgrade)Apple LLVM version 8.1.0 (clang-802.0.42)
Xcode Version 9.2 (9C40b)

Output:
I am answering my own question.

Incorrect CUDA installation on macOS could be  a nightmare. The versions of CUDA, Xcode, clang and macOS really matter. Here are some of the official tested ones:

+------+--------------+------------+---------------------------------+--------+
| CUDA |    Xcode     | Apple LLVM | Mac OSX Version (native x86_64) | Yes/No |
+------+--------------+------------+---------------------------------+--------+
| 8.0  | 7.2          | 7.0.3      | 10.11                           | YES    |
| 8.0  | 7.2          | 7.0.3      | 10.12                           | NO     |
| 8.0  | 8.2          | 8.0.0      | 10.11                           | NO     |
| 8.0  | 8.2          | 8.0.0      | 10.12                           | YES    |
| 9.0  | 8.3.3        | 8.1.0      | 10.12                           | YES    |
| 9.1  | 9.2          | 9.0.0      | 10.13.3                         | YES    |
| 9.2  | 9.2          | 9.0.0      | 10.13.5                         | YES    |
| 10.0 | 9.4          | 9.0.0      | 10.13.6                         | YES    |
| 10.1 | 10.1 (10B61) | 10.0.0     | 10.13.6 (17G2307)               | YES    |
+------+--------------+------------+---------------------------------+--------+

For CUDA Releases before 8.0, please search for NVIDIA CUDA INSTALLATION GUIDE FOR MAC OS X plus the CUDA version number, there should be a table of version matching in that PDF file.


Instruction: How to visualise filters in a CNN with PyTorch

Input:
I'm new to deep learning and Pytorch. I want to visual my filter in my CNN model so that I can iterate layer in the CNN model that I define. But I meet error like below.
error: 'CNN' object is not iterable
The CNN object is my model.
My iteration code is like below：
for index, layer in enumerate(self.model):
# Forward pass layer by layer
    x = layer(x)

my model code like below：
class CNN(nn.Module):
    def __init__(self):
        super(CNN,self).__init__()
        self.Conv1 = nn.Sequential( # input image size (1,28,20)
            nn.Conv2d(1, 16, 5, 1, 2), # outputize (16,28,20)
            nn.ReLU(),
            nn.MaxPool2d(2),           #outputize (16,14,10)
        )
        self.Conv2 = nn.Sequential( # input ize ? (16,,14,10)
            nn.Conv2d(16, 32, 5, 1, 2),   #output size(32,14,10)
            nn.ReLU(),
            nn.MaxPool2d(2),        #output size (32,7,5)
        )
        self.fc1 = nn.Linear(32 * 7 * 5, 800)
        self.fc2 = nn.Linear(800,500)
        self.fc3 = nn.Linear(500,10)
        #self.fc4 = nn.Linear(200,10)

    def forward(self,x):
        x = self.Conv1(x)
        x = self.Conv2(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = F.dropout(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.dropout(x)
        x = F.relu(x)
        x = self.fc3(x)
        #x = F.relu(x)
        #x = self.fc4(x)
        return x

So anyone can tell me how can I solve this problem.

Output:
Essentially, you will need to access the features in your model and transpose those matrices into the right shape first, then you can visualise the filters

    import numpy as np
    import matplotlib.pyplot as plt
    from torchvision import utils

    def visTensor(tensor, ch=0, allkernels=False, nrow=8, padding=1):
        n,c,w,h = tensor.shape

        if allkernels: tensor = tensor.view(n*c, -1, w, h)
        elif c != 3: tensor = tensor[:,ch,:,:].unsqueeze(dim=1)

        rows = np.min((tensor.shape[0] // nrow + 1, 64))
        grid = utils.make_grid(tensor, nrow=nrow, normalize=True, padding=padding)
        plt.figure( figsize=(nrow,rows) )
        plt.imshow(grid.numpy().transpose((1, 2, 0)))

    if __name__ == "__main__":
        layer = 1
        filter = model.features[layer].weight.data.clone()
        visTensor(filter, ch=0, allkernels=False)

        plt.axis('off')
        plt.ioff()
        plt.show()

You should be able to get a grid visual.

There are a few more visualisation techniques, you can study them here


Instruction: How to do numerical integration with pytorch similar to numpy's trapz function?

Input: Title says it all. Is there a convenient function in pytorch that can do something like np.trapz(y, x) (integrating over the the points in x and y via trapezoidal rule)?

Output: There is no built-in tool for that, but it should not be difficult to implement it yourself, especially using the numpy code as a guideline.


Instruction: Why is the derivative of f(x) with respect to 'x' 'x' and not 1 in pytorch?

Input:
I am trying to understand pytorch's autograd in full and I stumbled with this: let f(x)=x, from basic maths we know that f'(x)=1, however when I do that exercise in pytorch I get that f'(x) = x.

z = torch.linspace(-1, 1, steps=5, requires_grad=True)
y = z
y.backward(z)
print("Z tensor is: {} \n Gradient of y with respect to z is: {}".format(z, z.grad))

I would expect to get a tensor of size 5 full of 1 but instead I get:

Z tensor is: tensor([-1.0000, -0.5000,  0.0000,  0.5000,  1.0000], requires_grad=True)
 Gradient of y with respect to z is: tensor([-1.0000, -0.5000,  0.0000,  0.5000,  1.0000])

Why is this the behavior of pytorch?

Output:
First of all, given z = torch.linspace(-1, 1, steps=5, requires_grad=True) and y = z, the function is a vector-valued function, so the derivative of y w.r.t z is not as simple as 1 but a Jacobian matrix. Actually in your case z = [z1, z2, z3, z4, z5]T , the upper case T means z is a row vector. Here is what the official doc says:

Secondly, notice the official doc says: Now in this case y is no longer a scalar. torch.autograd could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to backward as argument link. In that case x.grad is not the actual gradient value (matrix) but the vector-Jacobian product.

EDIT:
x.grad is the actual gradient if your output y is a scalar.
See the example here:

z = torch.linspace(-1, 1, steps=5, requires_grad=True)
y = torch.sum(z)
y.backward()
z.grad

This will output:

tensor([1., 1., 1., 1., 1.])

As you can see, it is the gradient. Notice the only difference is that y is a scalar value here while a vector value in your example. grad can be implicitly created only for scalar outputs

You might wonder what if the gradient is not a constant, like dependent on input z as in this case

z = torch.linspace(-1, 1, steps=5, requires_grad=True)
y = torch.sum(torch.pow(z,2))
y.backward()
z.grad

The output is:

tensor([-2., -1.,  0.,  1.,  2.])

It is the same as

z = torch.linspace(-1, 1, steps=5, requires_grad=True)
y = torch.sum(torch.pow(z,2))
y.backward(torch.tensor(1.))
z.grad

The blitz tutorial is kind of brief so it is actually quite hard to understand for beginners.


Instruction: Why isn't there inplace flag in F.sigmoid in pytorch?

Input:
Both relu, leakyrelu have inplace flag, so why not sigmoid?

Signature: F.sigmoid(input)

F.relu(input, inplace=False)

Output:
According to docs:

nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.

If you need in-place version, use sigmoid_:
import torch
torch.manual_seed(0)

a = torch.randn(5)

print(a)

a.sigmoid_()

print(a)

tensor([ 1.5410, -0.2934, -2.1788,  0.5684, -1.0845])
tensor([0.8236, 0.4272, 0.1017, 0.6384, 0.2527])

sigmoid docs


Instruction: Why is torch.nn.Sigmoid a class instead of a method?

Input:
I'm trying to understand how pytorch works a little bit better. Usually, when defining a neural network class, in the init() constructor, people write self.sigmoid = nn.Sigmoid(), so that in the forward() method they can call the sigmoid function multiple times with having to reinstantiate nn.Sigmoid() every time.

But why isn't nn.Sigmoid just a method to begin with, instead of a class?

Also, I was curious what to refer to 'nn' in torch.nn as (package? library?).

Thanks!

Output:
My understanding is that the nn.Sigmoid exists to be composable with other nn layers, like this:

net = nn.Sequential(
      nn.Linear(3, 4),
      nn.Sigmoid())

If you don't need this, you can just use torch.sigmoid function.


Instruction: Evaluating pytorch models: `with torch.no_grad` vs `model.eval()`

Input: When I want to evaluate the performance of my model on the validation set, is it preferred to use with torch.no_grad: or model.eval()?

Output:
TL;DR:
Use both. They do different things, and have different scopes.

with torch.no_grad - disables tracking of gradients in autograd.
model.eval() changes the forward() behaviour of the module it is called upon
eg, it disables dropout and has batch norm use the entire population statistics

with torch.no_grad
The torch.autograd.no_grad documentation  says:

Context-manager that disabled [sic] gradient calculation.

Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.

model.eval()
The nn.Module.eval documentation says:

Sets the module in evaluation mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

The creator of pytorch said the documentation should be updated to suggest the usage of both, and I raised the pull request.


Instruction: Indexing a 3d tensor using a 2d tensor

Input:
I have a 3d tensor, source of shape (bsz x slen1 x nhd) and a 2d tensor, index of shape (bsz x slen2). More specifically, I have:

source = 32 x 20 x 768
index  = 32 x 16

Each value in the index tensor is in between [0, 19] which is the index of the desired vector according to the 2nd dim of the source tensor.

After indexing, I am expecting an output tensor of shape, 32 x 16 x 768.

Currently I am doing this:

bsz, _, nhid = source.size()
_, slen = index.size()

source = source.reshape(-1, nhid)
source = source[index.reshape(-1), :]
source = source.reshape(bsz, slen, nhid)

So, I am converting the 3d source tensor to a 2d tensor and 2d indexing tensor to a 1d tensor and then perform the indexing. Is this correct?

Is there any better way to do it?

Update

I checked that my code is not giving the expected result. To explain what I want, I am providing the following code snippet.

source = torch.FloatTensor([
    [[ 0.2413, -0.6667,  0.2621],
     [-0.4216,  0.3722, -1.2258],
     [-0.2436, -1.5746, -0.1270],
     [ 1.6962, -1.3637,  0.8820],
     [ 0.3490, -0.0198,  0.7928]],

    [[-0.0973,  2.3106, -1.8358],
     [-1.9674,  0.5381,  0.2406],
     [ 3.0731,  0.3826, -0.7279],
     [-0.6262,  0.3478, -0.5112],
     [-0.4147, -1.8988, -0.0092]]
     ])

index = torch.LongTensor([[0, 1, 2, 3],
                          [1, 2, 3, 4]])

And I want the output tensor as:

torch.FloatTensor([
    [[ 0.2413, -0.6667,  0.2621],
     [-0.4216,  0.3722, -1.2258],
     [-0.2436, -1.5746, -0.1270],
     [ 1.6962, -1.3637,  0.8820]],

    [[-1.9674,  0.5381,  0.2406],
     [ 3.0731,  0.3826, -0.7279],
     [-0.6262,  0.3478, -0.5112],
     [-0.4147, -1.8988, -0.0092]]
     ])

Output:
Update:

source[torch.arange(source.shape[0]).unsqueeze(-1), index]

Note that torch.arange(source.shape[0]).unsqueeze(-1) gives:

tensor([[0],
        [1]])  # 2 x 1

and index is:

tensor([[0, 1, 2, 3],
        [1, 2, 3, 4]])  # 2 x 4

The arange indexes the batch dimension while index simultaneously indexes the slen1 dimension. The unsqueeze call adds the extra x 1 dimension to the arange result so that the two can be broadcast together.


Instruction: Shape of tensor

Input:
I came across this piece of code

(x_train, y_train), (x_test, y_test) = mnist.load_data()
print("Shape of x_train: " + str(x_train.shape))
print("Shape of y_train: " + str(y_train.shape))

And found that the output looks like this

(60000, 28, 28)
(60000,)

For the first line of output

So far my understanding, does it means that in 1st dimension it can hold 60k items, then in next dimension, it can hold 28 "array of 60k items"
and finally, in the last dimension, it can hold 28 "array of 28 "array of 60k items""

What I want to clarify is, Is this 60k samples of 28x28 data or something else?

For the second line of output, it seems like its just a 1d array of 60k items. So what does it actually represents? (i know that in x_train it was handwritten numbers and each number represents the intensity of grey in that cell)

Please note I have taken this code from some online example(i don't remember and won't mind if you want your credit to be added to this) and public dataset
tf.keras.datasets.mnist

Output:
You are right the first line gives 60K items of 28x28 size data thus (60000, 28, 28).

The y_train are labels of the x_train. Thus they are a one dimensional and 60k in number.

For example: If the first item of the x_train is a handwritten image of 3, then the first item of y_train will be '3' which is the label.