Instruction: Periodic pattern in loss when training neural networks using pytorch

Input:
I am using Pytorch train a model on MNIST, the loss curve has the periodic pattern shown in the figure. I've double checked the data loader and the dataset is shuffled in each epoch. Any suggestions for the possible reason? Thanks.

Loss curve during training, training loss in blue and test loss in red

Output: What I understand from the figure is that your loss is oscillating. So try decreasing your learning rate and also some momentum term if available. I cannot guarantee you that this will work but hoping that it works it worth giving a try. Most of the things in Deep Learning can be explained, its just trial and error. Next time please ask such questions in https://ai.stackexchange.com or https://datascience.stackexchange.com.


Instruction: pytorch scoring set of images and evaluating results

Input:
In Pytorch by using GPU (cuda) need to score a set of images given trained NN.
The following code is meant to score a set of transformed images one by one.

model.to('cuda')
model.eval()

for ii, (inputs, classes) in enumerate(dataloaders['test']):
     inputs, classes = inputs, classes
     results = model.forward(inputs)
     ps = torch.exp(results)

Error Stack:

RuntimeError                              Traceback (most recent call last)
<ipython-input-24-948390e2b25a> in <module>()
      5 for ii, (inputs, classes) in enumerate(dataloaders['test']):
      6      inputs, classes = inputs, classes
----> 7      results = model(inputs)
      8      ps = torch.exp(results)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/models/vgg.py in forward(self, x)
     40
     41     def forward(self, x):
---> 42         x = self.features(x)
     43         x = x.view(x.size(0), -1)
     44         x = self.classifier(x)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     89     def forward(self, input):
     90         for module in self._modules.values():
---> 91             input = module(input)
     92         return input
     93

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    299     def forward(self, input):
    300         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 301                         self.padding, self.dilation, self.groups)
    302
    303

RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'weight'

Model made on a GPU (cuda).

Output:
This solves the problem and provides an Accuracy rate.

model.to('cuda')
model.eval()
accuracy = 0

for ii, (inputs, classes) in enumerate(dataloaders['test']):
        inputs, classes = inputs.to('cuda'), classes.to('cuda')
        # Forward and backward passes
        with torch.no_grad():
            output = model.forward(inputs)
        ps = torch.exp(output)
        equality = (classes.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

accuracy = accuracy/len(dataloaders['test'])
print(accuracy)


Instruction: PyTorch NotImplementedError in forward

Input:
import torch
import torch.nn as nn

device = torch.device('cuda' if torch.cuda.is_available() else
'cpu')

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2), # 16x16x650
            nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1), # 32x16x650
            nn.ReLU(),
            nn.Dropout2d(0.5),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1), # 64x16x650
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2), # 64x8x325
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU()) # 64x8x325

        self.fc = nn.Sequential(
            nn.Linear(64*8*325, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, 1),
        )

        def forward(self, x):
            out = self.layer1(x)
            out = self.layer2(out)
            out = out.reshape(out.size(0), -1)
            out = self.fc(out)
            return out

# HYPERPARAMETER
learning_rate = 0.0001
num_epochs = 15

import data

def main():
    model = Model().to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(),
lr=learning_rate)

    total_step = len(data.train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(data.train_loader):
            images = images.to(device)
            labels = labels.to(device)

            outputs = model(images)
            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))

    model.eval()
    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in data.test_loader:
            images = images.to(device)
            labels = labels.to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

        print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()

Error:

File "/home/rladhkstn8/Desktop/SWID/tmp/pycharm_project_853/model.py", line 82, in <module>
    main()
  File "/home/rladhkstn8/Desktop/SWID/tmp/pycharm_project_853/model.py", line 56, in main
    outputs = model(images)
  File "/home/rladhkstn8/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rladhkstn8/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 83, in forward
    raise NotImplementedError
NotImplementedError

I do not know where the problem is. I know that NotImplementedError should be implemented, but it happens when there is unimplemented code.

Output: please look carefully at the indentation of your __init__ function: your forward is part of __init__ not part of your module.


Instruction: Pytorch: join tensor and dictionary

Input:
Using Pytorch.
Got the following tensor:

(tensor([[-0.0030, -7.6063, -7.6334, -7.7098, -8.3540]], device='cuda:0'),
 tensor([[ 14,  85,  45,  82,  15]], device='cuda:0'))

Need to join it with the following dictionary:

{'14': 'a',
 '100': 'b',
 '45': 'c',
 '33': 'd',
 '15': 'e'}

In order to get the following results:
'a','c','e'

Output:
dictionary = {int(key):dictionary[key] for key in dictionary}
np.vectorize(cat_to_name.get)(best_5)


Instruction: pip - Unable to install Fastai

Input:
Whenever I run:

pip install fastai

I get the error

"Command "python setup.py egg_info" failed with error code 1 in C:\Users\seja9890\AppData\Local\Temp\pip-install-_cw7ve61\torch\".

Can someone please guide me where I might be going wrong?

Ps.: I have tried updating setuptools and it doesn't help in my case.

Output:
Fastai doesn't work with Python 2 so make sure you installed pip3 (sudo apt install python3-pip on Ubuntu).

Make sure Python3 is at least 3.6 this may change since Fastai may need 3.7. soon.

and then:
pip3 install git+https://github.com/fastai/fastai.git

or use pip3 install fastai, or in some cases you may need:
pip3 install --no-deps fastai

Note: At the moment I am writing this: PyTorch v1 and Python 3.6 are the minimal version requirements.


Instruction: Can neuroevolution of augmenting topologies (NEAT) neural networks be built in TensorFlow?

Input: I am making a machine learning program for time series data analysis and using NEAT could help the work. I started to learn TensorFlow not long ago but it seems that the computational graphs in TensorFlow are usually fixed. Is there tools in TensorFlow to help build a dynamically evolving neural network? Or something like Pytorch would be a better alternative? Thanks.

Output:
It can't be implemented in the static graph mode of TensorFlow without significant tradeoffs because the topology of the neural networks in the population changes. Static graphs are suited for models whose architecture doesn't change during training. However, it can be done in TensorFlow Eager or PyTorch because they support dynamic computation graphs.

Check this implementation in TensorFlow Eager: https://github.com/crisbodnar/TensorFlow-NEAT


Instruction: CUDA HOME in pytorch installation

Input:
I installed pytorch via conda with cuda 7.5

conda install pytorch=0.3.0 cuda75 -c pytorch

>>> import torch
>>> torch.cuda.is_available()
True

I didn't do any other installations for cuda other than this, since it looks like pytorch comes with cuda

Now, I am trying to setup yolo2 https://github.com/longcw/yolo2-pytorch

However, I am getting error in ./make.sh command

this is the error

  OSError: The nvcc binary could not be located in your $PATH. Either add it to your path, or set $CUDAHOME

I'm assuming I need to set CUDAHOME in my path, but I am not able to locate any cuda directory having nvcc binary. Any pointers on it?

Output:
The CUDA package which is distributed via anaconda is not a complete CUDA toolkit installation. It only includes the necessary libraries and tools to support numba and pyculib and other GPU accelerated binary packages they distribute, like tensorflow and pytorch.

If you need a fully functional CUDA toolkit (and it seems you do), you will need to install one yourself. Word to the wise -- install the same version that you have installed within anaconda. With a tiny bit of PATH modification, everything should just work.


Instruction: pytorch gradient / derivative / difference along axis like numpy.diff

Input:
I have been struggling with this for quite some time. All I want is a torch.diff() function. However, many matrix operations do not appear to be easily compatible with tensor operations.

I have tried an enormous amount of various pytorch operation combinations, yet none of them work.

Due to the fact that pytorch hasn't implemented this basic feature, I started by simply trying to subtract the element i+1 from element i along a specific axis.

However, you can't simply do this element-wise (due to the tensor limitations), so I tried to construct another tensor, with the elements shifted along one axis:

ix_plus_one = [0]+list(range(0,prediction.size(1)-1))
ix_differential_tensor = torch.LongTensor(ix_plus_one)
diff_one_tensor = prediction[:,ix_differential_tensor]

But now we have a different problem - indexing doesn't really work to mimic numpy in pytorch as it advertises, so you can't index with a "list-like" Tensor like this.  I also tried using the tensor scatter functions

So I'm still stuck with this simple problem of trying to get a gradient on a pytoch tensor.

All of my searching leads to the marvelous capabilities of pytorchs' "autograd" function - which has nothing to do with this problem.

Output:
A 1D convolution with a fixed filter should do the trick:

filter = torch.nn.Conv1d(in_channels=1, out_channels=1, kernel_size=2, stride=1, padding=1, groups=1, bias=False)
kernel = np.array([-1.0, 1.0])
kernel = torch.from_numpy(kernel).view(1,1,2)
filter.weight.data = kernel
filter.weight.requires_grad = False

Then use filter like you would any other layer in torch.nn.

Also, you might want to change padding to suit your specific needs.


Instruction: Python Jupyter notebook how to display matrix in vertical

Input:
When I try to index array, I use this code for printing column part with Numpy or Pytorch.

import numpy as np
a = np.random.randn(5,3)
a[:,1]

or

import torch
a = torch.Tensor(5,3)
a[:,1]

The output is displayed like this.

array([-0.07478094, -1.87787326,  0.50407517,  1.13335836,  0.23140931])

But I want to display output as column.(because i indexed column)

array([-0.07478094,
       -1.87787326,
        0.50407517,
        1.13335836,
        0.23140931])

Furthermore, When I make tensor with torch.ones(5), the result is

tensor([1., 1., 1., 1., 1.])

but I want to see the type of output on the buttom like this

tensor([1., 1., 1., 1., 1.])
[torch.FloatTensor of size 5]

The reason why i want to display this is that i can't distinguish tensor and numpy

Can anyone tell me how to do this? Thanks.

Output:
try this:

np.vstack(a)

Hope this helps..


Instruction: Understanding PyTorch Tensor Shape

Input:
I have a simple question regarding the shape of tensor we define in PyTorch. Let's say if I say:

input = torch.randn(32, 35)

This will create a matrix with 32 row and 35 columns.
Now when I define:

input2 = torch.randn(1,2,32, 35)

What can I say about the dimension of the new matrix input2?
How can I define the rows and columns here? I mean do I have two matrices with shapes 32*35 packed by the tensor?

I want to better understand the geometry behind this. Thanks.

Output:
Consider tensor shapes as the number of lists that a dimension holds. For instance, a tensor shaped (4, 4, 2) will have four elements, which will all contain 4 elements, which in turn have 2 elements.

The first holds 4 elements.
The second holds 4 elements.
The third dimension holds 2 elements.

Here's what the data would look like:
[[[0.86471446, 0.26302726],
  [0.04137454, 0.00349315],
  [0.06559607, 0.45617865],
  [0.0219786, 0.27513594]],

 [[0.60555118, 0.10853228],
  [0.07059685, 0.32746256],
  [0.99684617, 0.07496456],
  [0.55169005, 0.39024103]],

 [[0.55891377, 0.41151245],
  [0.3434965, 0.12956237],
  [0.74908291, 0.69889266],
  [0.98600141, 0.8570597]],

 [[0.7903229, 0.93017741],
  [0.54663242, 0.72318166],
  [0.6099451, 0.96090241],
  [0.63772238, 0.78605599]]]

In other words, four elements of four elements of two elements.


Instruction: Why will GPU usage run low in NN training?

Input:
I'm running a NN training on my GPU with pytorch.
But the GPU usage is strangely "limited" at about 50-60%.
That's a waste of computing resources but I can't make it a bit higher.

I'm sure that the hardware is fine because running 2 of my process at the same time,or training a simple NN (DCGAN,for instance) can both occupy 95% or more GPU.(which is how it supposed to be)
My NN contains several convolution layers and it should use more GPU resources.

Besides, I guess that the data from dataset has been feeding fast enough,because I used workers=64 in my dataloader instance and my disk works just fine.

I just confused about what is happening.

Dev details:

GPU : Nvidia GTX 1080 Ti

os:Ubuntu 64-bit

Output:
I can only guess without further research but it could be that your network is small in terms of layer-size (not number of layers) so each step of the training is not enough to occupy all the GPU resources. Or at least the ratio between the data size and the transfer speed (to the gpu memory) is bad and the GPU stays idle most of the time.

tl;dr: the gpu jobs are not long enough to justify the memory transfers


Instruction: 'no module named cv2' import error on pytorch

Input:
I am currently studying Pytorch and trying to use the cv2 module. I am using Jupyter notebook and Windows.

I have installed opencv like this:

!pip install opencv-python

When I choose the kernel (by change kernel option) Python3 and import cv2 then there is no problem.

But when I choose the kernel Pytorch and import cv2, then there is an error:

ModuleNotFoundError: No module named 'cv2'

This must be a basic question but I can't find out what the problem is.

Output:
I found the answer to this problem here:

  First do run these commands

  $ conda update anaconda-navigator

  $ conda update navigator-updater

  then the issue for the instruction below will be resolved

For windows if you have anaconda installed, you can simply do:

pip install opencv-python

or

conda install -c https://conda.binstar.org/menpo opencv

If you're on linux you can do :

pip install opencv-python

or

conda install opencv


Instruction: Derivative of ReLU

Input:
I'm learning PyTorch. Here is the first example in official tutorial. I got two questions, as shown in the block below,

a) I understand that derivative of a ReLU function is 0 when x < 0 and 1 when x > 0. Is that right? But the code seems to keep the x > 0 part unchanged and set x < 0 part to 0. Why is that?

b) Why transpose, i.e. x.T.mm(grad_h)? A transpose does't seem needed to me. I'm just confused. Thanks,

# -*- coding: utf-8 -*-

import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())

    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

Output:
1- It is true that derivative of a ReLU function is 0 when x < 0 and 1 when x > 0. But notice that gradient is flowing from output of the function to all the way back to h. When you get all the way back to calculate grad_h, it is calculated as:

grad_h = derivative of ReLu(x) * incoming gradient

As you said exactly, derivative of ReLu function is 1 so grad_h is just equal to incoming gradient.

2- Size of the x matrix is 64x1000 and grad_h matrix is 64x100. It is obvious that you can not directly multiply x with grad_h and you need to take transpose of x to get appropriate dimensions.


Instruction: What is the difference between parameters and children?

Input:
It looks like parameters and children show the same info, so what is the difference between them?

import torch

print('torch.__version__', torch.__version__)

m = torch.load('imagenet_resnet18.pth')

print(m.parameters)
print(m.children)

Output:
model.parameters() is a generator that returns tensors containing your model parameters.
model.children() is a generator that returns layers of the model from which you can extract your parameter tensors using <layername>.weight or <layername>.bias

Visit this link for a simple tutorial on accessing and freezing model layers.


Instruction: How to get weights shape for each layer?

Input:
There is a good question how to get model summary in pytorch
Model summary in pytorch but it doesn't output shape of weights.

Is it possible also to output shape of weights for each layer?

Output:
Looks like it's possible, here is an example:

import torch
from torchvision import models

m = models.resnet18()
print(m)
print('-'*60)
for l in list(m.named_parameters()):
    print(l[0], ':', l[1].detach().numpy().shape)

Which outputs:

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=512, out_features=1000, bias=True)
)
------------------------------------------------------------
conv1.weight : (64, 3, 7, 7)
bn1.weight : (64,)
bn1.bias : (64,)
layer1.0.conv1.weight : (64, 64, 3, 3)
layer1.0.bn1.weight : (64,)
layer1.0.bn1.bias : (64,)
layer1.0.conv2.weight : (64, 64, 3, 3)
layer1.0.bn2.weight : (64,)
layer1.0.bn2.bias : (64,)
layer1.1.conv1.weight : (64, 64, 3, 3)
layer1.1.bn1.weight : (64,)
layer1.1.bn1.bias : (64,)
layer1.1.conv2.weight : (64, 64, 3, 3)
layer1.1.bn2.weight : (64,)
layer1.1.bn2.bias : (64,)
layer2.0.conv1.weight : (128, 64, 3, 3)
layer2.0.bn1.weight : (128,)
layer2.0.bn1.bias : (128,)
layer2.0.conv2.weight : (128, 128, 3, 3)
layer2.0.bn2.weight : (128,)
layer2.0.bn2.bias : (128,)
layer2.0.downsample.0.weight : (128, 64, 1, 1)
layer2.0.downsample.1.weight : (128,)
layer2.0.downsample.1.bias : (128,)
layer2.1.conv1.weight : (128, 128, 3, 3)
layer2.1.bn1.weight : (128,)
layer2.1.bn1.bias : (128,)
layer2.1.conv2.weight : (128, 128, 3, 3)
layer2.1.bn2.weight : (128,)
layer2.1.bn2.bias : (128,)
layer3.0.conv1.weight : (256, 128, 3, 3)
layer3.0.bn1.weight : (256,)
layer3.0.bn1.bias : (256,)
layer3.0.conv2.weight : (256, 256, 3, 3)
layer3.0.bn2.weight : (256,)
layer3.0.bn2.bias : (256,)
layer3.0.downsample.0.weight : (256, 128, 1, 1)
layer3.0.downsample.1.weight : (256,)
layer3.0.downsample.1.bias : (256,)
layer3.1.conv1.weight : (256, 256, 3, 3)
layer3.1.bn1.weight : (256,)
layer3.1.bn1.bias : (256,)
layer3.1.conv2.weight : (256, 256, 3, 3)
layer3.1.bn2.weight : (256,)
layer3.1.bn2.bias : (256,)
layer4.0.conv1.weight : (512, 256, 3, 3)
layer4.0.bn1.weight : (512,)
layer4.0.bn1.bias : (512,)
layer4.0.conv2.weight : (512, 512, 3, 3)
layer4.0.bn2.weight : (512,)
layer4.0.bn2.bias : (512,)
layer4.0.downsample.0.weight : (512, 256, 1, 1)
layer4.0.downsample.1.weight : (512,)
layer4.0.downsample.1.bias : (512,)
layer4.1.conv1.weight : (512, 512, 3, 3)
layer4.1.bn1.weight : (512,)
layer4.1.bn1.bias : (512,)
layer4.1.conv2.weight : (512, 512, 3, 3)
layer4.1.bn2.weight : (512,)
layer4.1.bn2.bias : (512,)
fc.weight : (1000, 512)
fc.bias : (1000,)


Instruction: Is computing `loss.backward` for multiple losses performant in pytorch?

Input:
I would like to calculate the gradient of my model for several loss functions.

I would like to find out if calculating successive backwards calls with retain_graph=True is cheap or expensive.

In theory I would expect that the first call should be slower than those following the first, because the computational graph does not have to be reevaluated, but just a few matrix multiplications need to be made.

In practice I found it hard to benchmark.

My code:

# Code in file nn/two_layer_net_nn.py
import torch

D_in = 40
model = torch.load('model.pytorch')
device = torch.device('cpu')
def loss1(y_pred,x):
    return (y_pred*(0.5-x.clamp(0,1))).sum()

def loss2(y_pred,x):
    return (y_pred*(1-x.clamp(0,1))).sum()

# Predict random input
x = torch.rand(1,D_in, device=device,requires_grad=True)
y_pred = model(x)

# Is this
%%timeit
loss = loss1(y_pred,x)
loss.backward(retain_graph=True)

202 µs ± 4.34 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Slower than this?
%%timeit
loss = loss2(y_pred,x)
loss.backward(retain_graph=True)

216 µs ± 27.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# Are successive backwards calls cheap?
loss = lossX(y_pred,x)
loss.backward(retain_graph=True)

I think that %%timeit doesn't work because it will run several iterations and then average over it.

How can I measure whether successive calls to backward will be fast?
What does retain_graph=True actually mean for performance?

Output:
I think that you only asked if

the first backward pass is slower than subsequent ones?

There are two more questions that might as well be combined here:

is it more efficient to combine losses
does keeping the graph (if not its values) require more memory?

Before that, however, let's emphasize on what retain_graph is actually for: multiple passes through your graph if you computationally happen to have multiple outputs at multiple times. As an example, think joint multi-task-learning (see this question and its answers for a discussion on this).

Back to the questions: In general, I would expect that it does not really matter if you are retaining the graph. After all, it is just keeping partial computations in memory for future use, without "doing" anything with them.

That said - the first backwards pass will take longer, as pytorch will cache some of the computations needed when computing the gradients.

So here's proof:

import numpy as np
import torch
import torch.nn as nn
import time
import os
import psutil
D_in = 1024
model = nn.Sequential(nn.Linear(1024, 4096), nn.ReLU(), nn.Linear(4096, 4096), nn.ReLU(), nn.Linear(4096, 1024))
device = torch.device('cpu')
def loss1(y_pred,x):
    return (y_pred*(0.5-x.clamp(0,1))).sum()

def loss2(y_pred,x):
    return (y_pred*(1-x.clamp(0,1))).sum()

def timeit(func, repetitions):
    time_taken = []
    mem_used = []
    for _ in range(repetitions):
        time_start = time.time()
        mem_used.append(func())
        time_taken.append(time.time() - time_start)
    return np.round([np.mean(time_taken), np.min(time_taken), np.max(time_taken), \
           np.mean(mem_used), np.min(mem_used), np.max(mem_used)], 4).tolist()

# Predict random input
x = torch.rand(1,D_in, device=device,requires_grad=True)

def init():
    out = model(x)
    loss = loss1(out, x)
    loss.backward()

def func1():
    x = torch.rand(1, D_in, device=device, requires_grad=True)
    loss = loss1(model(x),x)
    loss.backward()
    loss = loss2(model(x),x)
    loss.backward()
    del x
    process = psutil.Process(os.getpid())
    return process.memory_info().rss

def func2():
    x = torch.rand(1, D_in, device=device, requires_grad=True)
    loss = loss1(model(x),x) + loss2(model(x),x)
    loss.backward()
    del x
    process = psutil.Process(os.getpid())
    return process.memory_info().rss

def func3():
    x = torch.rand(1, D_in, device=device, requires_grad=True)
    loss = loss1(model(x),x)
    loss.backward(retain_graph=True)
    loss = loss2(model(x),x)
    loss.backward(retain_graph=True)
    del x
    process = psutil.Process(os.getpid())
    return process.memory_info().rss

def func4():
    x = torch.rand(1, D_in, device=device, requires_grad=True)
    loss = loss1(model(x),x) + loss2(model(x),x)
    loss.backward(retain_graph=True)
    del x
    process = psutil.Process(os.getpid())
    return process.memory_info().rss

init()
print(timeit(func1, 100))
print(timeit(func2, 100))
print(timeit(func3, 100))
print(timeit(func4, 100))

The results are (sorry for my lazy formatting):

# time mean, time min, time max, memory mean, memory min, memory max
[0.1165, 0.1138, 0.1297, 383456419.84, 365731840.0, 384438272.0]
[0.127, 0.1233, 0.1376, 400914759.68, 399638528.0, 434044928.0]
[0.1167, 0.1136, 0.1272, 400424468.48, 399577088.0, 401223680.0]
[0.1263, 0.1226, 0.134, 400815964.16, 399556608.0, 434307072.0]

However, if you skip the first backwards pass (comment out the call to the init() function), the very first backwards run in func1 will take longer:

# time mean, time min, time max, memory mean, memory min, memory max
[0.1208, 0.1136, **0.1579**, 350157455.36, 349331456.0, 350978048.0]
[0.1297, 0.1232, 0.1499, 393928540.16, 350052352.0, 401854464.0]
[0.1197, 0.1152, 0.1547, 350787338.24, 349982720.0, 351629312.0]
[0.1335, 0.1229, 0.1793, 382819123.2, 349929472.0, 401776640.0]


Instruction: Hello World Convolution on Images in PyTorch

Input:
I am trying to verify some results with PyTorch's 2D convolution with the following:

Input matrix: X (10, 10, 3) [Dummy numpy image]
Weight matrix: W (3, 3, 3) [My Conv Filter to test]
Output matrix: Y (10, 10, 1)

I have the following code but I am not able to assign the weights properly and run the model without errors.
What am I doing wrong here??

import torch
import torch.nn as nn
import torchvision.transforms
import numpy as np

# Convert image to tensor
image2tensor = torchvision.transforms.ToTensor()

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        # Test layer
        self.layer1 = nn.Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

    def forward(self, x):
        out = self.layer1(x)
        return out

# Test image
image = np.ones((10, 10, 3))
tensor = image2tensor(image).unsqueeze(0)

# Create new model
conv = ConvNet()

# Assign test weight - NOT WORKING!!
weight = torch.nn.Parameter(torch.ones(3, 3, 3))
conv.layer1.weight.data = weight

# Run the model
output = conv(tensor)

Output:
Next time please post your corresponding error message.
Turns out the matrix dimension also has to match the batch size (i.e. needs an additional fourth dimension): Thus, you are initializing the weight matrix with the wrong parameters.

Correct would be this instead:

weight = torch.nn.Parameter(torch.ones(1, 3, 3, 3))

Additionally, for my version of PyTorch (0.4.1), I had to manually cast the tensor again to a float because it otherwise threw a different error. Avoid by doing it like so:

tensor = image2tensor(image).unsqueeze(0).float() # note the additional .float()

Then it runs successfully for me.


Instruction: Pytorch 0.4.1 invalid gradient at index 0 - expected shape[] but got [1]

Input:
I've been around this problem for the whole day.

torch.autograd.backward(loss_seq, grad_seq) will get an error.

Output:

Traceback (most recent call last):
  File "train_vgg.py", line 272, in <module>
    torch.autograd.backward(loss_seq, grad_seq)
  File "/root/anaconda3/lib/python3.6/site-
packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: invalid gradient at index 0 - expected shape [] but got [1]

Input:

loss_seq:[tensor(7.3761, device='cuda:1', grad_fn=<ThAddBackward>), tensor(4.3005, device='cuda:1', grad_fn=<ThAddBackward>), tensor(4.2209, device='cuda:1', grad_fn=<ThAddBackward>)]
grad_seq:[tensor([1.], device='cuda:1'), tensor([1.], device='cuda:1'), tensor([1.], device='cuda:1')]

```

Can someone tell how to fix it?

input code:

images = Variable(images).cuda(gpu)

label_yaw = Variable(labels[:,0]).cuda(gpu)
label_pitch = Variable(labels[:,1]).cuda(gpu)
label_roll = Variable(labels[:,2]).cuda(gpu)

pre_yaw, pre_pitch, pre_roll = model(images)

# Cross entropy loss
loss_yaw = criterion(pre_yaw, label_yaw)
loss_pitch = criterion(pre_pitch, label_pitch)
loss_roll = criterion(pre_roll, label_roll)

loss_yaw += 0.005 * loss_reg_yaw
loss_pitch += 0.005 * loss_reg_pitch
loss_roll += 0.005 * loss_reg_roll

loss_seq = [loss_yaw, loss_pitch, loss_roll]
grad_seq = [torch.ones(1).cuda(gpu) for _ in range(len(loss_seq))]

# crash here
torch.autograd.backward(loss_seq, grad_seq)

Output:
I have solved this problem. Only change:

grad_seq = [torch.ones(1).cuda(gpu) for _ in range(len(loss_seq))]

to:

grad_seq = [torch.tensor(1.0).cuda(gpu) for _ in range(len(loss_seq))]


Instruction: PyTorch and CUDA driver

Input:
I have CUDA 9.2 installed. For example:

(base) c:\>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Wed_Apr_11_23:16:30_Central_Daylight_Time_2018
Cuda compilation tools, release 9.2, V9.2.88

I installed PyTorch on Windows 10 using:

conda install pytorch cuda92 -c pytorch
pip3 install torchvision

I ran the test script:

(base) c:\>python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> import torch
>>> x = torch.rand(5, 3)
>>> print(x)
tensor([[0.7041, 0.5685, 0.4036],
        [0.3089, 0.5286, 0.3245],
        [0.3504, 0.8638, 0.1118],
        [0.6517, 0.9209, 0.6801],
        [0.0315, 0.1923, 0.8720]])
>>> quit()

So for, so good.  Then I ran:

(base) c:\>python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>>

Why did PyTorch say CUDA was not available?

The GPU is a compute capability 3.0 Quadro K3000M:

(base) C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe

Mon Oct 01 16:36:47 2018
NVIDIA-SMI 385.54 Driver Version: 385.54
-------------------------------+----------------------+----------------------
GPU Name TCC/WDDM              | Bus-Id Disp.A        | Volatile Uncorr.
ECC Fan Temp Perf Pwr:Usage/Cap| Memory-Usage         | GPU-Util Compute M.
0 Quadro K3000M WDDM           | 00000000:01:00.0 Off |
N/A N/A 35C  P0 N/A / N/A      | 29MiB / 2048MiB      | 0% Default

Output:
Ever since https://github.com/pytorch/pytorch/releases/tag/v0.3.1, PyTorch binary releases had removed support for old GPUs' with CUDA capability 3.0.  According to https://en.wikipedia.org/wiki/CUDA, the compute capability of Quadro K3000M is 3.0.

Therefore, you might have to build pytorch from source or try other packages.  Please refer to this thread for more information --  https://discuss.pytorch.org/t/pytorch-no-longer-supports-this-gpu-because-it-is-too-old/13803.


Instruction: Dataframe as datasource in torchtext

Input:
I have a dataframe, which has two columns (review and sentiment). I am using pytorch and torchtext library for preprocessing data.
Is it possible to use dataframe as source to read data from, in torchtext?
I am looking for something similar to, but not

data.TabularDataset.splits(path='./data')

I have performed some operation (clean, change to required format) on data and final data is in a dataframe.

If not torchtext, what other package would you suggest that would help in preprocessing text data present in a datarame. I could not find anything online. Any help would be great.

Output:
Adapting the Dataset and Example classes from torchtext.data
    from torchtext.data import Field, Dataset, Example
    import pandas as pd

     class DataFrameDataset(Dataset):
         """Class for using pandas DataFrames as a datasource"""
         def __init__(self, examples, fields, filter_pred=None):
             """
             Create a dataset from a pandas dataframe of examples and Fields
             Arguments:
                 examples pd.DataFrame: DataFrame of examples
                 fields {str: Field}: The Fields to use in this tuple. The
                     string is a field name, and the Field is the associated field.
                 filter_pred (callable or None): use only exanples for which
                     filter_pred(example) is true, or use all examples if None.
                     Default is None
             """
             self.examples = examples.apply(SeriesExample.fromSeries, args=(fields,), axis=1).tolist()
             if filter_pred is not None:
                 self.examples = filter(filter_pred, self.examples)
             self.fields = dict(fields)
             # Unpack field tuples
             for n, f in list(self.fields.items()):
                 if isinstance(n, tuple):
                     self.fields.update(zip(n, f))
                     del self.fields[n]

     class SeriesExample(Example):
         """Class to convert a pandas Series to an Example"""

         @classmethod
         def fromSeries(cls, data, fields):
             return cls.fromdict(data.to_dict(), fields)

         @classmethod
         def fromdict(cls, data, fields):
             ex = cls()

             for key, field in fields.items():
                 if key not in data:
                     raise ValueError("Specified key {} was not found in "
                     "the input data".format(key))
                 if field is not None:
                     setattr(ex, key, field.preprocess(data[key]))
                 else:
                     setattr(ex, key, data[key])
             return ex

Then, first define fields using torchtext.data fields. For example:
    TEXT = data.Field(tokenize='spacy')
    LABEL = data.LabelField(dtype=torch.float)
    TEXT.build_vocab(train, max_size=25000, vectors="glove.6B.100d")
    LABEL.build_vocab(train)
    fields = { 'sentiment' : LABEL, 'review' : TEXT }

before simply loading the dataframes:
    train_ds = DataFrameDataset(train_df, fields)
    valid_ds = DataFrameDataset(valid_df, fields)


Instruction: A better way to make pytorch code agnostic to running on a CPU or GPU?

Input:
The Migration guide recommends the following to make code CPU/GPU agnostic:

> # at beginning of the script
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
...
# then whenever you get a new Tensor or Module
# this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)

I did this and ran my code on a CPU-only device, but my model crashed when fed an input array, as saying it was expecting a CPU tensor not a GPU one.  Somehow my model was automatically converting the a CPU-input array to a GPU array.  Finally I traced it down to this command in my code:

model = torch.nn.DataParallel(model).to(device)

Even though I convert the model to 'cpu', the nn.DataParallel overrides this.  The best solution I came up with was a conditional:

if device.type=='cpu':
    model = model.to(device)
else:
    model = torch.nn.DataParallel(model).to(device)

This does not seem elegant.  Is there a better way?

Output:
How about

if torch.cuda.device_count() > 1:
    model = torch.nn.DataParallel(model)
model = model.to(device)

?

You don't need DataParallel if you have only one GPU.


Instruction: How to implement a location specific convolutional filters in Tensorflow or Pytorch?

Input:
I want to implement a convolutional layer, with a different convolutional filter for each output location. Specifically, think of the case when the output is of 16*16*128 (W * H * C). Instead of having a 3*3*128 filter we have 16*16 filters; each with size 3*3*128. This would lead to huge amount of parameters, but it can the case be that each of the 3*3*128 filter may be the same except scaled by a different constant, and the constants can be learned through a side network. In this way the number of parameters won't be too much.

The similar idea is briefly in Dynamic Filter Networks, but I cannot find an implementation of location specific filters. My question is, if we want a location specific convolutional filter how do I implement it in Tensorflow or Pytorch? Do I need to write my own operation or there is some smart way to use the functions provided? If I have to write an OP is there any trick that can easily achieve this idea? Any help is appreciated!

Output: A convolution, by definition, is not location specific - this is what makes it a convolution. If you wish to generalize convolution, bear in mind that eventually a convolution is a special case of a simple linear operation. Therefore, you can implement your "location specific" convolution as a fully-connected layer (nn.Linear) with a very specific sparse weights.


Instruction: How to derive weight and bias in a neural network?

Input:
This is the Network:

    class Net(torch.nn.Module):
        def __init__(self, n_feature, n_hidden, n_output):
            super(Net, self).__init__()
            self.hidden = torch.nn.Linear(n_feature, n_hidden)
            self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer
        def forward(self, x):
            x = F.relu(self.hidden(x))      # activation function for hidden layer
            x = self.predict(x)             # linear output
            return x

    net = Net(n_feature=1, n_hidden=10, n_output=1)
    pytorch_total_params = sum(p.numel() for p in net.parameters())
    print(pytorch_total_params)

    w = list(net.parameters())
    print(w)

This is the runnung result:
31
[Parameter containing:
tensor([[ 0.9534],
        [-0.0309],
        [-0.9570],
        [-0.4179],
        [-0.3757],
        [-0.4227],
        [-0.8866],
        [ 0.2107],
        [ 0.0222],
        [ 0.2531]], requires_grad=True), Parameter containing:
tensor([-0.0358, -0.2533,  0.2979,  0.9777,  0.9606,  0.9460,  0.9059,  0.7582,
        -0.5286,  0.3367], requires_grad=True), Parameter containing:
tensor([[-0.2863, -0.3157,  0.2086, -0.0011, -0.0415, -0.2574, -0.0683, -0.0788,
         -0.0339, -0.0195]], requires_grad=True), Parameter containing:
tensor([0.2031], requires_grad=True)]

I don't know why is the number of parameters be 31?
And also don't understand the numbers printed above.(whether it is weight or bias

Because I thought in Relu function, there will only be (2 parameters*10) which is weight and bias multiply 10 hidden layers.

Output:
If you print the named parameters you can see to which layer a parameter belongs.
Printing the named parameters:

for p in net.named_parameters():
    print(p)

Creates the following output:

('hidden.weight', Parameter containing:
tensor([[ 0.8324],
        [ 0.2166],
        [-0.9786],
        [ 0.3977],
        [ 0.9008],
        [-0.3102],
        [ 0.5052],
        [ 0.6589],
        [ 0.0828],
        [ 0.6505]], requires_grad=True))
('hidden.bias', Parameter containing:
tensor([ 0.6715,  0.5503, -0.6043,  0.1102, -0.2700,  0.7203, -0.6524, -0.6332,
        -0.2513, -0.1316], requires_grad=True))
('predict.weight', Parameter containing:
tensor([[ 0.1486,  0.1528, -0.0835, -0.3050,  0.1184, -0.0422, -0.2786, -0.2549,
         -0.1532, -0.0255]], requires_grad=True))
('predict.bias', Parameter containing:
tensor([0.2878], requires_grad=True))

As you can see the layers are connected by 10 weights each, as you expected, but there is one bias per neuron on the right side of a 'connection'.
So you have 10 bias-parameters between your input and your hidden layer and just one for the calculation of your final prediction.

You are calculating the input to each neuron in the l-th layer like this weighted sum:

So you need a weight for every connection between the neurons of the two layers, but only one bias per neuron in the l-th layer.

In your case:

input to hidden: 10 weights and 10 bias, because your hidden layer has 10 neurons
hidden to output/predict: 10 weights and 1 bias, because you output a single value

sums up to 31 parameters.


Instruction: Debugging in Google Colab

Input:
I am running the following code snippet in google colab in a single cell:

%debug
# Create tensors of shape (10, 3) and (10, 2).
x = torch.randn(10, 3)
y = torch.randn(10, 2)

# Build a fully connected layer.
linear = nn.Linear(3, 2)
print ('w: ', linear.weight)
print ('b: ', linear.bias)

I wish to debug a piece of code (step through it line by line) to understand what is going on. I wish to step inside the function nn.Linear.

However, when I step through, it does not enter the function at  all. Is there a way to step through nn.Linear line by line? Also, how exactly do I set a breakpoint in nn.Linear? Besides, I wish to step though the snippet line by line as well. However, as the picture shows, the step command automatically steps through and executes the print statement as well.

Output:
Since Python 3.7 you can use a built-in breakpoint function. If this is not available, you can use
import pdb
pdb.set_trace()

instead.
If you want to execute the next line you can try n (next) instead of s (step).


Instruction: PyTorch derivative has no degree

Input:
I am studying the tutorial from PyTorch official docs. I am trying to understand the content. Start from You can do many crazy things with autograd!

x = torch.randn(3, requires_grad=True)
y = x * 2

i = 0
while y.data.norm() < 100:
    y = y * 2
    i+= 1

print(x)
print(y)
print(i)

Output:

tensor([-0.6933,  0.1126,  0.3913], requires_grad=True)
tensor([-88.7455,  14.4082,  50.0871], grad_fn=<MulBackward>)
6

Find the derivative w.r.t to x at point [0.1, 1.0, 0.0001]

gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(gradients)
print(x.grad)

Output:

tensor([ 12.8000, 128.0000,   0.0128])

In my understanding i equals to 6. Then y = (2x)^7 and derivative is different from PyTorch. It has 7 as a factor when substitute the value
to my derivative.

The answer from the PyTorch is simply substitute x with given point to the dy/dx = 2^7 * x

Question:

How to derive the derivative?

References:

How to use PyTorch to calculate partial derivatives?

PyTorch Autograd automatic differentiation feature

Output: If you look closely at the expressions, it'd turn out that y = x * (2^7), the derivative of which is 2^7 * x .


Instruction: Problems about torch.nn.DataParallel

Input:
I am new in deep learning area. Now I am reproducing a paper’s codes. since they use several GPUs, there is a command torch.nn.DataParallel(model, device_ids= args.gpus).cuda() in codes. But I only have one GPU, what
should I change this code to match up my GPU?

Thank you!

Output:
DataParallel should work on a single GPU as well, but you should check if args.gpus only contains the id of the device that is to be used (should be 0) or None.
Choosing None will make the module use all available devices.

Also you could remove DataParallel as you do not need it and move the model to GPU only by calling model.cuda() or, as I prefer, model.to(device) where device is the device's name.

Example:

This example shows how to use a model on a single GPU, setting the device using .to() instead of .cuda().

from torch import nn
import torch

# Set device to cuda if cuda is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Create model
model = nn.Sequential(
  nn.Conv2d(1,20,5),
  nn.ReLU(),
  nn.Conv2d(20,64,5),
  nn.ReLU()
)

# moving model to GPU
model.to(device)

If you want to use DataParallel you could do it like this

# Optional DataParallel, not needed for single GPU usage
model1 = torch.nn.DataParallel(model, device_ids=[0]).to(device)
# Or, using default 'device_ids=None'
model1 = torch.nn.DataParallel(model).to(device)


Instruction: ValueError: Don't know how to translate op Unsqueeze when running converted PyTorch Model

Input:
I'm running into problems trying to use a PyTorch model exported as an ONNX model with Caffe2. Here is my export code

the_model = torchvision.models.densenet121(pretrained=True)
garbage, model_inputs = preprocessing("test.jpg")
torch_out = torch.onnx._export(the_model,
                           model_inputs,
                           "model_weights/chexnet-py.onnx",
                           export_params=True)

Now here is my testing code

model = onnx.load("model_weights/chexnet-py.onnx")
garbage, model_inputs = preprocessing("text.jpg")
prepared_backend = onnx_caffe2.backend.prepare(model)
W = {model.graph.input[0].name: model_inputs.numpy()}
c2_out = prepared_backend.run(W)[0]

This is returning the following error

ValueError: Don't know how to translate op Unsqueeze when running    converted PyTorch Model

Additional information
pytorch version 1.0.0a0+6f664d3
Caffe2 is latest version (attempted building from source, pip, and conda). All gave same result.

Output:
Try looking into this, if you have to edit package called onnx-caffe2 to add the mapping b/w Unsqueeze to ExpandDims
https://github.com/onnx/onnx/issues/1481

Look for the answer:

I found that the Caffe2 equivalence for Unsqueeze in ONNX is ExpandDims, and there is a special mapping in onnx_caffe2/backend.py around line 121 for those operators that are different only in their names and attribute names, but somehow Unsqueeze isn't presented there (have no idea why). So I manually added the mapping rules for it in the _renamed_operators and _per_op_renamed_attrs dicts and the code would look like:

_renamed_operators = {
    'Caffe2ConvTranspose':   'ConvTranspose',
    'GlobalMaxPool':         'MaxPool',
    'GlobalAveragePool':     'AveragePool',
    'Pad':                   'PadImage',
    'Neg':                   'Negative',
    'BatchNormalization':    'SpatialBN',
    'InstanceNormalization': 'InstanceNorm',
    'MatMul':                'BatchMatMul',
    'Upsample':              'ResizeNearest',
    'Equal':                 'EQ',
    'Unsqueeze':             'ExpandDims',  # add this line
}

_global_renamed_attrs = {'kernel_shape': 'kernels'}
_per_op_renamed_attrs = {
    'Squeeze':              {'axes': 'dims'},
    'Transpose':            {'perm': 'axes'},
    'Upsample':             {'mode': ''},
    'Unsqueeze':            {'axes': 'dims'},  # add this line
}

And everything works as expected.

I am not the OP, thanks to OP though.


Instruction: Segmentation fault (Core dumped) on importing torch with python3.5 in virtualenv

Input:
I have installed torch=0.4.1 without cuda in a virtualenv. I am using python3.5 on ubuntu 16.04. Whenever I import torch in an interactive python shell, it quits the python program by showing Segmentation fault (core dumped).

While, surprisingly I had earlier initialized a jupyter notebook and tried importing torch there and it was running fine. Can someone please help? I could not find a solution on official `PyTorch github' discussions.

Output:
I had the same issue, the solution was given on GitHub:

import cv2        # first import cv2
import torch


Instruction: spyder doesn't launch after installing pytorch

Input:
I installed pytorch but after that Spyder can no longer be launched. Here are the terminal info:

  conda install pytorch torchvision -c pytorch Solving environment: done

  ==> WARNING: A newer version of conda exists. <==   current version: 4.5.10   latest version: 4.5.11

  Please update conda by running

$ conda update -n base -c defaults conda

  Package Plan

  environment location: /anaconda3/envs/base_py36

  added / updated specs:
      - pytorch
      - torchvision

  The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
torchvision-0.2.1          |           py36_1         123 KB  pytorch
scipy-1.1.0                |   py36hf1f7d93_0        15.4 MB
scikit-learn-0.20.0        |   py36h4f467ca_1         5.4 MB
numpy-base-1.15.2          |   py36h8a80b8c_1         4.1 MB
numpy-1.11.3               |   py36heee0a97_5         3.4 MB
ninja-1.8.2                |   py36h04f5b5a_1          93 KB
pytorch-0.4.1              |py36_cuda0.0_cudnn0.0_1        10.0 MB  pytorch
------------------------------------------------------------
                                       Total:        38.5 MB

  The following NEW packages will be INSTALLED:

ninja:        1.8.2-py36h04f5b5a_1
pytorch:      0.4.1-py36_cuda0.0_cudnn0.0_1 pytorch
torchvision:  0.2.1-py36_1                  pytorch

  The following packages will be REMOVED:

accelerate:   2.3.1-np111py36_0

  The following packages will be UPDATED:

mkl:          11.3.3-0                              --> 2019.0-118
numexpr:      2.6.7-py36hde7755b_0                  --> 2.6.8-py36h1dc9127_0
numpy:        1.11.3-py36_nomklh8ecaf62_5           --> 1.11.3-py36heee0a97_5
numpy-base:   1.15.0-py36he97cb71_0                 --> 1.15.2-py36h8a80b8c_1
scikit-learn: 0.19.1-py36_nomklhde7755b_0           --> 0.20.0-py36h4f467ca_1
scipy:        1.1.0-py36_nomklh7cd7d8e_0            --> 1.1.0-py36hf1f7d93_0

  The following packages will be DOWNGRADED:

blas:         1.0-openblas                          --> 1.0-mkl

  Proceed ([y]/n)? y

  Downloading and Extracting Packages torchvision-0.2.1    | 123 KB    |

  ############################### | 100%  scipy-1.1.0          | 15.4 MB   | ##################################### | 100%  scikit-learn-0.20.0  | 5.4 MB    |

  ############################### | 100%  numpy-base-1.15.2    | 4.1 MB    | ##################################### | 100%  numpy-1.11.3         | 3.4 MB    | ##################################### | 100%

  ninja-1.8.2          | 93 KB     |

  ############################### | 100%  pytorch-0.4.1        | 10.0 MB   | ##################################### | 100%  Preparing transaction: done

Output:
I don't know if you have solved your issue, but in case you have and someone else comes across this question or you haven't and are still waiting:

I came across your post as a result of having the same thing happen to me...

It would seem that all I had to do was "conda update all" for it to start working again!


Instruction: How to take and restore snapshots of model training on another VM in Google Colab?

Input: There is a 12 hour time limit for training DL models on GPU, according to google colab. Other people have had similar questions in the past, but there has been no clear answer on how to save and load models halfway through training when the 12 hour limits get exceeded, including saving the number of epochs that has been completed/saving other parameters. Is there an automated script for me to save the relevant parameters and resume operations on another VM? I am a complete noob; clear cut answers will be much appreciated.

Output:
As far as I know, there is no way to automatically reconnect to another VM whenever you reach the 12 hours limit. So in any case, you have to manually reconnect when the time is up.

As Bob Smith points out, you can mount Google Drive in Colab VM so that you can save and load data from there. In particular, you can periodically save model checkpoints so that you can load the most recent one whenever you connect to a new Colab VM.

Mount Drive in your Colab VM:

from google.colab import drive
drive.mount('/content/gdrive')

Create a saver in your graph:

saver = tf.train.Saver()

Periodically (e.g. every epoch) save a checkpoint in Drive:

saver.save(session, CHECKPOINT_PATH)

When you connect to a new Colab VM (because of the timeout), mount Drive again in your VM and restore the most recent checkpoint before the training phase:

saver.restore(session, CHECKPOINT_PATH)
...
# Start training with the restored model.

Take a look at the documentation to read more about tf.train.Saver.


Instruction: Get CUDA_HOME environment path PYTORCH

Input:
I have cuda installed via anaconda on my system which has 2 GPUs which is getting recognized by my python.

import torch
torch.cuda.is_available()
true

However when I try to run a model via its C API, I m getting following error:

~/anaconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _join_cuda_home(*paths)
722     '''
723     if CUDA_HOME is None:
--> 724         raise EnvironmentError('CUDA_HOME environment variable is not set. '
725                                'Please set it to your CUDA install root.')
726     return os.path.join(CUDA_HOME, *paths)

OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

https://lfd.readthedocs.io/en/latest/install_gpu.html page gives instruction to set up CUDA_HOME path if cuda is installed via their method. Since I have installed cuda via anaconda I don't know which path to set.

I tried find method but it is returning me too many paths for cuda. Can somebody help me with the path for CUDA. Thanks in advance.

Output:
Solution to above issue!

As cuda installed through anaconda is not the entire package. Please install cuda drivers manually from Nvidia Website[ https://developer.nvidia.com/cuda-downloads ]

After installation of drivers, pytorch would be able to access the cuda path. You can test the cuda path using below sample code. Problem resolved!!!

CHECK INSTALLATION:

import os
print(os.environ.get('CUDA_PATH'))

OUTPUT:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1


Instruction: How to pretrain/select CNN for Biomedical Video Analysis

Input:
Data
I am trying to train a model in the biomedical domain with a rather specialized task (flow prediction).
My input consists of video clips and I would like to predict either a single image or a video. I have pixel wise ground truth for every frame of the video but the number of videos is very limited.
I do however have a lot of unlabeled data of related scenes, which is why I considered transfer learning/pretraining of some sort.

Architecture
Architecture wise I am considering a CNN, RNN combination where the CNN provides a representation of the input frames for the RNN to learn about the temporal relationship between input frames.

Now my question is: What kind of CNN do I use and on what do I pretrain it? Since I am working with biomedical data I would assume image net as well as most other image data sets do not really help as the image content is very different. Are there any data sets/tasks/networks I could use for this purpose?

Output: Some people recommend shallow autoencoders to pretrain networks if there is not much labeled data, but lots of unlabeled data. A reference would be Géron's book "Hands-On Machine Learning with Scikit-Learn and TensorFlow", but there are also many articles and tutorials online.


Instruction: Convolution Autoencoder Image Dimension Error

Input:
I have the following Convolutional Autoencoder setup:

class autoencoder(nn.Module):
def  __init__(self):
    super(autoencoder, self).__init__()
    self.encoder = nn.Sequential(
        nn.Conv2d(1, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
        nn.ReLU(True),
        nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
        nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
        nn.ReLU(True),
        nn.MaxPool2d(2, stride=1)  # b, 8, 2, 2
                                )

    self.decoder = nn.Sequential(
        nn.ConvTranspose2d(8, 16, 3, stride=2),  # b, 16, 5, 5
        nn.ReLU(True),
        nn.ConvTranspose2d(16, 8, 5, stride=3, padding=1),  # b, 8, 15, 15
        nn.ReLU(True),
        nn.ConvTranspose2d(8, 1, 2, stride=2, padding=1),  # b, 1, 28, 28
        nn.Tanh()
                                )

This is the main Loop:

for epoch in range(epochs):
running_loss = 0
for data in (train_loader):
    image,_=data
    inputs = image.view(image.size(0),-1)
    optimizer.zero_grad()
    #image = np.expand_dims(img, axis=0)

    outputs = net(inputs)
    loss = criterion(outputs,inputs)
    loss.backward()
    optimizer.step()
    running_loss += loss.data[0]
print('At Iteration : %d   ;  Mean-Squared Error : %f'%(epoch + 1,running_loss/(train_set.train_data.size(0)/batch_size)))

This is the Error:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [16, 1, 3, 3], but got input of size [1000, 784] instead

This has something to do with the flattening of the image but Im not exactly sure how to deflatten it.

Output:
Why are you "flattening" your input image (2nd line of main loop):

inputs = image.view(image.size(0),-1)

This line turns your 4 dimensional image (batch - channels - height - width) to a two dimensional "flat" vector (batch - c * h * w).
You autoencoder expects its inputs to be 4D and not "flat". just remove this line and you should be okay.


Instruction: How to install pytorch on Power 8 or PPC64 machine?

Input: I am trying to install pytorch using conda on power 8 IBM machine. Although, I read articles from IBM blog, I couldn't install with success because I got stuck with compiling magma.

Output:
Assuming that conda is already installed.
Simply run the following command

conda install -c engility pytorch
conda install -c engility torchvision

Note:
1. Goto this anaconda page
2. Search for pytorch
3. Scroll down to see which one has linux-ppc64le as platform
4. Click into that specific package
5. You will get the command to install pytorch


Instruction: Fastai on google colab

Input:
Two days back I ran my model using fastai 0.7.0 on google colab.
And for two days I got busy and now if I am trying to run it , its throwing me an error, on the execution of the line
*"from fastai.transforms import ’ .
the error is AttributeError: module ‘torch’ has no attribute ‘float32’.

Output:
The following should get you up and running with the 0.7.0 version of fast.ai (used by v2 of the course) on Google Colab:

!pip install -q "fastai==0.7.0" Pillow==4.1.1 torchtext==0.2.3
!apt-get -qq install -y libsm6 libxext6 && pip install -q -U opencv-python

from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
# !apt update -q
!apt install -y libsm6 libxext6

from os import path
accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'
torch_whl = f"http://download.pytorch.org/whl/{accelerator}/torch-0.3.0.post4-{platform}-linux_x86_64.whl"
!pip install -q {torch_whl} torchvision image


Instruction: What does the -1 mean in tensor.size(-1)?

Input:
I have seen something like this in the Pytorch documentation,

import torch

a = torch.tensor([1, 2])
a.size() # torch.Size([2])
a.size(-1) # 2

How does this work? I didn't find a description. Thanks,

Output:
a.size(-1) refers to the last dimension. For example, if the shape of x were (10,20) then x.size(-1) refers to second dimension i.e. 20.
Take a look at the following examples:

import torch
a= torch.zeros((2,5)) # a is matrix of 2 rows and 5 columns all elements are 0
#size gives a 1d tensor containing the shapes
a.size(-1)# refers to the last element in the tensor

This is equivalent to:

a_size= a.size()
a_size(-1)

Hope this helps you.


Instruction: Un-normalizing PyTorch data

Input:
Below code :

ux = torch.tensor(np.array([[255,1,255],[255,1,255]])).float()

print(ux)

ux = F.normalize(ux, p=2, dim=1)

print(ux)

prints :

tensor([[ 255.,    1.,  255.],
        [ 255.,    1.,  255.]])
tensor([[ 0.7071,  0.0028,  0.7071],
        [ 0.7071,  0.0028,  0.7071]])

How can I un-normalize the ux in order to return to values

tensor([[ 255.,    1.,  255.],
        [ 255.,    1.,  255.]])

from

tensor([[ 0.7071,  0.0028,  0.7071],
        [ 0.7071,  0.0028,  0.7071]])

There are various resources that detail this process such as https://discuss.pytorch.org/t/simple-way-to-inverse-normalize-a-batch-of-input-variable/12385/3 but do not detail unnormalizing result of F.normalize

Output:
F.normalize simply divides by the norm according to the documentation, so you simply need to multiply it by its magnitude.

This means you still need access to the magnitude of the original vector ux, otherwise, this is not possible, since the information about the magnitude cannot be recovered from the normalized vector.

Here's how this can be done:

# I modified the input to make it more interesting, but you can use any other value
ux = torch.tensor(np.array([[255,1,255],[101,10,123]])).float()

magnitude = ux.norm(p=2, dim=1, keepdim=True)    # NEW
ux = F.normalize(ux, p=2, dim=1)

ux_orig = ux * magnitude                         # NEW

print(ux_orig)
# Outputs:
# tensor([[255.,   1., 255.],
#         [101.,  10., 123.]])


Instruction: Pytorch adaptive_avg_pool2d algorithm

Input:
Anyone knows the algorithm for pytorch adaptive_avg_pool2d, for example,

adaptive_avg_pool2d(image,[14,14])

so question:
    I want to do the same in keras neural network, for any give inputs, want to get 14*14 output. Any suggestions?

Output:
I don't think this exists in Keras. You could get the dimension of your input and divide by 14 to get the desired pool_size.

For example if your inputs are 28x28 you can use:

keras.layers.AveragePooling2D(pool_size=(2, 2), strides=None, padding='valid', data_format=None)


Instruction: How to get probabilities when using Pytorch's densenet?

Input:
I want to do a binary classification and I used the DenseNet from Pytorch.

Here is my predict code:

densenet = torch.load(model_path)
densenet.eval()
output = densenet(input)
print(output)

And here is the output:

Variable containing:
54.4869 -54.3721
[torch.cuda.FloatTensor of size 1x2 (GPU 0)]

I want to get the probabilities of each class. What should I do?

I have noticed that torch.nn.Softmax() could be used when there are many categories, as discussed here.

Output:
import torch.nn as nn

Add a softmax layer to the classifier layer:
i.e. typical:

num_ftrs = model_ft.classifier.in_features
model_ft.classifier = nn.Linear(num_ftrs, num_classes)

 updated:

model_ft.classifier = nn.Sequential(nn.Linear(num_ftrs, num_classes),
nn.Softmax(dim=1))


Instruction: PyTorch: Loss remains constant

Input:
I've written a code in PyTorch with my own implemented loss function focal_loss_fixed. But my loss value stays fixed after every epoch. Looks like weights are not being updated. Here is my code snippet:

optimizer = optim.SGD(net.parameters(),
                          lr=lr,
                          momentum=0.9,
                          weight_decay=0.0005)

for epoch in T(range(20)):
    net.train()
    epoch_loss = 0
    for n in range(len(x_train)//batch_size):
        (imgs, true_masks) = data_gen_small(x_train, y_train, iter_num=n, batch_size=batch_size)
        temp = []
        for tt in true_masks:
            temp.append(tt.reshape(128, 128, 1))
        true_masks = np.copy(np.array(temp))
        del temp
        imgs = np.swapaxes(imgs, 1,3)
        imgs = torch.from_numpy(imgs).float().cuda()
        true_masks = torch.from_numpy(true_masks).float().cuda()
        masks_pred = net(imgs)
        masks_probs = F.sigmoid(masks_pred)
        masks_probs_flat = masks_probs.view(-1)
        true_masks_flat = true_masks.view(-1)
        print((focal_loss_fixed(tf.convert_to_tensor(true_masks_flat.data.cpu().numpy()), tf.convert_to_tensor(masks_probs_flat.data.cpu().numpy()))))
        loss = torch.from_numpy(np.array(focal_loss_fixed(tf.convert_to_tensor(true_masks_flat.data.cpu().numpy()), tf.convert_to_tensor(masks_probs_flat.data.cpu().numpy())))).float().cuda()
        loss = Variable(loss.data, requires_grad=True)
        epoch_loss *= (n/(n+1))
        epoch_loss += loss.item()*(1/(n+1))
        print('Step: {0:.2f}% --- loss: {1:.6f}'.format(n * batch_size* 100.0 / len(x_train), epoch_loss), end='\r')
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print('Epoch finished ! Loss: {}'.format(epoch_loss))

And this is my `focal_loss_fixed' function:

def focal_loss_fixed(true_data, pred_data):
    gamma=2.
    alpha=.25
    eps = 1e-7
    # print(type(y_true), type(y_pred))
    pred_data = K.clip(pred_data,eps,1-eps)
    pt_1 = tf.where(tf.equal(true_data, 1), pred_data, tf.ones_like(pred_data))
    pt_0 = tf.where(tf.equal(true_data, 0), pred_data, tf.zeros_like(pred_data))
    with tf.Session() as sess:
        return sess.run(-K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1))-K.sum((1-alpha) * K.pow( pt_0, gamma) * K.log(1. - pt_0)))

After each epoch the loss value stays constant(5589.60328). What's wrong with it?

Output:
I think the problem lies in your heavy weight decay.

Essentially, you are not reducing the weight by x, but rather you multiply the weights by x, which means that you are instantaneously only doing very small increments, leading to a (seemingly) plateauing loss function.

More explanation on this can be found in the PyTorch discussion forum (e.g., here, or here).
Unfortunately, the source for SGD alone also does not tell you much about its implementation.
Simply setting it to a larger value should result in better updates. You can start by leaving it out completely, and then iteratively reducing it (from 1.0), until you get more decent results.


Instruction: Creating and train only specified weights in TensorFlow or PyTorch

Input:
I am wondering if there is a way in TensorFlow, PyTorch or some other library to selectively connect neurons. I want to make a network with a very large number of neurons in each layer, but that has very few connections between layers.

Note that I do not think this is a duplicate of this answer: Selectively zero weights in TensorFlow?. I implemented a custom keras layer using essentially the same method that appears in that question - essentially by creating a dense layer where all but the specified weights are ignored in training and evaluation. This fulfills part of what I want to do by not training specified weights, and not using them for prediction. But, the problems is that I still waste memory saving the untrained weights, and I waste time calculating the gradients of the zeroed weights. What I would like is for the computation of the gradient matrices to involve only sparse matrices, so that I do not waste time and memory.

Is there a way to selectively create and train weights without wasting memory? If my question is unclear or there is more information that it would be helpful for me to provide, please let me know. I would like to be helpful as a question-asker.

Output:
The usual, simple solution is to initialize your weight matrices to have zeros where there should be no connection. You store a mask of the location of these zeros, and set the weights at these positions to zero after each weight update. You need to do this as the gradient for zero weights may be nonzero, and this would introduce nonzero weights (i.e. connectios) where you don't want any.

Pseudocode:

# setup network
weights = sparse_init()  # only nonzero for existing connections
zero_mask = where(weights == 0)

# train
for e in range(num_epochs):
    train_operation()  # may lead to introduction of new connections
    weights[zero_mask] = 0  # so we set them to zero again


Instruction: PyTorch: new_ones vs ones

Input:
In PyTorch what is the difference between new_ones() vs ones(). For example,

x2.new_ones(3,2, dtype=torch.double)

vs

torch.ones(3,2, dtype=torch.double)

Output:
For the sake of this answer, I am assuming that your x2 is a previously defined torch.Tensor. If we then head over to the PyTorch documentation, we can read the following on new_ones():

  Returns a Tensor of size size filled with 1. By default, the
  returned Tensor has the same torch.dtype and torch.device as this
  tensor.

Whereas ones()

  Returns a tensor filled with the scalar value 1, with the shape
  defined by the variable argument sizes.

So, essentially, new_ones allows you to quickly create a new torch.Tensor on the same device and data type as a previously existing tensor (with ones), whereas ones() serves the purpose of creating a torch.Tensor from scratch (filled with ones).


Instruction: What does the "greater than" operator ">" mean for PyTorch tensors?

Input:
I have a tensor it defined as:

import torch
it = torch.tensor([0,  0,  0,  0,  0,  0,  0,  0,  0,  0], device='cuda:0')

Given this definition, what does it > 0 then mean?

Output:
Using the > operator is the same as using the torch.gt() function.

In other words,

it > 0

is the same as

torch.gt(it, 0)

and it returns a ByteTensor (a Boolean tensor) of the same shape as it where out[i] is True if it[i] > 0 and False otherwise.


Instruction: is it possible make cuda deterministic?

Input:
I am trying to refactor an application and testing new against old version by comparing outputs. At the first pytorch conv layer using identical input I get identical output when using CPU. However using cuda on GPU the output tensors differ in the 5th decimal place on some cells. Is there any way to eliminate this difference? I am already setting:

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True

Output:
cpu and gpu can't produce the same result even if the seeds are set equal.
Refer to this and this.


Instruction: Pytorch : significance of grad_h_relu.clone() in Backprop

Input:
I am learning Pytorch, while I'm looking to the tutorial on their site I can't understand the significance of grad_h = grad_h_relu.clone().

h = x.mm(w1)
h_relu = h.clamp(min=0)
y_pred = h_relu.mm(w2)

# Compute and print loss
loss = (y_pred - y).pow(2).sum().item()
print(t, loss)

# Backprop to compute gradients of w1 and w2 with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_w2 = h_relu.t().mm(grad_y_pred)
grad_h_relu = grad_y_pred.mm(w2.t())

grad_h = grad_h_relu.clone() # what is the signifigance of this line?

grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)

# Update weights using gradient descent
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2

Output:
grad_h = grad_h_relu.clone()

It means that you are making a copy of the gradient of relu such that it does not share the memory with the original grad_h_relu. Then you perform some operation on it. As they are stored in two different locations, changing the value of grad_h by the following operation will not affect the grad_h_relu.

grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)

This grad_w1 is needed to update your parameters of your network.


Instruction: What is the equivalent of torch.nn.functional.grid_sample in Tensorflow / Numpy?

Input: I am new to pytorch and have been trying to convert some code . Can't find this particular functionality . Does it exist in tensorflow ?

Output:
I do not think there is anything like that provided in TensorFlow. Here is a possible implementation for the 2D case (I have not considered padding, but the code should behave like the border mode). Note that, unlike the PyTorch version, I am assuming the input dimension order is (batch_size, height, width, channels) (as is common in TensorFlow).

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def grid_sample_2d(inp, grid):
    in_shape = tf.shape(inp)
    in_h = in_shape[1]
    in_w = in_shape[2]

    # Find interpolation sides
    i, j = grid[..., 0], grid[..., 1]
    i = tf.cast(in_h - 1, grid.dtype) * (i + 1) / 2
    j = tf.cast(in_w - 1, grid.dtype) * (j + 1) / 2
    i_1 = tf.maximum(tf.cast(tf.floor(i), tf.int32), 0)
    i_2 = tf.minimum(i_1 + 1, in_h - 1)
    j_1 = tf.maximum(tf.cast(tf.floor(j), tf.int32), 0)
    j_2 = tf.minimum(j_1 + 1, in_w - 1)

    # Gather pixel values
    n_idx = tf.tile(tf.range(in_shape[0])[:, tf.newaxis, tf.newaxis], tf.concat([[1], tf.shape(i)[1:]], axis=0))
    q_11 = tf.gather_nd(inp, tf.stack([n_idx, i_1, j_1], axis=-1))
    q_12 = tf.gather_nd(inp, tf.stack([n_idx, i_1, j_2], axis=-1))
    q_21 = tf.gather_nd(inp, tf.stack([n_idx, i_2, j_1], axis=-1))
    q_22 = tf.gather_nd(inp, tf.stack([n_idx, i_2, j_2], axis=-1))

    # Interpolation coefficients
    di = tf.cast(i, inp.dtype) - tf.cast(i_1, inp.dtype)
    di = tf.expand_dims(di, -1)
    dj = tf.cast(j, inp.dtype) - tf.cast(j_1, inp.dtype)
    dj = tf.expand_dims(dj, -1)

    # Compute interpolations
    q_i1 = q_11 * (1 - di) + q_21 * di
    q_i2 = q_12 * (1 - di) + q_22 * di
    q_ij = q_i1 * (1 - dj) + q_i2 * dj

    return q_ij

# Test it
inp = tf.placeholder(tf.float32, [None, None, None, None])
grid = tf.placeholder(tf.float32, [None, None, None, 2])
res = grid_sample_2d(inp, grid)
with tf.Session() as sess:
    # Make test image
    im_grid_i, im_grid_j = np.meshgrid(np.arange(6), np.arange(10), indexing='ij')
    im = im_grid_i + im_grid_j
    im = im / im.max()
    im = np.stack([im] * 3, axis=-1)
    # Test grid 1: complete image
    grid1 = np.stack(np.meshgrid(np.linspace(-1, 1, 15), np.linspace(-1, 1, 18), indexing='ij'), axis=-1)
    # Test grid 2: lower right corner
    grid2 = np.stack(np.meshgrid(np.linspace(0, 1, 15), np.linspace(.5, 1, 18), indexing='ij'), axis=-1)
    # Run
    res1, res2 = sess.run(res, feed_dict={inp: [im, im], grid: [grid1, grid2]})
    # Plot image and sampled grids
    plt.figure()
    plt.imshow(im)
    plt.figure()
    plt.imshow(res1)
    plt.figure()
    plt.imshow(res2)

Here are the resulting images, first the input:

First grid result, which is the first image but with different shape:

Second grid result, which spans a region in the lower right corner:


Instruction: PyTorch autocomplete (code completion) in Sublime Text 3

Input:
I'm a fan of Sublime Text 3 and would like to get code autocompletion for PyTorch. However, I can't get this working yet. Any suggestions or starting points where I can begin to get this working?

I have searched in the packages repository of Sublime Text but unfortunately there's none.

Note: I have looked at a related question here IDE autocomplete for pytorch but that's only for VS Code.

Output:
Not a sublime user, but Jedi autocompletion handles PyTorch suggestions just fine.

Jedi is an autocompletion engine and should work for any Python package, including the ones in virtual environment if you set it up and including PyTorch/Tensorflow.

Anyway, sublime version seems to be missing a few things like results caching, hence you may have to wait a few seconds after issuing torch. (library is quite heavy, that's why you most probably will experience some lag). If you'd like to speed it up, I see no other possibility than changing it manually (vim plugin for jedi I am using has this option implemented, you may check how they've done it here, both  jedi plugins seem to be written in Python hence should be tunable/fixable).

Oh, and if something isn't working (or maybe you would like to submit a PR request to jedi team/sublime jedi team), the community around it is quite vibrant and you should get some help (definitely better and more in-depth than here on StackOverflow).


Instruction: Error while executing RNN code on pyTorch?

Input:
I am running a code on Binary Addition of two strings using PyTorch.

However, while training the model I am getting the following error:

can't convert np.ndarray of type numpy.object.
The only supported types are: double, float, float16, int64, int32, and uint8.

Can anyone help me? Here is my code:

featDim=2 # two bits each from each of the String
outputDim=1 # one output node which would output a zero or 1

lstmSize=10

lossFunction = nn.MSELoss()
model =Adder(featDim, lstmSize, outputDim)
print ('model initialized')
#optimizer = optim.SGD(model.parameters(), lr=3e-2, momentum=0.8)
optimizer=optim.Adam(model.parameters(),lr=0.001)
epochs=500
### epochs ##
totalLoss= float("inf")
while totalLoss > 1e-5:
  print(" Avg. Loss for last 500 samples = %lf"%(totalLoss))
  totalLoss=0
  for i in range(0,epochs): # average the loss over 200 samples

    stringLen=4
    testFlag=0
    x,y=getSample(stringLen, testFlag)

    model.zero_grad()

    x_var=autograd.Variable(torch.from_numpy(x).unsqueeze(1).float()) #convert to torch tensor and variable
    # unsqueeze() is used to add the extra dimension since
    # your input need to be of t*batchsize*featDim; you cant do away with the batch in pytorch
    seqLen=x_var.size(0)
    #print (x_var)
    x_var= x_var.contiguous()
    y_var=autograd.Variable(torch.from_numpy(y).float()) ##ERROR ON THIS LINE
    finalScores = model(x_var)
    #finalScores=finalScores.

    loss=lossFunction(finalScores,y_var)
    totalLoss+=loss.data[0]
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

  totalLoss=totalLoss/epochs

Output:
The main problem here is the type of your y. You haven't given any information about this, therefore this here will be more general:

But obviously your ndarray doesn't contain numeric data types. You have to use one of these mentioned in your error message:

  The only supported types are: double, float, float16, int64, int32,
  and uint8.

So here is a short example to demonstrate the issue:

If you use one of the previously mentioned data types it works just fine:

import torch
import numpy as np
a = np.ndarray(shape=(2,2), dtype=np.float) # data type np.float
print(a)
print(torch.autograd.Variable(torch.from_numpy(a).float()))

Output:

[[2.16641777e-314 2.16641777e-314]
 [2.16641777e-314 2.16641777e-314]]
Variable containing:
 0  0
 0  0
[torch.FloatTensor of size 2x2]

But if you use some other numpy data type (like np.object) you will get this error message:

import torch
import numpy as np
a = np.ndarray(shape=(2,2), dtype=np.object) # data type np.object
print(a)
print(torch.autograd.Variable(torch.from_numpy(a).float()))

This results in:

[[None None]
 [None None]]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-01e1e4bec020> in <module>()
      3 a = np.ndarray(shape=(2,2), dtype=np.object)
      4 print(a)
----> 5 print(torch.autograd.Variable(torch.from_numpy(a).float()))

RuntimeError: can't convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.

You probably haven't specified the data type np.object directly. I guess this is maybe a result of some nested arrays or so.

But your need to bring your numpy array y into proper shape with a numeric data type then it should work for you.


Instruction: How can I visualize what happens during loss.backward()?

Input:
I am confident in my understanding of the forward pass of my model, how can I control its backward pass?

This is not a theoretical question about what back-propagation is. The question is a practical one, about whether or not there are tools suited to visualize/track/control what happens during back-propagation.

Ideally, this tool would allow to visualize the structure of the computational graph of the model (a graph of the model's operations), its inputs and its trainable parameters.

Now, I do:

loss.backward()

and I would like to visualize what happens in that step.

Output:
There has been been already mention of pytorchviz which lets you visualize the graph.

Here is a small example that might help you to understand how pytorchviz does trace the graph using the grad_fn:

import torch
from torch import nn
d = 5
x = torch.rand(d, requires_grad=True)
print('Tensor x:', x)
y = torch.ones(d, requires_grad=True)
print('Tensor y:', y)
loss = torch.sum(x*y)*3

del x
print()
print('Tracing back tensors:')
def getBack(var_grad_fn):
    print(var_grad_fn)
    for n in var_grad_fn.next_functions:
        if n[0]:
            try:
                tensor = getattr(n[0], 'variable')
                print(n[0])
                print('Tensor with grad found:', tensor)
                print(' - gradient:', tensor.grad)
                print()
            except AttributeError as e:
                getBack(n[0])

loss.backward()
getBack(loss.grad_fn)

Output:

Tensor x: tensor([0.0042, 0.5376, 0.7436, 0.2737, 0.4848], requires_grad=True)
Tensor y: tensor([1., 1., 1., 1., 1.], requires_grad=True)

Tracing back tensors:
<MulBackward object at 0x1201bada0>
<SumBackward0 object at 0x1201bacf8>
<ThMulBackward object at 0x1201bae48>
<AccumulateGrad object at 0x1201badd8>
Tensor with grad found: tensor([0.0042, 0.5376, 0.7436, 0.2737, 0.4848], requires_grad=True)
 - gradient: tensor([3., 3., 3., 3., 3.])

<AccumulateGrad object at 0x1201bad68>
Tensor with grad found: tensor([1., 1., 1., 1., 1.], requires_grad=True)
 - gradient: tensor([0.0125, 1.6129, 2.2307, 0.8211, 1.4543])

Further you should definately take a look into how autograd functions (that are used by the backward()-function) are actually work !
Here is a tutorial from the pytorch site with an easy and short example:

PyTorch: Defining New autograd Functions

Hope this helps a bit!


Instruction: KL Divergence of Normal and Laplace isn't Implemented in TensorFlow Probability and PyTorch

Input:
In both TensorFlow Probability (v0.4.0) and PyTorch (v0.4.1) the KL Divergence of the Normal distribution (tfp, PyTorch) and the Laplace distribution (tfp, PyTorch) isn't implemented resulting in a NotImplementedError error being thrown.

>>> import tensorflow as tf
>>> import tensorflow_probability as tfp
>>> tfd = tfp.distributions
>>> import torch
>>>
>>> tf.__version__
'1.11.0'
>>> tfp.__version__
'0.4.0'
>>> torch.__version__
'0.4.1'
>>>
>>> p = tfd.Normal(loc=0., scale=1.)
>>> q = tfd.Laplace(loc=0., scale=1.)
>>> tfd.kl_divergence(p, q)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/tensorflow/python/ops/distributions/kullback_leibler.py", line 95, in kl_divergence
    % (type(distribution_a).__name__, type(distribution_b).__name__))
NotImplementedError: No KL(distribution_a || distribution_b) registered for distribution_a type Normal and distribution_b type Laplace
>>>
>>> a = torch.distributions.normal.Normal(loc=0., scale=1.)
>>> b = torch.distributions.laplace.Laplace(loc=0., scale=1.)
>>> torch.distributions.kl.kl_divergence(a,b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/torch/distributions/kl.py", line 161, in kl_divergence
    raise NotImplementedError
NotImplementedError

I assume as this is missing from both of these libraries that there is some good reason for this and that the user would be expected to implement it themselves with tfp.distributions.RegisterKL in TensorFlow Probability and torch.distributions.kl.register_kl in PyTorch.

Is this the correct assumption? If so, can someone explain why the KL Divergence wouldn't be implemented for given distribution classes? I think I am missing something very basic about this.

If my assumption is wrong, can someone explain how to properly have TensorFlow and PyTorch implement these operations?

For additional reference, using for this example an older version of TensorFlow that works with Edward,

pip install tensorflow==1.7
pip install edward

In this minimal example above, I'm trying to implement the equivalent of the following edward toy example code in tfp (or in torch).

import tensorflow as tf
import edward as ed

p = ed.models.Normal(loc=0., scale=1.)
s = tf.Variable(1.)
q = ed.models.Laplace(loc=0., scale=s)
inference = ed.KLqp({p: q})
inference.run(n_iter=5000)

Output:
IIRC, Edward's KLqp switches tries to use the Analytic form, and if not switches to using the sample KL.

For TFP, and I think PyTorch, kl_divergence only works for distributions
registered, and unlike Edward only computes the analytic KL. As you mention, these aren't implemented in TFP, and I would say that's more of because the common cases (such as KL(MultivariateNormal || MultivariateNormal) have been implemented.

To register the KL divergence, you would do something like: https://github.com/tensorflow/probability/blob/07878168731e0f6d3d0e7c878bdfd5780c16c8d4/tensorflow_probability/python/distributions/gamma.py#L275. (It would be great if you could file a PR at https://github.com/tensorflow/probability!).

If it turns out that there isn't a suitable analytic form of this (off the top of my head, I don't know if there is one), then one can form the sample KL and do optimization with that. That can be done explicitly in TFP (by sampling and computing the sample KL. Also please file a PR if you would like this to be done more automatically as well. This is something some of us on TFP are interested in.

It would be interesting to see for what cases analytic KL's can be automated. For instance, if q and p come from the same exponential family, then there is a nice form for the KL divergence in terms of sufficient statistics and the normalizer. But for KL's that are across exponential families (or even not exponential families), I'm not aware of results on classes of distributions where you can calculate the KL within the class semi-automatically.


Instruction: Pytorch is not using GPU even it detects the GPU

Input:
I made my windows 10 jupyter notebook as a server and running some trains on it.

I've installed CUDA 9.0 and cuDNN properly, and python detects the GPU. This is what I've got on the anaconda prompt.

>>> torch.cuda.get_device_name(0)
'GeForce GTX 1070'

And I also placed my model and tensors on cuda by .cuda()

model = LogPPredictor(1, 58, 64, 128, 1, 'gsc')

if torch.cuda.is_available():
    torch.set_default_tensor_type(torch.cuda.DoubleTensor)
    model.cuda()
else:
    torch.set_default_tensor_type(torch.FloatTensor)

list_train_loss = list()
list_val_loss = list()
acc = 0
mse = 0

optimizer = args.optim(model.parameters(),
                       lr=args.lr,
                       weight_decay=args.l2_coef)

data_train = DataLoader(args.dict_partition['train'],
                        batch_size=args.batch_size,
                        pin_memory=True,
                        shuffle=args.shuffle)

data_val = DataLoader(args.dict_partition['val'],
                     batch_size=args.batch_size,
                     pin_memory=True,
                     shuffle=args.shuffle)

for epoch in tqdm_notebook(range(args.epoch), desc='Epoch'):
    model.train()
    epoch_train_loss = 0
    for i, batch in enumerate(data_train):
        list_feature = torch.tensor(batch[0]).cuda()
        list_adj = torch.tensor(batch[1]).cuda()
        list_logP = torch.tensor(batch[2]).cuda()
        list_logP = list_logP.view(-1,1)

        optimizer.zero_grad()
        list_pred_logP = model(list_feature, list_adj)
        list_pred_logP.require_grad = False
        train_loss = args.criterion(list_pred_logP, list_logP)
        epoch_train_loss += train_loss.item()
        train_loss.backward()
        optimizer.step()

    list_train_loss.append(epoch_train_loss/len(data_train))

    model.eval()
    epoch_val_loss = 0
    with torch.no_grad():
        for i, batch in enumerate(data_val):
            list_feature = torch.tensor(batch[0]).cuda()
            list_adj = torch.tensor(batch[1]).cuda()
            list_logP = torch.tensor(batch[2]).cuda()
            list_logP = list_logP.view(-1,1)

            list_pred_logP = model(list_feature, list_adj)
            val_loss = args.criterion(list_pred_logP, list_logP)
            epoch_val_loss += val_loss.item()

    list_val_loss.append(epoch_val_loss/len(data_val))

data_test = DataLoader(args.dict_partition['test'],
                   batch_size=args.batch_size,
                   pin_memory=True,
                   shuffle=args.shuffle)

model.eval()
with torch.no_grad():
    logP_total = list()
    pred_logP_total = list()
    for i, batch in enumerate(data_val):
        list_feature = torch.tensor(batch[0]).cuda()
        list_adj = torch.tensor(batch[1]).cuda()
        list_logP = torch.tensor(batch[2]).cuda()
        logP_total += list_logP.tolist()
        list_logP = list_logP.view(-1,1)

    list_pred_logP = model(list_feature, list_adj)

    pred_logP_total += list_pred_logP.tolist()

mse = mean_squared_error(logP_total, pred_logP_total)

But on the Process Manager of Windows, whenever I start training, only CPU usage goes up to 25% and GPU usage remains 0. How can I fix this???

Output:
I had a similar problem with using PyTorch on Cuda. After looking for possible solutions, I found the following post by Soumith himself that found it very helpful.
https://discuss.pytorch.org/t/gpu-supposed-to-be-used-but-isnt/2883
The bottom line is, at least in my case, I could not put enough load on GPUs. There was a bottleneck in my application. Try another example, or increase batch size; it should be OK.


Instruction: Importing pytorch in Spyder crashes kernel after installing matplotlib

Input:
I created an environment in Anaconda3 and installed pytorch and spyder on a Linux machine. Here are the specifications:

spyder                    3.3.1
ipython                   7.0.1
python                    3.7.0
pytorch                   0.4.1
torchvision               0.2.1

When I open spyder and import torch, it works. Afterwards I installed matplotlib 3.0.1. Restarting spyder and importing again pytorch results in a message on the ipython window in spyder:

An error ocurred while starting the kernel
terminate called after throwing an instance of 'std::runtime_error'
what(): expected ) but found 'ident' here:
aten::_addmv(Tensor self, Tensor mat, Tensor vec, *, Scalar beta=1, Scalar alpha=1) ‑> Tensor
~~~~~~ <‑‑‑ HERE

On the bash terminal, I get the message:

js: Not allowed to load local resource:
file:///home/user/anaconda3/envs/myenv/lib/python3.7/site-packages/spyder/utils/help/static/css/default.css

I have been using all these packages in another environment for months (so they are in an older version), so it must be something with the new versions.
If I run ipython or python on the terminal, importing works, so I am concluding it has something to do with spyder.

The 'solution' is obvious: install older versions of the packages, but is there any other more sustainable solution?

Output:
I have ipython 7.0.1 and matplotlib 2.0.2 and the same problem, it seems like ipython crashes after the following two commands: %matplotlib auto followed by import torch.

This happens both in spyder as in jupyter notebook when the two commands are in seperate blocks.

What worked for me was:
First making sure that spyders backend graphics is set to inline:
Tools -> Preferences -> IPython console -> Graphics backed to Inline.
Then import torch followed by switching from inline to external plotting with %matplotlib auto.

Note that this does not happen any more with
ipython 7.2.0 and matplotlib 3.0.2


Instruction: Pytorch 3-GPUs, just can only use 2 of them to train

Input:
I have three 1080TI, but when train I can only use 2 of them..

Code:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.cuda()

criterion = nn.CrossEntropyLoss().cuda()
optimizer_conv = optim.SGD(model.classifier.parameters(), lr=0.0001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

train part:

outputs = nn.parallel.data_parallel(model,inputs,device_ids=[0,1,2])

With "CUDA_VISIBLE_DEVICES="1,2,3" python train.py"
Got this:

| 22%   35C    P8    10W / 250W |     12MiB / 11178MiB |      0%
| 43%   59C    P2    92W / 250W |   1169MiB / 11178MiB |     49%
| 44%   60C    P2    91W / 250W |   1045MiB / 11175MiB |     54%

With "CUDA_VISIBLE_DEVICES="0,1,2" python train.py"
Got this:

| 21%   38C    P2    95W / 250W |   1169MiB / 11178MiB |     78%      Default |
| 42%   63C    P2    93W / 250W |    777MiB / 11178MiB |     76%      Default |
| 43%   64C    P0    85W / 250W |    282MiB / 11175MiB |      0%      Default |

Output:
eeeee.. I found the reason:
my batchsize = 4 when there are three GPUs
so Change batchsize bigger can solve this "weird" problem