Instruction: Why randn doesn't always have a mean of 0 and variance of 1?

Input:
For the PyTorch.randn() method the documentation says:

  Returns a tensor filled with random numbers from a normal distribution
  with mean 0 and variance 1 (also called the standard normal
  distribution).

So here is an example tensor:

x = torch.randn(4,3)
tensor([[-0.6569, -0.7337, -0.0028],
        [-0.3938,  0.3223,  0.0497],
        [ 0.0129, -2.7546, -2.2488],
        [ 1.6754, -0.1497,  1.8202]])

When I print the mean:

x.mean()
tensor(-0.2550)

When I print the standard deviation:

x.std()
tensor(1.3225)

So why isn't the mean 0 and the standard deviation 1?

Bonus question: How do I generate a random tensor that always has a mean of 0?

Output: It would be a big coincidence that a finite sample of the distribution has exactly the same mean and exactly the same standard deviation. It is to be expected that the more numbers you generate, the closer the mean and deviation of the sample approaches the "true" mean and deviation of the distribution.


Instruction: How can I build an LSTM AutoEncoder with PyTorch?

Input:
I have my data as a DataFrame:

       dOpen     dHigh      dLow    dClose   dVolume  day_of_week_0  day_of_week_1  ...  month_6  month_7  month_8  month_9  month_10  month_11  month_12
639 -0.002498 -0.000278 -0.005576 -0.002228 -0.002229              0              0  ...        0        0        1        0         0         0         0
640 -0.004174 -0.005275 -0.005607 -0.005583 -0.005584              0              0  ...        0        0        1        0         0         0         0
641 -0.002235  0.003070  0.004511  0.008984  0.008984              1              0  ...        0        0        1        0         0         0         0
642  0.006161 -0.000278 -0.000281 -0.001948 -0.001948              0              1  ...        0        0        1        0         0         0         0
643 -0.002505  0.001113  0.005053  0.002788  0.002788              0              0  ...        0        0        1        0         0         0         0
644  0.004185  0.000556 -0.000559 -0.001668 -0.001668              0              0  ...        0        0        1        0         0         0         0
645  0.002779  0.003056  0.003913  0.001114  0.001114              0              0  ...        0        0        1        0         0         0         0
646  0.000277  0.004155 -0.002227 -0.002782 -0.002782              1              0  ...        0        0        1        0         0         0         0
647 -0.005540 -0.007448 -0.003348  0.001953  0.001953              0              1  ...        0        0        1        0         0         0         0
648  0.001393 -0.000278  0.001960 -0.003619 -0.003619              0              0  ...        0        0        1        0         0         0         0

My input will be 10 rows (already one-hot encoded). I want to create an n-dimensional auto encoded representation. So as I understand it, my input and output should be the same.

I've seen some examples to construct this, but am still stuck on the first step. Is my training data just a lot of those samples as to make a matrix? What then?

I apologize for the general nature of the question. Any questions, just ask and I will clarify in the comments.

Thank you.

Output:
It isn't quite clear from the question what you are trying to achieve. Based on what you wrote you want to create an autoencoder with the same input and output and that doesn't quite make sense to me when I see your data set. In the common case, the encoder part of the autoencoder creates a model which, based on a large set of input features produces a small output vector and decoder is performing an inverse operation of reconstruction of the plausible input features based on the full set of output and input features. A result of using an autoencoder is enhanced (in some meaning, like with noise removed, etc) input.

You can find a few examples here with the 3rd use case providing code for the sequence data, learning random number generation model. Here is another example, which looks closer to your application. A sequential model is constructed to encode a large data set with information loss. If that is what you are trying to achieve, you'll find the code there.

If the goal is a sequence prediction (like future stock prices), this and that example seem to be more appropriate as you likely only want to predict a handful of values in your data sequence (say dHigh and dLow) and you don't need to predict day_of_week_n or the month_n (even though that part of autoencoder model probably will train much more reliable as the pattern is pretty clear). This approach will allow you to predict a single consequent output feature value (tomorrow's dHigh and dLow)

If you want to predict a sequence of future outputs you can use a sequence of outputs, rather than a single one in your model.

In general, the structure of inputs and outputs is totally up to you


Instruction: Can you reverse a PyTorch neural network and activate the inputs from the outputs?

Input:
Can we activate the outputs of a NN to gain insight into how the neurons are connected to input features?

If I take a basic NN example from the PyTorch tutorials. Here is an example of a f(x,y) training example.

import torch

N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(500):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad

After I've finished training the network to predict y from x inputs. Is it possible to reverse the trained NN so that it can now predict x from y inputs?

I don't expect y to match the original inputs that trained the y outputs. So I expect to see what features the model activates on to match x and y.

If it is possible, then how do I rearrange the Sequential model without breaking all the weights and connections?

Output:
It is possible but only for very special cases. For a feed-forward network (Sequential) each of the layers needs to be reversible; that means the following arguments apply to each layer separately. The transformation associated with one layer is y = activation(W*x + b) where W is the weight matrix and b the bias vector. In order to solve for x we need to perform the following steps:

Reverse activation; not all activation functions have an inverse though. For example the ReLU function does not have an inverse on (-inf, 0). If we used tanh on the other hand we can use its inverse which is 0.5 * log((1 + x) / (1 - x)).
Solve W*x = inverse_activation(y) - b for x; for a unique solution to exist W must have similar row and column rank and det(W) must be non-zero. We can control the former by choosing a specific network architecture while the latter depends on the training process.

So for a neural network to be reversible it must have a very specific architecture: all layers must have the same number of input and output neurons (i.e. square weight matrices) and the activation functions all need to be invertible.

Code: Using PyTorch we will have to do the inversion of the network manually, both in terms of solving the system of linear equations as well as finding the inverse activation function. Consider the following example of a 1-layer neural network (since the steps apply to each layer separately extending this to more than 1 layer is trivial):

import torch

N = 10  # number of samples
n = 3   # number of neurons per layer

x = torch.randn(N, n)

model = torch.nn.Sequential(
    torch.nn.Linear(n, n), torch.nn.Tanh()
)

y = model(x)

z = y  # use 'z' for the reverse result, start with the model's output 'y'.
for step in list(model.children())[::-1]:
    if isinstance(step, torch.nn.Linear):
        z = z - step.bias[None, ...]
        z = z[..., None]  # 'torch.solve' requires N column vectors (i.e. shape (N, n, 1)).
        z = torch.solve(z, step.weight)[0]
        z = torch.squeeze(z)  # remove the extra dimension that we've added for 'torch.solve'.
    elif isinstance(step, torch.nn.Tanh):
        z = 0.5 * torch.log((1 + z) / (1 - z))

print('Agreement between x and z: ', torch.dist(x, z))


Instruction: Pytorch: [TypeError: __init__() takes 1 positional argument but 2 were given]

Input:
I searched StackOverflow and visited other websites for help, but I can´t find a solution to my problem.
I will leave the whole code to make it understandable for you. It´s about 110 lines, written with PyTorch.

Each time, I will compile and calculate a prediction, this error-code will show up:

Traceback (most recent call last):
File "/Users/MacBookPro/Dropbox/01 GST h_da Privat/BA/06_KNN/PyTorchV1/BesucherV5.py", line 108, in <module>
result = Network(test_exp).data[0][0].item()
TypeError: __init__() takes 1 positional argument but 2 were given

I know, other users had this too, but none of their solutions helped me out. I guess the mistake is either in my class "Network" or in variable "result".
I hope that someone of you had this problem and know how to fix it or can help me in a different way.

Short information about the Dataset:

My Dataset has 10 columns and gets splitted into two sets. X and Y. X has 9 columns, Y just one. These are then used to train the network.

Thank you in advance!

Kind regards
Christian Richter

My Code:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.optim as optim
import pandas as pd

### Dataset ###

dataset = pd.read_csv('./data/train_data_csv.csv')

x_temp = dataset.iloc[:, :-1].values

print(x_temp)
print()
print(x_temp.size)
print()

y_temp = dataset.iloc[:, 9:].values

print(y_temp)
print()
print(y_temp.size)
print()

x_train_tensor = torch.FloatTensor(x_temp)
y_train_tensor = torch.FloatTensor(y_temp)

### Network Architecture ###

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.linear1 = nn.Linear(9, 9)  #10 Input-Neurons, 10 Output-Neurons, Linearer Layer
        self.linear2 = nn.Linear(9, 1)

    def forward(self, x):
        pax_predict = F.relu(self.linear1(x))
        pax_predict = self.linear2(x)
        return pax_predict

    def num_flat_features(self, pax_predict):
        size = pax_predict.size()[1:]
        num = 1
        for i in size:
            num *= i
        return num

network = Network()
print(network)

criterion = nn.MSELoss()
target = Variable(y_train_tensor)
optimizer = torch.optim.SGD(network.parameters(), lr=0.0001)

### Training

for epoch in range(200):
    input = Variable(x_train_tensor)
    y_pred = network(input)

    loss = criterion(y_pred, target)

    optimizer.zero_grad()

    loss.backward()
    optimizer.step()

test_exp = torch.Tensor([[40116]])

result = Network(test_exp).data[0][0].item()

print('Result is: ', result)

Output:
The problem is quite simple and is in this line, I suppose:

result = Network(test_exp).data[0][0].item()

Here you should use network (the object) instead of Network (the class). As you defined, Network takes only 1 argument (i.e., self), but you are passing 2: self and test_exp.

Perhaps if you had chosen another name for your object (e.g., net), you'd have spotted this error more easily. Take that into consideration :)

And please, always post the full traceback.


Instruction: Algorithim of how Conv2d is implemented in PyTorch

Input:
I am working on an inference model of a pytorch onnx model which is why this question is being asked.

Assume, I have a image with dimensions 32 x 32 x 3 (CIFAR-10 dataset). I pass it through a Conv2d with dimensions : 3 x 192 x 5 x 5. The command I used is: Conv2d(3, 192, kernel_size=5, stride=1, padding=2)

Using the formula (stated here for reference pg12 https://arxiv.org/pdf/1603.07285.pdf) I should be getting an output image with dimensions 28 x 28 x 192 (input - kernel + 1 = 32 - 5 + 1).

Question is how has PyTorch implemented this 4d tensor 3 x 192 x 5 x 5 to get me an output of 28 x 28 x 192 ? The layer is a 4d tensor and the input image is a 2d one.

How is the kernel (5x5) spread in the image matrix 32 x 32 x 3 ? What does the kernel convolve with first -> 3 x 192  or 32 x 32?

Note : I have understood the 2d aspects of things. I am asking the above questions in 3 or more.

Output:
The input to Conv2d is a tensor of shape (N, C_in, H_in, W_in) and the output is of shape (N, C_out, H_out, W_out), where N is the batch size (number of images), C is the number of channels, H is the height and W is the width. The output height and width H_out, W_out are computed as follows (ignoring the dilation):

H_out = (H_in + 2*padding[0] - kernel_size[0]) / stride[0] + 1
W_out = (W_in + 2*padding[1] - kernel_size[1]) / stride[1] + 1

See cs231n for an explanation of how this formulas were obtained.

In your example N=1, H_in = 32, W_in = 32, C_in = 3, kernel_size = (5, 5), strides = (1, 1), padding = (0, 0), giving H_out = 28, W_out = 28.

The C_out=192 means that there are 192 different filters, each of shape (C_in, kernel_size[0], kernel_size[1]) = (3, 5, 5). Each filter independently performs convolution with the input image resulting in a 2D tensor of shape (H_out, W_out) = (28, 28), and since there are C_out = 192 filters and N = 1 images, the final output is of shape (N, C_out, H_out, W_out) = (1, 192, 28, 28).

To understand how exactly the convolution is performed see the convolution demo.


Instruction: element wise multiplication of 2D tensors as layer of neural network in pytorch

Input:
I have a 3D torch tensor with dimension of [Batch_size, n, n] which is the out put of a layer of my network and a constant 2D torch tensor with size of [n, n]. How can I perform element wise multiplication over the batch size which should resulted in a torch tensor with size of [Batch_size, n, n]?
I know it is possible to implement this operation using explicit loop but I am interested in the most efficient way.

Output: One option is that you can expand your weight matrix to have a matching batch dimension (without using any additional memory). E.g. twoDTensor.expand((batch_size, n, n)) returns the same underlying data, but representing a 3D tensor. You can see that the stride for the batch dim is zero.


Instruction: Torch installation results in not supported wheel on this platform

Input:
Tried running pip3 install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html first, taken from PyTorch website
which resulted in
No matching distribution found for torch===1.4.0
 and  Could not find a version that satisfies the requirement torch===1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

Finally downloaded the .whl file from downloads page and tried installing locally like so 'C:\Users\Raf\AppData\Local\Programs\Python\Python38\Scripts\pip.exe' install torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl
after which I get torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.

using 64 Python 3.8, on 64 bit Windows

Output:
  using 64 Python 3.8

but you downloaded the cp37 whl which is for python 3.7. There is currently no whl file available for python 3.8. So either install from source (probably not recommended), install a different python version or create a virtual environment with python 3.7

Update

There is now: https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp38-cp38-win_amd64.whl


Instruction: RuntimeError with copy_() and expand() - Unsupported operation: more than one element of the written-to tensor refers to a single memory location

Input:
I'm migrating a repository from Pytorch Nightly 1.0.0 to 1.3.1. Stripping the unnecessary details, it is basically performing the following sequence of operations:

mu = torch.tensor(0.005)
bar = torch.eye(5, 5)
foo = torch.eye(5).expand(5, 5, 5)

# update
bar.copy_(mu * bar)  # ok!
foo.copy_(mu * foo)  # error

bar.copy_(mu * bar) works, while when I try to foo.copy_() the result, it gives the following error:

  RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.

Output:
This is because expand() only creates a new view on the existing tensor, thus it doesn't allocate the full memory necessary to receive all the elements from the operation mu * foo, which has more elements than the original tensor foo. You can fix it by either using expand().clone() or repeat(), which will give you the full tensor.

foo = torch.eye(5).expand(5, 5, 5).clone()  # clone gives the full tensor
foo.copy_(mu * foo)  # ok!

albanD suggests that doing expand().clone() might still be faster than repeat().

See here and here for more details about expand() and repeat().


Instruction: PyTorch get indices of value in two-dimensional tensor

Input:
Given the following tensor (or any random tensor with two dimension), I want to get the index of '101':

tens = tensor([[  101,   146,  1176, 21806,  1116,  1105, 18621,   119,   102,     0,
             0,     0,     0],
        [  101,  1192,  1132,  1136,  1184,   146,  1354,  1128,  1127,   117,
          1463,   119,   102],
        [  101,  6816,  1905,  1132, 14918,   119,   102,     0,     0,     0,
             0,     0,     0]])

From the related answers I know that I can do something like this:

idxs = torch.tensor([(i == 101).nonzero() for i in tens])

But this seems messy and potentially quite slow. Is there a better way to do this that is fast and more torch-y?

Related questions discussing only one-dimensional tensor:

How Pytorch Tensor get the index of specific value
How Pytorch Tensor get the index of elements?

Output:
How about (tens == 101).nonzero()[:, 1]

In [20]: from torch import tensor

In [21]: tens = torch.tensor([[  101,   146,  1176, 21806,  1116,  1105, 18621,   119,   102,     0,
    ...:              0,     0,     0],
    ...:         [  101,  1192,  1132,  1136,  1184,   146,  1354,  1128,  1127,   117,
    ...:           1463,   119,   102],
    ...:         [  101,  6816,  1905,  1132, 14918,   119,   102,     0,     0,     0,
    ...:              0,     0,     0]])

In [22]: (tens == 101).nonzero()[:, 1]
Out[22]: tensor([0, 0, 0])


Instruction: modification from classification loss to regression loss

Input:
General
I am following this repo for object detection
https://github.com/yhenon/pytorch-retinanet
Motivation
Object detection networks usually perform 2 tasks.For every object in the image output a class confidence score and bounding box regression score.For my task along with these 2 outputs, for every object i want to output one more regression score which will be between 0-5.
Problem statement
The approach i have taken is that since the network already does classifcation i thought i would modify some of the parts and make it a regression loss.The loss used in the repo mentioned above is focal loss.Part of what the classification loss looks like is described below, i would like to modify this to be a regression loss.
     targets = torch.ones(classification.shape) * -1#Here classification shape is = torch.Size([114048, 1])
    targets = targets.cuda()
    targets[torch.lt(IoU_max, 0.4), :] = 0#Iou_max is a tensor of shape = torch.Size([114048]) which looks like tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:0')
    positive_indices = torch.ge(IoU_max, 0.5)
    num_positive_anchors = positive_indices.sum()
    assigned_annotations = bbox_annotation[IoU_argmax, :]#Here bbox_annotation has the shape torch.Size([6, 6]) and IoU_argmax has the shape torch.Size([114048])
    targets[positive_indices, :] = 0#Here positive indices has the shape torch.Size([114048])
    targets[positive_indices, assigned_annotations[positive_indices, 4].long()] = 1
    alpha_factor = torch.ones(targets.shape).cuda() * alpha#here alpha is 0.25
    alpha_factor = torch.where(torch.eq(targets, 1.), alpha_factor, 1. - alpha_factor)
    focal_weight = torch.where(torch.eq(targets, 1.), 1. - classification, classification)
    focal_weight = alpha_factor * torch.pow(focal_weight, gamma)#here gamma is 2.0

    bce = -(targets * torch.log(classification) + (1.0 - targets) * torch.log(1.0 - classification))
    cls_loss = focal_weight * bce
    cls_loss = torch.where(torch.ne(targets, -1.0), cls_loss,torch.zeros(cls_loss.shape).cuda())
  classification_losses.append(cls_loss.sum()/torch.clamp(num_positive_anchors.float(), min=1.0))

So i tried modifying some of the bits described above.But even though the code does not break This does not learn anything.Here erosion is just a tensor but erosion score is something we want to regress.
    targets = torch.ones(erosion.shape) * -1#Here erosion shape is torch.Size([114048, 1])
    targets = targets.cuda()
    targets[torch.lt(IoU_max, 0.4), :] = 0#This part remains the same as above
    positive_indices = torch.ge(IoU_max, 0.5)
    num_positive_anchors = positive_indices.sum()
    assigned_annotations = bbox_annotation[IoU_argmax, :]
    targets[positive_indices, :] = 0
    targets[assigned_annotations[positive_indices, 5].long()] = 1#Here i have indexed along the 5th dimension because the 5th dimension contains annotations for erosion.
    criterion = nn.MSELoss()#This part is where i use the mean squared loss for regression.
    loss = torch.sqrt(criterion(targets, erosion))
    erosion_losses.append(loss)

This will not throw an error , but it does not learn anything either.
Could someone please help me with the correct formulation of this loss to perform regression.
More info
        assigned_annotations_erosion = assigned_annotations[positive_indices, 5]
        erosion_pred =  erosion[positive_indices].view(-1)
        criterion = nn.MSELoss()#
        loss = torch.sqrt(criterion(assigned_annotations_erosion, erosion_pred))
        loss *torch.tensor(5.0).cuda()
        # import pdb;pdb.set_trace()
        erosion_losses.append(loss)

Output:
If I understand it correctly you want to add another regression parameter which should have values in [0-5].

Instead of trying to change the classification part you can just add it to the box regression part. So you add one more parameter to predict for every anchor.

Here is the loss calculation for the regression parameters. If you added one parameter to every anchor (probably by increasing the filter count in the last layer) you have to "extract" it here like with the other parameters. You also probably want to think about the activation for the value e.g. sigmoid so you get something between 0 and 1, you can rescale to 0-5 if you want.

After line 107 you want to add something like this

anchor_erosion_pi = anchor_erosion[positive_indices]

Then you have to get the gt somehow, maybe rescale it to 0-1, so you can just use the sigmoid activated output for this value.
Then you have to set the target. This is just:

targets_erosion = gt_erosion

If you are not dependent on the anchor which seems like it is the case.
After that the targets are stacked, where you have to add your new target value and a weighting is added, you have to decide and try out if you need a weighting here. I would probably add start by adding 0.2 or something to the tensor. Maybe have a look inside the paper how these other weightings are chosen.

targets = targets/torch.Tensor([[0.1, 0.1, 0.2, 0.2, 0.2]]).cuda()

After that the Smooth L1 loss is calculated where you probably don't need to change anything.

there is probably more you have to do. Just look at the code, see where the other parameters are and try to add your parameter to it.


Instruction: Autoencoder MaxUnpool2d missing 'Indices' argument

Input:
The following model returns the error:
TypeError: forward() missing 1 required positional argument: 'indices'

I've exhausted many online examples and they all look similar to my code. My maxpool layer returns both the input and the indices for the unpool layer. Any ideas on what's wrong?

class autoencoder(nn.Module):
def __init__(self):
    super(autoencoder, self).__init__()
    self.encoder = nn.Sequential(
        ...
        nn.MaxPool2d(2, stride=1, return_indices=True)
    )
    self.decoder = nn.Sequential(
        nn.MaxUnpool2d(2, stride=1),
        ...
    )

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

Output:
Similar to the question here, the solution seems to be to separate the maxunpool layer from the decoder and explicitly pass its required parameters. nn.Sequential only takes one parameter.

class SimpleConvAE(nn.Module):
def __init__(self):
    super().__init__()

    # input: batch x 3 x 32 x 32 -> output: batch x 16 x 16 x 16
    self.encoder = nn.Sequential(
        ...
        nn.MaxPool2d(2, stride=2, return_indices=True),
    )

    self.unpool = nn.MaxUnpool2d(2, stride=2, padding=0)

    self.decoder = nn.Sequential(
        ...
    )

def forward(self, x):
    encoded, indices = self.encoder(x)
    out = self.unpool(encoded, indices)
    out = self.decoder(out)
    return (out, encoded)


Instruction: Install specific version of PyTorch to conda environment

Input:
Using Anaconda Navigator I created a new environment for running someone's VAE code off GitHub that uses Python 3.6 and PyTorch 0.4.0. Unfortunately, Anaconda Navigator doesn't give me the option to install an older version of PyTorch on this environment, just the PyTorch version I have currently installed. How do I install PyTorch 0.4.0 only to this new Conda environment I created? If it's possible via Anaconda Navigator, great! But I assume it's going to be done via a Conda command. I definitely don't want to mess up my other environments.

Thanks!

Output:
Just navigate to the conda environment you want to install it, then use

  conda install pytorch=0.4.1 -c pytorch

More details here on how you can install previous PyTorch versions: https://pytorch.org/get-started/previous-versions/

According to this blog, conda navigator does not work, but you can follow it to install pytorch in a conda environment:
https://medium.com/@bryant.kou/how-to-install-pytorch-on-windows-step-by-step-cc4d004adb2a


Instruction: "ImportError: No module named torch" in ROS package

Input:
I've created a ROS package in which there are some python scripts. The python scripts are based on torch module (it is an inference code of pytorch models). when I try to run my scripts it gives me an error:
ImportError: No module named torch

To install ROS, I used  the instruction of ROS wiki. To validate my installation, I followed the sample code of ROS (a simple publisher and subscriber) and it works well. My sys-info is:

  python: 3.6.9
  torch: 1.1.0
  torchvision: 0.3.0
  OS kernel: Linux 4.15.0-74-generic
  OS distribution: Ubuntu 18.04.3

I want to import the below libs:

import torch
from cv_bridge import CvBridge
import cv2
import os
import numpy as np
from torch.autograd import Variable
from torchvision import transforms
import torch.nn.functional as F
import torch._utils
import time
from PIL import Image

My CMake file is as below:

cmake_minimum_required(VERSION 2.8.3)
project(inference_pytorch)

## Compile as C++11, supported in ROS Kinetic and newer
# add_compile_options(-std=c++11)

set(Torch_DIR ".local/lib/python3.6/site-packages/torch/share/cmake/Torch")

find_package(catkin REQUIRED COMPONENTS
  roscpp
  rospy
  std_msgs
  cv_bridge
)
find_package(Torch REQUIRED)
find_package(OpenCV 3 REQUIRED)
## System dependencies are found with CMake's conventions
# find_package(Boost REQUIRED COMPONENTS system)

###################################
## catkin specific configuration ##
###################################
catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES inference_pytorch
#  CATKIN_DEPENDS roscpp rospy std_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
  ${OPENCV_INCLUDE_DIRS}
  ${Torch_INSTALL_INCLUDE}
  ${Torch_DIR}
)

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

install(PROGRAMS
   scripts/inference_test.py
   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
 )

 catkin_install_python(PROGRAMS scripts/inference_test.py
                      DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION})

link_directories(
  ${Torch_INSTALL_LIB}
)

Now, how should I edit the CMAKE file to add the mentioned libs to my ROS package?

Output: Finally, I found out that the installed ROS is working with python2 so I installed the torch with pip2. Also, set the torch path in the cmake file. Now it works!


Instruction: how to create a pytorch NN with 2 hidden layer with nn.Sequential?

Input:
how to add an another hidden layer in this 1 layer model? :

model = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(D_in, H)),
                                    ('Tanh', nn.Tanh()),
                                    ('fc2', nn.Linear(H, D_out))]))

Output:
You an do this by separating the original hidden layer into two or adding another hidden layer:

model = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(D_in, H1)),
                                    ('act1', nn.Tanh()),
                                    ('fc2', nn.Linear(H1, H2)),
                                    ('act2', nn.Tanh()),
                                    ('fc3', nn.Linear(H2, D_out)) ]))

The only thing you got to do is take the 1st hidden layer (H1) as input to the next Linear layer which will output to another hidden layer (H2) then we add another Tanh activation layer and then lastly, we add a Linear layer which takes the H2 layer as input and the outputs to the number of output nodes.


Instruction: Understanding pytorch autograd

Input:
I am trying to understand how pytorch autograd works. If I have functions y = 2x and z = y**2, if I do normal differentiation, I get dz/dx at x = 1 as 8 (dz/dx = dz/dy * dy/dx = 2y*2 = 2(2x)*2 = 8x). Or, z = (2x)**2 = 4x^2 and dz/dx = 8x, so at x = 1, it is 8.

If I do the same with pytorch autograd, I get 4

x = torch.ones(1,requires_grad=True)
y = 2*x
z = y**2
x.backward(z)
print(x.grad)

which prints

tensor([4.])

where am I going wrong?

Output:
You're using Tensor.backward wrong. To get the result you asked for you should use

x = torch.ones(1,requires_grad=True)
y = 2*x
z = y**2
z.backward()  # <-- fixed
print(x.grad)

The call to z.backward() invokes the back-propagation algorithm, starting at z and working back to each leaf node in the computation graph. In this case x is the only leaf node. After calling z.backward() the computation graph is reset and the .grad member of each leaf node is updated with the gradient of z with respect to the leaf node (in this case dz/dx).

What's actually happening in your original code? Well, what you've done is apply back-propagation starting at x. With no arguments x.backward() would simply result in x.grad being set to 1 since dx/dx = 1. The additional argument (gradient) is effectively a scale to apply to the resulting gradient. In this case z=4 so you get x.grad = z * dx/dx = 4 * 1 = 4. If interested, you can check out this for more information on what the gradient argument does.


Instruction: Load custom data from folder in dir Pytorch

Input:
I am getting my hands dirty with Pytorch and I am trying to do what is apparently the hardest part in deep learning-> LOADING MY CUSTOM DATASET AND RUNNING THE PROGRAM<--    The problem is this " too many values to unpack (expected 2)" also I think I am loading the data wrong. Can someone please show me how to do this. How to use the Dataloader user one's own data.

import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms

# Hyper parameters
num_epochs = 20
batchsize = 100
lr = 0.001

EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "ImageFolder/images/train/"
TEST_DATA_PATH = "ImageFolder/images/test/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)

# Data Loader (Input Pipeline)
train_loader = torch.utils.data.DataLoader(dataset=TRAIN_DATA_PATH,
                                           batch_size=batchsize,
                                           shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=TEST_DATA_PATH,
                                          batch_size=batchsize,
                                          shuffle=False)

# CNN model
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conlayer1 = nn.Sequential(
            nn.Conv2d(1,6,3),
            nn.Sigmoid(),
            nn.MaxPool2d(2))
        self.conlayer2 = nn.Sequential(
            nn.Conv2d(6,16,3),
            nn.Sigmoid(),
            nn.MaxPool2d(2))
        self.fc = nn.Sequential(
            nn.Linear(400,120),
            nn.Relu(),
            nn.Linear(120,84),
            nn.Relu(),
            nn.Linear(84,10))

    def forward(self, x):
        out = self.conlayer1(x)
        out = self.conlayer2(out)
        out = out.view(out.size(0),-1)
        out = self.fc(out)
        return out

cnn = CNN()

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(cnn.parameters(), lr=lr)

# Train the Model
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = cnn(images)
        loss = criterion(outputs,labels)
        loss.backward()
        optimizer.step()

        if (i+1)%100 == 0:
            print('Epoch [%d/%d], Iter [%d/%d] Loss: %.4f' %
                  (epoch+1,num_epochs,i+1,len(train_dataset)//batchsize,loss.data[0]))

# Test the Model
cnn.eval()  # Change model to 'eval' mode (BN uses moving mean/var)
correct = 0
total = 0
for images, labels in test_loader:
    outputs = cnn(images)
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()
print('Test Accuracy of the model on test images: %.6f%%' % (100.0*correct/total))

#Save the Trained Model
torch.save(cnn.state_dict(),'cnn.pkl')

Output:
You just provided a string to dataloader:

dataset=TRAIN_DATA_PATH

Maybe use train_data, which is a data generator based on the filepath you provided


Instruction: What value should I set for `nn.Linear(1024, 256)` when the dimensions of the dataset are [64 x 25088]?

Input:
I keep getting an error in pytorch that the size I have entered is wrong:

  RuntimeError: size mismatch, m1: [64 x 25088], m2: [1024 x 256] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorMathBlas.cu:249

If I set nn.Linear to 64 x 25088, I get a similar error:

  RuntimeError: size mismatch, m1: [64 x 25088], m2: [64 x 25088] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorMathBlas.cu:249

What is going on?

This is the code that is generating the error:

model = models.vgg16(pretrained=True)

# Freeze parameters so we don't backprop through them
for param in model.parameters():
    param.requires_grad = False

model.classifier = nn.Sequential(nn.Linear(64, 25088),
                                 nn.ReLU(),
                                 nn.Dropout(0.2),
                                 nn.Linear(256, 2),
                                 nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()

# Only train the classifier parameters, feature parameters are frozen
optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)

# Use GPU if it's available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device);

epochs = 1
steps = 0
running_loss = 0
print_every = 5

with active_session():
    for epoch in range(epochs):
        for inputs, labels in trainloader:
            steps += 1
            # Move input and label tensors to the default device
            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            logps = model.forward(inputs)
            loss = criterion(logps, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()

            if steps % print_every == 0:
                test_loss = 0
                accuracy = 0
                model.eval()
                with torch.no_grad():
                    for inputs, labels in testloader:
                        inputs, labels = inputs.to(device), labels.to(device)
                        logps = model.forward(inputs)
                        batch_loss = criterion(logps, labels)

                        test_loss += batch_loss.item()

                        # Calculate accuracy
                        ps = torch.exp(logps)
                        top_p, top_class = ps.topk(1, dim=1)
                        equals = top_class == labels.view(*top_class.shape)
                        accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

                print(f"Epoch {epoch+1}/{epochs}.. "
                      f"Train loss: {running_loss/print_every:.3f}.. "
                      f"Test loss: {test_loss/len(testloader):.3f}.. "
                      f"Test accuracy: {accuracy/len(testloader):.3f}")
                running_loss = 0
                model.train()

Output:
In nn.Linear(), the first parameter is the size of the input, the second parameter is the size of the output. If your data'size is 64 x 25088, then you need to set nn.Linear(64*25088, output_size).

However this layer will be huge, so you should try to make your data smaller.


Instruction: Pytorch gradient computation

Input:
I am trying to figure out how exactly the function grad works.
This is my code :

A = torch.Tensor(2, 3).uniform_(-1, 1).requires_grad_()
B = torch.Tensor(3, 1).uniform_(-1, 1).requires_grad_()
o = torch.matmul(A,B)
print("A : ", A)
print("B : ", B)
do_dinput = torch.autograd.grad(o, A, grad_outputs=torch.ones(2, 1))
print('Size do/dA :', (do_dinput[0].size()))

I was expecting torch.Size([1, 3]) to be printed, because the derivative of AB w.r.t A is B^T. However, I got torch.Size([2, 3]).

Is there something wrong with my code, or am I missing something ?

Output:
What you get is the grad starting from o back-propagated through the computational graph to A. In the end you have the grad for every value in A.

It is the same as doing the following

A = torch.Tensor(2, 3).uniform_(-1, 1).requires_grad_()
B = torch.Tensor(3, 1).uniform_(-1, 1).requires_grad_()
o = torch.matmul(A,B).sum()
o.backward()
print("A : ", A)
print("B : ", B)
print(A.grad)

A.grad in this example and do_dinputare the same. If you look at the grad tensor it is just B^T in both rows.

To make it a bit more visual what happens. We have A and B as input and some function f(...) which takes all values from A and B as input and calculates some value. In this case the function is sum(AB).
Note: The summation doesn't change the gradients in any way.

A = x_1 x_2 x_3
    x_4 x_5 x_6
B = y_1
    y_2
    y_3
o = x_1 * y_1 + x_2 * y_2 + x_3 * y_3
    x_4 * y_1 + x_5 * y_2 + x_6 * y_3
f(x_1,...,x_6, y_1, y_2, y_3) = x_1 * y_1 + x_2 * y_2 + x_3 * y_3 + x_4 * y_1 + x_5 * y_2 + x_6 * y_3

If you now calculate the gradient you derive f(...) in respect to all variables. So for x_1 it would be

df/dx_1 = y_1

So the grad value in A for x_1 is equal to y_1. This is done for all other values. So in the end you get a grad value for all entries in A and B.

It works the same in your example you just skip the summing of the tensor.


Instruction: Python modify instance attribute with instance method

Input:
I am trying to use an instance method to modify one of the instance's attribute like this:

from torch.optim import SGD
from typing import Dict

class ChangeRateSgd(SGD):
    def __init__(self, params, lr: float, lr_change_instructions: Dict):
        super().__init__(params, lr)
        self.lr_change_instructions = lr_change_instructions

    def change_update_rate(self, input_epoch):
        update_mapping = self.lr_change_instructions
        if input_epoch in update_mapping.keys():
            new_lr = self.lr_change_instructions[input_epoch]
            self.lr = new_lr

However, my IDE flags the line self.lr = new_lr as not being ideal coding practice, with the warning Instance attribute lr defined outside __init__. What is the best way to do what I am trying to do with this instance method?

Output:
Try the below code , you need to define the lr in init method and access that any other methods.

from torch.optim import SGD
from typing import Dict

class ChangeRateSgd(SGD):
    def __init__(self, params, lr: float, lr_change_instructions: Dict):
        super().__init__(params, lr)
        self.lr =None
        self.lr_change_instructions = lr_change_instructions

    def change_update_rate(self, input_epoch):
        update_mapping = self.lr_change_instructions
        if input_epoch in update_mapping.keys():
            new_lr = self.lr_change_instructions[input_epoch]
            self.lr = new_lr


Instruction: Why do we need pack_padded_sequence() when we have pack_sequence()?

Input: After reading the answers to this question I'm still a bit confused about the whole PackedSequence object thing. As I understand it, this is an object optimized for parallel processing of variable sized sequences in recurrent models, a problem to which zero padding is one [imperfect] solution. It seems that given a PackedSequence object, a Pytorch RNN will process each sequence in the batch to its end, and not continue to process the padding. So why is padding needed here? Why are there both a pack_padded_sequence() and pack_sequence() methods?

Output:
Mostly for historical reasons; torch.nn.pack_padded_sequence() was created before torch.nn.pack_sequence() (the later appeared in 0.4.0 for the first time if I see correctly) and I suppose there was no reason to remove this functionality and break backward compatibility.

Furthermore, it's not always clear what's the best/fastest way to pad your input and it highly varies on data you are using. When data was somehow padded beforehand (e.g. your data was pre-padded and provided to you like that) it is faster to use pack_padded_sequence() (see source code of pack_sequence, it's calculating length of each data point for you and calls pad_sequence followed by pack_padded_sequence internally). Arguably pad_packed_sequence is rarely of use right now though.

Lastly, please notice enforce_sorted argument provided since 1.2.0 version for both of those functions. Not so long ago users had to sort their data (or batch) with the longest sequence first and shortest last, now it can be done internally when this parameter is set to False.


Instruction: Gradient calculation in A2C

Input:
In A2C, the actor and critic algorithm, the weights are updated via equations:

delta = TD Error and

theta = theta + alpha*delta*[Grad(log(PI(a|s,theta)))] and

w = w + beta*delta*[Grad(V(s,w))]

So my question is, when using neural networks to implement this,

how are the gradients calculated and
am I correct that the weights are updated via the optimization fmethods in TensorFlow or PyTorch?

Thanks, Jon

Output:
I'm not quite clear what you mean to update with w, but I'll answer the question for theta assuming it is denoting the parameters for the actor model.

1) Gradients can be calculated in a variety of ways, but if focussing on PyTorch, you can call .backward() on f(x)=alpha
* delta * log(PI(a|s,theta), which will df/dx for every parameter x that is chained to f(x) via autograd.

2) You are indeed correct that the weights are updated via the optimization methods in Pytorch (specifically, autograd). However, in order to complete the optimization step, you must call torch.optim.step with whatever optimizer you would like to use on the network's parameters (e.g. weights and biases).


Instruction: L1 regularization neural network in Pytorch does not yield sparse solution

Input:
I'm implementing a neural network with l1 regularization in pytorch. I directly add the l1 norm penalty to the loss function. The framework is basically the same as Lack of Sparse Solution with L1 Regularization in Pytorch, however, the solution is not sparse no matter how I adjust the tuning parameter. How do I make the solution sparse?

My code is pasted below.

class NeuralNet(nn.Module):
    """
    neural network class, with nn api
    """
    def __init__(self, input_size: int, hidden_size: List[int], output_size: int):
        """
        initialization function
        @param input_size: input data dimension
        @param hidden_size: list of hidden layer sizes, arbitrary length
        @param output_size: output data dimension
        """
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        """layers"""
        self.input = nn.Linear(self.input_size, self.hidden_size[0], bias=False)
        self.hiddens = nn.ModuleList([
            nn.Linear(self.hidden_size[h], self.hidden_size[h + 1]) for h in range(len(self.hidden_size) - 1)])
        self.output = nn.Linear(hidden_size[-1], output_size)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        forward propagation process, required by the nn.Module class
        @param x: the input data
        @return: the output from neural network
        """
        x = self.input(x)
        x = self.relu(x)
        for hidden in self.hiddens:
            x = hidden(x)
            x = self.relu(x)
        x = self.output(x)
        x = self.softmax(x)
        return x

def estimate(self, x, y, l1: bool = False, lam: float = None, learning_rate: float = 0.1, batch_size: int =
    32, epochs: int = 50):
    """
    estimates the neural network model
    @param x: training data
    @param y: training label
    @param l1: whether to use l1 norm regularization
    @param lam: tuning parameter
    @param learning_rate: learning rate
    @param batch_size: batch size
    @param epochs: number of epochs
    @return: null
    """
    input_size = x.shape[1]
    hidden_size = [50, 30, 10]
    output_size = 2
    model = NeuralNet(input_size, hidden_size, output_size)
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    trainset = []
    for i in range(x.shape[0]):
        trainset.append([x[i, :], y[i]])
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
    for e in range(epochs):
        running_loss = 0
        for data, label in trainloader:
            input_0 = data.view(data.shape[0], -1)
            optimizer.zero_grad()
            output = model(input_0.float())
            loss = torch.nn.CrossEntropyLoss()(output, label.squeeze(1))
            if l1:
                if lam is None:
                    raise ValueError("lam needs to be specified when l1 is True.")
                else:
                    for w in model.parameters():
                        if w.dim() > 1:
                            loss = loss + lam * w.norm(1)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

Doesn't pytorch support sparse solution?

Output:
The l1 regularization in pytorch does not handle the cases when the parameter will be shrunk to zero. You have to manually apply some soft-thresholding function like

$$s_{\lambda}(z)=sign(z)*(|z|-\lambda)_+$$

after each iteration to filter those parameters which are zero until convergence.


Instruction: Why does the parameters saved in the checkpoint are different from the ones in the fused model?

Input:
I trained a QAT (Quantization Aware Training) based model in Pytorch, the training went on smoothly. However when I tried to load the weights into the fused model and run a test on widerface dataset I faced lots of errors:

(base) marian@u04-2:/mnt/s3user/Pytorch_Retinaface_quantized# python test_widerface.py --trained_model ./weights/mobilenet0.25_Final_quantized.pth --network mobile0.25layers:
Loading pretrained model from ./weights/mobilenet0.25_Final_quantized.pth
remove prefix 'module.'
Missing keys:235
Unused checkpoint keys:171
Used keys:65
Traceback (most recent call last):
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/root/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/root/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/root/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/f3user/Pytorch_Retinaface_quantized/test_widerface.py", line 114, in <module>
    net = load_model(net, args.trained_model, args.cpu)
  File "/mnt/f3user/Pytorch_Retinaface_quantized/test_widerface.py", line 95, in load_model
    model.load_state_dict(pretrained_dict, strict=False)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 830, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RetinaFace:
        While copying the parameter named "ssh1.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).
        While copying the parameter named "ssh1.conv5X5_2.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
        While copying the parameter named "ssh1.conv7x7_3.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
        While copying the parameter named "ssh2.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).
        While copying the parameter named "ssh2.conv5X5_2.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
.....

The full list can be found here.
basically the weights cant be found. plus the scale and zero_point which are missing from the fused model.

in case it matters, the following snippet is the actual training loop which was used to train and save the model :

if __name__ == '__main__':
    # train()
    ...
    net = RetinaFace(cfg=cfg)
    print("Printing net...")
    print(net)

    net.fuse_model()
    ...

    net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
    torch.quantization.prepare_qat(net, inplace=True)
    print(f'quantization preparation done.')

    ...

    quantized_model = net
    for i in range(max_epoch):
        net = net.to(device)
        train_one_epoch(net, data_loader, optimizer, criterion, cfg, gamma, i, step_index, device)
        if i in stepvalues:
            step_index += 1
        if i > 3 :
            net.apply(torch.quantization.disable_observer)
        if i > 2 :
            net.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
        net=net.cpu()
        quantized_model = torch.quantization.convert(net.eval(), inplace=False)
        quantized_model.eval()
        # evaluate on test set ?!

    torch.save(net.state_dict(), save_folder + cfg['name'] + '_Final.pth')
    torch.save(quantized_model.state_dict(), save_folder + cfg['name'] + '_Final_quantized.pth')
    #torch.jit.save(torch.jit.script(quantized_model), save_folder + cfg['name'] + '_Final_quantized_jit.pth')

for testing the test_widerface.py is used which can be accessed here
You can view the keys here

Why has this happened? How should this be taken care of?

Update

I checked the name, and created a new state_dict dictionary and inserted the 112 keys that were in both checkpoint and model using the snippet below :

new_state_dict  = {}
checkpoint_state_dict = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
for (ck, cp) in checkpoint_state_dict.items():
    for (mk, mp) in model.state_dict().items():
        kname,kext = os.path.splitext(ck)
        mname,mext = os.path.splitext(mk)
        # check the two parameter and see if they are the same
        # then use models key naming scheme and use checkpoints weights
        if kname+kext == mname+mext or kname+'.0'+kext == mname+mext:
            new_state_dict[mname+mext] = cp
        else:
             if kext in ('.scale','.zero_point'):
                 new_state_dict[ck] = cp

and then use  this new state_dict! yet I'm getting  the ver same exact errors!
meaning errors like  this :

RuntimeError: Error(s) in loading state_dict for RetinaFace:
        While copying the parameter named "ssh1.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).

This is really frustrating and there is no documentation concerning this! I'm completely clueless here.

Output:
I finally found out the cause. The error messages with the form of :

  While copying the parameter named "xxx.weight", whose
  dimensions in the model are torch.Size([yyy]) and whose
  dimensions in the checkpoint are torch.Size([yyy]).

are actually generic messages, only returned when an exception has occured while copying the parameters in question.

Pytorch developers could easily add the actual exception args into this spurious yet unhelpful message, so it could actually help better debug the issue at hand. Anyway, looking at the exception which was by the way :

"copy_" not implemented for \'QInt8'

You'll now know what the actual issue is/was!


Instruction: 'NoneType' object has no attribute 'size' - how to face detection using mtcnn?

Input:
i'm trying to build face recognition in real time in neural network(facenet network) using pytorch and face detection using MTCNN
i've tried this for detecting faces in real time  (from webcam) but doesnt work
read frames then going through mtcnn detector

import cv2
capture = cv2.VideoCapture(0)
while(True):
   ret, frame = capture.read()
   frames_tracked = []
   print('\rTracking frame: {}'.format(i + 1), end='')
   boxes,_ = mtcnn.detect(frame)
   frame_draw = frame.copy()
   draw = ImageDraw.Draw(frame_draw)
   for box in boxes:
       draw.rectangle(box.tolist(), outline=(255, 0, 0), width=6)
       frames_tracked.append(frame_draw.resize((640, 360), Image.BILINEAR))
   d = display.display(frames_tracked[0], display_id=True)
   i = 1
   try:
      while True:
          d.update(frames_tracked[i % len(frames_tracked)])
          i += 1
   except KeyboardInterrupt:
      pass
   if cv2.waitKey('q') == 27:
     break

  capture.release()
  cv2.destroyAllWindows()

but it will rise this error :

this is the entire traceback http://dpaste.com/0HR58RQ

  AttributeError: 'NoneType' object has no attribute 'size'

is there a solution for this problem ? what caused this error? thanks for your advice

Output:
Let's take a look at that error again.

AttributeError: 'NoneType' object has no attribute 'size'

so, somewhere in your code you(or mtcnn) are trying to call size attribute from a None variable. you are passing frame to mtcnn using following command :

 boxes,_ = mtcnn.detect(frame)

this is exactly where you see that error. because you are passing a None variable to mtcnn. in order to prevent it, you can prevent it before calling this method. in other words :

ret, frame = capture.read()
if frame == None:
    continue


Instruction: Define a custom variable pytorch tensor

Input:
I have a problem similar to this one:

There is an input vector of the form (n1, n2), and I want a model where f(n1, n2) = f(n2, n1). I want this model to be linear. For instance a 2x2 matrix that I want to learn. The 2x2 matrix has 4 weights which the training will try to learn. However, the constriction allows a clever reduction of the problem.
f(nvec) = W*nvec, being nvec=(n1, n2).
So, let X = [[0, 1],[1, 0]] be a 2x2 matrix that flips vectors, then f(n2,n1) = W*X*(n1, n2) = f(n1, n2) = W (n1, n2).

Basically, the equality implies that W*X = W, or if W = [[w1, w2], [w3, w4]], then the equality implies W = [[w1, w1], [w3, w3]], i.e. the matrix has only 2 free parameters. How can I define a model like this one in pytorch?

Thanks.

Output:
If you have a special structure to your parameters, and you do not want to use the default parameters stored by the default layers, you can define a layer on your own and use nn.Parameter to store your parameters.

For example,

class CustomLayer(nn.Module):  # layer must be derived from nn.Module class
  def __init__(self):  # you can have arguments here...
    super(CustomLayer, self).__init__()
    self.params = nn.Parameter(data=torch.rand((2, 1), dtype=torch.float), requires_grad=True)

  def forward(self, x):
    w = self.params.repeat(1, 2)  # create a 2x2 matrix from the two parameters
    y = torch.bmm(x, w)
    return y


Instruction: Keeping gradients while rearranging data in a tensor, with pytorch

Input:
I have a scheme where I store a matrix with zeros on the diagonals as a vector. I want to later on optimize over that vector, so I require gradient tracking.
My challenge is to reshape between the two.

I want - for domain specific reasons - keep the order of data in the matrix so that transposed elements of the W matrix next to each other in the vector form.

The size of the W matrix is subject to change, so I start with enumering items in the top-left part of the matrix, and continue outwards.

I have come up with two ways to do this. See code snippet.

import torch
import torch.sparse

w = torch.tensor([10,11,12,13,14,15],requires_grad=True,dtype=torch.float)
i = torch.LongTensor([
    [0, 1,0],
    [1, 0,1],
    [0, 2,2],
    [2, 0,3],
    [1, 2,4],
    [2, 1,5],
])
v = torch.FloatTensor([1,      1,      1 ,1,1,1   ])
reshaper = torch.sparse.FloatTensor(i.t(), v, torch.Size([3,3,6])).to_dense()
W_mat_with_reshaper = reshaper @ w
W_mat_directly = torch.tensor([
  [0,    w[0],  w[2],],
  [w[1],    0,  w[4],],
  [w[3], w[5],     0,],
])
print(W_mat_with_reshaper)
print(W_mat_directly)

and this gives output

tensor([[ 0., 10., 12.],
        [11.,  0., 14.],
        [13., 15.,  0.]], grad_fn=<UnsafeViewBackward>)
tensor([[ 0., 10., 12.],
        [11.,  0., 14.],
        [13., 15.,  0.]])

As you can see, the direct way to reshape the vector into a matrix does not have a grad function, but the multiply-with-a-reshaper-tensor does. Creating the reshaper-tensor seems like it will be a hassle, but on the other hand, manually writing the matrix for is also infeasible.

Is there a way to do arbitrary reshapes in pytorch that keeps grack of gradients?

Output:
Instead of constructing W_mat_directly from the elements of w, try assigning w into W:

W_mat_directly = torch.zeros((3, 3), dtype=w.dtype)
W_mat_directly[(0, 0, 1, 1, 2, 2), (1, 2, 0, 2, 0, 1)] = w

You'll get

tensor([[ 0., 10., 11.],
        [12.,  0., 13.],
        [14., 15.,  0.]], grad_fn=<IndexPutBackward>)


Instruction: Pytorch dataloader for sentences

Input:
I have collected a small dataset for binary text classification and my goal is to train a model with the method proposed by Convolutional Neural Networks for Sentence Classification

I started my implementation by using the torch.util.data.Dataset. Essentially every sample in my dataset my_data looks like this (as example):

{"words":[0,1,2,3,4],"label":1},
{"words":[4,9,20,30,4,2,3,4,1],"label":0}

Next I took a look at Writing custom dataloaders with pytorch:
using:

dataloader = DataLoader(my_data, batch_size=2,
                    shuffle=False, num_workers=4)

I would suspect that enumerating over a batch would yield something the following:

{"words":[[0,1,2,3,4],[4,9,20,30,4,2,3,4,1]],"labels":[1,0]}

However it is more like this:

{"words":[[0,4],[1,9],[2,20],[3,30],[4,4]],"label":[1,0]}

I guess it has something to do that they are not equal size.
Do they need to be the same size and if so how can i achieve it? For people knwoing about this paper, what does your training data look like?

edit:

class CustomDataset(Dataset):
def __init__(self, path_to_file, max_size=10, transform=None):

    with open(path_to_file) as f:
        self.data = json.load(f)
    self.transform = transform
    self.vocab = self.build_vocab(self.data)
    self.word2idx, self.idx2word = self.word2index(self.vocab)

def get_vocab(self):
    return self.vocab

def get_word2idx(self):
    return self.word2idx, self.idx2word

def __len__(self):
    return len(self.data)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    inputs_ = word_tokenize(self.data[idx][0])
    inputs_ = [w for w in inputs_ if w not in stopwords]
    inputs_ = [w for w in inputs_ if w not in punctuation]
    inputs_ = [self.word2idx[w] for w in inputs_]  # convert words to index

    label = {"positive": 1,"negative": 0}
    label_ = label[self.data[idx][1]] #convert label to 0|1

    sample = {"words": inputs_, "label": label_}

    if self.transform:
        sample = self.transform(sample)

    return sample

def build_vocab(self, corpus):
    word_count = {}
    for sentence in corpus:
        tokens = word_tokenize(sentence[0])
        for token in tokens:
            if token not in word_count:
                word_count[token] = 1
            else:
                word_count[token] += 1
    return word_count

def word2index(self, word_count):
    word_index = {w: i for i, w in enumerate(word_count)}
    idx_word = {i: w for i, w in enumerate(word_count)}
    return word_index, idx_word

Output:
As you correctly suspected, this is mostly a problem of different tensor shapes. Luckily, PyTorch offers you several solutions of varying simplicity to achieve what you desire (batch sizes >= 1 for text samples):

The highest-level solution is probably torchtext, which provides several solutions out of the box to load (custom) datasets for NLP tasks. If you can make your training data fit in any one of the described loaders, this is probably the recommended option, as there is a decent documentation and several examples.
If you prefer to build a solution, there are padding solutions like torch.nn.utils.rnn.pad_sequence, in combination with torch.nn.utils.pack_padded_sequence, or the combination of both (torch.nn.utils.rnn.pack_sequence. This generally allows you a lot more flexibility, which may or may not be something that you require.

Personally, I have had good experiences using just pad_sequence, and sacrifice a bit of speed for a much clearer debugging state, and seemingly others have similar recommendations.


Instruction: Where is torch.cholesky and how torch refers to its methods?

Input:
I'm doing some research into Cholesky decomposition, which requires some insights into how torch.cholesky works. After a while of grep-ing and searching through ATen, I got stuck at TensorMethods.h, which interestingly has this following code:

inline Tensor Tensor::cholesky(bool upper) const {
#ifdef USE_STATIC_DISPATCH
    return TypeDefault::cholesky(const_cast<Tensor&>(*this), upper);
#else
    static c10::OperatorHandle op = c10::Dispatcher::singleton().findSchema({"aten::cholesky", ""}).value();
    return c10::Dispatcher::singleton().callUnboxed<Tensor, const Tensor &, bool>(
        op, impl::dispatchTypeId(at::detail::multi_dispatch_tensor_type_set(*this)), const_cast<Tensor&>(*this), upper);
#endif
}

This raised the question of how torch locates its methods. Thank you!

Output:
Take a look at aten/src/ATen/native/README.md which describes how functions are registered to the API.

  ATen "native" functions are the modern mechanism for adding operators
  and functions to ATen (they are "native" in contrast to legacy
  functions, which are bound via TH/THC cwrap metadata). Native
  functions are declared in native_functions.yaml and have
  implementations defined in one of the cpp files in this directory.

If we take a look at aten/src/ATen/native/native_functions.yaml and search for cholesky we find

- func: cholesky(Tensor self, bool upper=False) -> Tensor
  use_c10_dispatcher: full
  variants: method, function

To find the entry-point you basically just have to search the .cpp files in the aten/src/ATen/native directory and locate the function named cholesky. Currently it can be found at BatchLinearAlgebra.cpp:550

Tensor cholesky(const Tensor &self, bool upper) {
  if (self.size(-1) == 0) {
    return at::empty_like(self, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
  }
  squareCheckInputs(self);

  auto raw_cholesky_output = at::_cholesky_helper(self, upper);
  if (upper) {
    return raw_cholesky_output.triu_();
  } else {
    return raw_cholesky_output.tril_();
  }
}

From this point it's just a matter of following the C++ code to understand what's going on.


Instruction: Error training RNN with pytorch : RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Input:
Helo everyone, I am trying to create a model using the PyTorch RNN class and to train this model using minibatches. My dataset is a simple timeserie (one input one output). Here is what my model looks like :

class RNN_pytorch(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN_pytorch, self).__init__()

        self.hidden_size = hidden_size
        self.input_size = input_size

        self.rnn = nn.RNN(input_size, hidden_size, num_layers=1)
        self.linear =  nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden):

        batch_size = x.size(1)
#         print(batch_size)
        hidden = self.init_hidden(batch_size)

        out, hidden = self.rnn(x, hidden)
#         out = out.view(out.size(1), out.size(2))
        print("Input linear : ", out.size())
        out = self.linear(out)

        return out, hidden

    def init_hidden(self, batch_size):

        hidden = torch.zeros(1, batch_size, self.hidden_size)
#         print(hidden.size())

        return hidden

Then I process my dataset and split it like so :

batch_numbers = 13
batch_size = int(len(train_signal[:-1])/batch_numbers)
print("Train sample total size =", len(train_signal[:-1]))
print("Number of batches = ", batch_numbers)
print("Size of batches = {} (train_size / batch_numbers)".format(batch_size))

train_signal_batched = train_signal[:-1].reshape(batch_numbers, batch_size, 1)
train_label_batched = train_signal[1:].reshape(batch_numbers, batch_size, 1)

print("X_train shape =", train_signal_batched.shape)
print("Y_train shape =", train_label_batched.shape)

Returning :

Train sample total size = 829439
Number of batches =  13
Size of batches = 63803 (train_size / batch_numbers)
X_train shape = (13, 63803, 1)
Y_train shape = (13, 63803, 1)

So far so good, but then I try to train my model :

rnn_mod = RNN_pytorch(1, 16, 1)

criterion = nn.MSELoss()
optimizer = torch.optim.RMSprop(rnn_mod.parameters(), lr=0.01)

n_epochs = 3
hidden = rnn_mod.init_hidden(batch_size)
for epoch in range(1, n_epochs):
    for i, batch in enumerate(train_signal_batched):
        optimizer.zero_grad()
        x = torch.Tensor([batch]).float()
        print("Input : ",x.size())
        out, hidden = rnn_mod.forward(x, hidden)
        print("Output : ",out.size())
        label = torch.Tensor([train_label_batched[i]]).float()
        print("Label : ", label.size())
        loss = criterion(output, label)
        print("Loss : ", loss)
        loss.backward(retain_graph=True)
        optimizer.step()
        print("*", end="")

#     if epoch % 100 == 0:
    print("Step {} --- Loss {}".format(epoch, loss))

Which leads to the error :

Input :  torch.Size([1, 63803, 1])
Input linear :  torch.Size([1, 63803, 16])
Output :  torch.Size([1, 63803, 1])
Label :  torch.Size([1, 63803, 1])
Loss :  tensor(0.0051)

/home/kostia/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/nn/modules/loss.py:431: UserWarning: Using a target size (torch.Size([1, 63803, 1])) that is different to the input size (torch.Size([1, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-217-d019358438ff> in <module>
     17         loss = criterion(output, label)
     18         print("Loss : ", loss)
---> 19         loss.backward(retain_graph=True)
     20         optimizer.step()
     21         print("*", end="")

~/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119
    120     def register_hook(self, hook):

~/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Can anyone tell me what is the problem here because I honestly doesn't have a clue ?

Thanks in advance

Output: It is as the error says. The loss tensor doesn't require grad and doesn't have a gread_fn function. Your out in forward in RNN_pytorch  should also already have a grad_fn, start checking there. See if you somehow disabled gradients by for example having .eval() or .no_grad() somewhere.


Instruction: Pytorch CNN loss is not changing,

Input:
I making a CNN for a binary classification problem between bees and ants images.
Images are of 500x500 dimension with 3 channels.

Here is my code.

Dataloader:

def load_data(path):
    data = []
    ant = 0
    bee = 0
    for folder in os.listdir(path):
        print(folder)
        curfolder = os.path.join(path, folder)
        for file in os.listdir(curfolder):
            image = plt.imread(curfolder+'/'+file)
            image = cv2.resize(image, (500,500))
            if folder == 'ants':
                ant += 1
                data.append([np.array(image) , np.eye(2)[0]])
            elif folder == 'bees':
                bee += 1
                data.append([np.array(image) , np.eye(2)[1]])

    np.random.shuffle(data)
    np.save('train.npy',data)
    print('ants : ',ant)
    print('bees : ',bee)

training_data = np.load("train.npy",allow_pickle=True)
print(len(training_data))

CNN class

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(3, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(3,500,500).view(-1,3,500,500)
        self._to_linear = None
        self.convs(x)
        print(self._to_linear)
        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net()
print(net)

loss and optimizer

import torch.optim as optim

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

Data Flat

train_X = torch.Tensor([i[0] for i in training_data]).view(-1,3,500,500)
train_X = train_X/255.0
train_y = torch.Tensor([i[1] for i in training_data])

training the model

device = torch.device("cuda:0")
net = Net().to(device)
print(len(train_X))
epochs = 10
BATCH_SIZE = 1
for epoch in range(epochs):
        for i in range(0, len(train_X), BATCH_SIZE): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
            #print(f"{i}:{i+BATCH_SIZE}")
            batch_X = train_X[i:i+BATCH_SIZE]
            batch_y = train_y[i:i+BATCH_SIZE]

            batch_X, batch_y = batch_X.to(device), batch_y.to(device)
            net.zero_grad()
            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()     # Does the update

        print(f"Epoch : {epoch}. Loss: {loss}")

The loss does not update for every epoch. I have tried changing the learning rate but the problem still remains.

  Epoch : 0. Loss: 0.23345321416854858

  Epoch : 1. Loss: 0.23345321416854858

  Epoch : 2. Loss: 0.23345321416854858

  Epoch : 3. Loss: 0.23345321416854858

  Epoch : 4. Loss: 0.23345321416854858

  Epoch : 5. Loss: 0.23345321416854858

  Epoch : 6. Loss: 0.23345321416854858

  Epoch : 7. Loss: 0.23345321416854858

  Epoch : 8. Loss: 0.23345321416854858

  Epoch : 9. Loss: 0.23345321416854858

Thank you in advance.

Output: In your training loop, you should do optimizer.zero_grad() instead of net.zero_grad(). Also, you are using MSELoss() for a classification problem, you need something like BinaryCrossEntropy() or CrossEntropy() or NLLLoss().


Instruction: How to pass input dim from fit method to skorch wrapper?

Input:
I am trying to incorporate PyTorch functionalities into a scikit-learn environment (in particular Pipelines and GridSearchCV) and therefore have been looking into skorch. The standard documentation example for neural networks looks like

import torch.nn.functional as F
from torch import nn
from skorch import NeuralNetClassifier

class MyModule(nn.Module):
    def __init__(self, num_units=10, nonlin=F.relu):
        super(MyModule, self).__init__()

        self.dense0 = nn.Linear(20, num_units)
        self.nonlin = nonlin
        self.dropout = nn.Dropout(0.5)
        ...
        ...
        self.output = nn.Linear(10, 2)
    ...
    ...

where you explicitly pass the input and output dimensions by hardcoding them into the constructor. However, this is not really how scikit-learn interfaces work, where the input and output dimensions are derived by the fit method rather than being explicitly passed to the constructors. As a practical example consider

# copied from the documentation
net = NeuralNetClassifier(
    MyModule,
    max_epochs=10,
    lr=0.1,
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,
)

# any general Pipeline interface
pipeline = Pipeline([
        ('transformation', AnyTransformer()),
        ('net', net)
        ])

gs = GridSearchCV(net, params, refit=False, cv=3, scoring='accuracy')
gs.fit(X, y)

besides the fact that nowhere in the transformers must one specify the input and output dimensions, the transformers that are applied before the model may change the dimentionality of the training set (think at dimensionality reductions and similar), therefore hardcoding input and output in the neural network constructor just will not do.

Did I misunderstand how this is supposed to work or otherwise what would be a suggested solution (I was thinking of specifying the constructors into the forward method where you do have X available for fit already, but I am not sure this is good practice)?

Output:
This is a very good question and I'm afraid that there is best practice answer to this as PyTorch is normally written in a way where initialization and execution are separate steps which is exactly what you don't want in this case.

There are several ways forward which are all going in the same direction, namely introspecting the input data and re-initializing the network before fitting. The simplest way I can think of is writing a callback that sets the corresponding parameters during training begin:

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

This sets a module parameter during training begin which will re-initialize the PyTorch module with said parameter. This specific callback expects that the parameter for the first layer is called input_dim but you can change this if you want.

A full example:

import torch
import skorch
from sklearn.datasets import make_classification
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA

X, y = make_classification()
X = X.astype('float32')

class ClassifierModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 2)

    def forward(self, X):
        y = self.l0(X)
        y = self.l1(y)
        return torch.softmax(y, dim=-1)

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

net = skorch.NeuralNetClassifier(
    ClassifierModule,
    callbacks=[InputShapeSetter()],
)

pipe = Pipeline([
    ('pca', PCA(n_components=10)),
    ('net', net),
])

pipe.fit(X, y)
print(pipe.predict(X))


Instruction: How to debug invalid gradient at index 0? - PyTorch

Input:
I am trying to train an actor-critic model, but when I reach the backprop for the critic I get this error:
RuntimeError: invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.FloatTensor
I am failing to identify which gradient the error refers to. Can anyone help?

Here is the Stack trace:

Traceback (most recent call last):
  File "train.py", line 338, in <module>
    main()
  File "train.py", line 327, in main
    reinforce_trainer.train(opt.start_reinforce, opt.start_reinforce + opt.critic_pretrain_epochs - 1, True, start_time)
  File "/home/fbommfim/init-tests/treeLSTM/lib/train/reinforce_trainer.py", line 56, in train
    train_reward, critic_loss = self.train_epoch(epoch, pretrain_critic, no_update)
  File "/home/fbommfim/init-tests/treeLSTM/lib/train/reinforce_trainer.py", line 153, in train_epoch
    critic_loss = self.critic.backward(baselines.cuda(), rewards, critic_weights.cuda(), num_words, self.critic_loss_func, regression=True)
  File "/home/fbommfim/init-tests/treeLSTM/lib/model/encoder_decoder/hybrid2seq_model.py", line 67, in backward
    outputs.backward(grad_output)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.6_1/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.6_1/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.FloatTensor

and relevant code:
train_epoch from reinforce_trainer

def train_epoch(self, epoch, pretrain_critic, no_update):
        self.actor.train() # may also have self.critic.train() ?
        total_reward, report_reward = 0, 0
        total_critic_loss, report_critic_loss = 0, 0
        total_sents, report_sents = 0, 0
        total_words, report_words = 0, 0
        last_time = time.time()
        batch_count = len(self.train_data)
        batch_order = torch.randperm(batch_count)
        with tqdm(total = (batch_count)) as prog:
            for i in range(batch_count):
                batch = self.train_data[i] # batch_order[i]
                if self.opt.data_type == 'code':
                    targets = batch[2]
                    attention_mask = batch[1][2][0].data.eq(lib.Constants.PAD).t()
                elif self.opt.data_type == 'text':
                    targets = batch[2]
                    attention_mask = batch[0][0].data.eq(lib.Constants.PAD).t()
                elif self.opt.data_type == 'hybrid':
                    targets = batch[2]
                    attention_mask_code = batch[1][2][0].data.eq(lib.Constants.PAD).t()
                    attention_mask_txt = batch[0][0].data.eq(lib.Constants.PAD).t()

                batch_size = targets.size(1)

                self.actor.zero_grad()
                self.critic.zero_grad()

                # Sample translations
                if self.opt.has_attn:
                    if self.opt.data_type == 'code' or self.opt.data_type == 'text':
                        self.actor.decoder.attn.applyMask(attention_mask)
                    elif self.opt.data_type == 'hybrid':
                        self.actor.decoder.attn.applyMask(attention_mask_code, attention_mask_txt)
                samples, outputs = self.actor.sample(batch, self.max_length)

                # Calculate rewards
                rewards, samples = self.sent_reward_func(samples.t().tolist(), targets.data.t().tolist())
                reward = sum(rewards)

                # Perturb rewards (if specified).
                if self.pert_func is not None:
                    rewards = self.pert_func(rewards)

                samples = torch.LongTensor(samples).t().contiguous()
                rewards = torch.FloatTensor([rewards] * samples.size(0)).contiguous()
                if self.opt.cuda:
                    samples = samples.cuda()
                    rewards = rewards.cuda()

                # Update critic.
                critic_weights = samples.ne(lib.Constants.PAD).float()
                num_words = critic_weights.data.sum()
                if not no_update:
                    if self.opt.data_type == 'code':
                        baselines = self.critic((batch[0], batch[1], samples, batch[3]), eval=False, regression=True)
                    elif self.opt.data_type == 'text':
                        baselines = self.critic((batch[0], batch[1], samples, batch[3]), eval=False, regression=True)
                    elif self.opt.data_type == 'hybrid':
                        baselines = self.critic((batch[0], batch[1], samples, batch[3]), eval=False, regression=True)

                    critic_loss = self.critic.backward(baselines, rewards, critic_weights, num_words, self.critic_loss_func, regression=True)
                    self.critic_optim.step()
                else:
                    critic_loss = 0

                # Update actor
                if not pretrain_critic and not no_update:
                    # Subtract baseline from reward
                    norm_rewards = (rewards - baselines).data
                    actor_weights = norm_rewards * critic_weights
                    # TODO: can use PyTorch reinforce() here but that function is a black box.
                    # This is an alternative way where you specify an objective that gives the same gradient
                    # as the policy gradient's objective, which looks much like weighted log-likelihood.
                    actor_loss = self.actor.backward(outputs, samples, actor_weights, 1, self.actor_loss_func)
                    self.optim.step()
                else:
                    actor_loss = 0

                # Gather stats
                total_reward += reward
                report_reward += reward
                total_sents += batch_size
                report_sents += batch_size
                total_critic_loss += critic_loss
                report_critic_loss += critic_loss
                total_words += num_words
                report_words += num_words
                self.opt.iteration += 1
                print ("iteration: %s, loss: %s " % (self.opt.iteration, actor_loss))
                print ("iteration: %s, reward: %s " % (self.opt.iteration, (report_reward / report_sents) * 100))

                if i % self.opt.log_interval == 0 and i > 0:
                    print("""Epoch %3d, %6d/%d batches; actor reward: %.4f; critic loss: %f; %5.0f tokens/s; %s elapsed""" %
                          (epoch, i, batch_count, (report_reward / report_sents) * 100,
                          report_critic_loss / report_words,
                          report_words / (time.time() - last_time),
                          str(datetime.timedelta(seconds=int(time.time() - self.start_time)))))

                    report_reward = report_sents = report_critic_loss = report_words = 0
                    last_time = time.time()
                prog.update(1)

        return total_reward / total_sents, total_critic_loss / total_words

and backward for hybrid2seq_model.py:

def backward(self, outputs, targets, weights, normalizer, criterion, regression=False):
        grad_output, loss = self.generator.backward(outputs, targets, weights, normalizer, criterion, regression)
        outputs.cuda()
        grad_output.cuda()
        outputs.backward(grad_output)
        return loss

Output:
The error says it's expecting a cuda tensor and got a non-cuda tensor, so that's what I'd look for.

Calls like grad_output.cuda() returns a cuda tensor. It's not an inplace operation. You probably wanted grad_output = grad_output.cuda(), so I'd start by fixing calls like that.


Instruction: Having trouble writing training loop in pytorch

Input:
I load data from a CSV FILE OF 20+6 Columns(Features and Labels). Im trying to run my data through Convolutional Neural Network in pytorch. I get error saying it expects a 3D input and Im giving it 1D input. I am using Conv1d .

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import pandas as pd
from torch.utils.data import Dataset,DataLoader
from sklearn.model_selection import train_test_split

#Read Data
data=pd.read_csv('Data.csv')
Features=data[data.columns[0:20]]
Labels=data[data.columns[20:]]
#Split Data
X_train, X_test, y_train, y_test = train_test_split( Features, Labels, test_size=0.33, shuffle=True)
#Create Tensors
train_in=torch.tensor(X_train.values)
train_out=torch.tensor(y_train.values)
test_in=torch.tensor(X_test.values)
test_out=torch.tensor(y_test.values)
#Model CNN
class CNN(nn.Module):
        def __init__(self):
                super(CNN,self).__init__()
                self.layer1 = nn.Sequential(
                                nn.Conv1d(20,40,kernel_size=5,stride=1,padding=2),
                                nn.ReLU(),
                                nn.MaxPool1d(kernel_size=2,stride=2)
                                )
                self.layer2 = nn.Sequential(
                                nn.Conv1d(40,60,kernel_size=5,stride=1,padding=2),
                                nn.ReLU(),
                                nn.MaxPool1d(kernel_size=2,stride=2)
                                )
                self.drop_out = nn.Dropout()
                self.fc1 = nn.Linear(60,30)
                self.fc2 = nn.Linear(30,15)
                self.fc3 = nn.Linear(15,6)

        def forward(self,x):
                out=self.layer1(x)
                out=self.layer2(out)
                out=self.drop_out(out)
                out=self.fc1(out)
                out=self.fc2(out)
                out=self.fc3(out)
                return out

Epochs=10
N_labels=len(Labels.columns)
N_features=len(Features.columns)
batch_size=100
learning_rate=0.001
#TRAIN MODEL
model = CNN()
#LOSS AND OPTIMIZER
criterion = torch.nn.SmoothL1Loss()
optimizer = torch.optim.Adam(model.parameters(),lr=learning_rate)
#TRAIN MODEL
model.train()
idx=0
for i in train_in:
        y=model(i)
        loss=criterion(y,train_out[idx])
        idx+=1
        loss.backward()
        optimizer.step()

How do I write the Training and Eval loop? All the examples I see on the internet all use images and they also use DataLoader.

Output:
Conv1D takes as input a tensor with 3 dimensions (N, C, L) where N is the batchsize, C is the number of channels and L size of the 1D data. In your case it seems like one sample has 20 entries and you have one channel. You have a batch_size variable but it is not used in the code posted.

 nn.Conv1d(20,40,kernel_size=5,stride=1,padding=2)

This lines creates a convolution which takes a input with 20 channels (you have 1) and outputs 40 channels. So you have to change the 20 to a 1 and you might wanna change the 40 to something smaller. Since convolutions are applied to the whole input (controlled by stride, patting and kernel size), there is no need to specify the size of a sample.

Also you might wanna add some logic to build minibatches. Right now it seems like you just want to input every sample by itself. Maybe read a bit about dataset classes and data loaders in pytorch.


Instruction: What does model.eval() do in pytorch?

Input:
When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training?
Example training code using .eval().

Output:
model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn off them during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() in pair with model.eval() to turn off gradients computation:

# evaluate model:
model.eval()

with torch.no_grad():
    ...
    out_data = model(data)
    ...

BUT, don't forget to turn back to training mode after eval step:

# training step
...
model.train()
...


Instruction: PyTorch C++ - how to know the recommended version of cuDNN?

Input:
I've previously inferenced TensorFlow graphs from C++.  Now I'm embarking on working out how to inference PyTorch graphs via C++.

My first question is, how can I know the recommended version of cuDNN to use with LibTorch, or if I'm doing my own PyTorch compile?

Determining the recommended CUDA version is easy.  Upon going to https://pytorch.org/ and choosing the options under Quick Start Locally (PyTorch Build, Your OS, etc.) the site makes it pretty clear that CUDA 10.1 is recommended, but there is no mention of cuDNN version and upon Googling I'm unable to find a definitive answer for this.

From what I understand about PyTorch on ubuntu, if you use the Python version you have to install the CUDA driver (ex. so nvidia-smi works, version 440 currently), but the CUDA and cuDNN install are not actually required beyond the driver because they are included in the pip3 package, is this correct?  If so, then is there a command I can run in a Python script that shows the version of CUDA (expected to be 10.1) and cuDNN that the pip pre-compiled .whl uses?  I suspect there is such a command but I'm not familiar enough with PyTorch yet to know what that may be or how to look it up.

I've ran into compile and inferencing errors using C++ with TensorFlow when I was not using the specific recommended version of cuDNN for a certain version of TensorFlow and CUDA so I'm aware these version can be sensitive and I have to make the right choices from the get-go.  If anybody can assist in determining the recommended version of cuDNN for a certain version of PyTorch that would be great.

Output:
CUDA is supported via the graphics card driver, AFAIK there's no separate "CUDA driver". The system graphics card driver pretty much just needs to be new enough to support the CUDA/cudNN versions for the selected PyTorch version. To the best of my knowledge backwards compatibility is included in most drivers. For example a driver that supports CUDA 10.1 (reported via nvidia-smi) will also likely support CUDA 8, 9, 10.0

If you installed with pip or conda then a version of CUDA and cudNN are included with the install. You can query the actual versions being used in python with torch.version.cuda and torch.backends.cudnn.version().


Instruction: Errow Showing :'ResNet' object has no attribute 'classifier'

Input:
I download Resnet18 model to train a model.

When I type

model

it shows

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )

  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=1000, bias=True)
  (classifer): Sequential(
    (fc1): Linear(in_features=512, out_features=256, bias=True)
    (relu): ReLU()
    (fc5): Linear(in_features=128, out_features=2, bias=True)
    (output): LogSoftmax()
  )
)

As you can see it clearly shows the classifier

but when I do

optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

it shows an error

AttributeError: 'ResNet' object has no attribute 'classifier'

I don't know what mistake I am doing, if you can help that would be great. I can provide some extra details if you want.

Output:
Remove classifier and keep it model.parameters() only.

optimizer = optim.Adam(model.parameters(), lr=0.001)

To construct an Optimizer you have to give it an iterable containing the parameters to optimize.


Instruction: Pytorch Module: Why do we pass the class and object to the parent class initializer, in the __init__ method?

Input:
Why do we pass the class and the object (self) to the parent's init method for pytorch Module? For e.g.

class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
    super(RNN, self).__init__()

Why is the class RNN, as well as the object (self) passed to the parent's init?

Output:
Every method receives the instance of the class invoking the method as its first argument; __init__ is no exception. foo = RNN(...) causes foo.__init__(...) to be called, which is equivalent to RNN.__init__(foo, ...).

super returns a "proxy" for the instance indicated by its arguments. The class indicates the starting point in the MRO to decide which class the proxy represents, and the second argument indicates which instance. You virtually  never pass anything other than the class and self, and in Python 3 this is the default: super().__init__().