Instruction: Equal output values given for Multiclass Classification

Input:
I'm trying to build a CNN for predicting the number of fingers in an image, using PyTorch. The network:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.Layer1 = nn.Sequential(
        nn.Conv2d(in_channels=3, out_channels=16, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=256, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=256, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=128, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=128, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        )

        self.Layer2 = nn.Sequential(
        nn.Linear(1536, 100),
        nn.Tanh(),
        nn.Linear(100, 6),
        nn.Softmax()
        )
        self.optimizer = optimizers.Adadelta(self.parameters())

    def forward(self, X):
        X = self.Layer1(X)
        print(X.shape)
        X = self.Layer2(X.reshape(1, 1536))
        X = X.squeeze()

        return X

    def calc_loss(self, X, num):
        out = self.forward(X).unsqueeze(dim=0)
        print("Output: "+str(out))
        target = torch.tensor([num], dtype=torch.int64).cuda()
        criterion = nn.CrossEntropyLoss()
        loss = criterion(out, target)
        return loss

    def train_step(self, X, Y):
        loss = self.calc_loss(X, Y)
        print(loss)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

However, after the training is complete, all predictions have almost the same values (around 0.15 ~ 0.18).

It appears as though the network averages out the output probabilities to minimize loss, instead of learning the actual values.

I get the same result whether I use Softmax for the last layer with cross-entropy loss, or Sigmoid with binary cross-entropy, MSE, or SmoothL1Loss.

In case of using Adam optimizer, I get similar results, only in the range of 1e-12 ~ 1e-14.

What am I missing?

Output:
If you are using CrossEntropyLoss, you don't need to use Softmax in your forward. It is already included to CrossEntropyLoss, so you need the "raw" output. But, if you need Softmax during inference time, use NLLLoss + 'Softmax' instead.

You can find more info here


Instruction: 'ToPILImage' object has no attribute 'show'

Input:
I'm doing an image processing task and I want to concat two sites of pictures. For concatting, I first converted the image to tensor, then converted the tensor to a PIL image to display it, but it was reported incorrectly.Could someone please help me?
Here is my code:

import skimage.io as io
import torch
from torchvision import  models, transforms
from PIL import Image
import matplotlib.pyplot as plt
from torchvision.transforms import ToPILImage
import numpy as np
from skimage import data_dir,io,color

coll1 = io.ImageCollection('F:\\code1/*.jpg')
coll2 = io.ImageCollection('F:\\code2/*.jpg')
a = torch.tensor(coll1)
print(a)
print(a.shape)
b = torch.tensor(coll2)
print(b)
print(b.shape)
c=torch.cat((a,b),1)
print(c.shape)
print(c)
img= transforms.ToPILImage()
img.show()

and here is the error code:

  Traceback (most recent call last):   File "F:/filelist.py", line 39,
  in
      img.show() AttributeError: 'ToPILImage' object has no attribute 'show'

Output:
The ToPILImage method accepts a tensor or an ndarray as input, source.

You will need to cast a single image tensor to the ToPILImage method. From your post, I suspect you are passing batches of image tensor instead of one, hence the error.

Assumed if you want to visualize image from tensor c,

img = transforms.ToPILImage()(c)
img.show()


Instruction: Deal with negative values resulting from my nn model

Input:
I have a simple nn model that looks like this

class TestRNN(nn.Module):
    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_outputs):
        super(TestRNN, self).__init__()
        ...
        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)
        self.FC = nn.Linear(self.n_neurons, self.n_outputs)

    def forward(self, X):
        ...
        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)

and I am using criterion = nn.CrossEntropyLoss() for calculating my error.  The operation order goes something like this:

# get the inputs
x, y = data

# forward + backward + optimize
outputs = model(x)
loss = criterion(outputs, y)

Where my training data x is normalized and looks like this:

tensor([[[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [2.6164e-02, 2.6164e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 1.3108e-05],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [9.5062e-01, 3.1036e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [0.0000e+00, 1.3717e-05, 3.2659e-07,  ..., 0.0000e+00,
          0.0000e+00, 3.2659e-07]],

        [[5.1934e-01, 5.4041e-01, 6.8083e-06,  ..., 0.0000e+00,
          0.0000e+00, 6.8083e-06],
         [5.2340e-01, 6.0007e-01, 2.7062e-06,  ..., 0.0000e+00,
          0.0000e+00, 2.7062e-06],
         [8.1923e-01, 5.7346e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0714e-01, 7.0708e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 7.0407e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        ...,

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.1852e-01, 2.3411e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0775e-01, 7.0646e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 3.9888e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.9611e-01, 5.8796e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0710e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.7538e-01, 2.4842e-01, 1.7787e-06,  ..., 0.0000e+00,
          0.0000e+00, 1.7787e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.2433e-01, 5.2433e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [1.3155e-01, 1.3155e-01, 0.0000e+00,  ..., 8.6691e-02,
          9.7871e-01, 0.0000e+00],
         [7.4412e-01, 6.6311e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 9.6093e-07]]])

While a typical output and y passed to the criterion function look like this:

tensor([[-0.0513],
        [-0.0445],
        [-0.0514],
        [-0.0579],
        [-0.0539],
        [-0.0323],
        [-0.0521],
        [-0.0294],
        [-0.0372],
        [-0.0518],
        [-0.0516],
        [-0.0501],
        [-0.0312],
        [-0.0496],
        [-0.0436],
        [-0.0514],
        [-0.0518],
        [-0.0465],
        [-0.0530],
        [-0.0471],
        [-0.0344],
        [-0.0502],
        [-0.0536],
        [-0.0594],
        [-0.0356],
        [-0.0371],
        [-0.0513],
        [-0.0528],
        [-0.0621],
        [-0.0404],
        [-0.0403],
        [-0.0562],
        [-0.0510],
        [-0.0580],
        [-0.0516],
        [-0.0556],
        [-0.0063],
        [-0.0459],
        [-0.0494],
        [-0.0460],
        [-0.0631],
        [-0.0525],
        [-0.0454],
        [-0.0509],
        [-0.0522],
        [-0.0426],
        [-0.0527],
        [-0.0423],
        [-0.0572],
        [-0.0308],
        [-0.0452],
        [-0.0555],
        [-0.0479],
        [-0.0513],
        [-0.0514],
        [-0.0498],
        [-0.0514],
        [-0.0471],
        [-0.0505],
        [-0.0467],
        [-0.0485],
        [-0.0520],
        [-0.0517],
        [-0.0442]], device='cuda:0', grad_fn=<ViewBackward>)
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')

When the criterion is being applied I get the following error (running with CUDA_LAUNCH_BLOCKING=1):

/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered

The fact that my model outputs negative values is causing the error message above, how can I resolve this issue?

Output:
TL;DR

You have two options:

Make the second dimension of outputs be of size 2 instead of 1.
Use nn.BCEWithLogitsLoss instead of nn.CrossEntropyLoss

I think that the problem is not the negative numbers. It is the shape of outputs.

Looking at your array y, I see that you have 2 different classes (maybe even more, but let's suppose that it's 2). This means that the last dimension of outputs should be 2. The reason is that outputs needs to give a "score" to each one of the 2 different classes (see the documentation). The score can be negative, zero or positive. But the shape of your outputs is [64,1], and not [64,2] as required.

One of the steps of the nn.CrossEntropyLoss() object will be to convert these scores to a probability distribution over the two classes. This is done using a softmax operation. However, When doing binary classification (that is, classification with only 2 classes, as in our current case), there is another option: Give a score for only one class, convert it to a probability for that class using a sigmoid function, and then perform "1-p" on this to get the probability for the other class. This option means that outputs needs to give a score for only one of the two classes, as in you current case. To chose this options, you will need to change nn.CrossEntropyLoss with nn.BCEWithLogitsLoss. You can then pass to it outputs and y as you are currently doing (note however that the shape of outputs needs to be precisely the shape of y, so in your example you will need to pass outputs[:,0] instead of outputs. Also you will need to convert y to a float: y.float(). Thus the call is criterion(outputs[:,0], y.float()))


Instruction: How to install torch audio on Windows 10 conda?

Input:
In Anaconda Python 3.6.7 with PyTorch installed, on Windows 10, I do this sequence:

conda install -c conda-forge librosa
conda install -c groakat sox

then in a fresh download from https://github.com/pytorch/audio I do

python setup.py install

and it runs for a while and ends like this:

torchaudio/torch_sox.cpp(3): fatal error C1083: Cannot open include file: 'sox.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.15.26726\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

I am trying to reproduce this OpenNMT-py speech training demo on Windows: http://opennmt.net/OpenNMT-py/speech2text.html

Output:
I managed to compile torchaudio with sox in Windows 10, but is a bit tricky.

Unfortunately the sox_effects are not usable, this error shows up:

RuntimeError: Error opening output memstream/temporary file

But you can use the other torchaudio functionalities.

The steps I followed for Windows 10 64bit are:

TORCHAUDIO WINDOWS10 64bit

Note: I mix some command lines unix-like syntax, you can use file explorer or whatever

preliminar arrangements

Download sox sources

$ git clone git://git.code.sf.net/p/sox/code sox

Download other sox source to get lpc10

$ git clone https://github.com/chirlu/sox/tree/master/lpc10 sox2
$ cp -R sox2/lpc10 sox

IMPORTANT get VisualStudio2019 and BuildTools installed

lpc10 lib

4.0. Create a VisualStudio CMake project for lpc10 and build it

Start window -> open local folder -> sox/lpc10
(it reads CMakeLists.txt automatically)
Build->build All

4.2. Copy lpc10.lib to sox

$ mkdir -p sox/src/out/build/x64-Debug
$ cp sox/lpc10/out/build/x64-Debug/lpc10.lib sox/src/out/build/x64-Debug

gsm lib

5.0. Create a CMake project for libgsm and compile it as before with lpc10

5.1. Copy gsm.lib to sox

$ mkdir -p sox/src/out/build/x64-Debug
$ cp sox/libgsm/out/build/x64-Debug/gsm.lib sox/src/out/build/x64-Debug

sox lib

6.0. Create a CMake project for sox in VS

6.1. Edit some files:

CMakeLists.txt: (add at the very beginning)

project(sox)

sox_i.h: (add under stdlib.h include line)

#include <wchar.h> /* For off_t not found in stdio.h */
#define UINT16_MAX  ((int16_t)-1)
#define INT32_MAX  ((int32_t)-1)

sox.c: (add under time.h include line)

`#include <sys/timeb.h>`

6.2. Build sox with VisualStudio

6.3. Copy the libraries where python will find them, I use a conda environment:

$ cp sox/src/out/build/x64-Debug/libsox.lib envs\<envname>\libs\sox.lib
$ cp sox/src/out/build/x64-Debug/gsm.lib envs\<envname>\libs
$ cp sox/src/out/build/x64-Debug/lpc10.lib envs\<envname>\libs

torchaudio

$ activate <envname>

7.0. Download torchaudio from github

$ git clone https://github.com/pytorch/audio thaudio

7.1. Update setup.py, after the "else:" statement of "if IS_WHEEL..."

$ vi thaudio/setup.py

if IS_WHEEL...

else:
    audio_path = os.path.dirname(os.path.abspath(__file__))

    # Add include path for sox.h, I tried both with the same outcome
    include_dirs += [os.path.join(audio_path, '../sox/src')]
    #include_dirs += [os.path.join(audio_path, 'torchaudio/sox')]

    # Add more libraries

    #libraries += ['sox']
    libraries += ['sox','gsm','lpc10']

7.2. Edit sox.cpp from torchaudio because dynamic arrays are not allowed:

$ vi thaudio/torchaudio/torch_sox.cpp

 //char* sox_args[max_num_eopts];
 char* sox_args[20]; //Value of MAX_EFFECT_OPTS

7.3. Build and install

$ cd thaudio
$ python setup.py install

It will print out tons of warnings about type conversion and some library conflict with MSVCRTD but "works".

And thats all.


Instruction: How can I use pytorch pre-trained model without installing pytorch?

Input:
I only want to use pre-trained model in pytorch without installing the whole package.
Can I just copy the model module from pytorch?

Output: I'm afraid you cannot do that: in order to run the model, you need not only the trained weights ('.pth.tar' file) but also the "structure" of the net: that is, the layers, how they are connected to each other etc. This network structure is coded in python and requires pytorch to be installed.


Instruction: How to to drop a specific labeled pixels in semantic segmentation

Input:
I am new to semantic segmentation. I used the FCN to train my dataset. In the data set there are some pixels for the unknown class. I would like to exclude this class from my loss. So I defined a weight based on the class distribution of whole dataset and set the weight for the unknown class to zero as following. But I am still getting prediction for this class. Do you have any idea how to properly exclude one specific class?

loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits
                      (logits=logits, labels=tf.squeeze(annotation,
                                                        squeeze_dims=[3]),name="entropy"))
weighted_losses = (loss * weights)
train_op = optimizer.minimize(weighted_losses,
                                  var_list=tf.trainable_variables(),
                                  global_step=tf.train.get_global_step())

I do not know pytorch, but I heard that there is some thing for this purpose "ignore_index" in loss function and you can ignore a specific class. If this is right approach to my problem, do you know if there is some thing equivalent in tensorflow?

Output:
For semantic segmentation you have 2 "special" labels: the one is "background" (usually 0), and the other one is "ignore" (usually 255 or -1).

"Background" is like all other semantic labels meaning "I know this pixel does not belong to any of the semantic categories I am working with". It is important for your model to correctly output "background" whenever applicable.
"Ignore" label is not a label that your model can predict - it is "outside" its range. This label only exists in the training annotation meaning "we were unsure how this pixel should be labeled, so just ignore it".

When there are "ignore" pixels in your target labels, your model cannot (and should not) output "ignore" labels. Nevertheless, your model should output something. The fact that this pixel is labeled "ignore" means that whatever your model outputs for that pixel will be ignored by the loss function (assuming you told the loss to ignore "ignore" pixels). Moreover, if your test/validation sets have "ignore" labels means that whatever your model outputs for these pixels, it would simply be ignored by the scoring mechanism and won't be counted as either a correct or incorrect prediction.

To summarize: even when the ground truth has "ignore" labels, the model cannot and should not output "ignore". It simply outputs whatever valid label it feels like and it is perfectly okay.

for tensorflow you can checkout this thread.


Instruction: TextLMDataBunch Memory issue Language Model Fastai

Input:
I have a dataset with 45 million rows of data. I have three 6gb ram gpu. I am trying to train a language model on the data.

For that, I am trying to load the data as the fastai data bunch. But this part always fails because of the memory issue.

data_lm = TextLMDataBunch.from_df('./', train_df=df_trn,
valid_df=df_val, bs=10)

How do I handle this issue?

Output:
When you use this function, your Dataframe is loaded in memory. Since you have a very big dataframe, this causes your memory error. Fastai handles tokenization with a chunksize, so you should still be able to tokenize your text.

Here are two things you should try :

Add a chunksize argument (the default value is 10k) to your TextLMDataBunch.from_df, so that the tokenization process needs less memory.
If this is not enough, I would suggest not to load your whole dataframe into memory. Unfortunately, even if you use TextLMDataBunch.from_folder, it just loads the full DataFrame and pass it to TextLMDataBunch.from_df, you might have to create your own DataBunch constructor. Feel free to comment if you need help on that.


Instruction: Using PyTorch on AWS Lambda

Input: Has anyone had any luck being able to use PyTorch on AWS Lambda for feature extraction from images or just using the framework at all? I finally got PyTorch, numpy, and pillow zipped in a folder under the uncompressed size limit (which is actually around 262 MB) but I had to build PyTorch from source to do this. The problem I am having now is that Lambda has a very old version of gcc running on it (4.8.3) which is very buggy and missing whole header files altogether. I believe the Pytorch docs state you should be using at least gcc 7 or later but I'm hoping someone may have found a way around this? I built the source using gcc 7.5 but then when I tried to import torch Lambda obviously used it's installed version of 4.8.3 causing an error on import: Floating point exception (core dumped) which stems from the old version of gcc. Is there a possible solution around this? I've been at this for a day and a half now so any help would be great. I think the bottom line is I am facing this similar issue. Better yet does anyone have a Pytorch lambda layer I could use?

Output:
I was able to utilize the below layers for using pytorch on AWS Lambda:

arn:aws:lambda:AWS_REGION:934676248949:layer:pytorchv1-py36:1   PyTorch 1.0.1
arn:aws:lambda:AWS_REGION:934676248949:layer:pytorchv1-py36:2   PyTorch 1.1.0

Found these on Fastai production deployment page, thanks to Matt McClean


Instruction: RuntimeError when changing the values of specific parts of a `torch.Tensor`

Input:
Say I have a 3 dimentional tensor x initialized with zeros:

x = torch.zeros((2, 2, 2))

and an other 3 dimentional tensor y

y = torch.ones((2, 1, 2))

I am trying to change the values of the first line of x[0] and x[1] like this

x[:, 0, :] = y

but I get this error:

RuntimeError: expand(torch.FloatTensor{[2, 1, 2]}, size=[2, 2]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (3)

It is as if the tensor y was getting squeezed somehow.  Is there a way around this?

Output:
I found a straight forward way to do it:

x[:, 0, :] = y[:, 0, :]


Instruction: PyTorch Datasets: Converting entire Dataset to NumPy

Input:
I'm trying to convert the Torchvision MNIST train and test datasets into NumPy arrays but can't find documentation to actually perform the conversion.

My goal would be to take an entire dataset and convert it into a single NumPy array, preferably without iterating through the entire dataset.

I've looked at How do I turn a Pytorch Dataloader into a numpy array to display image data with matplotlib? but it doesn't address my issue.

So my question is, utilizing torch.utils.data.DataLoader, how would I go about converting the datasets (train/test) into two NumPy arrays such that all of the examples are present?

Note: I've left the batch size as the default of 1 for now; I could set it to 60,000 for train and 10,000 for test, but I'd prefer to not use magic numbers of that sort.

Thank you.

Output:
If I understand you correctly, you want to get the whole train dataset of MNIST images (in total 60000 images, each image of size 1x28x28 array with 1 for color channel) as a numpy array of size (60000, 1, 28, 28)?
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Transform to normalized Tensors
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.1307,), (0.3081,))])

train_dataset = datasets.MNIST('./MNIST/', train=True, transform=transform, download=True)
# test_dataset = datasets.MNIST('./MNIST/', train=False, transform=transform, download=True)

train_loader = DataLoader(train_dataset, batch_size=len(train_dataset))
# test_loader = DataLoader(test_dataset, batch_size=len(test_dataset))

train_dataset_array = next(iter(train_loader))[0].numpy()
# test_dataset_array = next(iter(test_loader))[0].numpy()

This is the result:
>>> train_dataset_array

array([[[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]],

       [[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]],

       [[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]],

       ...,

       [[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]],

       [[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]],

       [[[-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         ...,
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296],
         [-0.42421296, -0.42421296, -0.42421296, ..., -0.42421296,
          -0.42421296, -0.42421296]]]], dtype=float32)

Edit: You can also get the labels by next(iter(train_loader))[1].numpy(). Alternatively you can use train_dataset.data.numpy() and train_dataset.targets.numpy(), but note that the data will not be transformed by transform as is done when using the dataloader.


Instruction: DataLoader Class Errors Pytorch

Input:
I am beginner pytorch user, and I am trying to use dataloader.

Actually, I am trying to implement this into my network but it takes a very long time to load. And so, I debugged my network to see if the network itself has the problem, but it turns out it has something to with my dataloader class. Here is the code:

 from torch.utils.data import Dataset, DataLoader
 import numpy as np
 import pandas as pd

class DiabetesDataset(Dataset):

  def __init__(self, csv):
      self.xy = pd.read_csv(csv)

  def __len__(self):
      return len(self.xy)

  def __getitem__(self, index):
       self.x_data = torch.Tensor(xy.iloc[:, 0:-1].values)
       self.y_data = torch.Tensor(xy.iloc[:, [-1]].values)
       return self.x_data[index], self.y_data[index]

 dataset = DiabetesDataset("trial.csv")
 train_loader = DataLoader(dataset=dataset,
                      batch_size=1,
                      shuffle=True,
                      num_workers=2)`

 for a in train_loader:
    print(a)

To verify that the dataloader causes all the delay, I created a dummy csv file with 2 columns of 1s and 2s, for a total of 10 samples for each columns. Then, I looped over the train_loader object, it has been more than 1 hr and it is still running, considering that the sample size is small and batch size is set to 1.

I am not sure as to what the error to my code is and it is causing this issue.

Any comments/inputs are greatly appreciated!

Output:
There are some bugs in your code - could you check if this works (it is working on my computer with your toy example):

from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
import torch

class DiabetesDataset(Dataset):

    def __init__(self, csv):
        self.xy = pd.read_csv(csv)

    def __len__(self):
        return len(self.xy)

    def __getitem__(self, index):
        x_data = torch.Tensor(self.xy.iloc[:, 0:-1].values)
        y_data = torch.Tensor(self.xy.iloc[:, [-1]].values)
        return x_data[index], y_data[index]

dataset = DiabetesDataset("trial.csv")

train_loader = DataLoader(
    dataset=dataset,
    batch_size=1,
    shuffle=True,
    num_workers=2)

if __name__ == '__main__':
    for a in train_loader:
        print(a)

Edit: Your code is not working because you are missing a self in the __getitem__ method (self.xy.iloc...) and because you do not have a if __name__ == '__main__ at the end of your script. For the second error, see RuntimeError on windows trying python multiprocessing


Instruction: How to load images in the same folder in Pytorch?

Input:
I want to load all of the images from the folder /img and /mask respectively. The data structure can be shown as follows:

data
    img
        0.png
        1.png
        2.png
        3.png
        ...
   mask
       label_0.png
       label_1.png
       label_2.png
       ...

Hopefully for help.

Output:
If you want to load all the images from the two folders then you can try cv2

import cv2

img = []
for i in range(n): # n = number of images in img folder
    img_path = f'~data\img\{i}.png' # replace ~ with full path
    img.append(cv2.imread(img_path))

for i in range(n): # n = number of images in mask folder
    img_path = f'~data\mask\lable_{i}.png' # replace ~ with full path
    img.append(cv2.imread(img_path))


Instruction: how to modify rnn cells in pytorch?

Input:
If I want to change the compute rules in a RNN cell (e.g. GRU cell), what should I do?
I do not want to implement it via for or while loop considering the issue of efficiency.
I have viewed the source code of pytorch, but it seems that the major components of rnn cells are implement in c code which I cannot find and modify.
You can answer this question through an example: implement GRU cell without the existing version.

thank you ~

Output:
Yes, you implement it "via for or while loop".
Since Pytorch 1.0 there is JIT https://pytorch.org/docs/stable/jit.html that works pretty well (probably better to use the latest git version of PyTorch because of recent improvements to JIT), and depending on your network and implementation in can as fast as a native PyTorch C++ implementation (but still slower than CuDNN).

You can see example implementations at https://github.com/pytorch/benchmark/blob/master/rnns/fastrnns/custom_lstms.py


Instruction: Understanding Feature Maps in Convolutional Layers (PyTorch)

Input:
I've got this segment of code in a discriminator network for MNIST:

nn.Conv2d(1, 64, 4, 2, 1),

From my understanding, there is 1 input channel (the MNIST image), then we apply a 4x4 kernel to the image in strides of 2 to produce 64 feature maps. Does this mean that we actually have 64 kernels at this layer? Because in order to get 64 different feature maps, we would need 64 separate kernels to convolve over the image?

Then after some ReLu, we have another convolution:

nn.Conv2d(64, 128, 4, 2, 1),

How do we get from 64 to 128? From my understanding of the first example, we have 64 seperate kernels that can produce 64 seperate feature maps. But here we go from 64 feature maps to 128 feature maps? Does that mean that we only have two kernels?

I hope someone can shine some light on whether my understanding is correct!

Output:
Your understanding in the first example is correct, you have 64 different kernels to produce 64 different feature maps.

In case of the second example, so the number of input channels not beeing one, you still have as "many" kernels as the number of output feature maps (so 128), which each are trained on a linear combination of the input feature maps. So in your case each of these kernels would have 4x4x64 trainable weights.


Instruction: Custom weight initialisation causing error - pytorch

Input:
%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

num_epochs = 20

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])

num_epochs = 200

x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([0,1,1,0]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

device = 'cpu'

input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = .0001

torch.manual_seed(24)

def weights_init(m):
    m.weight.data.normal_(0.0, 1)

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size , num_classes)

    def forward(self, x) :
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)
model.apply(weights_init)

criterionCE = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i in range(0 , 1) :

        total_step = len(train_loader)
        for epoch in range(num_epochs) :
            for i,(images , labels) in enumerate(train_loader) :
                images = images.to(device)
                labels = labels.to(device)

                outputs = model(images)
                loss = criterionCE(outputs , labels)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

        outputs = model(x)

        print(outputs.data.max(1)[1])

I'm using to initialize the weights :

def weights_init(m):
    m.weight.data.normal_(0.0, 1)

But following error is thrown :

~/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536
    537     def __setattr__(self, name, value):

AttributeError: 'ReLU' object has no attribute 'weight'

Is this the correct method to initialize the weights ?

Also, should object be of type nn.Module , not Relu ?

Output:
You are trying to set the weights of a weight-free layer (ReLU).

Inside weights_init, you should check the type of layers before initializing weights. For instance:

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

See How to initialize weights in PyTorch?.


Instruction: What is the class definition of nn.Linear in PyTorch?

Input:
What is self.hidden in the following code?
import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(784, 256)
        self.output = nn.Linear(256, 10)

    def forward(self, x):
        x = F.sigmoid(self.hidden(x))
        x = F.softmax(self.output(x), dim=1)
        return x

self.hidden is nn.Linear and it can take a tensor x as argument.

Output:
What is the class definition of nn.Linear in pytorch?

From documentation:

CLASS torch.nn.Linear(in_features, out_features, bias=True)
Applies a linear transformation to the incoming data: y = x*W^T + b
Parameters:

in_features – size of each input sample (i.e. size of x)
out_features – size of each output sample (i.e. size of y)
bias – If set to False, the layer will not learn an additive bias. Default: True

Note that the weights W have shape (out_features, in_features) and biases b have shape (out_features). They are initialized randomly and can be changed later (e.g. during the training of a Neural Network they are updated by some optimization algorithm).
In your Neural Network, the self.hidden = nn.Linear(784, 256) defines a hidden (meaning that it is in between of the input and output layers), fully connected linear layer, which takes input x of shape (batch_size, 784), where batch size is the number of inputs (each of size 784) which are passed to the network at once (as a single tensor), and transforms it by the linear equation y = x*W^T + b into a tensor y of shape (batch_size, 256). It is further transformed by the sigmoid function, x = F.sigmoid(self.hidden(x)) (which is not a part of the nn.Linear but an additional step).
Let's see a concrete example:
import torch
import torch.nn as nn

x = torch.tensor([[1.0, -1.0],
                  [0.0,  1.0],
                  [0.0,  0.0]])

in_features = x.shape[1]  # = 2
out_features = 2

m = nn.Linear(in_features, out_features)

where x contains three inputs (i.e. the batch size is 3), x[0], x[1] and x[3], each of size 2, and the output is going to be of shape (batch size, out_features) = (3, 2).
The values of the parameters (weights and biases) are:
>>> m.weight
tensor([[-0.4500,  0.5856],
        [-0.1807, -0.4963]])

>>> m.bias
tensor([ 0.2223, -0.6114])

(because they were initialized randomly, most likely you will get different values from the above)
The output is:
>>> y = m(x)
tensor([[-0.8133, -0.2959],
        [ 0.8079, -1.1077],
        [ 0.2223, -0.6114]])

and (behind the scenes) it is computed as:
y = x.matmul(m.weight.t()) + m.bias  # y = x*W^T + b

i.e.
y[i,j] == x[i,0] * m.weight[j,0] + x[i,1] * m.weight[j,1] + m.bias[j]

where i is in interval [0, batch_size) and j in [0, out_features).


Instruction: Is it possible to freeze only certain embedding weights in the embedding layer in pytorch?

Input:
When using GloVe embedding in NLP tasks, some words from the dataset might not exist in GloVe. Therefore, we instantiate random weights for these unknown words.

Would it be possible to freeze weights gotten from GloVe, and train only the newly instantiated weights?

I am only aware that we can set:
model.embedding.weight.requires_grad = False

But this makes the new words untrainable..

Or are there better ways to extract semantics of words..

Output:
1. Divide embeddings into two separate objects
One approach would be to use two separate embeddings one for pretrained, another for the one to be trained.
The GloVe one should be frozen, while the one for which there is no pretrained representation would be taken from the trainable layer.
If you format your data that for pretrained token representations it is in smaller range than the tokens without GloVe representation it could be done. Let's say your pretrained indices are in the range [0, 300], while those without representation are [301, 500]. I would go with something along those lines:
import numpy as np
import torch

class YourNetwork(torch.nn.Module):
    def __init__(self, glove_embeddings: np.array, how_many_tokens_not_present: int):
        self.pretrained_embedding = torch.nn.Embedding.from_pretrained(glove_embeddings)
        self.trainable_embedding = torch.nn.Embedding(
            how_many_tokens_not_present, glove_embeddings.shape[1]
        )
        # Rest of your network setup

    def forward(self, batch):
        # Which tokens in batch do not have representation, should have indices BIGGER
        # than the pretrained ones, adjust your data creating function accordingly
        mask = batch > self.pretrained_embedding.num_embeddings

        # You may want to optimize it, you could probably get away without copy, though
        # I'm not currently sure how
        pretrained_batch = batch.copy()
        pretrained_batch[mask] = 0

        embedded_batch = self.pretrained_embedding(pretrained_batch)

        # Every token without representation has to be brought into appropriate range
        batch -= self.pretrained_embedding.num_embeddings
        # Zero out the ones which already have pretrained embedding
        batch[~mask] = 0
        non_pretrained_embedded_batch = self.trainable_embedding(batch)

        # And finally change appropriate tokens from placeholder embedding created by
        # pretrained into trainable embeddings.
        embedded_batch[mask] = non_pretrained_embedded_batch[mask]

        # Rest of your code
        ...

Let's say your pretrained indices are in the range [0, 300], while those without representation are [301, 500].
2. Zero gradients for specified tokens.
This one is a bit tricky, but I think it's pretty concise and easy to implement. So, if you obtain the indices of tokens which got no GloVe representation, you can explicitly zero their gradient after backprop, so those rows will not get updated.
import torch

embedding = torch.nn.Embedding(10, 3)
X = torch.LongTensor([[1, 2, 4, 5], [4, 3, 2, 9]])

values = embedding(X)
loss = values.mean()

# Use whatever loss you want
loss.backward()

# Let's say those indices in your embedding are pretrained (have GloVe representation)
indices = torch.LongTensor([2, 4, 5])

print("Before zeroing out gradient")
print(embedding.weight.grad)

print("After zeroing out gradient")
embedding.weight.grad[indices] = 0
print(embedding.weight.grad)

And the output of the second approach:
Before zeroing out gradient
tensor([[0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0833, 0.0833, 0.0833],
        [0.0417, 0.0417, 0.0417],
        [0.0833, 0.0833, 0.0833],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417]])
After zeroing out gradient
tensor([[0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0417, 0.0417, 0.0417]])


Instruction: Pytorch CNN error: Expected input batch_size (4) to match target batch_size (64)

Input:
I've been teaching myself this since November and any help on this would be really appreciated, thank you for looking, as I seem to be going round in circles. I am trying to use a Pytorch CNN example that was used with the Mnist dataset. Now I am trying to modify the CNN for facial key point recognition.  I am using the Kaggle dataset (CSV) of 7048 training images and key points (15 key points per face) and 1783 test images.  I split training dataset and converted the images to jpeg, made separate file for the key points (shape 15, 2). I have made dataset and data loader and can iterate through and display images and plot key points. When I run the CNN I am getting this error.

> Net(
  (conv1): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  (conv2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  (conv2_drop): Dropout2d(p=0.5)
  (fc1): Linear(in_features=589824, out_features=100, bias=True)
  (fc2): Linear(in_features=100, out_features=30, bias=True)
)
Data and target shape:  torch.Size([64, 96, 96])   torch.Size([64, 15, 2])
Data and target shape:  torch.Size([64, 1, 96, 96])   torch.Size([64, 15, 2])

Traceback (most recent call last):
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 416, in <module>
    main()
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 412, in main
    train(args, model, device, train_loader, optimizer, epoch)
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 324, in train
    loss = F.nll_loss(output, target)
  File "/home/keith/Desktop/PycharmProjects/fkp/FacialLandMarks/lib/python3.6/site-packages/torch/nn/functional.py", line 1788, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (4) to match target batch_size (64).

Process finished with exit code 1

Here are some links I have read, I could not figure out the problem
but may help some one else.

    https://github.com/pytorch/pytorch/issues/11762
        How do I modify this PyTorch convolutional neural network to accept a 64 x 64 image and properly output predictions?
            pytorch-convolutional-neural-network-to-accept-a-64-x-64-im
        Pytorch Validating Model Error: Expected input batch_size (3) to match target batch_size (4)
            model-error-expected-input-batch-size-3-to-match-target-ba

Here is my code:

    class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=(2, 2))
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=(2, 2))
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(64 * 96 * 96, 100)
        self.fc2 = nn.Linear(100, 30)  # 30 is x and y key points

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 64 * 96 * 96)
        # x = x.view(x.size(0), -1)
        # x = x.view(x.size()[0], 30, -1)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, batch in enumerate(train_loader):
        data = batch['image']
        target = batch['key_points']
        print('Data and target shape: ', data.shape, ' ', target.shape)
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        data = data.unsqueeze(1).float()

        print('Data and target shape: ', data.shape, ' ', target.shape)

        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

# def test(args, model, device, test_loader):
#     model.eval()
#     test_loss = 0
#     correct = 0
#     with torch.no_grad():
#         for data, target in test_loader:
#             data, target = data.to(device), target.to(device)
#             output = model(data)
#             test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss
#             pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
#             correct += pred.eq(target.view_as(pred)).sum().item()
#
#     test_loss /= len(test_loader.dataset)
#     print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
#         test_loss, correct, len(test_loader.dataset),
#         100. * correct / len(test_loader.dataset)))

def main():
    # Training settings
    parser = argparse.ArgumentParser(description='Project')
    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10, metavar='N',   # ========  epoch
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                        help='how many batches to wait before logging training status')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    train_data_set = FaceKeyPointDataSet(csv_file='faces/Kep_points_and_id.csv',
                                         root_dir='faces/',
                                         transform=transforms.Compose([
                                             # Rescale(96),
                                             ToTensor()
                                         ]))

    train_loader = DataLoader(train_data_set, batch_size=args.batch_size,
                              shuffle=True)

    print('Number of samples: ', len(train_data_set))
    print('Number of train_loader: ', len(train_loader))

    model = Net().to(device)
    print(model)
    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        # test(args, model, device, test_loader)

if __name__ == '__main__':
    main()

Output:
to understand what went wrong you can print shape after every step in forward :

# Input data
torch.Size([64, 1, 96, 96])
x = F.relu(F.max_pool2d(self.conv1(x), 2))
torch.Size([64, 32, 48, 48])
x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
torch.Size([64, 64, 24, 24])
x = x.view(-1, 64 * 96 * 96)
torch.Size([4, 589824])
x = F.relu(self.fc1(x))
torch.Size([4, 100])
x = F.dropout(x, training=self.training)
torch.Size([4, 100])
x = self.fc2(x)
torch.Size([4, 30])
return F.log_softmax(x, dim=1)
torch.Size([4, 30])

Your maxpool2d layers reduce the height and width of your feature maps.
The 'view' should be x = x.view(-1, 64 * 24 * 24)
the first linear layer of size : self.fc1 = nn.Linear(64 * 24 * 24, 100)

this will give your output = model(data) final shape of torch.Size([64, 30])

But this code will still face a problem in calculating the Negative Log Likelihood Loss :

  The input is expected to contain scores for each class. input has to
  be a 2D Tensor of size (minibatch, C). This criterion expects a class
  index (0 to C-1) as the target for each value of a 1D tensor of size
  minibatch

where class indices are just labels :

  values representing a class. For example:

  0 - class0,  1 - class1,

Since your last nn layer outputs a softmax over 30 classes, i'm assuming that is the output classes you want to classify into,
so transformation for target :

target = target.view(64, -1) # gives 64X30 ie, 30 values per channel
loss = F.nll_loss(x, torch.max(t, 1)[1]) # takes max amongst the 30 values as class label

This is when the target is a probability distribution over 30 classes, if not can do a soft-max before that. Thus the maximum value in the 30 values will represent the highest probability - thus that class which is exactly what your output represents and thus you calculate a nll between the two values. .


Instruction: cudnn error while running a pyorch code on gpu

Input:
I have the following error:
Traceback (most recent call last):

  File "odenet_mnist.py", line 343, in <module>
    logits = model(x)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

This is not my code. I am just trying to run the code of a recent paper "Neural Ordinary Differential Equations"-by Chen et al.
Here is the link to the code.

gpu: nvidia 2080Ti

pytorch version:'1.0.1.post2'

cuda 9.0

python 3.7.2

cudnn:

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 4
#define CUDNN_PATCHLEVEL 2
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

I am new to pytorch. Why am i getting this error and how do i fix it?

Output: RTX2080Ti needs CUDA10 to work properly.Install the PyTorch binaries containing CUDA10


Instruction: python - tensor : access a value

Input:
Given below is the output of VGG16 model. The command VGG16.classifier[6] output shows Linear(in_features=25088, out_features=4096, bias=True) I'm not able to understand how this works. ALso,how can I print the values of linear

    VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

Output:
VGG16 model is divided into two groups of layers named features and classifier. You can access them as VGG16.features and VGG16.classifier:

>>> VGG16 = torchvision.models.vgg16(pretrained=True)
>>> VGG16.classifier
Sequential(
  (0): Linear(in_features=25088, out_features=4096, bias=True)
  (1): ReLU(inplace)
  (2): Dropout(p=0.5)
  (3): Linear(in_features=4096, out_features=4096, bias=True)
  (4): ReLU(inplace)
  (5): Dropout(p=0.5)
  (6): Linear(in_features=4096, out_features=1000, bias=True)
)

Further, you can access each layer of these groups of layers using indices. For example, to access the first layer of classifier portion of model, you can do:

>>> VGG16.classifier[0]  # first layer of classifier portion
Linear(in_features=25088, out_features=4096, bias=True)
# and so on...
>>> VGG16.classifier[3]  # fourth layer of classifier portion
Linear(in_features=4096, out_features=4096, bias=True)


Instruction: Scatter homogenous list of values to PyTorch tensor

Input:
Consider the following list:

[[3], [1, 2], [4], [0], [2]]

And zeros tensor of size (5, 5)

I want to fill these indices according to their index in the list to the tensor with 1.

So, the expected output should be:

tensor([[0., 0., 0., 1., 0.],
        [0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 1.],
        [1., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0.]])

What happened above is this:

at the index [0, 3] put 1 (the case for the first element in my list).

A very similar case is achievable through using Tensor.scatter_. However, since it takes a tensor as the argument (index); you cannot create a tensor from a list if it contains a sub-list with a different size than the other elements, which is the case with [1, 2] in my list (this is actually the problem).

The scatter method could be used if the list is all of same size as the following:

tensor.scatter_(1, torch.tensor(index), 1)

Numpy solutions are acceptable

Output:
You can solve this by modifying your index list to have the same number of indices in each element.

max_length = max([len(l) for l in index])
index = [l + l[-1:] * (max_length - len(l)) for l in index]

This code will repeat the last element of each sub-list until they are all the same size. You can then pass it to the scatter_ function as you wrote in your question.


Instruction: How to access pytorch model parameters by index

Input:
If I have network with let's say 10 layers including biases, how can I access its i'th layer parameters just by index?

Currently, what I am doing is something like this

for parameter in myModel.parameters():
    parameter.data /= 5

How could I access parameter.data with an index? For example I'd like to access 9th layer without iterating, such as myModel.parameter.data[8] or something similar.

Output:
simply do a :

layers=[x.data for x in myModel.parameters()]

Now it will be a list of weights and biases, in order to access weights of the first layer you can do:

print(layers[0])

in order to access biases of the first layer:

print(layers[1])

and so on.

Remember if bias is false for any particular layer it will have no entries at all, so for example if bias is false for second layer, then layers[3] will actually give weights of the third layer.


Instruction: Understanding ELMo's number of presentations

Input:
I am trying my hand at ELMo by simply using it as part of a larger PyTorch model. A basic example is given here.

  This is a torch.nn.Module subclass that computes any number of ELMo
  representations and introduces trainable scalar weights for each. For
  example, this code snippet computes two layers of representations (as
  in the SNLI and SQuAD models from our paper):

from allennlp.modules.elmo import Elmo, batch_to_ids

options_file = "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json"
weight_file = "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5"

# Compute two different representation for each token.
# Each representation is a linear weighted combination for the
# 3 layers in ELMo (i.e., charcnn, the outputs of the two BiLSTM))
elmo = Elmo(options_file, weight_file, 2, dropout=0)

# use batch_to_ids to convert sentences to character ids
sentences = [['First', 'sentence', '.'], ['Another', '.']]
character_ids = batch_to_ids(sentences)

embeddings = elmo(character_ids)

# embeddings['elmo_representations'] is length two list of tensors.
# Each element contains one layer of ELMo representations with shape
# (2, 3, 1024).
#   2    - the batch size
#   3    - the sequence length of the batch
#   1024 - the length of each ELMo vector

My question concerns the 'representations'. Can you compare them to normal word2vec output layers? You can choose how many ELMo will give back (increasing an n-th dimension), but what is the difference between these generated representations and what is their typical use?

To give you an idea, for the above code, embeddings['elmo_representations'] returns a list of two items (the two representation layers) but they are identical.

In short, how can one define the 'representations' in ELMo?

Output:
See Section 3.2 of the original paper.

  ELMo  is  a  task  specific  combination  of  the  intermediate layer representations in the biLM. For each token, a L-layer biLM computes a set of 2L+ 1representations

Previously in Section 3.1, it is said that:

  Recent  state-of-the-art  neural  language  models compute a context-independent token representation (via token embeddings or a CNN over characters) then pass it through L layers of forward LSTMs.  At each position k, each LSTM  layer  outputs  a  context-dependent  representation. The top layer LSTM output is used to predict the next token with a Softmax layer.

To answer your question, the representations are these L LSTM-based context-dependent representations.


Instruction: Pytorch torchvision MNIST download

Input:
I'm new to Pytorch and torchvision. I followed a tutorial that is roughly a year old where he tried to download mnist via python and torchvision.

This is how:

import torch
from torchvision import datasets, transforms

kwargs = {'num_workers': 1, 'pin_memory': True}
train = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True,
                   transform=transforms.Compose([transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)
test = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=False,
                   transform=transforms.Compose([transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

Now my problem is that I get this error:

  Traceback (most recent call last):

  Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data\MNIST\raw\train-images-idx3-ubyte.gz

  File "C:/Users/Nico/PycharmProjects/PyTorch/mnist.py", line 13, in
      transforms.Normalize((0.1307,), (0.3081,))])),

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\mnist.py", line 68, in init
      self.download()

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\mnist.py", line 143, in download
      download_url(url, root=self.raw_folder, filename=filename, md5=None)

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\utils.py", line 73, in download_url
      reporthook=gen_bar_updater(tqdm())

  TypeError: init() missing 1 required positional argument: 'total'

Do any of you guys know what I have to change, or how I can download/use them? As I said earlier I'm new to it and I don't have any clue.

I hope you guys can help me, thanks in advance.

Greetings Nico aka. Myridor

Output:
So the problem wasn't the code or the naming or anything.

It was the torchvision version. I had 0.2.2.post2 and it worked with 0.2.1!


Instruction: How to sort a dataset in pytorch

Input:
I would like to sort my dataset by the numerical values in the labels.

Is there a function from pytorch to handle this efficiently?

my dataset type() is in this from:

 <class 'torchvision.datasets.mnist.MNIST'>

Output:
There is no generic way to do this efficiently, as the dataset class is only implements a __getitem__ and __len__ method, and don't necessarily have any "stored" information about the labels.

In the case of the MNIST dataset class however you can sort the dataset from the label list.

For example when you want to list the indices that have the label 5.

mnist = torchvision.datasets.mnist.MNIST("/")
labels = mnist.train_labels
fives = (labels == 5).nonzero()


Instruction: Pytorch Indexing

Input:
I have a tensor [[1,2],[4,5],[7,8]] and a tensor with indices [0,1,0].

I want to apply them to second dimension so that it returns: [1,5,8].

How should I do that?

Thanks!

Output:
import torch

arr=torch.tensor([[1,2],[4,5],[7,8]])
indices_arr=torch.tensor([0,1,0])

ret=arr[[0,1,2],indices_arr]
# print(ret)
# tensor([1, 5, 7])


Instruction: How does pytorch backprop through argmax?

Input:
I'm building Kmeans in pytorch using gradient descent on centroid locations, instead of expectation-maximisation. Loss is the sum of square distances of each point to its nearest centroid.  To identify which centroid is nearest to each point, I use argmin, which is not differentiable everywhere.  However, pytorch is still able to backprop and update weights (centroid locations), giving similar performance to sklearn kmeans on the data.

Any ideas how this is working, or how I can figure this out within pytorch? Discussion on pytorch github suggests argmax is not differentiable: https://github.com/pytorch/pytorch/issues/1339.

Example code below (on random pts):

import numpy as np
import torch

num_pts, batch_size, n_dims, num_clusters, lr = 1000, 100, 200, 20, 1e-5

# generate random points
vector = torch.from_numpy(np.random.rand(num_pts, n_dims)).float()

# randomly pick starting centroids
idx = np.random.choice(num_pts, size=num_clusters)
kmean_centroids = vector[idx][:,None,:] # [num_clusters,1,n_dims]
kmean_centroids = torch.tensor(kmean_centroids, requires_grad=True)

for t in range(4001):
    # get batch
    idx = np.random.choice(num_pts, size=batch_size)
    vector_batch = vector[idx]

    distances = vector_batch - kmean_centroids # [num_clusters, #pts, #dims]
    distances = torch.sum(distances**2, dim=2) # [num_clusters, #pts]

    # argmin
    membership = torch.min(distances, 0)[1] # [#pts]

    # cluster distances
    cluster_loss = 0
    for i in range(num_clusters):
        subset = torch.transpose(distances,0,1)[membership==i]
        if len(subset)!=0: # to prevent NaN
            cluster_loss += torch.sum(subset[:,i])

    cluster_loss.backward()
    print(cluster_loss.item())

    with torch.no_grad():
        kmean_centroids -= lr * kmean_centroids.grad
        kmean_centroids.grad.zero_()

Output:
As alvas noted in the comments, argmax is not differentiable. However, once you compute it and assign each datapoint to a cluster, the derivative of loss with respect to the location of these clusters is well-defined. This is what your algorithm does.

Why does it work? If you had only one cluster (so that the argmax operation didn't matter), your loss function would be quadratic, with minimum at the mean of the data points. Now with multiple clusters, you can see that your loss function is piecewise (in higher dimensions think volumewise) quadratic - for any set of centroids [C1, C2, C3, ...] each data point is assigned to some centroid CN and the loss is locally quadratic. The extent of this locality is given by all alternative centroids [C1', C2', C3', ...] for which the assignment coming from argmax remains the same; within this region the argmax can be treated as a constant, rather than a function and thus the derivative of loss is well-defined.

Now, in reality, it's unlikely you can treat argmax as constant, but you can still treat the naive "argmax-is-a-constant" derivative as pointing approximately towards a minimum, because the majority of data points are likely to indeed belong to the same cluster between iterations. And once you get close enough to a local minimum such that the points no longer change their assignments, the process can converge to a minimum.

Another, more theoretical way to look at it is that you're doing an approximation of expectation maximization. Normally, you would have the "compute assignments" step, which is mirrored by argmax, and the "minimize" step which boils down to finding the minimizing cluster centers given the current assignments. The minimum is given by d(loss)/d([C1, C2, ...]) == 0, which for a quadratic loss is given analytically by the means of data points within each cluster. In your implementation, you're solving the same equation but with a gradient descent step. In fact, if you used a 2nd order (Newton) update scheme instead of 1st order gradient descent, you would be implicitly reproducing exactly the baseline EM scheme.


Instruction: How to add augmented images to original dataset using Pytorch?

Input:
From my understanding, RandomHorizontalFlip etc. replace image rather than adding new images to dataset. How do I increase my dataset size by adding augmented images to dataset using PyTorch?

I have gone through the links posted & haven't found a solution. I want to increase the data size by adding flipped/rotated images - but the post addresses the in-place processing of images.

Thanks.

Output:
Why do you want it? Generally speaking, it is enough to increase the number of epochs over the dataset, and your model will see the original and the augmented version of every image at least once (assuming a relatively high number of epochs).
Explanation:
For instance, if your augmentation has a chance of 50% to be applied, after 100 epochs, for every sample you will get ~50 samples of the original image and ~50 augmented samples. So, increasing the dataset size is equivalent to add epochs but (maybe) less efficient in terms of memory (need to store the images in memory to have high performances).


Instruction: RuntimeError: size mismatch, m1: [4 x 3136], m2: [64 x 5] at c:\a\w\1\s\tmp_conda_3.7_1

Input:
I used  python 3 and when i insert  transform random crop size 224 it gives miss match error.

here my code

what did i wrong ?

Output:
Your code makes variations on resnet: you changed the number of channels, the number of bottlenecks at each "level", and you removed a "level" entirely. As a result, the dimension of the feature map you have at the end of layer3 is not 64: you have a larger spatial dimension than you anticipated by the nn.AvgPool2d(8). The error message you got actually tells you that the output of level3 is of shape 64x56x56 and after avg pooling with kernel and stride 8 you have 64x7x7=3136 dimensional feature vector, instead of only 64 you are expecting.

What can you do?
As opposed to "standard" resnet, you removed stride from conv1 and you do not have max pool after conv1. Moreover, you removed layer4 which also have a stride. Therefore, You can add pooling to your net to reduce the spatial dimensions of layer3.
Alternatively, you can replace nn.AvgPool(8) with nn.AdaptiveAvgPool2d([1, 1]) an avg pool that outputs only one feature regardless of the spatial dimensions of the input feature map.


Instruction: What is loss_cls and loss_bbox and why are they always zero in training

Input:
I'm trying to train a custom dataset on using faster_rcnn using the Pytorch implementation of Detectron here. I have made changes to the dataset and configuration according to the guidelines in the repo.

The training process is carried out successfully, but the loss_cls and loss_bbox values are 0 from the beginning and even though the training is completed, final output cannot be used to make an evaluation or an inference.
I would like to know what these two mean and how to get those values to change during the training. The exact model I'm using here is e2e_faster_rcnn_R-50-FPN_1x

Any help regarding this would be appreciated. I' using Ubuntu 16.04 with Python 3.6 on Anaconda, CUDA 9, cuDNN 7.

Output:
What are the two losses?
When training a multi-object detector, you usually have (at least) two types of losses:

loss_bbox: a loss that measures how "tight" the predicted bounding boxes are to the ground truth object (usually a regression loss, L1, smoothL1 etc.).
loss_cls: a loss that measures the correctness of the classification of each predicted bounding box: each box may contain an object class, or a "background". This loss is usually called cross entropy loss.

Why are the losses always zero?
When training a detector, the model predict quite a few (~1K) possible boxes per image. Most of them are empty (i.e. belongs to "background" class). The loss function associate each of the predicted boxes with the ground truth boxes annotation of the image.
If a predicted box has a significant overlap with a ground truth box then loss_bbox and loss_cls are computed to see how well the model is able to predict the ground truth box.
On the other hand, if a predicted box has no overlap with any ground truth box, than only loss_cls is computed for the "background" class.
However, if there is only very partial overlap with ground truth the predicted box is "discarded" and no loss is computed. I suspect, for some reason, this is the case for your training session.
I suggest you check the parameters that determines the association between predicted boxed and ground truth annotations. Moreover, look at the parameters of your "anchors": these parameters determines the scale and aspect ratios of the predicted boxes.


Instruction: Why feature extraction of text don't return all possible feature names?

Input:
Here is the snippet of code from the book
Natural Language Processing with PyTorch:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import seaborn as sns

corpus = ['Time flies flies like an arrow.', 'Fruit flies like a banana.']
one_hot_vectorizer = CountVectorizer()
vocab = one_hot_vectorizer.get_feature_names()

The value of vocab :

vocab = ['an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time']

Why is not there an 'a' among the extracted feature names? If it is excluded as too common word automatically, why "an" is not excluded for the same reasons? How to make .get_feature_names() filter other words as well?

Output:
Very good question! Though this is not a pytorch question but a sklearn one =)

I encourage to first go through this https://www.kaggle.com/alvations/basic-nlp-with-nltk, esp. the "Vectorization with sklearn" section

TL;DR

If we use the CountVectorizer,

from io import StringIO
from sklearn.feature_extraction.text import CountVectorizer

sent1 = "The quick brown fox jumps over the lazy brown dog."
sent2 = "Mr brown jumps over the lazy fox."

with StringIO('\n'.join([sent1, sent2])) as fin:
    # Create the vectorizer
    count_vect = CountVectorizer()
    count_vect.fit_transform(fin)

# We can check the vocabulary in our vectorizer
# It's a dictionary where the words are the keys and
# The values are the IDs given to each word.
print(count_vect.vocabulary_)

[out]:

{'brown': 0,
 'dog': 1,
 'fox': 2,
 'jumps': 3,
 'lazy': 4,
 'mr': 5,
 'over': 6,
 'quick': 7,
 'the': 8}

We didn't tell the vectorizer to remove punctuation and tokenize and lowercase, how did they do it?

Also, the is in the vocabulary, it's a stopword, we want it gone...
And jumps isn't stemmed or lemmatized!

If we look at the documentation of the CountVectorizer in sklearn, we see:

CountVectorizer(
    input=’content’, encoding=’utf-8’,
    decode_error=’strict’, strip_accents=None,
    lowercase=True, preprocessor=None,
    tokenizer=None, stop_words=None,
    token_pattern=’(?u)\b\w\w+\b’, ngram_range=(1, 1),
    analyzer=’word’, max_df=1.0, min_df=1,
    max_features=None, vocabulary=None,
    binary=False, dtype=<class ‘numpy.int64’>)

And more specifically:

  analyzer : string, {‘word’, ‘char’, ‘char_wb’} or callable

  Whether the feature should be made of word or character n-grams.
  Option ‘char_wb’ creates character n-grams only from text inside word
  boundaries; n-grams at the edges of words are padded with space. If a
  callable is passed it is used to extract the sequence of features out
  of the raw, unprocessed input.

  preprocessor : callable or None (default)

  Override the preprocessing (string transformation) stage while
  preserving the tokenizing and n-grams generation steps.

  tokenizer : callable or None (default)

  Override the string tokenization step while preserving the
  preprocessing and n-grams generation steps. Only applies if analyzer
  == 'word'.

  stop_words : string {‘english’}, list, or None (default)

  If ‘english’, a built-in stop word list for English is used. If a
  list, that list is assumed to contain stop words, all of which will be
  removed from the resulting tokens. Only applies if analyzer == 'word'.
  If None, no stop words will be used.

  lowercase : boolean, True by default

  Convert all characters to lowercase before tokenizing.

But in the case of the example from http://shop.oreilly.com/product/0636920063445.do, it's not exactly the stopwords causing the issue.

If we explicitly use the English stopwords from https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> one_hot_vectorizer = CountVectorizer(stop_words='english')

>>> one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words='english',
        strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)

>>> one_hot_vectorizer.get_feature_names()
['arrow', 'banana', 'flies', 'fruit', 'like', 'time']

So what exactly is happening in the case where the stop_words argument is left as None?

Lets try an experiment where I add some single character words to the input:

>>> corpus = ['Time flies flies like an arrow 1 2 3.', 'Fruit flies like a banana x y z.']

>>> one_hot_vectorizer = CountVectorizer()

>>> one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)
>>> one_hot_vectorizer.get_feature_names()
['an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time']

They're all gone again!!!

Now if we dig into the docs, https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L738

  token_pattern : string
          Regular expression denoting what constitutes a "token", only used
          if analyzer == 'word'. The default regexp select tokens of 2
          or more alphanumeric characters (punctuation is completely ignored
          and always treated as a token separator).

Ah ha, that's why all the single character tokens gets deleted!

The default pattern for CountVectorizer is token_pattern=r"(?u)\b\w\w+\b", to enable it to take single character, you can try:

>>> one_hot_vectorizer = CountVectorizer(token_pattern=r"(?u)\b\w+\b")
>>> one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern='(?u)\\b\\w+\\b', tokenizer=None,
        vocabulary=None)
>>> one_hot_vectorizer.get_feature_names()
['1', '2', '3', 'a', 'an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time', 'x', 'y', 'z']


Instruction: After Pytorch Upgrade , my model is giving almost random output

Input:
I trained, tested and still using a model in "Pytorch 0.4.1". It was, and is still working fine (output is what it should be) if I use pitch 0.4.1.
But as i upgrade to version 1.0.1, every time, I try to evaluate same input image, I get different output (Its a regression).

I tried to see what has been changed in those versions, but since i am not getting any errors, warnings anything, i am not sure what should I look for specifically.

PS: I checked the weights, they are also same when I load the model

Output:
It was because of drop out layer. Model.eval disables the dropout layer. Pretty simple.
But Now in Pytorh upgrade, if Dropout is not defined specifically in model init function, it will not get disable during eval.
Atleast this was reason for my case.


Instruction: How does pytorch's nn.Module register submodule?

Input:
  When I read the source code(python) of torch.nn.Module , I found the
  attribute self._modules has been used in many functions like
  self.modules(), self.children(), etc. However, I didn't find any functions
  updating it. So, where will the self._modules be updated?
  Furthermore, how does pytorch's nn.Module register submodule?

class Module(object):
    def __init__(self):
        self._backend = thnn_backend
        self._parameters = OrderedDict()
        self._buffers = OrderedDict()
        self._backward_hooks = OrderedDict()
        self._forward_hooks = OrderedDict()
        self._forward_pre_hooks = OrderedDict()
        self._modules = OrderedDict()
        self.training = True

    def named_modules(self, memo=None, prefix=''):
        if memo is None:
            memo = set()
        if self not in memo:
            memo.add(self)
            yield prefix, self
            for name, module in self._modules.items():
                if module is None:
                    continue
                submodule_prefix = prefix + ('.' if prefix else '') + name
                for m in module.named_modules(memo, submodule_prefix):
                    yield m

Output:
The modules and parameters are usually registered by setting an attribute for an instance of nn.module.
Particularly, this kind of behavior is implemented by customizing the __setattr__ method:
def __setattr__(self, name, value):
        def remove_from(*dicts):
            for d in dicts:
                if name in d:
                    del d[name]

        params = self.__dict__.get('_parameters')
        if isinstance(value, Parameter):
            if params is None:
                raise AttributeError(
                    "cannot assign parameters before Module.__init__() call")
            remove_from(self.__dict__, self._buffers, self._modules)
            self.register_parameter(name, value)
        elif params is not None and name in params:
            if value is not None:
                raise TypeError("cannot assign '{}' as parameter '{}' "
                                "(torch.nn.Parameter or None expected)"
                                .format(torch.typename(value), name))
            self.register_parameter(name, value)
        else:
            modules = self.__dict__.get('_modules')
            if isinstance(value, Module):
                if modules is None:
                    raise AttributeError(
                        "cannot assign module before Module.__init__() call")
                remove_from(self.__dict__, self._parameters, self._buffers)
                modules[name] = value
            elif modules is not None and name in modules:
                if value is not None:
                    raise TypeError("cannot assign '{}' as child module '{}' "
                                    "(torch.nn.Module or None expected)"
                                    .format(torch.typename(value), name))
                modules[name] = value
            else:
                buffers = self.__dict__.get('_buffers')
                if buffers is not None and name in buffers:
                    if value is not None and not isinstance(value, torch.Tensor):
                        raise TypeError("cannot assign '{}' as buffer '{}' "
                                        "(torch.Tensor or None expected)"
                                        .format(torch.typename(value), name))
                    buffers[name] = value
                else:
                    object.__setattr__(self, name, value)

See https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py to find this method.


Instruction: pdb cannot debug into backward hooks

Input:
Here is my code.

import torch
v = torch.tensor([0., 0., 0.], requires_grad=True)
x = 1
def f(grad):
    global x
    x = 2
    return grad * 2
h = v.register_hook(f)  # double the gradient
v.backward(torch.tensor([1., 2., 3.]))
h.remove()
print(v.grad)

When I debug with pdb, I find that I cannot break in function f (I set a breakpoint inside f at the statement x = 2).

Does anyone know how to solve this?

Note: if I use pycharm, I can break into the function. But on the remote server, I would like to use pdb.

Output: You can try ipdb https://pypi.org/project/ipdb/ instead of pdb.


Instruction: Pytorch TypeError: eq() received an invalid combination of arguments

Input:
num_samples = 10
def predict(x):
    sampled_models = [guide(None, None) for _ in range(num_samples)]
    yhats = [model(x).data for model in sampled_models]
    mean = torch.mean(torch.stack(yhats), 0)
    return np.argmax(mean.numpy(), axis=1)

print('Prediction when network is forced to predict')
correct = 0
total = 0
for j, data in enumerate(test_loader):
    images, labels = data
    predicted = predict(images.view(-1,28*28))
    total += labels.size(0)
    correct += (predicted == labels).sum().item()
print("accuracy: %d %%" % (100 * correct / total))

Error:

correct += (predicted == labels).sum().item() TypeError:
eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
* (Tensor other)
  didn't match because some of the arguments have invalid types: (!numpy.ndarray!)
* (Number other)
  didn't match because some of the arguments have invalid types: (!numpy.ndarray!)

*

Output:
You are trying to compare predicted and labels. However, your predicted is an np.array while labels is a torch.tensor therefore eq() (the == operator) cannot compare between them.
Replace the np.argmax with torch.argmax:

 return torch.argmax(mean, dim=1)

And you should be okay.


Instruction: Why does creating a single tensor on the GPU take 2.5 seconds in PyTorch?

Input:
I'm just going through the beginner tutorial on PyTorch and noticed that one of the many different ways to put a tensor (basically the same as a numpy array) on the GPU takes a suspiciously long amount compared to the other methods:

import time
import torch

if torch.cuda.is_available():
    print('time =', time.time())
    x = torch.randn(4, 4)
    device = torch.device("cuda")
    print('time =', time.time())
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU  => 2.5 secs??
    print('time =', time.time())
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!
    a = torch.ones(5)
    print(a.cuda())
    print('time =', time.time())
else:
    print('I recommend you get CUDA to work, my good friend!')

Output (just times):

time = 1551809363.28284
time = 1551809363.282943
time = 1551809365.7204516  # (!)
time = 1551809365.7236063

Version details:

1 CUDA device: GeForce GTX 1050, driver version 415.27
CUDA          = 9.0.176
PyTorch       = 1.0.0
cuDNN         = 7401
Python        = 3.5.2
GCC           = 5.4.0
OS            = Linux Mint 18.3
Linux kernel  = 4.15.0-45-generic

As you can see this one operation ("y = ...") takes much longer (2.5 seconds) than the rest combined (.003 seconds). I'm confused about this as I expect all these methods to basically do the same. I've tried making sure the types in this line are 32 bit or have different shapes but that didn't change anything.

Output: When I re-order the commands, whatever command is on top takes 2.5 seconds. So this leads me to believe there is a delayed one-time setup of the device happening here, and future on-GPU allocations will be faster.


Instruction: Trying to print class names for dog breed but it keeps saying list index out of range

Input:
I am using a resnet model to classify dog breeds but when I try to print out an image with the label of dog breed it says list index out of range.
Here is my code:

import torchvision.models as models
import torch.nn as nn

model_transfer = models.resnet18(pretrained=True)

if use_cuda:
    model_transfer = model_transfer.cuda()

model_transfer.fc.out_features = 133

Then I train the model and get over 70% accuracy on the dog breeds.

Then here is my code to classify dog and print the dog breed:

data_transfer = {'train':
 datasets.ImageFolder('/data/dog_images/train',transform=transforms.Compose([transforms.RandomResizedCrop(224),transforms.ToTensor()]))}
class_names[0]
class_names = [item[4:].replace("_", " ") for item in data_transfer['train'].classes]

def predict_breed_transfer(img_path):

    image = Image.open(img_path)

    # large images will slow down processing

    in_transform = transforms.Compose([
                        transforms.CenterCrop(224),
                        transforms.ToTensor(),
                        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                             std=[0.229, 0.224, 0.225])])

    # discard the transparent, alpha channel (that's the :3) and add the batch dimension
    image = in_transform(image)[:3,:,:].unsqueeze(0)

    image = image

    output = model_transfer(image)
    pred = torch.argmax(output)

    return class_names[pred]
    predict_breed_transfer('images/Labrador_retriever_06455.jpg')

The code always predicts the dog wrong for some reason
Then when I try to print out the image and the label:

import matplotlib.pyplot as plt
def run_app(img_path):
    img = Image.open(img_path)
    dog = dog_detector(img_path)
    if not dog:
        print('hello, human!')
        plt.imshow(img)
        print('You look like a ... ')
        print(predict_breed_transfer(img_path))
    if dog:
        print('hello, dog!')
        print('Your predicted breed is ....')
        print(predict_breed_transfer(img_path))
        plt.imshow(img)
    else:
        print('Niether human nor dog')

And run a for loop that calls it on some dog images it will print some of the breeds out then it will say list index out of range and doesn't show any of the images.

The length of class_names is 133
And when I print out the resnet model the output is only 133 nodes does anyone know why it is saying list index out of range or why it is so inaccurate.

`IndexError                                Traceback (most recent
call last)
<ipython-input-26-473a9ba884b5> in <module>()
      5 ## suggested code, below
      6 for file in np.hstack((human_files[:3], dog_files[:3])):
----> 7     run_app(file)
      8
 <ipython-input-25-1d44200e44cc> in run_app(img_path)
      10         plt.show(img)
      11         print('You look like a ... ')
 ---> 12         print(predict_breed_transfer(img_path))
      13     if dog:
      14         print('hello, dog!')

 <ipython-input-20-a51fb205659e> in predict_breed_transfer(img_path)
      26     pred = torch.argmax(output)
      27
 ---> 28     return class_names[pred]
      29
predict_breed_transfer('images/Labrador_retriever_06455.jpg')
      30

IndexError: list index out of range`

Here is the full error

Output:
I suppose you have several issues that can be fixed using 13 chars.

First, I suggest what @Alekhya Vemavarapu suggested - run your code with a debugger to isolate each line and inspect the output. This is one of the greatest benefits of dynamic graphs with pytorch.

Secondly, the most probable cause for your issue is the argmax statement that you use incorrectly. You do not specify the dimension you perform the argmax on, and therefore PyTorch automatically flattens the image and perform the operation on the full length vector. Thus, you get a number between 0 and MB_Size x num_classes -1. See Official doc on this method.

So, due to your fully connected layer I assume your output is of shape (MB_Size, num_classes). If so, you need to change your code to the following line:

pred = torch.argmax(output,dim=1)

and thats it. Otherwise, just choose the dimension of the logits.

Third thing you want to consider is the dropout and other influences that a training configuration may cause to the inference. For instance, the dropout in some frameworks may require multiplying the ouptut by 1/(1-p) in the inference (or not since it can be done while training), batch normalization may be cancelled since the batch size is different, and so on. Additionally, to reduce memory consumption, no gradients should be computed. Luckily, PyTorch developers are very thoughtful and provided us with torch.no_grad() and model.eval() for that.

I strongly suggest to have a practice for that, possibly changing your code with a few letters:

output = model_transfer.eval()(image)

and your done!

Edit:
This is a simple use case of wrong usage of the PyTorch framework, not reading the docs and not debugging your code. The following code is purely incorrect:

model_transfer.fc.out_features = 133

This line does not actually creates a new fully connected layer. It just changes the property of that tensor. Try in your console:

import torch
a = torch.nn.Linear(1,2)
a.out_features = 3
print(a.bias.data.shape, a.weight.data.shape)

Output:

torch.Size([2]) torch.Size([2, 1])

which indicates that the actual matrix of the weights and the biases vector remain in their original dimension.
A correct way to perform transfer learning is to keep the backbone (usually the convolutional layers until the fully connected ones in these type of models) and overwriting the head (FC layer in this case) with yours. If it is just one fully connected layer that exists in the original model, you do not have to change the forward pass of your model and you're good to go.
Since this answer is already long enough, just visit the Transfer learning tutorial in PyTorch docs to see how it can be done.

Good luck to ya'll.


Instruction: Why doesn't the learning rate (LR) go below 1e-08 in pytorch?

Input:
I am training a model. To overcome overfitting I have done optimization, data augmentation etc etc. I have an updated LR (I tried for both SGD and Adam), and when there is a plateu (also tried step), the learning rate is decreased by a factor until it reaches LR 1e-08 but won't go below than that and my model's validation gets stuck after this point. I tried passing the epsilon parameter to Adam to suggest a smaller value, but it still got stuck at LR 1e-08. I also pass a weight decay, but it doesn't change the situation. Neither did setting the amsgrad to true.

I did some research and people suggest that Adam optimizer has inherent problems but nothing is mentioned about the learning rate - and every discussion added that with SGD, there is no problem.

Why is this? Is it a bug or is it designed so because authors think it is meaninglessly a small value after that? It seems like it would really help to have a smaller learning rate for my dataset because all seems well up until learning rate is down to LR 1e-08.

Output:
Personally I'm not aware of a lower limit on the learning rate (other than 0.0). But you can achieve the effect of a lower learning rate by reducing the loss before computing the backwards pass:

outputs = model(batch)
loss = criterion(outputs, targets)

# Equivalent to lowering the learning rate by a factor of 100
loss = loss / 100

self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()