Instruction: How to store switch locations during max-pooling layer

Input: I am implementing the article https://arxiv.org/abs/1311.2901 by Zeiler and Fergus on visualizing and understanding convolutional Networks. To be able reflect hidden layers back to the image space we need deconvolution kernels, rectified linear functions and switch locations. I couldn't find how to store switch locations during max-pooling. I would be glad if you can explain how to do it in pytorch or tensorflow. Thanks in advance.

Output: The pytorch max pool operation takes an optional argument return_indices which is set to False by default. If you set that to True, the output will be the max pooled tensor as well the indices of the maximum items.


Instruction: what is the first initialized weight in pytorch convolutional layer

Input:
I do self-studying in Udacity PyTorch
Regarding to the last paragraph

Learning
In the code you've been working with, you've been setting the values of filter weights explicitly, but neural networks will actually learn the best filter weights as they train on a set of image data. You'll learn all about this type of neural network later in this section, but know that high-pass and low-pass filters are what define the behavior of a network like this, and you know how to code those from scratch!
In practice, you'll also find that many neural networks learn to detect the edges of images because the edges of object contain valuable information about the shape of an object.

I have studied all through the last 44th sections. But I couldn't be able to answer the following questions

What is the initialized weight when I do torch.nn.Conv2d? And how to define it myself?
How does PyTorch update weights in the convolutional layer?

Output:
When you declared nn.Conv2d the weights are initialized via this code.

In particular, if you give bias it uses initialization as proposed by Kaiming et.al. It initializes as uniform distribution between (-bound, bound) where bound=\sqrt{6/((1+a^2)fan_in)} (See here).

You can initialize weight manually too. This has been answered elsewhere (See here) and I won't repeat it.

When you call optimizer.step and optimizer has parameters of convolutional filter registered they are updated.


Instruction: How can i process multi loss in pytorch?

Input:
Such as this, I want to using some auxiliary loss to promoting my model performance.
Which type code can implement it in pytorch?

#one
loss1.backward()
loss2.backward()
loss3.backward()
optimizer.step()
#two
loss1.backward()
optimizer.step()
loss2.backward()
optimizer.step()
loss3.backward()
optimizer.step()
#three
loss = loss1+loss2+loss3
loss.backward()
optimizer.step()

Thanks for your answer!

Output:
First and 3rd attempt are exactly the same and correct, while 2nd approach is completely wrong.
In Pytorch, low layer gradients are Not  "overwritten" by subsequent backward() calls, rather they are accumulated, or summed. This makes first and 3rd approach identical, though 1st approach might be preferable if you have low-memory GPU/RAM (a batch size of 1024 with one backward() + step() call is same as having 8 batches of size 128 and 8 backward() calls, with one step() call in the end).
To illustrate the idea, here is a simple example. We want to get our tensor x close to 40,50 and 60 simultaneously:
x = torch.tensor([1.0],requires_grad=True)
loss1 = criterion(40,x)
loss2 = criterion(50,x)
loss3 = criterion(60,x)

Now the first approach: (we use tensor.grad to get current gradient for our tensor x)
loss1.backward()
loss2.backward()
loss3.backward()

print(x.grad)

This outputs : tensor([-294.]) (EDIT: put retain_graph=True in first two backward calls for more complicated computational graphs)
The third approach:
loss = loss1+loss2+loss3
loss.backward()
print(x.grad)

Again the output is : tensor([-294.])
2nd approach is different because we don't call opt.zero_grad after calling step() method. This means in all 3 step calls gradients of first backward call is used. For example, if 3 losses provide gradients 5,1,4 for same weight, instead of having 10 (=5+1+4), now your weight will have 5*3+1*2+4*1=21 as gradient.
For further reading : Link 1,Link 2


Instruction: axes don't match array in pytorch

Input:
I am new to pytorch and i am stuck in this for more than a week now.
i am trying to use AlexNet to make a 'gta san Andreas' self driving car and i am having alot of problems with preparing the data.
for now i am getting this error.

Traceback (most recent call last):
  File "training_script.py", line 19, in <module>
    transformed_data = transform(all_data)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 49, in __call__
    img = t(img)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\functional.py", line 48, in to_tensor
    img = torch.from_numpy(pic.transpose((2, 0, 1)))
ValueError: axes don't match array

this is the training script

from AlexNetPytorch import*
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.utils.data
import numpy as np
import torch

AlexNet = AlexNet()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(AlexNet.parameters(), lr=0.001, momentum=0.9)

all_data = np.load('training_data.npy')
transform = transforms.Compose([
    # you can add other transformations in this list
    transforms.ToTensor()
])
transformed_data = transform(all_data)
# # data_set = torchvision.datasets.ImageFolder('training_data.npy' ,transform = transforms.ToTensor() )
data_loader = torch.utils.data.DataLoader(training_data, batch_size=4,shuffle=True, num_workers=2)
# training_data = all_data[:-500]lk
# testing_data = all_data[-500:]

if __name__ == '__main__':
 for epoch in range(8):
  runing_loss = 0.0
  for i,data in enumerate(data_loader , 0):
     inputs= data[0]
     inputs = torch.FloatTensor(inputs)
     labels= data[1]
     labels = torch.FloatTensor(labels)
     optimizer.zero_grad()
     outputs = AlexNet(inputs)
     loss = criterion(outputs , labels)
     loss.backward()
     optimizer.step()

     runing_loss +=loss.item()
     if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
 print('finished')

and this is how i am preparing the data

import cv2
from PIL import ImageGrab
import numpy as np
import time
from directKeys import PressKey,W,A,S,D
from getKeys import key_check
import os

def keys_to_output(keys):
    output = [0,0,0]

    if 'A' in keys:
        output[0] = 1
    elif 'D' in keys:
        output[2] = 1
    else:
        output[1] = 1
    return output

file_name = "training_data.npy"
if os.path.isfile(file_name):
   print("file exists , loading previous data!")
   training_data = list(np.load(file_name))
else:
   print("file does not exist , starting fresh")
   training_data = []

last_time = time.time()

while True:

   kernel = np.ones((15 , 15) , np.float32)/225
   get_screen = ImageGrab.grab(bbox=(10,10,1280,720))
   screen_shot = np.array(get_screen)
   hsv = cv2.cvtColor(screen_shot , cv2.COLOR_BGR2HSV)
   lower_color = np.array([90 , 0 , 70])
   upper_color = np.array([100 , 100 ,  100])

   output = cv2.inRange(hsv , lower_color , upper_color)

   kernel = np.ones((1,20), np.uint8)  # note this is a horizontal kernel
   dilation = cv2.dilate(output, kernel, iterations=1)
   output = cv2.erode(dilation, kernel, iterations=1)
   # output = cv2.Canny(output , threshold1 = 50 , threshold2 = 300)
   # output = cv2.GaussianBlur(output , (15,15) , 0)

   resized = cv2.resize(output , (640 , 480))

   print('loop took {} seconds'.format(time.time()-last_time))
   last_time = time.time()

   cv2.imshow('manipulated' , resized)
   screen_output = cv2.resize(output , (32 ,32))
   keys = key_check()
   Keys_output = keys_to_output(keys)
   training_data.append([screen_output,Keys_output])

   if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

   if len(training_data) % 500 == 0:
       print(len(training_data))
       np.save(file_name,training_data)

I tried a lot of solutions but none of them work, but I feel like I am missing something.
i am way over my head so please help

Output: You are applying the transformation to a list of numpy arrays instead of to a single PIL image (which is usually what ToTensor() transforms expects).


Instruction: How does the "number of workers" parameter in PyTorch dataloader actually work?

Input:
If num_workers is 2, Does that mean that it will put 2 batches in the RAM and send 1 of them to the GPU or Does it put 3 batches in the RAM then sends 1 of them to the GPU?
What does actually happen when the number of workers is higher than the number of CPU cores? I tried it and it worked fine but How does it work? (I thought that the maximum number of workers I can choose is the number of cores).
If I set num_workers to 3 and during the training there were no batches in the memory for the GPU, Does the main process waits for its workers to read the batches or Does it read a single batch (without waiting for the workers)?

Output:
When num_workers>0, only these workers will retrieve data, main process won't. So when  num_workers=2 you have at most 2 workers simultaneously putting data into RAM, not 3.
Well our CPU can usually run like 100 processes without trouble and these worker processes aren't special in anyway, so having more workers than cpu cores is ok. But is it efficient? it depends on how busy your cpu cores are for other tasks, speed of cpu, speed of your hard disk etc. In short, its complicated, so setting workers to number of cores is a good rule of thumb, nothing more.
Nope. Remember DataLoader doesn't just randomly return from what's available in RAM right now, it uses batch_sampler to decide which batch to return next. Each batch is assigned to a worker, and main process will wait until the desired batch is retrieved by assigned worker.

Lastly to clarify, it isn't DataLoader's job to send anything directly to GPU, you explicitly call cuda() for that.

EDIT: Don't call cuda() inside Dataset's __getitem__() method, please look at @psarka's comment for the reasoning


Instruction: How to use AlexNet with one channel

Input:
I am new to pytorch and had a problem with channels in AlexNet.
I am using it for a ‘gta san andreas self driving car’ project, I collected the dataset from a black and white image that has one channel and trying to train AlexNet using the script:

from AlexNetPytorch import*
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.utils.data
import numpy as np
import torch
from IPython.core.debugger import set_trace

AlexNet = AlexNet()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(AlexNet.parameters(), lr=0.001, momentum=0.9)

all_data = np.load('training_data.npy')
inputs= all_data[:,0]
labels= all_data[:,1]
inputs_tensors = torch.stack([torch.Tensor(i) for i in inputs])
labels_tensors = torch.stack([torch.Tensor(i) for i in labels])

data_set = torch.utils.data.TensorDataset(inputs_tensors,labels_tensors)
data_loader = torch.utils.data.DataLoader(data_set, batch_size=3,shuffle=True, num_workers=2)

if __name__ == '__main__':
 for epoch in range(8):
  runing_loss = 0.0
  for i,data in enumerate(data_loader , 0):
     inputs= data[0]
     inputs = torch.FloatTensor(inputs)
     labels= data[1]
     labels = torch.FloatTensor(labels)
     optimizer.zero_grad()
     # set_trace()
     inputs = torch.unsqueeze(inputs, 1)
     outputs = AlexNet(inputs)
     loss = criterion(outputs , labels)
     loss.backward()
     optimizer.step()

     runing_loss +=loss.item()
     if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
 print('finished')

I am using AlexNet from the link:
https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py

But changed line 18 from :

nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2)

To :

nn.Conv2d(1, 64, kernel_size=11, stride=4, padding=2)

Because I am using only one channel in training images, but I get this error:

 File "training_script.py", line 44, in <module>
    outputs = AlexNet(inputs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Documents\AI_projects\gta\AlexNetPytorch.py", line 34, in forward
    x = self.features(x)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 91, in forward
    input = module(input)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\pooling.py", line 142, in forward
    self.return_indices)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\functional.py", line 396, in max_pool2d
    ret = torch._C._nn.max_pool2d_with_indices(input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: Given input size: (256x1x1). Calculated output size: (256x0x0). Output size is too small at c:\programdata\miniconda3\conda-bld\pytorch-cpu_1532499824793\work\aten\src\thnn\generic/SpatialDilatedMaxPooling.c:67

I don't know what is wrong, is it wrong to change the channel size like this, and if it is wrong can you please lead me to a neural network that work with one channel , as I said I am a newbie in pytorch and I don't want to write the nn myself.

Output:
Your error is not related to using gray-scale images instead of RGB. Your error is about the spatial dimensions of the input: while "forwarding" an input image through the net, its size (in feature space) became zero - this is the error you see. You can use this nice guide to see what happens to the output size of each layer (conv/pooling) as a function of kernel size, stride and padding.
Alexnet expects its input images to be 224 by 224 pixels - make sure your inputs are of the same size.

Other things you overlooked:

You are using Alexnet architecture, but you are initializing it to random weights instead of using pretrained weights (trained on imagenet). To get a trained copy of alexnet you'll need to instantiate the net like this

AlexNet = alexnet(pretrained=True)

Once you decide to use pretrained net, you cannot change its first layer from 3 input channels to three (the trained weight simply won't fit). The easiest fix is to make your input images "colorful" by simply repeating the single channel three times. See repeat() for more info.


Instruction: How do I implement a PyTorch Dataset for use with AWS SageMaker?

Input:
I have implemented a PyTorch Dataset that works locally (on my own desktop), but when executed on AWS SageMaker, it breaks. My Dataset implementation is as follows.

class ImageDataset(Dataset):
    def __init__(self, path='./images', transform=None):
        self.path = path
        self.files = [join(path, f) for f in listdir(path) if isfile(join(path, f)) and f.endswith('.jpg')]
        self.transform = transform
        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((128, 128)),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
            ])

    def __len__(self):
        return len(files)

    def __getitem__(self, idx):
        img_name = self.files[idx]

        # we may infer the label from the filename
        dash_idx = img_name.rfind('-')
        dot_idx = img_name.rfind('.')
        label = int(img_name[dash_idx + 1:dot_idx])

        image = Image.open(img_name)

        if self.transform:
            image = self.transform(image)

        return image, label

I am following this example and this one too, and I run the estimator as follows.

inputs = {
 'train': 'file://images',
 'eval': 'file://images'
}
estimator = PyTorch(entry_point='pytorch-train.py',
                            role=role,
                            framework_version='1.0.0',
                            train_instance_count=1,
                            train_instance_type=instance_type)
estimator.fit(inputs)

I get the following error.

  FileNotFoundError: [Errno 2] No such file or directory: './images'

In the example that I am following, they upload the CFAIR dataset (which is downloaded locally) to S3.

inputs = sagemaker_session.upload_data(path='data', bucket=bucket, key_prefix='data/cifar10')

If I take a peek at inputs, it is just a string literal s3://sagemaker-us-east-3-184838577132/data/cifar10. The code to create a Dataset and a DataLoader is shown here, which does not help unless I track down the source and step through the logic.

I think what needs to happen inside my ImageDataset is to supply the S3 path and use the AWS CLI or something to query the files and acquire their content. I do not think the AWS CLI is the right approach as this relies on the console and I will have to execute some sub-process commands and then parse through.

There must be a recipe or something to create a custom Dataset backed by S3 files, right?

Output:
I was able to create a PyTorch Dataset backed by S3 data using boto3. Here's the snippet if anyone is interested.

class ImageDataset(Dataset):
    def __init__(self, path='./images', transform=None):
        self.path = path
        self.s3 = boto3.resource('s3')
        self.bucket = self.s3.Bucket(path)
        self.files = [obj.key for obj in self.bucket.objects.all()]
        self.transform = transform
        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((128, 128)),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
            ])

    def __len__(self):
        return len(files)

    def __getitem__(self, idx):
        img_name = self.files[idx]

        # we may infer the label from the filename
        dash_idx = img_name.rfind('-')
        dot_idx = img_name.rfind('.')
        label = int(img_name[dash_idx + 1:dot_idx])

        # we need to download the file from S3 to a temporary file locally
        # we need to create the local file name
        obj = self.bucket.Object(img_name)
        tmp = tempfile.NamedTemporaryFile()
        tmp_name = '{}.jpg'.format(tmp.name)

        # now we can actually download from S3 to a local place
        with open(tmp_name, 'wb') as f:
            obj.download_fileobj(f)
            f.flush()
            f.close()
            image = Image.open(tmp_name)

        if self.transform:
            image = self.transform(image)

        return image, label


Instruction: How to set a variable as an attribute?

Input:
I'm getting error when trying to set variable as an atribute.

parser = argparse.ArgumentParser()

parser.add_argument('--arch', action='store',
                dest='arch', default='alexnet',
                help='Store a simple value')

args = parser.parse_args()

model = models.args.arch(pretrained=True)

I know models.args.arch prodocues an error but how syntax should looke like to set a variable as an attribute? I could do it with if statements but it would be lot's of code and I guess it's possible in 1 line.

Output:
You want to access the internal dict to update:

model = models.__dict__[args.arch](pretrained=True)

or using getattr:

 getattr(models, args.arch)(pretrained=True)


Instruction: pytorch dataloader stucked if using opencv resize method

Input:
I can run all the cells of the tutorial notebook of Pytorch about dataloading (pytorch tutorial).
But when I use OpenCV in place of Skimage to resize the image, the dataloader gets stuck, i.e nothing happens.

In the Rescale class:

class Rescale(object):
    .....
    def __call__(self, sample):
       ....
       #img = transform.resize(image, (new_h, new_w))
       img = cv2.resize(image, (new_h, new_w))
       .....

The dataloader and the for loop are defined with:

dataloader = DataLoader(transformed_dataset, batch_size=4,
                        shuffle=True, num_workers=4)

for i_batch, sample_batched in enumerate(dataloader):
    print(i_batch, sample_batched['image'].size(),
          sample_batched['landmarks'].size())

I can get the iterator to print something if num_workers=0. It looks like opencv does not play well with the Multiprocessing of pytorch.
I would really prefer to use same package to transform the images at train time and test time (and I am already using OpenCV for the image rescale at test time).
Any suggestions would be greatly appreciated.

Output:
I had a very similar problem and that's how I solved it:

when you import cv2 set cv2.setNumThreads(0)
and then you can set num_workers>0 in the dataloader in PyTorch.

Seems like OpenCV tries to multithread and somewhere something goes into a deadlock.

Hope it helps.


Instruction: Selection/Filter with indices using pytorch

Input:
I have a 3-dimensional numpy array, for example:

 x = [[[0.3, 0.2, 0.5],
       [0.1, 0.2, 0.7],
       [0.2, 0.2, 0.6]]]

The indices array is also 3-dimensional, like:

indices = [[[0],
            [1],
            [2]]]

I expect the output is:

 output= [[[0.3],
           [0.2],
           [0.6]]]

I tried the torch.index_select and torch.gather function, but I couldn't find a right way to deal with the dimension. Thanks for any help!

Output: How about using x.gather(dim=2, indices)? That works for me.


Instruction: What does the & operator do in PyTorch and why does it change the shape?

Input:
I have code that contains x and y, both of the type torch.autograd.variable.Variable. Their shape is

torch.Size([30, 1, 9])
torch.Size([1, 9, 9])

What I don't understand is, why the following results in a different size/shape

z = x & y
print(z.shape)

which outputs

torch.Size([30, 9, 9])

Why is the shape of z 30*9*9, after x & y? The shape of x is 30*1*9, and the shape of y is 1*9*9, what does & do in x & y?

Output:
This has nothing to do with the & operator, but with how broadcasting works in Python. To quote Eric Wieser's excellent documentation on broadcasting in NumPy:

  In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.

See the following image from the quoted page as an example:

This translates to your problem as follows:

a has the shape 30 x 1 x 9
b has the shape  1 x 9 x 9

Therefore the result is created like this:

result1 is a1, because a1 > b1
result2 is b2, because a2 < b2
result3 is both a3 and b3, because a3 = b3

Therefore result has the shape 30 x 9 x 9.

Please also note that the & operator implements logical conjunction on a binary encoding of the tensors' items.


Instruction: The correct way to build a binary classifier for CNN

Input:
I created a neural network on pytorch using the pretraining model VGG16 and added my own extra layer to define belonging to one of two classes. For example bee or ant.

model = models.vgg16(pretrained=True)
# Freeze early layers
for param in model.parameters():
    param.requires_grad = False
n_inputs = model.classifier[6].in_features
# Add on classifier
model.classifier[6] = nn.Sequential(
    nn.Linear(n_inputs, 256), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(256, 2), nn.LogSoftmax(dim=1))

The model works well with two classes, but if to upload a crocodile image into it, it is likely to take it for a bee)
Now I want to make a binary classifier based on this model, which defines for example a bee or not a bee (absolutely any image without a bee)
I am just starting to understand neural networks and I need advice on whether the correct way would be training in two groups of images in one of which will only bees and in the other several thousand random images. Or should it be done in another way?

Output:
It is indeed not surprising that a 2-class classifier fails with an image not belonging to any class.

To train your new one-class classifier, yes, use in our test set bees images and a set of non bee images. You need to accommodate for the imbalance between the classes as well to avoid overfitting just the bees images you have. The test accuracy would show such a bias.


Instruction: A vector and matrix rows cosine similarity in pytorch

Input: In pytorch, I have multiple (scale of hundred thousand) 300 dim vectors (which I think I should upload in a matrix), I want to sort them by their cosine similarity with another vector and extract the top-1000. I want to avoid for loop as it is time consuming. I was looking for an efficient solution.

Output:
You can use torch.nn.functional.cosine_similarity function for computing cosine similarity. And torch.argsort to extract top 1000.

Here is an example:

x = torch.rand(10000,300)
y = torch.rand(1,300)
dist = F.cosine_similarity(x,y)
index_sorted = torch.argsort(dist)
top_1000 = index_sorted[:1000]

Please note the shape of y, don't forget to reshape before calling similarity function. Also note that argsort simply returns the indexes of closest vectors. To access those vectors themselves, just write x[top_1000], which will return a matrix shaped (1000,300).


Instruction: There is an error with LSTM Hidden state dimension: RuntimeError: Expected hidden[0] size (4, 1, 256), got (1, 256)

Input:
I'm experimenting with seq2seq_tutorial in PyTorch. There appears to be a dimension error with the encoder's lstm hidden state size.

With bidirectional=True and num_layers = 2, the hidden state's shape is supposed to be (num_layers*2, batch_size, hidden_size).

However, an error occurs with the following message:

RuntimeError: Expected hidden[0] size (4, 1, 256), got (1, 256)

I have tried reshaping the hidden state to initializing the hidden state with a different shape, to begin with, but nothing seems to work.

Here is the train method of my code:

def train(self, input, target, encoder, decoder, encoder_optim, decoder_optim, criterion):
    enc_optimizer = encoder_optim
    dec_optimizer = decoder_optim
    enc_optimizer.zero_grad()
    dec_optimizer.zero_grad()

    pair = (input, target)
    input_len = input.size(0)
    target_len = target.size(0)
    enc_output_tensor = torch.zeros(self.opt['max_seq_len'], encoder.hidden_size, device=device)
    enc_hidden = encoder.cuda().initHidden(device)

    for word_idx in range(input_len):
        print('Input:', input[word_idx], '\nHidden shape:', enc_hidden.size())
        enc_output, enc_hidden = encoder(input[word_idx], enc_hidden)
        enc_output_tensor[word_idx] = enc_output[0,0]

Here is the encoder method of my code:

class EncoderBRNN(nn.Module):
    # A bidirectional rnn based encoder
    def __init__(self, input_size, hidden_size, emb_size, batch_size=1, num_layers=2, bidir=True):
        super(EncoderBRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.embedding_dim = emb_size
        self.num_layers = num_layers
        self.bidir = bidir
        self.embedding_layer = nn.Embedding(self.input_size, self.embedding_dim)
        self.enc_layer = nn.LSTM(self.embedding_dim, self.hidden_size, num_layers=self.num_layers, bidirectional=self.bidir)

    def forward(self, input, hidden):
        embed = self.embedding_layer(input).view(1, 1, -1)
        output, hidden = self.enc_layer(embed, hidden)
        return output, hidden

    def initHidden(self, device):
        if self.bidir:
            num_stacks = self.num_layers * 2
        else:
            num_stacks = self.num_layers
        return torch.zeros(num_stacks, self.batch_size, self.hidden_size, device=device)

Output:
I know this was asked a while ago but I think I found the answer to this in this torch discussion. Relevant info:

  LSTM takes a tuple of hidden states: self.rnn(x, (h_0, c_0)) it looks like you haven’t sent in the second hidden state?

You can also see this in the documentation for LSTM


Instruction: Gradually decay the weight of loss function

Input:
I am not sure is the right place to ask this question, feel free to tell me if I need to remove the post.

I am quite new in pyTorch and currently working with CycleGAN (pyTorch implementation) as a part of my project and I understand most of the implementation of cycleGAN.

I read the paper with the name ‘CycleGAN with better Cycles’ and I am trying to apply the modification which mentioned in the paper. One of modification is Cycle consistency weight decay which I don’t know how to apply.

optimizer_G.zero_grad()

# Identity loss
loss_id_A = criterion_identity(G_BA(real_A), real_A)
loss_id_B = criterion_identity(G_AB(real_B), real_B)

loss_identity = (loss_id_A + loss_id_B) / 2

# GAN loss
fake_B = G_AB(real_A)
loss_GAN_AB = criterion_GAN(D_B(fake_B), valid)
fake_A = G_BA(real_B)
loss_GAN_BA = criterion_GAN(D_A(fake_A), valid)

loss_GAN = (loss_GAN_AB + loss_GAN_BA) / 2

# Cycle consistency loss
recov_A = G_BA(fake_B)
loss_cycle_A = criterion_cycle(recov_A, real_A)
recov_B = G_AB(fake_A)
loss_cycle_B = criterion_cycle(recov_B, real_B)

loss_cycle = (loss_cycle_A + loss_cycle_B) / 2

# Total loss
loss_G =    loss_GAN +
            lambda_cyc * loss_cycle + #lambda_cyc is 10
            lambda_id * loss_identity #lambda_id is 0.5 * lambda_cyc

loss_G.backward()
optimizer_G.step()

My question is how can I gradually decay the weight of cycle consistency loss?

Any help in implementing this modification would be appreciated.

This is from the paper:
Cycle consistency loss helps to stabilize training a lot in early stages but becomes an obstacle towards realistic images in later stages. We propose to gradually decay the weight of cycle consistency loss λ as training progress. However, we should still make sure that λ is
not decayed to 0 so that generators won’t become unconstrained and go completely wild.

Thanks in advance.

Output:
Below is a prototype function you can use!

def loss (other params, decay params, initial_lambda, steps):
    # compute loss
    # compute cyclic loss
    # function that computes lambda given the steps
    cur_lambda  = compute_lambda(step, decay_params, initial_lamdba)

    final_loss = loss + cur_lambda*cyclic_loss
    return final_loss

compute_lambda function for linearly decaying from 10 to 1e-5 in 50 steps

def compute_lambda(step, decay_params):
    final_lambda = decay_params["final"]
    initial_lambda = decay_params["initial"]
    total_step = decay_params["total_step"]
    start_step = decay_params["start_step"]

    if (step < start_step+total_step and step>start_step):
        return initial_lambda + (step-start_step)*(final_lambda-initial_lambda)/total_step
    elif (step < start_step):
        return initial_lambda
    else:
        return final_lambda
# Usage:
compute_lambda(i, {"final": 1e-5, "initial":10, "total_step":50, "start_step" : 50})


Instruction: pytorch:RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)

Input:
When i using the cross-entropy loss as a loss function,i get this dimension out of range error.

This is my code:

self.ce = nn.CrossEntropyLoss()
    def forward(self, pred, y):
        loss = 0
        for w_, p_, y_ in zip(self.weights, pred, y):
            loss += w_ * self.ce(p_, y_)
        return loss

when i run this code :

the value of p_:tensor(1.00000e-02 *[-0.7625,  5.8737], device='cuda:0')
the value of w_:tensor(1., device='cuda:0')
the value of y_:tensor(0, device='cuda:0')

Output:
For cross entropy there should be the same number of labels as predictions.

In your specific case the dimensions of y_ and p_ should match which they don't as y_ is a 0 dimensional scalar and p_ is 1x2.


Instruction: Pytorch: AttributeError: cannot assign module before Module.__init__() call even if initialized

Input:
I'm getting the following error:

  AttributeError: cannot assign module before Module.init() call

I'm trying to create an instance of my class :

class ResNetGenerator(nn.Module):
    def __init__(self, input_nc=3, output_nc=3, n_residual_blocks=9, use_dropout=False):
        # super(ResNetGenerator, self).__init__()
        super().__init__()

I'm calling super().__init__() but in vain.

What I am doing wrong here?

Complete Traceback:

File "train.py", line 40, in <module>
    model = ColorizationCycleGAN(args)
File "/path/cycle_gan.py", line 27, in __init__
    self.G_A2B = ResNetGenerator(input_nc=self.input_nc, output_nc=self.output_nc, n_residual_blocks=9, use_dropout=False)
File "/path/.local/lib/python3.6/site packages/torch/nn/modules/module.py", line 544, in __setattr__
    "cannot assign module before Module.__init__() call")
AttributeError: cannot assign module before Module.__init__() call

Output:
In fact, I realized that i wasn't calling super().__init__() in the main class ColorizationCycleGAN. Adding this solved the problem.

I hope that this answer will have the effect of reminding you to check to call the super().__init__() function in all classes that inherits from nn.Module.


Instruction: Index pytorch tensor with different dimension index array

Input:
I have the following function, which does what I want using numpy.array, but breaks when feeding a torch.Tensor due to indexing errors.

import torch
import numpy as np

def combination_matrix(arr):
    idxs = np.arange(len(arr))
    idx = np.ix_(idxs, idxs)
    mesh = np.stack(np.meshgrid(idxs, idxs))

    def np_combination_matrix():
        output = np.zeros((len(arr), len(arr), 2, *arr.shape[1:]), dtype=arr.dtype)
        num_dims = len(output.shape)
        output[idx] = arr[mesh].transpose((2, 1, 0, *np.arange(3, num_dims)))
        return output

    def torch_combination_matrix():
        output = torch.zeros(len(arr), len(arr), 2, *arr.shape[1:], dtype=arr.dtype)
        num_dims = len(output.shape)
        print(arr[mesh].shape)  # <-- This is wrong/different to numpy!
        output[idx] = arr[mesh].permute(2, 1, 0, *np.arange(3, num_dims))
        return output

    if isinstance(arr, np.ndarray):
        return np_combination_matrix()
    elif isinstance(arr, torch.Tensor):
        return torch_combination_matrix()

The problem is that arr[mesh] results in different dimensions, depending on numpy and torch. Apparently, pytorch does not support indexing with index arrays of different dimensionality than the array being indexed. Ideally, the following should work:

features = np.arange(9).reshape(3, 3)
np_combs = combination_matrix(features)
features = torch.from_numpy(features)
torch_combs = combination_matrix(features)
assert np.array_equal(np_combs, torch_combs.numpy())

But the dimensions are different:

(2, 3, 3, 3)
torch.Size([3, 3])

Which results in an error (logically):

Traceback (most recent call last):
  File "/home/XXX/util.py", line 226, in <module>
    torch_combs = combination_matrix(features)
  File "/home/XXX/util.py", line 218, in combination_matrix
    return torch_combination_matrix()
  File "/home/XXX/util.py", line 212, in torch_combination_matrix
    output[idx] = arr[mesh].permute(2, 1, 0, *np.arange(3, num_dims))
RuntimeError: number of dims don't match in permute

How do I match the torch behavior to numpy?
I've read various questions on the torch forums (e.g. this one with only one dimension), but could find how to apply this here. Similarly, index_select only works for one dimension, but I need it to work for at least 2 dimensions.

Output:
It is actually embarrassingly easy. You just need to flatten the indices, then reshape and permute the dimensions.
This is the full working version:

import torch
import numpy as np

def combination_matrix(arr):
    idxs = np.arange(len(arr))
    idx = np.ix_(idxs, idxs)
    mesh = np.stack(np.meshgrid(idxs, idxs))

    def np_combination_matrix():
        output = np.zeros((len(arr), len(arr), 2, *arr.shape[1:]), dtype=arr.dtype)
        num_dims = len(output.shape)
        output[idx] = arr[mesh].transpose((2, 1, 0, *np.arange(3, num_dims)))
        return output

    def torch_combination_matrix():
        output_shape = (2, len(arr), len(arr), *arr.shape[1:])  # Note that this is different to numpy!
        return arr[mesh.flatten()].reshape(output_shape).permute(2, 1, 0, *range(3, len(output_shape)))

    if isinstance(arr, np.ndarray):
        return np_combination_matrix()
    elif isinstance(arr, torch.Tensor):
        return torch_combination_matrix()

I used pytest to run this on random arrays of different dimensions, and it seems to work in all cases:

import pytest

@pytest.mark.parametrize('random_dims', range(1, 5))
def test_combination_matrix(random_dims):
    dim_size = np.random.randint(1, 40, size=random_dims)
    elements = np.random.random(size=dim_size)
    np_combs = combination_matrix(elements)
    features = torch.from_numpy(elements)
    torch_combs = combination_matrix(features)

    assert np.array_equal(np_combs, torch_combs.numpy())

if __name__ == '__main__':
    pytest.main(['-x', __file__])


Instruction: Indexing the max elements in a multidimensional tensor in PyTorch

Input:
I'm trying to index the maximum elements along the last dimension in a multidimensional tensor. For example, say I have a tensor

A = torch.randn((5, 2, 3))
_, idx = torch.max(A, dim=2)

Here idx stores the maximum indices, which may look something like

>>>> A
tensor([[[ 1.0503,  0.4448,  1.8663],
     [ 0.8627,  0.0685,  1.4241]],

    [[ 1.2924,  0.2456,  0.1764],
     [ 1.3777,  0.9401,  1.4637]],

    [[ 0.5235,  0.4550,  0.2476],
     [ 0.7823,  0.3004,  0.7792]],

    [[ 1.9384,  0.3291,  0.7914],
     [ 0.5211,  0.1320,  0.6330]],

    [[ 0.3292,  0.9086,  0.0078],
     [ 1.3612,  0.0610,  0.4023]]])
>>>> idx
tensor([[ 2,  2],
    [ 0,  2],
    [ 0,  0],
    [ 0,  2],
    [ 1,  0]])

I want to be able to access these indices and assign to another tensor based on them. Meaning I want to be able to do

B = torch.new_zeros(A.size())
B[idx] = A[idx]

where B is 0 everywhere except where A is maximum along the last dimension. That is B should store

>>>>B
tensor([[[ 0,  0,  1.8663],
     [ 0,  0,  1.4241]],

    [[ 1.2924,  0,  0],
     [ 0,  0,  1.4637]],

    [[ 0.5235,  0,  0],
     [ 0.7823,  0,  0]],

    [[ 1.9384,  0,  0],
     [ 0,  0,  0.6330]],

    [[ 0,  0.9086,  0],
     [ 1.3612,  0,  0]]])

This is proving to be much more difficult than I expected, as the idx does not index the array A properly. Thus far I have been unable to find a vectorized solution to use idx to index A.

Is there a good vectorized way to do this?

Output:
An ugly hackaround is to create a binary mask out of idx and use it to index the arrays. The basic code looks like this:

import torch
torch.manual_seed(0)

A = torch.randn((5, 2, 3))
_, idx = torch.max(A, dim=2)

mask = torch.arange(A.size(2)).reshape(1, 1, -1) == idx.unsqueeze(2)
B = torch.zeros_like(A)
B[mask] = A[mask]
print(A)
print(B)

The trick is that torch.arange(A.size(2)) enumerates the possible values in idx and mask is nonzero in places where they equal the idx. Remarks:

If you really discard the first output of torch.max, you can use torch.argmax instead.
I assume that this is a minimal example of some wider problem, but be aware that you are currently reinventing torch.nn.functional.max_pool3d with kernel of size (1, 1, 3).
Also, be aware that in-place modification of tensors with masked assignment can cause issues with autograd, so you may want to use torch.where as shown here.

I would expect that somebody comes up with a cleaner solution (avoiding the intermedia allocation of the mask array), likely making use of torch.index_select, but I can't get it to work right now.


Instruction: Weight decay loss

Input:
I need to write a code to gradually decay the weight of my loss function by computes lambda with given steps, But I don't have any idea. Any help will be appreciated.

This is my Loss function:

loss_A = criterion(recov_A, real_A)
loss_Final = lambda_A * loss_A + #lambda_A is a fixed number: 10

I don't want the lambda_A to be fixed. I need to gradually decay the lambda after passing the specified number of steps

# write function that computes lambda given the steps
cur_lambda  = compute_lambda(step, decay_params, initial_lamdba)
Loss_Final = cur_lambda * loss_A

Output:
To decay the fixed number depends on the number of steps or even the number of epochs you can use the following code or you can write the code as a function and call it whenever you want.

final_value = 1e-3 # Small number because dont end up with 0
initial_value = 20
starting_step = 25
total_step = 100
for i in range(total_step):
  if i <= starting_step:
     print(i, initial_value)
  else:
    print (i, initial_value + i * (final_value-initial_value)/total_step)


Instruction: RuntimeError: Error(s) in loading state_dict for ResNet:

Input:
I am loading my model using the following code.

def load_model(checkpoint_path):
  '''
  Function that loads a checkpoint and rebuilds the model
  '''

  checkpoint = torch.load(checkpoint_path, map_location = 'cpu')

  if checkpoint['architecture'] == 'resnet18':
    model = models.resnet18(pretrained=True)

  # Freezing the parameters
    for param in model.parameters():
        param.requires_grad = False

  else:
    print('Wrong Architecture!')
    return None

  model.class_to_idx = checkpoint['class_to_idx']

  classifier = nn.Sequential(OrderedDict([
                            ('fc1', nn.Linear(512, 1024)),
                            ('relu1', nn.ReLU()),
                            ('dropout', nn.Dropout(0.2)),
                            ('fc2', nn.Linear(1024, 102))
                            ]))

  model.fc = classifier

  model.load_state_dict(checkpoint['state_dict'])

  return model

And while running

# Load your model to this variable
model = load_model('checkpoint.pt')

I get the following error,

  RuntimeError                              Traceback (most recent call
  last)
       in ()
            1 # Load your model to this variable
      ----> 2 model = load_model('checkpoint.pt')
            3
            4 # If you used something other than 224x224 cropped images, set the
      correct size here
            5 image_size = 224

<ipython-input-11-81aef50793cb> in load_model(checkpoint_path)
     30   model.fc = classifier
     31
---> 32   model.load_state_dict(checkpoint['state_dict'])
     33
     34   return model

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in
load_state_dict(self, state_dict, strict)
    719         if len(error_msgs) > 0:
    720             raise RuntimeError('Error(s) in loading state_dict for
{}:\n\t{}'.format(
--> 721                                self.__class__.__name__,
"\n\t".join(error_msgs)))
    722
    723     def parameters(self):

RuntimeError: Error(s) in loading state_dict for ResNet:
  Unexpected key(s) in state_dict: "bn1.num_batches_tracked",
"layer1.0.bn1.num_batches_tracked", "layer1.0.bn2.num_batches_tracked",
"layer1.1.bn1.num_batches_tracked", "layer1.1.bn2.num_batches_tracked",
"layer2.0.bn1.num_batches_tracked", "layer2.0.bn2.num_batches_tracked",
"layer2.0.downsample.1.num_batches_tracked",
"layer2.1.bn1.num_batches_tracked", "layer2.1.bn2.num_batches_tracked",
"layer3.0.bn1.num_batches_tracked", "layer3.0.bn2.num_batches_tracked",
"layer3.0.downsample.1.num_batches_tracked",
"layer3.1.bn1.num_batches_tracked", "layer3.1.bn2.num_batches_tracked",
"layer4.0.bn1.num_batches_tracked", "layer4.0.bn2.num_batches_tracked",
"layer4.0.downsample.1.num_batches_tracked",
"layer4.1.bn1.num_batches_tracked", "layer4.1.bn2.num_batches_tracked".

Output:
I was using Pytorch 0.4.1 but Jupyter Notebook which I loaded uses 0.4.0. So I added strict=False attribute to load_state_dict().
model.load_state_dict(checkpoint['state_dict'], strict=False)


Instruction: I can't adapt my dataset to VGG-net, getting size mismatch

Input:
I’m trying to implement the pre-trained VGG net to my script, in order to recognize faces from my dataset in RGB [256,256], but I’m getting a “size mismatch, m1: [1 x 2622], m2: [4096 x 2]” even if i'm resizing my images it doesn't work, as you can see my code work with resnet and alexnet.

I've tryed resizing the images with the function interpolate but the size mismatch persist.

def training(model_conv, learning_rate, wd, net):

criterion = nn.CrossEntropyLoss(weight= torch.FloatTensor([1,1]))
optimizer = torch.optim.Adam(model_conv.fc.parameters(),         lr=learning_rate, weight_decay = wd)
total_step = len(train_loader)
loss_list = []
acc_list = []
print("Inizio il training")

for epoch in range(num_epochs):
    for i, (im, labels) in enumerate(train_loader):

        images = torch.nn.functional.interpolate(im, 224, mode = 'bilinear')
        outputs = model_conv(images)
        loss = criterion(outputs, labels)
        loss_list.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))
torch.save(model_conv, 'TrainedModel.pt')
return images, labels

def main():

net = "vgg"
learning_rate = 10e-6
wd = 10e-4

if net == "vgg":
    print("Hai selezionato VGG")
    model_conv = VGG_FACE.vgg_face
    data = torch.load("VGG_FACE.pth")
    model_conv.load_state_dict(data)

    model_conv.fc = nn.Linear(4096, 2)
    model_conv[-1] = model_conv.fc

if __name__ == '__main__':
main()

For example this is another code where I used correctly my VGG with some random images

def test():
N=5
net = VGG_FACE.vgg_face
data = torch.load("VGG_FACE.pth")
net.load_state_dict(data)
net.eval()
names = open("names.txt").read().split()

with torch.no_grad():
    mean = np.array([93.5940, 104.7624, 129.1863])
    images = scipy.misc.imread("cooper2.jpg", mode="RGB")
    images = scipy.misc.imresize(images, [224, 224])
    images = images.astype(np.float32)
    images -= mean[np.newaxis, np.newaxis, :]
    images = np.transpose(images, (2, 0, 1))
    images = images[np.newaxis, ...]
    images = torch.tensor(images, dtype=torch.float32)

    y = net(images)
    y = torch.nn.functional.softmax(y, 1)
    rank = torch.topk(y[0, :], N)
    for i in range(N):
        index = rank[1][i].item()
        score = rank[0][i].item()
        print("{}) {} ({:.2f})".format(i + 1, names[index], score))
    print()

numero_classi = 2
net[-1] = torch.nn.Linear(4096, numero_classi)

if __name__ == "__main__":
test()

the error i'm gettin is

  File "/Users/danieleligato/PycharmProjects/parametral/VGGTEST.py", line 53, in training
outputs = model_conv(images)
RuntimeError: size mismatch, m1: [4 x 2622], m2: [4096 x 2] at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:2070

THIS IS THE VGG NET THAT I'M USING

class LambdaBase(nn.Sequential):
def __init__(self, fn, *args):
    super(LambdaBase, self).__init__(*args)
    self.lambda_func = fn

def forward_prepare(self, input):
    output = []
    for module in self._modules.values():
        output.append(module(input))
    return output if output else input

class Lambda(LambdaBase):
def forward(self, input):
    return self.lambda_func(self.forward_prepare(input))

class LambdaMap(LambdaBase):
def forward(self, input):
    return map(self.lambda_func,self.forward_prepare(input))

class LambdaReduce(LambdaBase):
def forward(self, input):
    return reduce(self.lambda_func,self.forward_prepare(input))

vgg_face = nn.Sequential( # Sequential,
nn.Conv2d(3,64,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(64,64,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.MaxPool2d((2, 2),(2, 2),(0, 0),ceil_mode=True),
nn.Conv2d(64,128,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(128,128,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.MaxPool2d((2, 2),(2, 2),(0, 0),ceil_mode=True),
nn.Conv2d(128,256,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(256,256,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(256,256,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.MaxPool2d((2, 2),(2, 2),(0, 0),ceil_mode=True),
nn.Conv2d(256,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(512,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(512,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.MaxPool2d((2, 2),(2, 2),(0, 0),ceil_mode=True),
nn.Conv2d(512,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(512,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.Conv2d(512,512,(3, 3),(1, 1),(1, 1)),
nn.ReLU(),
nn.MaxPool2d((2, 2),(2, 2),(0, 0),ceil_mode=True),
Lambda(lambda x: x.view(x.size(0),-1)), # View,
nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(25088,4096)), # Linear,
nn.ReLU(),
nn.Dropout(0.5),
nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(4096,4096)), # Linear,
nn.ReLU(),
nn.Dropout(0.5),
nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(4096,2622)), # Linear,
)

Output:
The error comes from this line:

model_conv.fc = nn.Linear(4096, 2)

Change to:

model_conv.fc = nn.Linear(2622, 2)


Instruction: Gradient disappearing after first epoch in manual linear regression

Input:
I'm new to Pytorch and I've been working through the tutorials and playing around with toy examples. I wanted to just make a super simple model to get a better handle on autograd, but I'm running into issues.

I'm trying to train a linear regression model but I keep running into the following error,

----------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)
<ipython-input-80-ba5ca34a3a54> in <module>()
      9   loss = torch.dot(delta, delta)
     10
---> 11   loss.backward()
     12   with torch.no_grad():
     13     w, b = w - learning_rate*w.grad.data, b - learning_rate*b.grad.data

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
     90
     91

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

And for reference, the code is here,

# dataset for training
X = torch.randn(100, 3)
y = -3*X[:,0] + 2.2*X[:,1] + 0.002*X[:,2] + 1

w = torch.randn(3, requires_grad=True, dtype=torch.float)  # model weights
b = torch.randn(1, requires_grad=True, dtype=torch.float)  # model bias

num_epochs = 10
learning_rate = 1e-4

for i in range(num_epochs):

  y_hat = torch.mv(X, w) + b
  delta = y_hat - y
  loss = torch.dot(delta, delta)

  loss.backward()
  with torch.no_grad():
    w, b = w - learning_rate*w.grad, b - learning_rate*b.grad

The issue seems to be that after the first epoch the gradient attribute is set to None, but I'm a little confused why this would be the case.

If I try to zero the gradient after updating the weights, then I get a similar error.

Output:
The answer lies in locally disabling gradient computation. As you can see in the first example, computations carried out with the torch.no_grad() context manager result in tensors for which requires_grad == False. Since you create "fresh" w and b instead of updating them in place, these tensors lose the requires_grad property after the first iteration and you get the error on 2nd iteration. A simple fix is to reenable gradients

with torch.no_grad():
    w, b = w - learning_rate*w.grad, b - learning_rate*b.grad
    w.requires_grad_(True)
    b.requires_grad_(True)

If you look up the source of optimizers in pytorch optim module, such as SGD, you will see that they use the in-place operators such as add_. You can rewrite your loop in this manner

with torch.no_grad():
    w.sub_(learning_rate*w.grad)
    b.sub_(learning_rate*b.grad)

which will not touch the requires_grad flag, since the tensors keep their "identity" - just change values. In this case, you will need to remember to call w.zero_grad() and b.zero_grad() in each iteration or the gradient values will keep additively growing.


Instruction: Why does this semantic segmentation network have no softmax classification layer in Pytorch?

Input:
I am trying to use the following CNN architecture for semantic pixel classification. The code I am using is here

However, from my understanding this type of semantic segmentation network typically should have a softmax output layer for producing the classification result.

I could not find softmax used anywhere within the script. Here is the paper I am reading on this segmentation architecture. From Figure 2, I am seeing softmax being used. Hence I would like to find out why this is missing in the script. Any insight is welcome.

Output:
You are using quite a complex code to do the training/inference. But if you dig a little you'll see that the loss functions are implemented here and your model is actually trained using cross_entropy loss. Looking at the doc:

  This criterion combines log_softmax and nll_loss in a single function.

For numerical stability it is better to "absorb" the softmax into the loss function and not to explicitly compute it by the model.
This is quite a common practice having the model outputs "raw" predictions (aka "logits") and then letting the loss (aka criterion) do the softmax internally.
If you really need the probabilities you can add a softmax on top when deploying your model.


Instruction: Bicubic interpolation in pytorch

Input:
I'm trying to do bicubic interpolation on a torch.Tensor.

I know about torch.nn.functional.interpolate, but that method doesn't support bicubic interpolation yet.

I know that PIL images support bicubic interpolation, so I created this snippet (part of torch.nn.Module).

def build_transform(self, shape):
    h, w = shape[2] * self.scale_factor, shape[3] * self.scale_factor
    return Compose([
        ToPILImage(),
        Resize((h, w), interpolation=PIL.Image.BICUBIC),
        ToTensor()
      ])

def forward(self, x: Tensor):
    if self.transform is None:
        self.transform = self.build_transform(x.shape)

    shape = x.shape[0], x.shape[1], x.shape[2] * self.scale_factor, x.shape[3] * self.scale_factor
    new = Tensor(size=shape, device=x.device)

    for i in range(x.shape[0]):
        new[i] = self.transform(x[i])
    new.requires_grad_()

    return new

My assumption is that this method is slower than interpolating the Tensor in one pass.

My Question: Is there no other way to compute this without the loop? I assume that the difference would be noticeable (correct me if I'm wrong).

Output: torch.nn.functional.interpolate() with mode='bicubic' is supported since pytorch 1.2.0.


Instruction: Expect FloatTensors but got LongTensors in MNIST-like task

Input:
I am performing a MNIST-like task, the input is 10-class images, and the expected output is the predicted class of the images.

But now the output is like [-2.3274, -2.2723, ...], which the length is the batch_size. And the target is [4., 2., 2., 8., ...]

Error message: RuntimeError: expected object for scalar type Long but got scalar type float for argument #2 'target'

class Net(nn.Module):
    ...
    ...
    def forward(self, x):
        ...
        ...
        return F.log_softmax(x, dim = 1)

criterion = torch.nn.NLLLoss()

Can anyone give me some advice? Thanks.

Output:
The error you got refers to the second (#2) argument of the loss: the target.
NLLLoss expects (for each element) to have a float vector of probabilities, and a single long (i.e., integer) target per element.
In your case, your "target" values are [4., 2., 2., 8., ...] which are of type float. you need to convert your target to long:

target = target.to(dtype=torch.long)


Instruction: In PyTorch, what makes a tensor have non-contiguous memory?

Input:
According to this SO and this PyTorch discussion, PyTorch's view function works only on contiguous memory, while reshape does not. In the second link, the author even claims:

  [view] will raise an error on a non-contiguous tensor.

But when does a tensor have non-contiguous memory?

Output:
This is a very good answer, which explains the topic in the context of NumPy. PyTorch works essentially the same. Its docs don't generally mention whether function outputs are (non)contiguous, but that's something that can be guessed based on the kind of the operation (with some experience and understanding of the implementation). As a rule of thumb, most operations preserve contiguity as they construct new tensors. You may see non-contiguous outputs if the operation works on the array inplace and change its striding. A couple of examples below

import torch

t = torch.randn(10, 10)

def check(ten):
    print(ten.is_contiguous())

check(t) # True

# flip sets the stride to negative, but element j is still adjacent to
# element i, so it is contiguous
check(torch.flip(t, (0,))) # True

# if we take every 2nd element, adjacent elements in the resulting array
# are not adjacent in the input array
check(t[::2]) # False

# if we transpose, we lose contiguity, as in case of NumPy
check(t.transpose(0, 1)) # False

# if we transpose twice, we first lose and then regain contiguity
check(t.transpose(0, 1).transpose(0, 1)) # True

In general, if you have non-contiguous tensor t, you can make it contiguous by calling t = t.contiguous(). If t is contiguous, call to t.contiguous() is essentially a no-op, so you can do that without risking a big performance hit.


Instruction: How to fix 'cannot initialize type TensorProto DataType' error while importing torch?

Input:
I installed pytorch using pip3 command for my windows pc without GPU support.
But when I tried to import torch it is giving an error.

At first, there was a different error saying numpy version not matching and I updated the numpy to the latest version.

import torch
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-c031d3dd82fc> in <module>()
----> 1 import torch

C:\Users\iamuraptha\Anaconda3\lib\site-packages\torch\__init__.py in <module>()
     82     pass
     83
---> 84 from torch._C import *
     85
     86 __all__ += [name for name in dir(_C)

RuntimeError: generic_type: cannot initialize type "TensorProtoDataType": an object with that name is already defined

Output: I reinstalled anaconda and then created a virtual environment for pytorch.Now everything works fine


Instruction: Implementing dropout from scratch

Input:
This code attempts to utilize a custom implementation of dropout :

%reset -f

import torch
import torch.nn as nn
# import torchvision
# import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F

num_epochs = 1000

number_samples = 10

from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
# generate 2d classification dataset
X, y = make_moons(n_samples=number_samples, noise=0.1)
# scatter plot, dots colored by class value

x_data = [a for a in enumerate(X)]
x_data_train = x_data[:int(len(x_data) * .5)]
x_data_train = [i[1] for i in x_data_train]
x_data_train

y_data = [y[i[0]] for i in x_data]
y_data_train = y_data[:int(len(y_data) * .5)]
y_data_train

x_test = [a[1] for a in x_data[::-1][:int(len(x_data) * .5)]]
y_test = [a for a in y_data[::-1][:int(len(y_data) * .5)]]

x = torch.tensor(x_data_train).float() # <2>
print(x)

y = torch.tensor(y_data_train).long()
print(y)

x_test = torch.tensor(x_test).float()
print(x_test)

y_test = torch.tensor(y_test).long()
print(y_test)

class Dropout(nn.Module):
    def __init__(self, p=0.5, inplace=False):
#         print(p)
        super(Dropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, "
                             "but got {}".format(p))
        self.p = p
        self.inplace = inplace

    def forward(self, input):
        print(list(input.shape))
        return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

    def __repr__(self):
        inplace_str = ', inplace' if self.inplace else ''
        return self.__class__.__name__ + '(' \
            + 'p=' + str(self.p) \
            + inplace_str + ')'

class MyLinear(nn.Linear):
    def __init__(self, in_feats, out_feats, drop_p, bias=True):
        super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
        self.custom_dropout = Dropout(p=drop_p)

    def forward(self, input):
        dropout_value = self.custom_dropout(self.weight)
        return F.linear(input, dropout_value, self.bias)

my_train = data_utils.TensorDataset(x, y)
train_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

my_test = data_utils.TensorDataset(x_test, y_test)
test_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

# Device configuration
device = 'cpu'
print(device)

# Hyper-parameters
input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = 0.0001

pred = []

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p):
        super(NeuralNet, self).__init__()
#         self.drop_layer = nn.Dropout(p=p)
#         self.drop_layer = MyLinear()
#         self.fc1 = MyLinear(input_size, hidden_size, p)
        self.fc1 = MyLinear(input_size, hidden_size , p)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
#         out = self.drop_layer(x)
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes, p=0.9).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % 100 == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Custom dropout is implemented as :

class Dropout(nn.Module):
    def __init__(self, p=0.5, inplace=False):
#         print(p)
        super(Dropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, "
                             "but got {}".format(p))
        self.p = p
        self.inplace = inplace

    def forward(self, input):
        print(list(input.shape))
        return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

    def __repr__(self):
        inplace_str = ', inplace' if self.inplace else ''
        return self.__class__.__name__ + '(' \
            + 'p=' + str(self.p) \
            + inplace_str + ')'

class MyLinear(nn.Linear):
    def __init__(self, in_feats, out_feats, drop_p, bias=True):
        super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
        self.custom_dropout = Dropout(p=drop_p)

    def forward(self, input):
        dropout_value = self.custom_dropout(self.weight)
        return F.linear(input, dropout_value, self.bias)

It seems I've implemented the dropout function incorrectly ? :

np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))

How to modify in order to correctly utilize dropout ?

These posts were useful in getting to this point :

Hinton's Dropout in 3 Lines of Python :
https://iamtrask.github.io/2015/07/28/dropout/

Making a Custom Dropout Function : https://discuss.pytorch.org/t/making-a-custom-dropout-function/14053/2

Output:
It seems I've implemented the dropout function incorrectly?
np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1 dropout_percent)[0] * (1.0/(1-self.p))

In fact, the above implementation is known as Inverted Dropout. Inverted Dropout is how Dropout is implemented in practice in the various deep learning frameworks.
What is inverted dropout?
Before jump into the inverted dropout, it can be helpful to see how Dropout works for a single neuron:

Since during train phase a neuron is kept on with probability q (=1-p), during the testing phase we have to emulate the behavior of the ensemble of networks used in the training phase. To this end, the authors suggest scaling the activation function by a factor of q during the test phase in order to use the expected output produced in the training phase as the single output required in the test phase (Section 10, Multiplicative Gaussian Noise). Thus:

Inverted dropout is a bit different. This approach consists in the scaling of the activations during the training phase, leaving the test phase untouched. The scale factor is the inverse of the keep probability 1/1-p = 1/q, thus:

Inverted dropout helps to define the model once and just change a parameter (the keep/drop probability) to run train and test on the same model. Direct Dropout, instead, force you to modify the network during the test phase because if you don’t multiply by q the output the neuron will produce values that are higher respect to the one expected by the successive neurons (thus the following neurons can saturate or explode): that’s why Inverted Dropout is the more common implementation.
References:

Dropout Regularization, coursera by Andrew NG

What is inverted dropout?

Dropout: scaling the activation versus inverting the dropout

Analysis of Dropout

How implement inverted dropout Pytorch?
class MyDropout(nn.Module):
    def __init__(self, p: float = 0.5):
        super(MyDropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
        self.p = p

    def forward(self, X):
        if self.training:
            binomial = torch.distributions.binomial.Binomial(probs=1-self.p)
            return X * binomial.sample(X.size()) * (1.0/(1-self.p))
        return X

How to implement in Numpy?
import numpy as np

pKeep = 0.8
weights = np.ones([1, 5])
binary_value = np.random.rand(weights.shape[0], weights.shape[1]) < pKeep
res = np.multiply(weights, binary_value)
res /= pKeep  # this line is called inverted dropout technique
print(res)

How to implement in Tensorflow?
import tensorflow as tf
tf.enable_eager_execution()

weights = tf.ones(shape=[1, 5])
keep_prob = 0.8
random_tensor = keep_prob
random_tensor += tf.random_uniform(weights.shape)
# 0. if [keep_prob, 1.0) and 1. if [1.0, 1.0 + keep_prob)
binary_tensor = tf.floor(random_tensor)
ret = tf.div(weights, keep_prob) * binary_tensor
print(ret)


Instruction: Why listing model components in pyTorch is not useful?

Input:
I am trying to create Feed forward neural networks with N layers
So idea is suppose If I want 2 inputs 3 hidden and 2 outputs than I will just pass [2,3,2] to neural network class and neural network model will get created so if I want [100,1000,1000,2]
where in this case 100 is inputs, two hidden layers contains 1000 neuron each and 2 outputs so I want fully connected neural network where I just wanted to pass list which contains number of neuron in each layer.
So for that I have written following code

class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, layers):
        super(FeedforwardNeuralNetModel, self).__init__()
        self.fc=[]
        self.sigmoid=[]
        self.activationValue = []
        self.layers = layers
        for i in range(len(layers)-1):
            self.fc.append(nn.Linear(layers[i],layers[i+1]))
            self.sigmoid.append(nn.Sigmoid())

    def forward(self, x):
        out=x
        for i in range(len(self.fc)):
            out=self.fc[i](out)
            out = self.sigmoid[i](out)
        return out

when I tried to use it I found it kind of empty model

model=FeedforwardNeuralNetModel([3,5,10,2])

print(model)

>>FeedforwardNeuralNetModel()

and when I used following code

class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(FeedforwardNeuralNetModel, self).__init__()
        # Linear function
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        # Non-linearity
        self.tanh = nn.Tanh()
        # Linear function (readout)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Linear function
        out = self.fc1(x)
        # Non-linearity
        out = self.tanh(out)
        # Linear function (readout)
        out = self.fc2(out)
        return out

and when I tried to print this model I found following result

print(model)

>>FeedforwardNeuralNetModel(
(fc1): Linear(in_features=3, out_features=5, bias=True)
(sigmoid): Sigmoid()
(fc2): Linear(in_features=5, out_features=10, bias=True)
)

in my code I am just creating lists that is what difference
I just wanted to understand why in torch listing model components is not useful?

Output:
If you do print(FeedForwardNetModel([1,2,3]) it gives the following error

AttributeError: 'FeedforwardNeuralNetModel' object has no attribute '_modules'

which basically means that the object is not able to recognize modules that you have declared.

Why does this happen?

Currently, modules are declared in self.fc which is list and hence torch has no way of knowing if it is a model unless it does a deep search which is bad and inefficient.

How can we let torch know that self.fc is a list of modules?

By using nn.ModuleList (See modified code below). ModuleList and ModuleDict are python list and dictionaries respectively, but they tell torch that the list/dict contains a nn module.

#modified init function
def __init__(self, layers):
    super().__init__()
    self.fc=nn.ModuleList()
    self.sigmoid=[]
    self.activationValue = []
    self.layers = layers
    for i in range(len(layers)-1):
        self.fc.append(nn.Linear(layers[i],layers[i+1]))
        self.sigmoid.append(nn.Sigmoid())


Instruction: pytorch instance tensor not moved to gpu even with explicit cuda() call

Input:
I'm working on a project where the model requires access to a tensor that i declare in the constructor init of the class (im sub-classing torch.nn.Module class)  and then i need to use this tensor in the forward() method via a simple matmul() , the model is sent to gpu via a cuda() call:

model = Model()
model.cuda()

However when i do forward-propagation of a simple input X through:

model(X) # or model.forward(X)

I get

  RuntimeError: Expected object of type torch.cuda.FloatTensor but found
  type torch.FloatTensor for argument #2 'mat2'

Indicating that the second argument of matmul(the instance tensor i declared) is on CPU and it was expected on GPU (as the rest of the model and data).

In matmul, the tensor is transposed via matrix.t()

I even tried overriding the cuda() method thorugh:

def cuda(self):
    super().cuda()
    self.matrix.cuda()

The data is already in the GPU ,meaning the following line of code was already executed:

X = X.cuda()

Also the error explcitly says argument 2 of matmul which for this case is the tensor(called matrix) not X.

Output:
Let's assume the following:

X is moved correctly to the GPU

The tensor declared in the Model class is a simple attribute.
i.e. Something like the following:

class Model(nn.Module):
   def __init__(self):
       super().__init__()
       self.matrix = torch.randn(784, 10)

   def forward(self, x):
       return torch.matmul(x, self.matrix)

If so, your first attempt wouldn't work because the nn.Module.cuda() method only moves all of the Parameters and Buffers to the GPU.
You would need to make Model.matrix a Parameter instead of regular attribute.
Wrap it in the parameter class.
Something like:
self.matrix = nn.Parameter(torch.randn(784, 10))

Now, instead of automatically casting to the GPU like above, you tried to manually call the .cuda() method on Model.matrix within the override.
This doesn't work either because of a subtle difference between the nn.Module.cuda() method and the torch.Tensor.cuda() method.
While nn.Module.cuda() moves all the Parameters and Buffers of the Module to GPU and returns itself, torch.Tensor.cuda() only returns a copy of the tensor on the GPU.
The original tensor is unaffected.

In summary, either:

Wrap your matrix attribute as a Parameter or
Assign the GPU copy back to matrix via:

self.matrix = self.matrix.cuda()

In your override.
I would suggest the first.


Instruction: RuntimeError: The shape of the mask [1682] at index 0 does not match the shape of the indexed tensor [1, 1682] at index 0

Input:
I am designing an stacked autoencoder trying to train my neural network on movie rating if the user doesnt rate any movie it will not consider it

My training set runs perfectly but when i run test set it shows me this error

RuntimeError: The shape of the mask [1682] at index 0 does not match the shape of the indexed tensor [1, 1682] at index 0
I got error at the end test block i have commented there

CODE:-

# Auto Encoder

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable

# Importing dataset
movies= pd.read_csv('ml-1m/movies.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')
users= pd.read_csv('ml-1m/users.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')
ratings = pd.read_csv('ml-1m/ratings.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')

# preparing the training set and the dataset

training_set =pd.read_csv('ml-100k/u1.base',delimiter ='\t')
training_set =np.array(training_set, dtype= 'int')

test_set =pd.read_csv('ml-100k/u1.test',delimiter ='\t')
test_set =np.array(test_set, dtype= 'int')

# Getting the number of users and  movies
# we are taking the maximum no of values from training set and test set

nb_users = int(max(max(training_set[:,0]), max(test_set[:,0])))
nb_movies = int(max(max(training_set[:,1]), max(test_set[:,1])))

# converting the data into an array within users in lines and movies in columns

def convert(data):
    new_data = []
    for id_users in range(1, nb_users +1):
        id_movies = data[:,1][data[:,0]==id_users]#movies id from data
        id_ratings = data[:,2][data[:,0]==id_users] #ratings
        ratings= np.zeros(nb_movies)
        ratings[id_movies-1] = id_ratings  # -1 for making it start from 1
        new_data.append(list(ratings))
    return new_data

training_set =convert(training_set)
test_set =convert(test_set)

# Converting the data into Torch tensor
training_set = torch.FloatTensor(training_set)
test_set = torch.FloatTensor(test_set)

# creating the architecture of the neural network
class SAE(nn.Module):

    def  __init__(self, ): # after comma it will consider parameters of module ie parent class
        super(SAE,self).__init__()#parent class inheritence
        self.fc1 = nn.Linear(nb_movies, 20)  #20 nodes in hidden layer
        self.fc2= nn.Linear(20,10)
        self.fc3 = nn.Linear(10,20)  #decoding
        self.fc4= nn.Linear(20, nb_movies) #decoding
        self.activation= nn.Sigmoid()
            #self.myparameters= nn.ParameterList(self.fc1,self.fc2,self.fc3,self.fc4,self.activation)

    def forward(self, x):
        x=self.activation(self.fc1(x))#encoding
        x=self.activation(self.fc2(x))#encoding
        x=self.activation(self.fc3(x)) #decoding
        x=self.fc4(x) #last layer machine understand automaically
        return x

sae= SAE()
criterion = nn.MSELoss()

optimizer= optim.RMSprop(sae.parameters(), lr= 0.01 , weight_decay =0.5)

# Training the SAE
nb_epoch = 200
for epoch in range(1, nb_epoch + 1):
    train_loss = 0
    s = 0.
    for id_user in range(nb_users):
        input = Variable(training_set[id_user]).unsqueeze(0)
        target = input.clone()
        if torch.sum(target.data > 0) > 0:
            output = sae(input)
            target.require_grad = False
            output[target == 0] = 0
            loss = criterion(output, target)
            mean_corrector = nb_movies/float(torch.sum(target.data > 0) + 1e-10)
            loss.backward()
            train_loss += np.sqrt(loss.data.item()*mean_corrector)
            s += 1.
            optimizer.step()
    print('epoch: '+str(epoch)+' loss: '+str(train_loss/s))

# Testing the SAE
test_loss = 0
s = 0.
for id_user in range(nb_users):
    input = Variable(training_set[id_user]).unsqueeze(0)
    target = Variable(test_set[id_user])
    if torch.sum(target.data > 0) > 0:
        output = sae(input)
        target.require_grad = False
        output[target == 0] = 0      # I get error at this line
        loss = criterion(output, target)
        mean_corrector = nb_movies/float(torch.sum(target.data > 0) + 1e-10)
        test_loss += np.sqrt(loss.data.item()*mean_corrector)
        s += 1.
print('test loss: '+str(test_loss/s))

Output:
Change:

output[target == 0] = 0      # I get error at this line

To:

output[(target == 0).unsqueeze(0)] = 0

Reason:

The torch.Tensor returned by target == 0 is of the shape [1682].

(target == 0).unsqueeze(0) will convert it to [1, 1682]


Instruction: How to use a numpy function as the loss function in PyTorch and avoid getting errors during run time?

Input:
For my task, I do not need to compute gradients. I am simply replacing nn.L1Loss with a numpy function (corrcoef) in my loss evaluation but I get the following error:

RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

I couldn’t figure out how exactly I should detach the graph (I tried torch.Tensor.detach(np.corrcoef(x, y)) but I still get the same error. I eventually wrapped everything using with torch.no_grad as follow:

with torch.no_grad():
    predFeats = self.forward(x)
    targetFeats = self.forward(target)
    loss = torch.from_numpy(np.corrcoef(predFeats.cpu().numpy().astype(np.float32), targetFeats.cpu().numpy().astype(np.float32))[1][1])

But this time I get the following error:

TypeError: expected np.ndarray (got numpy.float64)

I wonder, what am I doing wrong?

Output:
TL;DR

with torch.no_grad():
    predFeats = self(x)
    targetFeats = self(target)
    loss = torch.tensor(np.corrcoef(predFeats.cpu().numpy(),
                                    targetFeats.cpu().numpy())[1][1]).float()

You would avoid the first RuntimeError by detaching the tensors (predFeats and targetFeats) from the computational graph.
i.e. Getting a copy of the tensor data without the gradients and the gradient function (grad_fn).

So, instead of

torch.Tensor.detach(np.corrcoef(x.numpy(), y.numpy())) # Detaches a newly created tensor!
# x and y still may have gradients. Hence the first error.

which does nothing, do

# Detaches x and y properly
torch.Tensor(np.corrcoef(x.detach().numpy(), y.detach().numpy()))

But let's not bother with all the detachments.

Like you rightfully fixed, it, let's disable the gradients.

torch.no_grad()

Now, compute the features.

predFeats = self(x) # No need for the explicit .forward() call
targetFeats = self(target)

I found it helpful to break your last line up.

loss = np.corrcoef(predFeats.numpy(), targetFeats.numpy()) # We don't need to detach

# Notice that we don't need to cast the arguments to fp32
# since the `corrcoef` casts them to fp64 anyway.

print(loss.shape, loss.dtype) # A 2-dimensional fp64 matrix

loss = loss[1][1]
print(type(loss)) # Output: numpy.float64
# Loss now just a simple fp64 number

And that is the problem!

Because, when we do

loss = torch.from_numpy(loss)

we're passing in a number (numpy.float64) while it expects a numpy tensor (np.ndarray).

If you're using PyTorch 0.4 or up, there's inbuilt support for scalars.

Simply replace the from_numpy() method with the universal tensor() creation method.

loss = torch.tensor(loss)

P.S. You might also want to look at setting rowvar=False in corrcoef since the rows in PyTorch tensors usually represent the observations.


Instruction: grad_outputs in torch.autograd.grad (CrossEntropyLoss)

Input:
I’m trying to get d(loss)/d(input). I know I have 2 options.

First option:

    loss.backward()
    dlossdx = x.grad.data

Second option:

    # criterion = nn.CrossEntropyLoss(reduce=False)
    # loss = criterion(y_hat, labels)
    # No need to call backward.
    dlossdx = torch.autograd.grad(outputs = loss,
                                  inputs = x,
                                  grad_outputs = ? )

My question is: if I use cross-entropy loss, what should I pass as grad_outputs in the second option?

Do I put d(CE)/d(y_hat)? Since pytorch crossentropy contains softmax, this will require me to pre-calculate softmax derivative using Kronecker delta.

Or do I put d(CE)/d(CE) which is torch.ones_like?

A conceptual answer is fine.

Output:
Let's try to understand how both the options work.

We will use this setup

import torch
import torch.nn as nn
import numpy as np
x = torch.rand((64,10), requires_grad=True)
net = nn.Sequential(nn.Linear(10,10))
labels = torch.tensor(np.random.choice(10, size=64)).long()
criterion = nn.CrossEntropyLoss()

First option

loss = criterion(net(x), labels)
loss.backward(retain_graph=True)
dloss_dx = x.grad

Note that you passed no options to gradient because the loss is a scalar quantity if you compute loss as a vector, then you have to pass

Second option

dloss_dx2 = torch.autograd.grad(loss, x)

This will return a tuple and you can use the first element as the gradient of x.

Note that torch.autograd.grad return sum of dout/dx if you pass multiple outputs as tuples. But since loss is scalar, you don't need to pass grad_outputs as by default it will consider it to be one.


Instruction: PyTorch - RuntimeError: bool value of Tensor with more than one value is ambiguous

Input:
I trained a GAN on the MNIST dataset, and I'm trying to make a very simple UI that has a button to generate and display new images. When I press a button I make a call to the generator and pass a new latent vector to the forward method and keep getting this error message.

def update_picture():
    print('press')
    _, img = netG.forward(create_noise(1))
    img = img.detach().cpu().numpy()[0][0]
    img = ((img - img.min()) * (1 / img.max() - img.min()) * 255)
    photo = ImageTk.PhotoImage(image=Image.fromarray(img))
    label = Label(image=photo).grid(row=0, column=0)

tk = Tk()

photo = ImageTk.PhotoImage(image=Image.fromarray(img))
label = Label(image=photo).grid(row=0, column=0)

create = Button(text="update", command=update_picture).grid(row=1, column=0)

tk.mainloop()

And when I press the button to generate a new picture I keep getting this error:

Traceback (most recent call last):
  File "C:\Users\daman\AppData\Local\Programs\Python\Python37\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
  File "C:/Users/daman/PycharmProjects/untitled4/DCGAN_MNIST.py", line 243, in update_picture
_, img = netG.forward(create_noise(1))
  File "C:/Users/daman/PycharmProjects/untitled4/DCGAN_MNIST.py", line 104, in create_noise
return Variable(torch.zeros(b, feature_space, 1, 1).normal_(0, 1))
  File "C:\Users\daman\AppData\Local\Programs\Python\Python37\lib\tkinter\__init__.py", line 315, in __init__
if not master:
RuntimeError: bool value of Tensor with more than one value is ambiguous

The error traces back to my create noise function:

def create_noise(b):
    return Variable(torch.zeros(b, feature_space, 1, 1).normal_(0, 1))

Any ideas why this is happening and what does that error actually mean? I can post more code if needed.

Output:
I think I got the problem.

Variable is a name reserved in torch and tkinter. If you are doing from ... import * you may get Variable from tkinter. Since the error is comming from this line, the Variable in your code is from tkinter. However, since you are calling it with a Tensor inside, I'm guessing that you wanted the deprecated version of torch's Variable.

Simply removing Variable in create_noise should do the work.

def create_noise(b):
    return torch.zeros(b, feature_space, 1, 1).normal_(0, 1)


Instruction: Pytorch detach() function failed to be excuated on different GPU severs

Input:
Recently, our lab bought a new server with 9 GPUs and I want to run my programming on this machine. However, I do not change my right code and I got an unexpected error like the following.

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
  File "main.py", line 166, in <module>
    p_img.copy_(netG(p_z).detach())
  File "/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py", line 100, in forward
    out,p1 = self.attn1(out)
  File "/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py", line 32, in forward
    energy =  torch.bmm(proj_query,proj_key) # transpose check
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCBlas.cu:411

However, I can run my programming successfully on the old machine with 4 GPUs. I am not sure what the problem is and it seems that the error is caused by the detach() function. My code is as the following.

z_b = torch.FloatTensor(opt.batch_size, opt.z_dim).to(device)
img_b = torch.FloatTensor(opt.batch_size, 3, 64, 64).to(device)
img_a = torch.FloatTensor(opt.batch_size, 3, 64, 64).to(device)
p_z = torch.FloatTensor(pool_size, opt.z_dim).to(device)
p_img = torch.FloatTensor(pool_size, 3, 64, 64).to(device)

## evaluation placeholder
show_z_b = torch.FloatTensor(100, opt.z_dim).to(device)
eval_z_b = torch.FloatTensor(250, opt.z_dim).to(device) # 250/batch * 120 --> 300000

optim_D = optim.Adam(netD.parameters(), lr=opt.lr_d) # other param?
optim_G = optim.Adam(netG.parameters(), lr=opt.lr_g) #?suitable
criterion_G = nn.MSELoss()

eta = 1
loss_GD = []
pre_loss = 0
cur_loss = 0
G_epoch = 1
for epoch in range(start_epoch, start_epoch + opt.num_epoch):
    print('Start epoch: %d' % epoch)
    ## input_pool: [pool_size, opt.z_dim] -> [pool_size, 32, 32]
    netD.train()
    netG.eval()
    p_z.normal_()
    # print(netG(p_z).detach().size())
    p_img.copy_(netG(p_z).detach())

    for t in range(opt.period):

        for _ in range(opt.dsteps):

            t = time.time()
            ### Update D
            netD.zero_grad()
            ## real
            real_img, _ = next(iter(dataloader)) # [batch_size, 1, 32, 32]
            img_b.copy_(real_img.squeeze().to(device))
            real_D_err = torch.log(1 + torch.exp(-netD(img_b))).mean()
            print("D real loss", netD(img_b).mean())
            # real_D_err.backward()

            ## fake
            z_b_idx = random.sample(range(pool_size), opt.batch_size)
            img_a.copy_(p_img[z_b_idx])
            fake_D_err = torch.log(1 + torch.exp(netD(img_a))).mean() # torch scalar[]
            loss_gp = calc_gradient_penalty(netD, img_b, img_a)
            total_loss = real_D_err + fake_D_err + loss_gp
            print("D fake loss", netD(img_a).mean())
            total_loss.backward()

            optim_D.step()

        ## update input pool
        p_img_t = p_img.clone().to(device)
        p_img_t.requires_grad_(True)
        if p_img_t.grad is not None:
            p_img_t.grad.zero_()
        fake_D_score = netD(p_img_t)

        fake_D_score.backward(torch.ones(len(p_img_t)).to(device))

        p_img = img_truncate(p_img + eta * p_img_t.grad)
        print("The mean of gradient", torch.mean(p_img_t.grad))

Output: The error is caused by the version mismatching between the RTX GPU cards and the CUDA driver.


Instruction: Torch C++: Getting the value of a int tensor by using *.data()

Input:
In the C++ version of Libtorch, I found that I can get the value of a float tensor by *tensor_name[0].data<float>(), in which instead of 0 I can use any other valid index. But, when I have defined an int tensor by adding option at::kInt into the tensor creation, I cannot use this structure to get the value of the tensor, i.e. something like *tensor_name[0].data<at::kInt>() or *tensor_name[0].data<int>() does not work and the debugger keeps saying that Couldn't find method at::Tensor::data<at::kInt> or Couldn't find method at::Tensor::data<int>.
I can get the values by auto value_array = tensor_name=accessor<int,1>(), but it was easier to use *tensor_name[0].data<int>(). Can you please let me know how I can use data<>() to get the value of an int tensor?
I also have a same problem with bool type.

Output:
Use item<dtype>() to get a scalar out of a Tensor.

int main() {
  torch::Tensor tensor = torch::randint(20, {2, 3});
  std::cout << tensor << std::endl;
  int a = tensor[0][0].item<int>();
  std::cout << a << std::endl;
  return 0;
}

~/l/build ❯❯❯ ./example-app
  3  10   3
  2   5   8
[ Variable[CPUFloatType]{2,3} ]
3

The following code prints 0 (tested on Linux with the stable libtorch):

#include <torch/script.h>
#include <iostream>

int main(int argc, const char* argv[])
{
    auto indx = torch::zeros({20},at::dtype(at::kLong));
    std::cout << indx[0].item<long>() << std::endl;

    return 0;
}


Instruction: size mismatch, m1: [3584 x 28], m2: [784 x 128] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:940

Input:
I have executed the following code and getting the error shown at extreme bottom. I would like to know how to resolve this. thanks

import torch.nn as nn
import torch.nn.functional as F
from torch import optim

from torchvision import transforms
_tasks = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

from torchvision.datasets import MNIST
mnist = MNIST("data", download=True, train=True, transform=_tasks)

from torch.utils.data import DataLoader
from torch.utils.data.sampler import SubsetRandomSampler

create training and validation split
split = int(0.8 * len(mnist))

index_list = list(range(len(mnist)))
train_idx, valid_idx = index_list[:split], index_list[split:]

create sampler objects using SubsetRandomSampler
tr_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(valid_idx)

create iterator objects for train and valid datasets
trainloader = DataLoader(mnist, batch_size=256, sampler=tr_sampler)
validloader = DataLoader(mnist, batch_size=256, sampler=val_sampler)

Creating model for execution

class Model(nn.Module):
  def init(self):
    super().init()
    self.hidden = nn.Linear(784, 128)
    self.output = nn.Linear(128, 10)

  def forward(self, x):
    x = self.hidden(x)
    x = F.sigmoid(x)
    x = self.output(x)
    return x

model = Model()

loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay= 1e-6, momentum = 0.9, nesterov = True)

for epoch in range(1, 11): ## run the model for 10 epochs
  train_loss, valid_loss = [], []

  #training part
  model.train()
  for data, target in trainloader:
    optimizer.zero_grad()

    #1. forward propagation
    output = model(data)

    #2. loss calculation
    loss = loss_function(output, target)

    #3. backward propagation
    loss.backward()

    #4. weight optimization
    optimizer.step()

    train_loss.append(loss.item())

  # evaluation part
  model.eval()
  for data, target in validloader:
     output = model(data)
     loss = loss_function(output, target)
     valid_loss.append(loss.item())

Executing this I am getting the following error :

  RuntimeError Traceback (most recent call last) in ()
  ----> 1 output = model(data) 2 3 ## 2. loss calculation 4 loss = loss_function(output, target) 5

  /usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in
  call(self, *input, **kwargs) 487 result = self._slow_forward(*input,
  **kwargs)

  /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in
  linear(input, weight, bias) 1352 ret =
  torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t()) 1353
  else:
  -> 1354 output = input.matmul(weight.t()) 1355 if bias is not None: 1356 output += torch.jit._unwrap_optional(bias)

  RuntimeError: size mismatch, m1: [3584 x 28], m2: [784 x 128] at
  /pytorch/aten/src/TH/generic/THTensorMath.cpp:940

Output:
Your input MNIST data has shape [256, 1, 28, 28] corresponding to [B, C, H, W]. You need to flatten the input images into a single 784 long vector before feeding it to the Linear layer Linear(784, 128) such that the input becomes [256, 784] corresponding to [B, N], where N is 1x28x28, your image size. This can be done as follows:

for data, target in trainloader:

        # Flatten MNIST images into a 784 long vector
        data = data.view(data.shape[0], -1)

        optimizer.zero_grad()
        ...

The same is needed to be done in the validation loop.


Instruction: How to vectorize computation of mean over specific set of indices given as matrix rows?

Input:
I have a problem vectorizing some code in pytorch.
A numpy solution would also help, but a pytorch solution would be better.
I'm going to use array and Tensor interchangeably.

The problem I am facing is this:

Given an 2D float array X of size (n, x), and a boolean 2D array A of size (n, n), compute the mean over rows in X indexed by rows in A.
The problem is that the rows in A contain a variable number of True indices.

Example (numpy):

import numpy as np
A = np.array([[0, 1, 0, 0, 0, 0],
              [1, 0, 1, 0, 0, 0],
              [0, 1, 0, 0, 0, 0],
              [0, 0, 0, 0, 1, 0],
              [0, 0, 0, 1, 0, 0],
              [0, 1, 1, 1, 0, 0]])
X = np.arange(6 * 3, dtype=np.float32).reshape(6, 3)

# Compute the mean in numpy with a for loop
means_np = np.array([X[A.astype(np.bool)[i]].mean(axis=0) for i in np.arange(len(A)])

So this examples works, but this formulation has 3 problems:

The for loop is slow for larger A and X. I need to loop over a few 10 thousand indices.
It can happen that A[i] contains no True indices. This results in np.mean(np.array([])), which is NaN. I want this to be 0 instead.
Implementing it this way in pytorch results in SIGFPE (Floating point error) during the backwards pass of backpropagation through this function. The cause is when nothing being selected.

The workaround that I am using now is (also see code below):

Set the diagonal elements of A to True so that there is always at least one element to select
sum of all selected elements, subtract the values in X from that sum (the diagonal is guaranteed to be False in the beginning), and divide by the number of True elements - 1 clamped to at least 1 in each row.

This works, is differentiable in pytorch and does not produce NaN, but I still need a loop over all indices.
How can I get rid of this loop?

This is my current pytorch code:

 import torch
 A = torch.from_numpy(A).bytes()
 X = torch.from_numpy(X)
 A[np.diag_indices(len(A)] = 1  # Set the diagonal to 1
 means = [(X[A[i]].sum(dim=0) - X[i]) / torch.clamp(A[i].sum() - 1, min=1.)  # Compute the mean safely
          for i in range(len(A))]  # Get rid of the loop somehow
 means = torch.stack(means)

I don't mind if your version looks completely different, as long as it is differentiable and produces the same result.

Output:
We can leverage matrix-multiplication -

c = A.sum(1,keepdims=True)
means_np = np.where(c==0,0,A.dot(X)/c)

We can optimize it further by converting A to float32 dtype if it's not already so and if the loss of precision is okay there, as shown below -

In [57]: np.random.seed(0)

In [58]: A = np.random.randint(0,2,(1000,1000))

In [59]: X = np.random.rand(1000,1000).astype(np.float32)

In [60]: %timeit A.dot(X)
10 loops, best of 3: 27 ms per loop

In [61]: %timeit A.astype(np.float32).dot(X)
100 loops, best of 3: 10.2 ms per loop

In [62]: np.allclose(A.dot(X), A.astype(np.float32).dot(X))
Out[62]: True

Thus, use A.astype(np.float32).dot(X) to replace A.dot(X).

Alternatively, to solve for the case where the row-sum is zero, and that requires us to use np.where, we could assign any non-zero value, say 1 into c and then simply divide by it, like so -

c = A.sum(1,keepdims=True)
c[c==0] = 1
means_np = A.dot(X)/c

This would also avoid the warning that we would otherwise get from np.where in those zero row sum cases.