Instruction: Why unintialized tensor in Pytorch have initial values?

Input:
The torch command

x = torch.Tensor(4, 3)

is supposed to create an uninitialized tensor (based on documentations).
But when we try to print the content of x, there are values there.

>>>from __future__ import print_function
>>>print(x)

 0.0000e+00 -8.5899e+09  6.1021e-38
 8.5920e+09  1.7470e-21  4.5806e-41
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 4x3]

So what is the meaning of uninitialized here?

Output:
It means that PyTorch just reserves a certain area within the memory for the tensor, without changing its content.

This part of the memory was before occupied by something else (an other tensor or or maybe something completely different like Browser, Code Editor .. if you use CPU memory). The values inside are not cleared afterwards for performance reasons.

The content (what previously might be something entirely different) is just interpreted as values for tensor.

Writing zeros or some other initialization requires computational power, so just reserving the area in memory is much faster.

But the values are also completely uncontrolled, values can grow very high, so in many cases you might do additional initialization.


Instruction: Package Cuda 8 not found for PyTorch

Input:
I am trying to follow the tutorial for chatbots at facebook (https://github.com/facebookresearch/end-to-end-negotiator) and I am stuck here :
conda install pytorch torchvision cuda80 -c soumith
 I managed to get all packages except for cuda80 which is not found. I also tried
conda install magma-cuda80 -c soumith
conda install cuda80 -c soumith
conda install magma-cuda80 -c pytorch

with no success.
PackagesNotFoundError: The following packages are not available from current channels:
  - magma-cuda80

Where can I find the package ? I am on Windows10.

Output:
Open Anaconda Prompt and run -

conda install -c peterjc123 pytorch cuda80

OR

conda install -c peterjc123 pytorch cuda90


Instruction: Install Pytorch on Windows

Input:
I am trying to install Pytorch on Windows8.1. I am using Python 3.6.4 and no GPU. I've tried already the Anaconda package provided by peterjc123 by running conda install -c peterjc123 pytorch_legacy cuda80 using a virtual environment. While the installation goes smooth (without errors), after import torch I get the following error.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\myuser\AppData\Local\conda\conda\envs\myenv\lib\site-packages\torch\__init__.py", line 76, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

Can somebody help me to install it?

Output:
You are installing the GPU version with the command.
Check the link for the github repo.

In short, you should run something like
conda install -c peterjc123 pytorch.
Be sure to install the required dependencies before attempting to install the main framework. For more details, check the link.


Instruction: What does .contiguous() do in PyTorch?

Input: What does x.contiguous() do for a tensor x?

Output:
There are a few operations on Tensors in PyTorch that do not change the contents of a tensor, but change the way the data is organized. These operations include:

narrow(), view(), expand() and transpose()

For example: when you call transpose(), PyTorch doesn't generate a new tensor with a new layout, it just modifies meta information in the Tensor object so that the offset and stride describe the desired new shape. In this example, the transposed tensor and original tensor share the same memory:
x = torch.randn(3,2)
y = torch.transpose(x, 0, 1)
x[0, 0] = 42
print(y[0,0])
# prints 42

This is where the concept of contiguous comes in. In the example above, x is contiguous but y is not because its memory layout is different to that of a tensor of same shape made from scratch. Note that the word "contiguous" is a bit misleading because it's not that the content of the tensor is spread out around disconnected blocks of memory. Here bytes are still allocated in one block of memory but the order of the elements is different!
When you call contiguous(), it actually makes a copy of the tensor such that the order of its elements in memory is the same as if it had been created from scratch with the same data.
Normally you don't need to worry about this. You're generally safe to assume everything will work, and wait until you get a RuntimeError: input is not contiguous where PyTorch expects a contiguous tensor to add a call to contiguous().


Instruction: PyTorch Sparse Tensors number of dimensions must be nDimI + nDimV

Input:
I'm trying to insert the value in gd to coordinate [1,0]. Below are the matrices. When I try this, I get a RuntimeError.

>>> import torch
>>> cd = [[1, 0]]
>>> gd = [0.39613232016563416]
>>> i = torch.LongTensor(cd)
>>> v = torch.FloatTensor(gd)
>>> p = torch.rand(2)
>>> i

 1  0
[torch.LongTensor of size 1x2]

>>> v

 0.3961
[torch.FloatTensor of size 1]

>>> p

 0.4678
 0.0996
[torch.FloatTensor of size 2]

>>> torch.sparse.FloatTensor(i.t(), v, torch.Size(list(p.size()))).to_dense()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: invalid argument 2: number of dimensions must be nDimI + nDimV at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/THS/generic/THSTensor.c:169

Output:
Two things.

1) Right now p is a Tensor of rank 1. To insert something in position [1,0] it needs to be a Tensor of rank 2.

2) You don't need to do complicated things with sparse tensors. Simply p[cd[0], cd[1]] = v[0] should work. Where cd = torch.LongTensor([row_idx, col_idx])

So:

>>> cd = torch.LongTensor([1,0])
>>> gd = [0.39613232016563416]
>>> v = torch.FloatTensor(gd)
>>> p = torch.rand((2,2))
>>> p

 0.9342  0.8539
 0.7044  0.0823

  [torch.FloatTensor of size 2x2]

>>> p[cd[0], cd[1]] = v[0]
>>> p

0.9342  0.8539
0.3961  0.0823

  [torch.FloatTensor of size 2x2]

That simple.


Instruction: multiprocessing error with pytorch on windows 10

Input:
i get the following error when i try to execute my code, which clearly shows its an mulitprocessing error:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

On Linux, the following code runs fine, but Im wondering why i cant get it running on Windows 10. However, here is the code:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

kwargs = {'num_workers': 1, 'pin_memory': True}
train_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=True,                 download=True,
                                            transform=transforms.Compose([transforms.ToTensor,
                                                            transforms.Normalize((0.1307,), (0.3081,))])),
                                 batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=False,
                                                transform=transforms.Compose([transforms.ToTensor,
                                                            transforms.Normalize((0.1307,), (0.3081,))])),
                                 batch_size=64, shuffle=True, **kwargs)

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(x.size())
        exit()

model = Netz()
model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)

def train(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(train_data):
        data = data.cuda()
        target = target.cuda()
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = F.nll_loss
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, batch_id * len(data), len(train_data.dataset),
                                                                   100. * batch_id / len(train_data), loss.data[0]))

for epoch in range(1, 30):
    print(epoch)
    train(epoch)

I tried to fix it with:

if __name__ == '__main__':
    for epoch in range(1, 30):
        train(epoch)

But also without success.
Anyone has an idea how i can fix this multiprocessing error?
Any help would be appreciated.
(Yes, i know pytorch is not officially released for windows, but i dont think this is causing the error here.)

Thanks!

Output:
I found it out myself.

I had to put the whole code into if name == 'main':

Also i forgot the brakets in the end at the transforms.ToTensor part.


Instruction: PyTorch - How to use "toPILImage" correctly

Input:
I would like to know, whether I used toPILImage from torchvision correctly. I want to use it, to see how the images look after initial image transformations are applied to the dataset.

When I use it like in the code below, the image that comes up has weird colors like this one. The original image is a regular RGB image.

This is my code:

import os
import torch
from PIL import Image, ImageFont, ImageDraw
import torch.utils.data as data
import torchvision
from torchvision import transforms
import matplotlib.pyplot as plt

# Image transformations
normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
    )
transform_img = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    normalize ])

train_data = torchvision.datasets.ImageFolder(
    root='./train_cl/',
    transform=transform_img
    )
test_data = torchvision.datasets.ImageFolder(
    root='./test_named_cl/',
    transform=transform_img
    )

train_data_loader = data.DataLoader(train_data,
    batch_size=4,
    shuffle=True,
    num_workers=4) #num_workers=args.nThreads)

test_data_loader = data.DataLoader(test_data,
    batch_size=32,
    shuffle=False,
    num_workers=4)

# Open Image from dataset:
to_pil_image = transforms.ToPILImage()
my_img, _ = train_data[248]
results = to_pil_image(my_img)
results.show()

Edit:

I had to use .data on the Torch Variable to get the tensor.
Also I needed to rescale the numpy array before transposing. I found a working solution here, but it doesn't always work well. How can I do this better?

for i, data in enumerate(train_data_loader, 0):
    img, labels = data
    img = Variable(img)
    break

image = img.data.cpu().numpy()[0]

# This worked for rescaling:
image = (1/(2*2.25)) * image + 0.5

# Both of these didn't work:
# image /= (image.max()/255.0)
# image *= (255.0/image.max())

image = np.transpose(image, (1,2,0))
plt.imshow(image)
plt.show()

Output:
You can use PIL image but you're not actually loading the data as you would normally.
Try something like this instead:
import numpy as np
import matplotlib.pyplot as plt

for img,labels in train_data_loader:
    # load a batch from train data
    break

# this converts it from GPU to CPU and selects first image
img = img.cpu().numpy()[0]
#convert image back to Height,Width,Channels
img = np.transpose(img, (1,2,0))
#show the image
plt.imshow(img)
plt.show()

As an update (02-10-2021):
import torchvision.transforms.functional as F
# load the image (creating a random image as an example)
img_data = torch.ByteTensor(4, 4, 3).random_(0, 255).numpy()
pil_image = F.to_pil_image(img_data)

Alternatively
import torchvision.transforms as transforms
img_data = torch.ByteTensor(4, 4, 3).random_(0, 255).numpy()
pil_image = transforms.ToPILImage()(img_data)

The second form can be integrated with dataset loader in pytorch or called directly as so.
I added a modified to_pil_image here
essentially it does what I suggested back in 2018 but it is integrated into pytorch now.


Instruction: Pytorch softmax: What dimension to use?

Input:
The function torch.nn.functional.softmax takes two parameters: input and dim.  According to its documentation, the softmax operation is applied to all slices of input along the specified dim, and will rescale them so that the elements lie in the range (0, 1) and sum to 1.

Let input be:

input = torch.randn((3, 4, 5, 6))

Suppose I want the following, so that every entry in that array is 1:

sum = torch.sum(input, dim = 3) # sum's size is (3, 4, 5, 1)

How should I apply softmax?

softmax(input, dim = 0) # Way Number 0
softmax(input, dim = 1) # Way Number 1
softmax(input, dim = 2) # Way Number 2
softmax(input, dim = 3) # Way Number 3

My intuition tells me that is the last one, but I am not sure. English is not my first language and the use of the word along seemed confusing to me because of that.

I am not very clear on what "along" means, so I will use an example that could clarify things.  Suppose we have a tensor of size (s1, s2, s3, s4), and I want this to happen

Output:
The easiest way I can think of to make you understand is: say you are given a tensor of shape (s1, s2, s3, s4) and as you mentioned you want to have the sum of all the entries along the last axis to be 1.

sum = torch.sum(input, dim = 3) # input is of shape (s1, s2, s3, s4)

Then you should call the softmax as:

softmax(input, dim = 3)

To understand easily, you can consider a 4d tensor of shape (s1, s2, s3, s4) as a 2d tensor or matrix of shape (s1*s2*s3, s4). Now if you want the matrix to contain values in each row (axis=0) or column (axis=1) that sum to 1, then, you can simply call the softmax function on the 2d tensor as follows:

softmax(input, dim = 0) # normalizes values along axis 0
softmax(input, dim = 1) # normalizes values along axis 1

You can see the example that Steven mentioned in his answer.


Instruction: Change Tanh activation in LSTM to ReLU

Input: The default non-linear activation function in LSTM class is tanh. I wish to use ReLU for my project. Browsing through the documentation and other resources, I'm unable to find a way to do this in a simple manner. The only way I could find was to define my own custom LSTMCell, but here the author says that custom LSTMCells don't support GPU acceleration capabilities(or has that changed since the article was published?). I need to use CUDA to speed up my training. Any help would be appreciated.

Output:
Custom LSTMCells don't support GPU acceleration capabilities - this statement probably means GPU acceleration capabilities become limited if you use LSTMCells. And definitely, you can write your own implementation of LSTM but you need to sacrifice runtime.

For example, once I implemented an LSTM (based on linear layers) as follows which used to take 2~3 times more time than LSTM (provided in PyTorch) when used as a part of a deep neural model.

class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size, nlayers, dropout):
        """"Constructor of the class"""
        super(LSTMCell, self).__init__()

        self.nlayers = nlayers
        self.dropout = nn.Dropout(p=dropout)

        ih, hh = [], []
        for i in range(nlayers):
            ih.append(nn.Linear(input_size, 4 * hidden_size))
            hh.append(nn.Linear(hidden_size, 4 * hidden_size))
        self.w_ih = nn.ModuleList(ih)
        self.w_hh = nn.ModuleList(hh)

    def forward(self, input, hidden):
        """"Defines the forward computation of the LSTMCell"""
        hy, cy = [], []
        for i in range(self.nlayers):
            hx, cx = hidden[0][i], hidden[1][i]
            gates = self.w_ih[i](input) + self.w_hh[i](hx)
            i_gate, f_gate, c_gate, o_gate = gates.chunk(4, 1)

            i_gate = F.sigmoid(i_gate)
            f_gate = F.sigmoid(f_gate)
            c_gate = F.tanh(c_gate)
            o_gate = F.sigmoid(o_gate)

            ncx = (f_gate * cx) + (i_gate * c_gate)
            nhx = o_gate * F.tanh(ncx)
            cy.append(ncx)
            hy.append(nhx)
            input = self.dropout(nhx)

        hy, cy = torch.stack(hy, 0), torch.stack(cy, 0)
        return hy, cy

I would be happy to know if the runtime of custom implementation of LSTM can be improved!


Instruction: How could I feed a custom image into this model?

Input:
I've been following a course online and one of the exercises was to create a simple image detection model (using MNIST data) to detect written numbers. I've been trying to load a custom image I drew in (128x128 jpg) but I can't seem to figure it out. I'm really close, but I think I'm just confused about what parameters the model takes in. Any help would be appreciated!!

Here is my code

Output:
Simply convert your image to an 128x128 numpy array with values between 0 and 1.

Then:

image = Variable(torch.from_numpy(image))[None, :, :]
classification = model(image)

classification is then a pytorch Variable containing probabilities of belonging to each class.


Instruction: Implementing RNN and LSTM into DQN Pytorch code

Input:
I have some troubles finding some example on the great www to how i implement a recurrent neural network with LSTM layer into my current Deep q-network in Pytorch so it become a DRQN.. Bear with me i am just getting started..
Futhermore, I am NOT working with images processing, thereby CNN so do not worry about this. My states are purely temperatures values.

Here is my code that i am currently train my DQN with:

# Importing the libraries

import numpy as np
import random # random samples from different batches (experience replay)
import os # For loading and saving brain
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim # for using stochastic gradient descent
import torch.autograd as autograd # Conversion from tensor (advanced arrays) to avoid all that contains a gradient
# We want to put the tensor into a varaible taht will also contain a
# gradient and to this we need:
from torch.autograd import Variable
# to convert this tensor into a variable containing the tensor and the gradient

# Creating the architecture of the Neural Network
class Network(nn.Module): #inherinting from nn.Module

    #Self - refers to the object that will be created from this class
    #     - self here to specify that we're referring to the object
    def __init__(self, input_size, nb_action): #[self,input neuroner, output neuroner]
        super(Network, self).__init__() #inorder to use modules in torch.nn
        # Input and output neurons
        self.input_size = input_size
        self.nb_action = nb_action
        # Full connection between different layers of NN
        # In this example its one input layer, one hidden layer and one output layer
        # Using self here to specify that fc1 is a variable of my object
        self.fc1 = nn.Linear(input_size, 40)
        self.fc2 = nn.Linear(40, 30)
        #Example of adding a hiddenlayer
        # self.fcX = nn.Linear(30,30)
        self.fc3 = nn.Linear(30, nb_action) # 30 neurons in hidden layer

    # For function that will activate neurons and perform forward propagation
    def forward(self, state):
        # rectifier function
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        q_values = self.fc3(x)
        return q_values

# Implementing Experience Replay
# We know that RL is based on MDP
# So going from one state(s_t) to the next state(s_t+1)
# We gonna put 100 transition between state into what we call the memory
# So we can use the distribution of experience to make a decision
class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity #100 transitions
        self.memory = [] #memory to save transitions

    # pushing transitions into memory with append
    #event=transition
    def push(self, event):
        self.memory.append(event)
        if len(self.memory) > self.capacity: #memory only contain 100 events
            del self.memory[0] #delete first transition from memory if there is more that 100

    # taking random sample
    def sample(self, batch_size):
        #Creating variable that will contain the samples of memory
        #zip =reshape function if list = ((1,2,3),(4,5,6)) zip(*list)= (1,4),(2,5),(3,6)
        #                (state,action,reward),(state,action,reward)
        samples = zip(*random.sample(self.memory, batch_size))
        #This is to be able to differentiate with respect to a tensor
        #and this will then contain the tensor and gradient
        #so for state,action and reward we will store the seperately into some
        #bytes which each one will get a gradient
        #so that eventually we'll be able to differentiate each one of them
        return map(lambda x: Variable(torch.cat(x, 0)), samples)

# Implementing Deep Q Learning

class Dqn():

    def __init__(self, input_size, nb_action, gamma, lrate, T):
        self.gamma = gamma #self.gamma gets assigned to input argument
        self.T = T
        # Sliding window of the evolving mean of the last 100 events/transitions
        self.reward_window = []
        #Creating network with network class
        self.model = Network(input_size, nb_action)
        #creating memory with memory class
        #We gonna take 100000 samples into memory and then we will sample from this memory to
        #to get a snakk number of random transitions
        self.memory = ReplayMemory(100000)
        #creating optimizer (stochastic gradient descent)
        self.optimizer = optim.Adam(self.model.parameters(), lr = lrate) #learning rate
        #input vector which is batch of input observations
        #by unsqeeze we create a fake dimension to this is
        #what the network expect for its inputs
        #have to be the first dimension of the last_state
        self.last_state = torch.Tensor(input_size).unsqueeze(0)
        #Inilizing
        self.last_action = 0
        self.last_reward = 0

    def select_action(self, state):
        #Q value depends on state
        #Temperature parameter T will be a positive number and the closer
        #it is to ze the less sure the NN will when taking an action
        #forexample
        #softmax((1,2,3))={0.04,0.11,0.85} ==> softmax((1,2,3)*3)={0,0.02,0.98}
        #to deactivate brain then set T=0, thereby it is full random
        probs = F.softmax((self.model(Variable(state, volatile = True))*self.T),dim=1) # T=100
        #create a random draw from the probability distribution created from softmax
        action = probs.multinomial()
        print(probs.multinomial())
        return action.data[0,0]

    # See section 5.3 in AI handbook
    def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
        outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
        #next input for target see page 7 in attached AI handbook
        next_outputs = self.model(batch_next_state).detach().max(1)[0]
        target = self.gamma*next_outputs + batch_reward
        #Using hubble loss inorder to obtain loss
        td_loss = F.smooth_l1_loss(outputs, target)
        #using  lass loss/error to perform stochastic gradient descent and update weights
        self.optimizer.zero_grad() #reintialize the optimizer at each iteration of the loop
        #This line of code that backward propagates the error into the NN
        #td_loss.backward(retain_variables = True) #userwarning
        td_loss.backward(retain_graph = True)
        #And this line of code uses the optimizer to update the weights
        self.optimizer.step()

    def update(self, reward, new_signal):
        #Updated one transition and we have dated the last element of the transition
        #which is the new state
        new_state = torch.Tensor(new_signal).float().unsqueeze(0)
        self.memory.push((self.last_state, new_state, torch.LongTensor([int(self.last_action)]), torch.Tensor([self.last_reward])))
        #After ending in a state its time to play a action
        action = self.select_action(new_state)
        if len(self.memory.memory) > 100:
            batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(100)
            self.learn(batch_state, batch_next_state, batch_reward, batch_action)
        self.last_action = action
        self.last_state = new_state
        self.last_reward = reward
        self.reward_window.append(reward)
        if len(self.reward_window) > 1000:
            del self.reward_window[0]
        return action

    def score(self):
        return sum(self.reward_window)/(len(self.reward_window)+1.)

    def save(self):
        torch.save({'state_dict': self.model.state_dict(),
                    'optimizer' : self.optimizer.state_dict(),
                   }, 'last_brain.pth')

    def load(self):
        if os.path.isfile('last_brain.pth'):
            print("=> loading checkpoint... ")
            checkpoint = torch.load('last_brain.pth')
            self.model.load_state_dict(checkpoint['state_dict'])
            self.optimizer.load_state_dict(checkpoint['optimizer'])
            print("done !")
        else:
            print("no checkpoint found...")

I hope there is someone out there that can help me and could implement a RNN and a LSTM layer into my code! I believe in you stackflow!

Best regards Søren Koch

Output:
From my point of view, I think you could add RNN, LSTM layer to the Network#__init__,Network#forward; shape of data should be reshaped into sequences...

For more detail, I think you should read these two following articles; after that implementing RNN, LSTM not hard as it seem to be.

http://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#sphx-glr-beginner-nlp-sequence-models-tutorial-py

http://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html


Instruction: An analog of weighted_cross_entropy_with_logits in PyTorch

Input:
I'm trying to train a model with PyTorch. Is there any simple way to create a loss like weighted_cross_entropy_with_logits from Tensorflow?

There are pos_weight argument in weighted_cross_entropy_with_logits that can help with balancing. But there are only weights for labels in the list of arguments in BCEWithLogitsLoss.

Output:
You can write your own custom loss function as you want. For example, you can write:

def weighted_cross_entropy_with_logits(logits, target, pos_weight):
    return targets * -logits.sigmoid().log() * pos_weight +
               (1 - targets) * -(1 - logits.sigmoid()).log()

This is a basic implementation. You should follow the steps mentioned here to ensure stability and avoid overflow. Just use the final formulation that they derived.


Instruction: PyTorch: Testing with torchvision.datasets.ImageFolder and DataLoader

Input:
I'm a newbie trying to make this PyTorch CNN work with the Cats&Dogs dataset from kaggle. As there are no targets for the test images, I manually classified some of the test images and put the class in the filename, to be able to test (maybe should have just used some of the train images).

I used the torchvision.datasets.ImageFolder class to load the train and test images. The training seems to work.

But what do I need to do to make the test-routine work? I don't know, how to connect my test_data_loader with the test loop at the bottom, via test_x and test_y.

The Code is based on this MNIST example CNN. There, something like this is used right after the loaders are created. But I failed to rewrite it for my dataset:

test_x = Variable(torch.unsqueeze(test_data.test_data, dim=1), volatile=True).type(torch.FloatTensor)[:2000]/255.   # shape from (2000, 28, 28) to (2000, 1, 28, 28), value in range(0,1)
test_y = test_data.test_labels[:2000]

The Code:

import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms

EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "./train_cl/"
TEST_DATA_PATH = "./test_named_cl/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)

class CNN(nn.Module):
    # omitted...

if __name__ == '__main__':

    print("Number of train samples: ", len(train_data))
    print("Number of test samples: ", len(test_data))
    print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure

    model = CNN()
    optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
    loss_func = nn.CrossEntropyLoss()

    # Training and Testing
    for epoch in range(EPOCHS):
        for step, (x, y) in enumerate(train_data_loader):
            b_x = Variable(x)   # batch x (image)
            b_y = Variable(y)   # batch y (target)
            output = model(b_x)[0]
            loss = loss_func(output, b_y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            # Test -> this is where I have no clue
            if step % 50 == 0:
                test_x = Variable(test_data_loader)
                test_output, last_layer = model(test_x)
                pred_y = torch.max(test_output, 1)[1].data.squeeze()
                accuracy = sum(pred_y == test_y) / float(test_y.size(0))
                print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)

Output: Looking at the data from Kaggle and your code, it seems that there are problems in your data loading, both train and test set. First of all, the data should be in a different folder per label for the default PyTorch ImageFolder to load it correctly. In your case, since all the training data is in the same folder, PyTorch is loading it as one class and hence learning seems to be working. You can correct this by using a folder structure like - train/dog, - train/cat, - test/dog, - test/cat and then passing the train and the test folder to the train and test ImageFolder respectively. The training code seems fine, just change the folder structure and you should be good. Take a look at the official documentation of ImageFolder which has a similar example.


Instruction: Image Captioning Example input size of Decoder LSTM Pytorch

Input:
I'm new to Pytorch, there is a doubt that am having in the Image Captioning example code . In DcoderRNN class the lstm is defined as ,

self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)

in the forward function ,

embeddings = self.embed(captions)
embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)

we first embed the captions and then concat the embeddings with the context feature from the EncoderCNN, but the concat increases the size from embed size how we can forward that to the lstm? as the input size of lstm is already defined as embed_size.

Am I missing something here? Thanks in advance .

Output:
You can analyze the shape of all input and output tensors and then it will become easier for you to understand what changes you need to make.

Let's say: captions = B x S where S = sentence (caption) length.

embeddings = self.embed(captions)

Now, embeddings = B x S x E where E = embed_size.

embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)

Here, embeddings = B x (S + 1) X E.

My understanding says you are doing wrong here. I guess you should concatenate features along axis=2. Because probably you want to concatenate the image features along with the word embeddings for each word in the caption. So, if you do:

embeddings = torch.cat((features.unsqueeze(1), embeddings), 2)

It results in, embeddings = B X S X (E + F) where E + F = embed_size + img_feat_size

Then you need to revise your LSTM definition as follows.

self.lstm = nn.LSTM(embed_size+img_feat_size, hidden_size, num_layers, batch_first=True)

My experience says, usually, people concatenate image features with word features and pass it to the LSTM layer.


Instruction: Is a torch.FloatTensor not a Tensor?

Input:
Although this example has no training, this is an adapted part of a larger program where training does occur. I simply want the generator network to spew out a random image in this case:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms
from PIL import Image

class Generator(nn.Module):

    def __init__(self):
        """
        Generator component of GAN. requires an input slightly bigger
        than 300 x 300 (precisely 308 x 308)
        """
        super(Generator, self).__init__()

        # 5 x 5 square convolution.
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 4, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        return x

def main():
    # Generate example image.
    generator = Generator()
    img = generator(Variable(torch.randn(1, 3, 308, 308))).data
    img_pil = transforms.ToPILImage()(img)
    img_pil.save("test.png")

if __name__ == "__main__":
    main()

Running this program gives the following:

(mgan-Csuh5VLx) ➜  mgan git:(broken) ✗ python test.py
Traceback (most recent call last):
  File "test.py", line 34, in <module>
    main()
  File "test.py", line 30, in main
    img_pil = transforms.ToPILImage()(img)
  File "/home/christopher/.local/share/virtualenvs/mgan-Csuh5VLx/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 92, in __call__
    return F.to_pil_image(pic, self.mode)
  File "/home/christopher/.local/share/virtualenvs/mgan-Csuh5VLx/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 96, in to_pil_image
    raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(type(pic)))
TypeError: pic should be Tensor or ndarray. Got <class 'torch.FloatTensor'>.

I thought FloatTensors were essentially Tensors. Is there a way to get around this error?

(Note: I have four output channels for RGBA, but even switching to 3 output channels gives the same error.)

Output:
Simply changing the second last line of your main function solves the problem:

img_pil = transforms.ToPILImage()(img.squeeze())

img.squeeze() makes the tensor shape (1, 4, 300, 300) to (4, 300, 300).


Instruction: where is the definition of _cudnn_convolution_full_forward in pytorch

Input:
as the title mentioned, I want to find the definition of _cudnn_convolution_full_forward, but I search through all the project
in pytorch and failed. And I cannot find and doc about this function.

any one can help me?

Output:
All the cudnn convolution functions are defined here:
https://github.com/pytorch/pytorch/blob/1848cad10802db9fa0aa066d9de195958120d863/aten/src/ATen/native/cudnn/Conv.cpp

This function doesn't exist anymore in the latest versions of pytorch. The closest thing that there is there is cudnn_convolution_forward. In version 0.1.12, the function is in the same file:
https://github.com/pytorch/pytorch/blob/v0.1.12/torch/csrc/cudnn/Conv.cpp

I would recommend against using using an unpublic api (one starting with _) and use a public method instead, but you probably already know that.

In otherwords you should be using

torch.backends.cudnn.enabled = True

and then conv2d or conv3d depending on your use.


Instruction: Indexing on axis by list in PyTorch

Input:
I have Variables lengths_X of size (10L,) and A of size (10L, 16L, 5L).

I want to use lengths_X to index along the second axis of A. In other words, I want to get a new tensor predicted_Y of size (10L, 5L) that indexes axis 1 at i for all entries with index i in axis 0.

What is the best way to do this in PyTorch?

Output:
What you are looking for is actually called batched_index_select and I looked for such functionality before but couldn't find any native function in PyTorch that can do the job. But we can simply use:

A = torch.randn(10, 16, 5)
index = torch.from_numpy(numpy.random.randint(0, 16, size=10))
B = torch.stack([a[i] for a, i in zip(A, index)])

You can see the discussion here. You can also check out the function batched_index_select provided in the AllenNLP library. I would be happy to know if there is a better solution.


Instruction: Issue training RNN model with pytorch with trivial goal

Input:
I'm trying to train a simple RNN model with a trivial goal where the output matches a fixed vector regardless of the input

import torch
import torch.nn as nn

from torch.autograd import Variable
import numpy as np

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        print "i2h WEIGHT size ", list(self.i2h.weight.size())
        print "i2h bias size ", list(self.i2h.bias.size())
        self.i2o = nn.Linear(hidden_size, output_size)
        print "i2o WEIGHT size ", list(self.i2o.weight.size())
        print "i2o bias size ", list(self.i2o.bias.size())
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(hidden)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

n_hidden = 20
rnn = RNN(10, n_hidden, 3)

learning_rate = 1e-3
loss_fn = torch.nn.MSELoss(size_average=False)
out_target = Variable( torch.FloatTensor([[0.0 , 1.0, 0.0]] ) , requires_grad=False)

print "target output::: ", out_target
def train(category_tensor, line_tensor):
    hidden = rnn.initHidden()

    rnn.zero_grad()

    for i in range(line_tensor.size()[0]):
        #print "train iteration ", i, ": input data: ", line_tensor[i]
        output, hidden = rnn(line_tensor[i], hidden)

    loss = loss_fn(output, out_target)
    loss.backward()

    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        #print "parameter: ", p, " gradient: ", p.grad.data
        p.data.add_(-learning_rate, p.grad.data)

    return output, loss.data[0]

current_loss = 0
n_iters = 500

for iter in range(1, n_iters + 1):
    inp = Variable(torch.randn(100,1,10) + 5)
    output, loss = train(out_target, inp)
    current_loss += loss
    if iter % 1 == 0:
      print "weights: ",rnn.i2h.weight
      print "LOSS: ", loss
      print output

As it shows, the loss stays above 6 and never goes down. Notice also that I am biasing all the random inputs normal distributions by 5, so they are mostly positive numbers, so there should exist a weight distribution that approaches the goal output

What am I doing wrong in this example that is failing to output to approach the goal?

Output:
Your fixed output is:

torch.FloatTensor([[0.0, 1.0, 0.0]])

But you are using the following as the final layer in your RNN:

self.softmax = nn.LogSoftmax(dim=1)

Does LogSoftmax returns value in [0, 1]? Althouhgh, you can use the Softmax but I would recommend you to use the sign function and transform -1 to 0.


Instruction: how to apply gradients manually in pytorch

Input:
Starting to learn pytorch and was trying to do something very simple, trying to move a randomly initialized vector of size 5 to a target vector of value [1,2,3,4,5].

But my distance is not decreasing!! And my vector x just goes crazy. No idea what I am missing.

import torch
import numpy as np
from torch.autograd import Variable

# regress a vector to the goal vector [1,2,3,4,5]

dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

x = Variable(torch.rand(5).type(dtype), requires_grad=True)
target = Variable(torch.FloatTensor([1,2,3,4,5]).type(dtype),
requires_grad=False)
distance = torch.mean(torch.pow((x - target), 2))

for i in range(100):
  distance.backward(retain_graph=True)
  x_grad = x.grad
  x.data.sub_(x_grad.data * 0.01)

Output:
There are two errors in your code that prevents you from getting the desired results.

The first error is that you should put the distance calculation in the loop. Because the distance is the loss in this case. So we have to monitor its change in each iteration.

The second error is that you should manually zero out the x.grad because pytorch won't zero out the grad in variable by default.

The following is an example code which works as expected:

import torch
import numpy as np
from torch.autograd import Variable
import matplotlib.pyplot as plt

# regress a vector to the goal vector [1,2,3,4,5]

dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

x = Variable(torch.rand(5).type(dtype), requires_grad=True)
target = Variable(torch.FloatTensor([1,2,3,4,5]).type(dtype),
requires_grad=False)

lr = 0.01 # the learning rate

d = []
for i in range(1000):
  distance = torch.mean(torch.pow((x - target), 2))
  d.append(distance.data)
  distance.backward(retain_graph=True)

  x.data.sub_(lr * x.grad.data)
  x.grad.data.zero_()

print(x.data)

fig, ax = plt.subplots()
ax.plot(d)
ax.set_xlabel("iteration")
ax.set_ylabel("distance")
plt.show()

The following is the graph of distance w.r.t iteration

We can see that the model converges at about 600 iterations. If we set the learning rate to be higher (e.g, lr=0.1), the model will converge much faster (it takes about 60 iterations, see image below)

Now, x becomes something like the following

  0.9878
   1.9749
   2.9624
   3.9429
   4.9292

which is pretty close to your target of [1, 2, 3, 4, 5].


Instruction: How to get around in place operation error if index leaf variable for gradient update?

Input:
I am encountering In place operation error when I am trying to index a leaf variable to update gradients with customized Shrink function. I cannot work around it. Any help is highly appreciated!

import torch.nn as nn
import torch
import numpy as np
from torch.autograd import Variable, Function

# hyper parameters
batch_size = 100 # batch size of images
ld = 0.2 # sparse penalty
lr = 0.1 # learning rate

x = Variable(torch.from_numpy(np.random.normal(0,1,(batch_size,10,10))), requires_grad=False)  # original

# depends on size of the dictionary, number of atoms.
D = Variable(torch.from_numpy(np.random.normal(0,1,(500,10,10))), requires_grad=True)

# hx sparse representation
ht = Variable(torch.from_numpy(np.random.normal(0,1,(batch_size,500,1,1))), requires_grad=True)

# Dictionary loss function
loss = nn.MSELoss()

# customized shrink function to update gradient
shrink_ht = lambda x: torch.stack([torch.sign(i)*torch.max(torch.abs(i)-lr*ld,0)[0] for i in x])

### sparse reprsentation optimizer_ht single image.
optimizer_ht = torch.optim.SGD([ht], lr=lr, momentum=0.9) # optimizer for sparse representation

## update for the batch
for idx in range(len(x)):
    optimizer_ht.zero_grad() # clear up gradients
    loss_ht = 0.5*torch.norm((x[idx]-(D*ht[idx]).sum(dim=0)),p=2)**2
    loss_ht.backward() # back propogation and calculate gradients
    optimizer_ht.step() # update parameters with gradients
    ht[idx] = shrink_ht(ht[idx])  # customized shrink function.

RuntimeError Traceback (most recent call last) in ()
15 loss_ht.backward() # back propogation and calculate gradients
16 optimizer_ht.step() # update parameters with gradients
—> 17 ht[idx] = shrink_ht(ht[idx]) # customized shrink function.
18
19

/home/miniconda3/lib/python3.6/site-packages/torch/autograd/variable.py in setitem(self, key, value)
85 return MaskedFill.apply(self, key, value, True)
86 else:
—> 87 return SetItem.apply(self, key, value)
88
89 def deepcopy(self, memo):

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

Specifically, this line of code below seems give error as it index and update leaf variable at the same time.

ht[idx] = shrink_ht(ht[idx])  # customized shrink function.

Thanks.

W.S.

Output: I just found: In order to update the variable, it needs to be ht.data[idx] instead of ht[idx]. We can use .data to access the tensor directly.


Instruction: Copying a PyTorch Variable to a Numpy array

Input:
Suppose I have a PyTorch Variable in GPU:

var = Variable(torch.rand((100,100,100))).cuda()

What's the best way to copy (not bridge) this variable to a NumPy array?

var.clone().data.cpu().numpy()

or

var.data.cpu().numpy().copy()

By running a quick benchmark, .clone() was slightly faster than .copy(). However, .clone() + .numpy() will create a PyTorch Variable plus a NumPy bridge, while .copy() will create a NumPy bridge + a NumPy array.

Output:
This is a very interesting question. According to me, the question is little bit opinion-based and I would like to share my opinion on this.

From the above two approaches, I would prefer the first one (use clone()). Since your goal is to copy information, essentially you need to invest extra memory. clone() and copy() should take a similar amount of storage since creating numpy bridge doesn't cause extra memory. Also, I didn't understand what you meant by, copy() will create two numPy arrays. And as you mentioned, clone() is faster than copy(), I don't see any other problem with using clone().

I would love to give a second thought on this if anyone can provide some counter arguments.


Instruction: Check the total number of parameters in a PyTorch model

Input: How do I count the total number of parameters in a PyTorch model? Something similar to model.count_params() in Keras.

Output:
PyTorch doesn't have a function to calculate the total number of parameters as Keras does, but it's possible to sum the number of elements for every parameter group:

pytorch_total_params = sum(p.numel() for p in model.parameters())

If you want to calculate only the trainable parameters:

pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

Answer inspired by this answer on PyTorch Forums.

Note: I'm answering my own question. If anyone has a better solution, please share with us.


Instruction: How to use pack_padded_sequence with multiple variable-length input with the same label in pytorch

Input:
I have a model which takes three variable-length inputs with the same label. Is there a way I could use pack_padded_sequence somehow? If so, how should I sort my sequences?

For example,

a = (([0,1,2], [3,4], [5,6,7,8]), 1) # training data is in length 3,2,4; label is 1
b = (([0,1], [2], [6,7,8,9,10]), 1)

Both a and b will be fed into three separated LSTMs and the result will be merged to predict the target.

Output:
Let's do it step by step.

  Input Data Processing

a = (([0,1,2], [3,4], [5,6,7,8]), 1)

# store length of each element in an array
len_a = np.array([len(a) for a in a[0]])
variable_a  = np.zeros((len(len_a), np.amax(len_a)))
for i, a in enumerate(a[0]):
    variable_a[i, 0:len(a)] = a

vocab_size = len(np.unique(variable_a))
Variable(torch.from_numpy(variable_a).long())
print(variable_a)

It prints:

Variable containing:
 0  1  2  0
 3  4  0  0
 5  6  7  8
[torch.DoubleTensor of size 3x4]

  Defining embedding and RNN layer

Now, let's say, we have an Embedding and RNN layer class as follows.

class EmbeddingLayer(nn.Module):

    def __init__(self, input_size, emsize):
        super(EmbeddingLayer, self).__init__()
        self.embedding = nn.Embedding(input_size, emsize)

    def forward(self, input_variable):
        return self.embedding(input_variable)

class Encoder(nn.Module):

    def __init__(self, input_size, hidden_size, bidirection):
        super(Encoder, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bidirection = bidirection
        self.rnn = nn.LSTM(self.input_size, self.hidden_size, batch_first=True,
                                    bidirectional=self.bidirection)

    def forward(self, sent_variable, sent_len):
        # Sort by length (keep idx)
        sent_len, idx_sort = np.sort(sent_len)[::-1], np.argsort(-sent_len)
        idx_unsort = np.argsort(idx_sort)

        idx_sort = torch.from_numpy(idx_sort)
        sent_variable = sent_variable.index_select(0, Variable(idx_sort))

        # Handling padding in Recurrent Networks
        sent_packed = nn.utils.rnn.pack_padded_sequence(sent_variable, sent_len, batch_first=True)
        sent_output = self.rnn(sent_packed)[0]
        sent_output = nn.utils.rnn.pad_packed_sequence(sent_output, batch_first=True)[0]

        # Un-sort by length
        idx_unsort = torch.from_numpy(idx_unsort)
        sent_output = sent_output.index_select(0, Variable(idx_unsort))

        return sent_output

  Embed and encode the processed input data

We can embed and encode our input as follows.

emb = EmbeddingLayer(vocab_size, 50)
enc = Encoder(50, 100, False, 'LSTM')

emb_a = emb(variable_a)
enc_a = enc(emb_a, len_a)

If you print the size of enc_a, you will get torch.Size([3, 4, 100]). I hope you understand the meaning of this shape.

Please note, the above code runs only on CPU.


Instruction: pytorch error: multi-target not supported in CrossEntropyLoss()

Input:
I am on a project using acceleration data to predict some activities.
But I have problems on the loss calculation. I am using CrossEntropyLoss for it.

Data is used for it like below
I use the first 4 data of each rows to predict the index like the last one of each rows.

1 84 84 81 4
81 85 85 80 1
81 82 84 80 1
1 85 84 2 0
81 85 82 80 1
81 82 84 80 1
81 25 84 80 5

The error messages are like below.

minoh@minoh-VirtualBox:~/cow$ python lec5.py
Traceback (most recent call last):
  File "lec5.py", line 97, in <module>
    train(epoch)
  File "lec5.py", line 74, in train
    loss = criterion(y_pred, labels)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 679, in forward
    self.ignore_index, self.reduce)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1161, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1052, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THNN/generic/ClassNLLCriterion.c:22

My code is based on Sung Kim's pytorch

import numpy as np
import torch
from torch.autograd import Variable
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

class CowDataset(Dataset):
    def __init__(self):
        xy_str = np.loadtxt('cow_test', delimiter = ' ', dtype = np.str)
        xy = xy_str.astype(np.float32)
        xy_int = xy_str.astype(np.int)
        self.len = xy.shape[0]
        self.x_data = torch.from_numpy(xy[:, 0:4])
        self.y_data = torch.from_numpy(xy_int[:, [4]])

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.len

dataset = CowDataset()
train_loader = DataLoader(dataset = dataset, batch_size = 32, shuffle = True)

class CowTestset(Dataset):
        def __init__(self):
                xy_str = np.loadtxt('cow_test2', delimiter = ' ', dtype =np.str)
                xy = xy_str.astype(np.float32)
                xy_int = xy_str.astype(np.int)
                self.len = xy.shape[0]
                self.x_data = torch.from_numpy(xy[:, 0:4])
                self.y_data = torch.from_numpy(xy_int[:, [4]])

        def __getitem__(self, index):
                return self.x_data[index], self.y_data[index]

        def __len__(self):
                return self.len

testset = CowTestset()
test_loader = DataLoader(dataset = testset, batch_size = 32, shuffle = True)

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(4,5)
        self.l2 = torch.nn.Linear(5,7)
        self.l3 = torch.nn.Linear(7,6)
        self.sigmoid = torch.nn.Sigmoid()

    def forward(self, x):
        out1 = self.sigmoid(self.l1(x))
        out2 = self.sigmoid(self.l2(out1))
        y_pred = self.sigmoid(self.l3(out2))
        return y_pred

model = Model()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.1, momentum = 0.5)

def train(epoch):
    model.train()
    for batch_idx, (inputs, labels) in enumerate(train_loader):
        inputs, labels = Variable(inputs), Variable(labels)
        optimizer.zero_grad()
        y_pred = model(inputs)
        loss = criterion(y_pred, labels)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        data, target = Variable(data, volatile = True), Variable(target)
        print(target)
        output = model(data)
        test_loss += criterion(output, target).data[0]
        pred = output.data.max(1, keepdim = True)[1]
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()
    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, len(test_loader.dataset), 100.* correct / len(test_loader.dataset)))

for epoch in range(1,7):
    train(epoch)
    test()

Output:
Ok. So I reproduced your problem and after some search and reading the API of CrossEntropyLoss(), I have found it's because you have a wrong label dimension.

Offical docs of CrossEntropyLoss here. And you can see

  Input: (N,C) where C = number of classes
  Target: (N) where each value is 0≤targets[i]≤C−1

While here, in your criterion() function,  you have a batchSize x 7 input and batchSize x 1 label. The confusing point is, say your batchSize is 10, a 10x1 tensor can not be regarded as a size-10 tensor, which is what the loss function expectes. You must explictly do the size conversion.

Solution:
Add labels = labels.squeeze_() before you call loss = criterion(y_pred, labels) and do the same thing in your test code. The squeeze_() funciton removes size-1 dimensions inplace. So you have a batchSize-size label now.


Instruction: Is there an efficient way to create a random bit mask in Pytorch?

Input:
I want to have a random bit mask that has some specified percent of 0s. The function I devised is:

def create_mask(shape, rate):
    """
    The idea is, you take a random permutations of numbers. You then mod then
    mod it by the [number of entries in the bitmask] / [percent of 0s you
    want]. The number of zeros will be exactly the rate of zeros need. You
    can clamp the values for a bitmask.
    """
    mask = torch.randperm(reduce(operator.mul, shape, 1)).float().cuda()
    # Mod it by the percent to get an even dist of 0s.
    mask = torch.fmod(mask, reduce(operator.mul, shape, 1) / rate)
    # Anything not zero should be put to 1
    mask = torch.clamp(mask, 0, 1)
    return mask.view(shape)

To illustrate:

>>> x = create_mask((10, 10), 10)
>>> x

    1     1     1     1     1     1     1     1     1     1
    1     1     1     1     1     1     0     1     1     1
    0     1     1     1     1     0     1     1     1     1
    0     1     1     1     1     1     1     1     1     1
    1     1     1     1     1     1     1     1     1     0
    1     1     1     1     1     1     1     1     1     1
    1     1     1     0     1     1     1     0     1     1
    0     1     1     1     1     1     1     1     1     1
    1     1     1     0     1     1     0     1     1     1
    1     1     1     1     1     1     1     1     1     1
[torch.cuda.FloatTensor of size 10x10 (GPU 0)]

The main issue I have with this method is it requires the rate to divide the shape. I want a function that accepts an arbitrary decimal and gives approximately rate percent of 0s in the bitmask. Furthermore, I am trying to find a relatively efficient way of doing so. Hence, I would rather not move a numpy array from the CPU to the GPU. Is there an effiecient way of doing so that allows for a decimal rate?

Output:
For anyone running into this, this will create a bitmask with approximately 80% zero's directly on GPU. (PyTorch 0.3)

torch.cuda.FloatTensor(10, 10).uniform_() > 0.8


Instruction: Difference between 1 LSTM with num_layers = 2 and 2 LSTMs in pytorch

Input:
I am new to deep learning and currently working on using LSTMs for language modeling. I was looking at the pytorch documentation and was confused by it.

If I create a

nn.LSTM(input_size, hidden_size, num_layers)

where hidden_size = 4 and num_layers = 2, I think I will have an architecture something like:

op0    op1 ....
LSTM -> LSTM -> h3
LSTM -> LSTM -> h2
LSTM -> LSTM -> h1
LSTM -> LSTM -> h0
x0     x1 .....

If I do something like

nn.LSTM(input_size, hidden_size, 1)
nn.LSTM(input_size, hidden_size, 1)

I think the network architecture will look exactly like above. Am I wrong? And if yes, what is the difference between these two?

Output:
The multi-layer LSTM is better known as stacked LSTM where multiple layers of LSTM are stacked on top of each other.

Your understanding is correct. The following two definitions of stacked LSTM are same.

nn.LSTM(input_size, hidden_size, 2)

and

nn.Sequential(OrderedDict([
    ('LSTM1', nn.LSTM(input_size, hidden_size, 1),
    ('LSTM2', nn.LSTM(hidden_size, hidden_size, 1)
]))

Here, the input is feed into the lowest layer of LSTM and then the output of the lowest layer is forwarded to the next layer and so on so forth. Please note, the output size of the lowest LSTM layer and the rest of the LSTM layer's input size is hidden_size.

However, you may have seen people defined stacked LSTM in the following way:

rnns = nn.ModuleList()
for i in range(nlayers):
    input_size = input_size if i == 0 else hidden_size
    rnns.append(nn.LSTM(input_size, hidden_size, 1))

The reason people sometimes use the above approach is that if you create a stacked LSTM using the first two approaches, you can't get the hidden states of each individual layer. Check out what LSTM returns in PyTorch.

So, if you want to have the intermedia layer's hidden states, you have to declare each individual LSTM layer as a single LSTM and run through a loop to mimic the multi-layer LSTM operations. For example:

outputs = []
for i in range(nlayers):
    if i != 0:
        sent_variable = F.dropout(sent_variable, p=0.2, training=True)
    output, hidden = rnns[i](sent_variable)
    outputs.append(output)
    sent_variable = output

In the end, outputs will contain all the hidden states of each individual LSTM layer.


Instruction: What is the difference between softmax and log-softmax?

Input:
The difference between these two functions that has been described in this pytorch post: What is the difference between log_softmax and softmax?
 is: exp(x_i) / exp(x).sum()
and log softmax is: log(exp(x_i) / exp(x).sum()).

But for the Pytorch code below why am I getting different output:

>>> it = autograd.Variable(torch.FloatTensor([0.6229,0.3771]))
>>> op = autograd.Variable(torch.LongTensor([0]))
>>> m  = nn.Softmax()
>>> log = nn.LogSoftmax()
>>> m(it)
Variable containing:
`0.5611  0.4389`
[torch.FloatTensor of size 1x2]
>>>log(it)
Variable containing:
-0.5778 -0.8236
[torch.FloatTensor of size 1x2]

However, the value log(0.5611) is -0.25095973129 and log(0.4389) is -0.35763441915

Why is there such discrepancy?

Output:
By default, torch.log provides the natural logarithm of the input, so the output of PyTorch is correct:

ln([0.5611,0.4389])=[-0.5778,-0.8236]

Your last results are obtained using the logarithm with base 10.


Instruction: Pytorch beginner : tensor.new method

Input:
everyone, I have a small question.

What is the purpose of the method tensor.new(..) in Pytorch, I didn't find anything in the documentation. It looks like it creates a new Tensor (like the name suggests), but why we don't just use torch.Tensor constructors instead of using this new method that requires an existing tensor.

Thank you in advance.

Output:
As the documentation of tensor.new() says:

  Constructs a new tensor of the same data type as self tensor.

Also note:

  For CUDA tensors, this method will create new tensor on the same device as this tensor.


Instruction: How does batching work in a seq2seq model in pytorch?

Input:
I am trying to implement a seq2seq model in Pytorch and I am having some problem with the batching.
For example I have a batch of data whose dimensions are

  [batch_size, sequence_lengths, encoding_dimension]

where the sequence lengths are different for each example in the batch.

Now, I managed to do the encoding part by padding each element in the batch to the length of the longest sequence.

This way if I give as input to my net a batch with the same shape as said, I get the following outputs:

  output, of shape [batch_size, sequence_lengths, hidden_layer_dimension]

  hidden state, of shape [batch_size, hidden_layer_dimension]

  cell state, of shape [batch_size, hidden_layer_dimension]

Now, from the output, I take for each sequence the last relevant element, that is the element along the sequence_lengths dimension corresponding to the last non padded element of the sequence. Thus the final output I get is of shape [batch_size, hidden_layer_dimension].

But now I have the problem of decoding it from this vector. How do I handle a decoding of sequences of different lengths in the same batch? I tried to google it and found this, but they don't seem to address the problem. I thought of doing element by element for the whole batch, but then I have the problem to pass the initial hidden states, given that the ones from the encoder will be of shape [batch_size, hidden_layer_dimension], while the ones from the decoder will be of shape [1, hidden_layer_dimension].

Am I missing something? Thanks for the help!

Output:
You are not missing anything. I can help you since I have worked on several sequence-to-sequence application using PyTorch. I am giving you a simple example below.

class Seq2Seq(nn.Module):
    """A Seq2seq network trained on predicting the next query."""

    def __init__(self, dictionary, embedding_index, args):
        super(Seq2Seq, self).__init__()

        self.config = args
        self.num_directions = 2 if self.config.bidirection else 1

        self.embedding = EmbeddingLayer(len(dictionary), self.config)
        self.embedding.init_embedding_weights(dictionary, embedding_index, self.config.emsize)

        self.encoder = Encoder(self.config.emsize, self.config.nhid_enc, self.config.bidirection, self.config)
        self.decoder = Decoder(self.config.emsize, self.config.nhid_enc * self.num_directions, len(dictionary),
                                   self.config)

    @staticmethod
    def compute_decoding_loss(logits, target, seq_idx, length):
        losses = -torch.gather(logits, dim=1, index=target.unsqueeze(1)).squeeze()
        mask = helper.mask(length, seq_idx)  # mask: batch x 1
        losses = losses * mask.float()
        num_non_zero_elem = torch.nonzero(mask.data).size()
        if not num_non_zero_elem:
        return losses.sum(), 0 if not num_non_zero_elem else losses.sum(), num_non_zero_elem[0]

    def forward(self, q1_var, q1_len, q2_var, q2_len):
        # encode the query
        embedded_q1 = self.embedding(q1_var)
        encoded_q1, hidden = self.encoder(embedded_q1, q1_len)

        if self.config.bidirection:
            if self.config.model == 'LSTM':
                h_t, c_t = hidden[0][-2:], hidden[1][-2:]
                decoder_hidden = torch.cat((h_t[0].unsqueeze(0), h_t[1].unsqueeze(0)), 2), torch.cat(
                    (c_t[0].unsqueeze(0), c_t[1].unsqueeze(0)), 2)
            else:
                h_t = hidden[0][-2:]
                decoder_hidden = torch.cat((h_t[0].unsqueeze(0), h_t[1].unsqueeze(0)), 2)
        else:
            if self.config.model == 'LSTM':
                decoder_hidden = hidden[0][-1], hidden[1][-1]
            else:
                decoder_hidden = hidden[-1]

        decoding_loss, total_local_decoding_loss_element = 0, 0
        for idx in range(q2_var.size(1) - 1):
            input_variable = q2_var[:, idx]
            embedded_decoder_input = self.embedding(input_variable).unsqueeze(1)
            decoder_output, decoder_hidden = self.decoder(embedded_decoder_input, decoder_hidden)
            local_loss, num_local_loss = self.compute_decoding_loss(decoder_output, q2_var[:, idx + 1], idx, q2_len)
            decoding_loss += local_loss
            total_local_decoding_loss_element += num_local_loss

        if total_local_decoding_loss_element > 0:
            decoding_loss = decoding_loss / total_local_decoding_loss_element

        return decoding_loss

You can see the complete source code here. This application is about predicting users' next web-search query given the current web-search query.

The answerer to your question:

  How do I handle a decoding of sequences of different lengths in the same batch?

You have padded sequences, so you can consider as all the sequences are of the same length. But when you are computing loss, you need to ignore loss for those padded terms using masking.

I have used a masking technique to achieve the same in the above example.

Also, you are absolutely correct on: you need to decode element by element for the mini-batches. The initial decoder state [batch_size, hidden_layer_dimension] is also fine. You just need to unsqueeze it at dimension 0, to make it [1, batch_size, hidden_layer_dimension].

Please note, you do not need to loop over each example in the batch, you can execute the whole batch at a time, but you need to loop over the elements of the sequences.


Instruction: Tying weights in neural machine translation

Input:
I want to tie weights of the embedding layer and the next_word prediction layer of the decoder. The embedding dimension is set to 300 and the hidden size of the decoder is set to 600. Vocabulary size of the target language in NMT is 50000, so embedding weight dimension is 50000 x 300 and weight of the linear layer which predicts the next word is 50000 x 600.

So, how can I tie them? What will be the best approach to achieve weight tying in this scenario?

Output: You could use linear layer to project the 600 dimensional space down to 300 before you apply the shared projection. This way you still get the advantage that the entire embedding (possibly) has a non-zero gradient for each mini-batch but at the risk of increasing the capacity of the network slightly.


Instruction: PyTorch: Comparing predicted label and target label to compute accuracy

Input:
I'm trying to implement this loop to get the accuracy of my PyTorch CNN (The complete code of it is here)
My version of the loop is so far:

correct = 0
    test_total = 0
    for itera, testdata2 in enumerate(test_loader, 0):
        test_images2, test_labels2 = testdata2
        if use_gpu:
            test_images2 = Variable(test_images2.cuda())
        else:
            test_images2 = Variable(test_images2)
        outputs = model(test_images2)
        _, predicted = torch.max(outputs.data, 1)
        test_total += test_labels2.size(0)
        test_labels2 = test_labels2.type_as(predicted)
        correct += (predicted == test_labels2[0]).sum()
    print('Accuracy of the network on all the test images: %d %%' % (
        100 * correct / test_total))

If I run it like this, I get:

> Traceback (most recent call last):   File
> "c:/python_code/Customized-DataLoader-master_two/multi_label_classifier_for2classes.py",
> line 186, in <module>
>     main()   File "c:/python_code/Customized-DataLoader-master_two/multi_label_classifier_for2classes.py",
> line 177, in main
>     correct += (predicted == test_labels2[0]).sum()   File "C:\anaconda\envs\pytorch_cuda\lib\site-packages\torch\tensor.py",
> line 360, in __eq__
>     return self.eq(other) RuntimeError: invalid argument 3: sizes do not match at
> c:\anaconda2\conda-bld\pytorch_1519501749874\work\torch\lib\thc\generated\../THCTensorMathCompareT.cuh:65

I used test_labels2 = test_labels2.type_as(predicted) to have both tensors as LongTensors, which seems to work fine to avert the "Expected this...but got..." errors. They look like this now:

test_labels2 after conversion:
 0  1
 1  0
 1  0
[torch.cuda.LongTensor of size 3x2 (GPU 0)]

predicted:
 1
 1
 1
[torch.cuda.LongTensor of size 3 (GPU 0)]

I supppose the problem now is, that test_labels2[0] is returning a row but not the column.

How do I get this to work?

Output:
Indexing in pytorch works mostly like indexing in numpy. To index all rows of a certain column j use:

tensor[:, j]

Alternatively, the select function from pytorch can be used.


Instruction: Pytorch model weight type conversion

Input:
I'm trying to do inference on FlowNet2-C model loading from file.
However, I met some data type problem.  How can I resolve it?

Source code
FlowNet2-C pre-trained model

$ python main.py

Initializing Datasets
  [0.000s] Loading checkpoint '/notebooks/data/model/FlowNet2-C_checkpoint.pth.tar'
  [1.293s] Loaded checkpoint '/notebooks/data/model/FlowNet2-C_checkpoint.pth.tar' (at epoch 0)
(1L, 6L, 384L, 512L)
<class 'torch.autograd.variable.Variable'>
  [1.642s] Operation failed

Traceback (most recent call last):
  File "main.py", line 102, in <module>
    main()
  File "main.py", line 98, in main
    summary(input_size, model)
  File "main.py", line 61, in summary
    model(x)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/notebooks/data/vinet/FlowNetC.py", line 75, in forward
    out_conv1a = self.conv1(x1)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 282, in forward
    self.padding, self.dilation, self.groups)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/functional.py", line 90, in conv2d
    return f(input, weight, bias)
RuntimeError: Input type (CUDAFloatTensor) and weight type (CPUFloatTensor) should be the same

Output:
Maybe that is because your model and input x to the model has different data types. It seems that the model parameters have been moved to GPU, but your input x is on GPU.

You can try to use model.cuda() after line 94, which will put the model on the GPU. Then the error should disappear.


Instruction: math operator difference *= or +=

Input:
I found a weird thing when I used the operator e.g. *= or +=

The code:

aa = Variable(torch.FloatTensor([[1,2],[3,4]]))
bb = aa
bb = bb*2
print(bb)
print(aa)

cc = Variable(torch.FloatTensor([[1,2],[3,4]]))
dd = cc
dd *= 2
print(cc)
print(dd)

The results showed as below:

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

Variable containing:
 1  2
 3  4
[torch.FloatTensor of size 2x2]

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

As you see, when I used bb=bb*2, aa was not be affected. However, if using dd *= 2, cc seems to point towards (share) the same address as cc, it is changed.

Their respective previous line is the same, e.g. bb = aa and dd = cc. It seems that the *= operator changed the original deep copy to shallow copy and the change was made after the copy line itself.

I am wondering if this is a bug. If it is, it is important since it affects the basic mathematics operations. Generally, I thought that just use the built-in operation functions e.g. torch.add() is a good solution.

OS: Mac OS X
PyTorch version: 3.0
How you installed PyTorch (conda, pip, source): conda
Python version: 3.6
CUDA/cuDNN version: None
GPU models and configuration: None

*---------------------------------------

I understand dd *= 2 is multiplication in-place, but how the value of dd transfer into cc? but if I used dd = dd * 2, the new values did not transfer to cc? There is no difference in the previous line: dd = cc and bb =aa.
BTW, in python (not pytorch Variable or Tensor), dd *= 2 and dd = dd * 2 both will not affect cc value.

Output:
When you do dd = cc, both dd and cc are now references to the same object (same for bb = aa). Nothing is being copied!

When you do bb = bb * 2, the * operator creates a new object and bb now refers to that object. No existing object is changed.

When you do dd *= 2, the object that dd refers to (and which cc also refers to) is changed.

So the difference is that * creates a new object and = makes a variable refer to a new object (rather than changing the object in any way), whereas *= changes the object.

It may be counter-intuitive that x *= y behaves differently than x = x * y, but those are the semantics of the language, not a bug.


Instruction: How to take shape [1,1,256] from [1,4,256] cuda.FloatTensor?

Input:
I have an batch of output hidden vector from GRU. It's shape is [1,4,256]

( 0 ,.,.) =
-0.9944  1.0000  0.0000  ...  -1.0000  0.0000 -1.0000
-1.0000  1.0000  0.0000  ...  -1.0000  0.0000 -1.0000
-1.0000  1.0000  0.0000  ...  -1.0000  0.0000 -1.0000
-1.0000  1.0000  0.0000  ...  -1.0000  0.0000 -1.0000
[torch.cuda.FloatTensor of size (1,4,256) (GPU 0)]

I need a shape of [1,1,256] to pass to another model. How can I take it? Through this line I can only have a shape of [1,256]

decoder_hidden = encoder_hidden[:, index]

resize and wrap a new FloatTensor didn't work.

Output:
You can unsqueeze() in dimension 1 to achieve this.

encoder_hidden = torch.randn(1, 4, 256)
print(encoder_hidden.size())

for idx in range(encoder_hidden.size(1)):
    decoder_hidden = encoder_hidden[:, idx, :].unsqueeze(1)
    print(decoder_hidden.size())

It prints:

torch.Size([1, 4, 256])
torch.Size([1, 1, 256])
torch.Size([1, 1, 256])
torch.Size([1, 1, 256])
torch.Size([1, 1, 256])


Instruction: Understanding PyTorch CNN Channels

Input:
I'm a bit confused at how CNNs and channels work. Specifically, how come these two implementations are not equal? Isn't the # of output channels just applying however many # of filters?

    self.conv1 = nn.Conv2d(1, 10, kernel_size=(3, self.embeds_size))
    self.conv2 = nn.ModuleList([nn.Conv2d(1, 1, kernel_size=(3, self.embeds_size)) for f in range(10)])
    ...

    conv1s = self.conv1(x)
    conv2s = [conv(x) for conv in self.conv2]
    conv2s = torch.stack(conv2s, 1).squeeze(2)
    print(torch.equal(conv1s, conv2s))

Output:
Check the state dicts of the different modules. Unless you're doing something fancy that you didn't tell us about, PyTorch will initialize the weights randomly. Specifically, try this:

print(self.conv1.state_dict()["weight"][0])
print(self.conv2[0].state_dict()["weight"][0])

They will be different.


Instruction: Non linear mapping to vector of higher dimension

Input:
I am learning Keras and need help on the following. I currently have a sequence of floats in lists X and Y. What I need to do is to have a non-linear mapping to map each element to a vector of higher dimension following the below equation.

pos(i) = tanh(W.[concat(X[i],Y[i]])
#where W is a learnable weight matrix, concat performs the concatenation and pos(i) is a vector of 16x1. (I'm trying to create 16 channel inputs for a CNN).

I found that Pytorch implementation for the above is

m = nn.linear(2,16)
input = torch.cat(X[i],Y[i])
torch.nn.functional.tanh(m(input))

Currently I've tried the concat and tanh in numpy and it seems that is not what I want here.

Can you help me implement the above using Keras.

Output:
Based on what you have there.

This is what I would do in keras. Im going to assume that you just want your to concatenate your inputs before you feed them into the model.

So we'll do it with numpy. Note

something like :

import numpy as np
from keras.model import Dense, Model,Input
X = np.random.rand(100, 1)
Y = np.random.rand(100, 1)
y = np.random.rand(100, 16)
# concatenate along the features in numpy
XY = np.cancatenate(X, Y, axis=1)

# write model
in = Input(shape=(2, ))
out = Dense(16, activation='tanh')(in)
# print(out.shape) (?, 16)
model = Model(in, out)
model.compile(loss='mse', optimizer='adam')
model.fit(XY, y)

....


Instruction: Cross Entropy in PyTorch

Input:
Cross entropy formula:

But why does the following give loss = 0.7437 instead of loss = 0 (since 1*log(1) = 0)?

import torch
import torch.nn as nn
from torch.autograd import Variable

output = Variable(torch.FloatTensor([0,0,0,1])).view(1, -1)
target = Variable(torch.LongTensor([3]))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

Output:
In your example you are treating output [0, 0, 0, 1] as probabilities as required by the mathematical definition of cross entropy.  But PyTorch treats them as outputs, that don’t need to sum to 1, and need to be first converted into probabilities for which it uses the softmax function.
So H(p, q) becomes:
H(p, softmax(output))

Translating the output [0, 0, 0, 1] into probabilities:
softmax([0, 0, 0, 1]) = [0.1749, 0.1749, 0.1749, 0.4754]

whence:
-log(0.4754) = 0.7437


Instruction: How do I re-use trained fastai models?

Input:
How do I load pretrained model using fastai implementation over PyTorch? Like in SkLearn I can use pickle to dump a model in file then load and use later. I've use .load() method after declaring learn instance like bellow to load previously saved weights:

arch=resnet34
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=False)
learn.load('resnet34_test')

Then to predict the class of an image:

trn_tfms, val_tfms = tfms_from_model(arch,100)
img = open_image('circle/14.png')
im = val_tfms(img)
preds = learn.predict_array(im[None])
print(np.argmax(preds))

But It gets me the error:

ValueError: Expected more than 1 value per channel when training, got input size [1, 1024]

This code works if I use learn.fit(0.01, 3) instead of learn.load(). What I really want is to avoid the training step In my application.

Output:
This error occurs whenever a batch of your data contains a single element.

Solution 1:
Call learn.predict() after learn.load('resnet34_test')

Solution 2:
Remove 1 data point from your training set.

Pytorch issue

Fastai forum issue description


Instruction: RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'weight'

Input:
My input tensor is torch.DoubleTensor type. But I got the RuntimeError below:

RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'weight'

I didn't specify the type of the weight explicitly(i.e. I did not init my weight by myself. The weight is created by pytorch). What will influence the type of weight in the forward process?

Thanks a lot!!

Output:
The default type for weights and biases are torch.FloatTensor. So, you'll need to cast either your model to torch.DoubleTensor or cast your inputs to torch.FloatTensor. For casting your inputs you can do

X = X.float()

or cast your complete model to DoubleTensor as

model = model.double()

You can also set the default type for all tensors using

pytorch.set_default_tensor_type('torch.DoubleTensor')

It is better to convert your inputs to float rather than converting your model to double, because mathematical computations on double datatype is considerably slower on GPU.


Instruction: Replace diagonal elements with vector in PyTorch

Input:
I have been searching everywhere for something equivalent of the following to PyTorch, but I cannot find anything.

L_1 = np.tril(np.random.normal(scale=1., size=(D, D)), k=0)
L_1[np.diag_indices_from(L_1)] = np.exp(np.diagonal(L_1))

I guess there is no way to replace the diagonal elements in such an elegant way using Pytorch.

Output:
I do not think that such a functionality is implemented as of now. But, you can implement the same functionality using mask as follows.

# Assuming v to be the vector and a be the tensor whose diagonal is to be replaced
mask = torch.diag(torch.ones_like(v))
out = mask*torch.diag(v) + (1. - mask)*a

So, your implementation will be something like

L_1 = torch.tril(torch.randn((D, D)))
v = torch.exp(torch.diag(L_1))
mask = torch.diag(torch.ones_like(v))
L_1 = mask*torch.diag(v) + (1. - mask)*L_1

Not as elegant as numpy, but not too bad either.


Instruction: How do I initialize weights in PyTorch?

Input: How do I initialize weights and biases of a network (via e.g. He or Xavier initialization)?

Output:
Single layer
To initialize the weights of a single layer, use a function from torch.nn.init. For instance:
conv1 = torch.nn.Conv2d(...)
torch.nn.init.xavier_uniform(conv1.weight)

Alternatively, you can modify the parameters by writing to conv1.weight.data (which is a torch.Tensor). Example:
conv1.weight.data.fill_(0.01)

The same applies for biases:
conv1.bias.data.fill_(0.01)

nn.Sequential or custom nn.Module
Pass an initialization function to torch.nn.Module.apply. It will initialize the weights in the entire nn.Module recursively.

apply(fn): Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch-nn-init).

Example:
def init_weights(m):
    if isinstance(m, nn.Linear):
        torch.nn.init.xavier_uniform(m.weight)
        m.bias.data.fill_(0.01)

net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
net.apply(init_weights)


Instruction: Understanding the code in pyTorch

Input:
I am having problems with understanding the following part of the code from ResNet architecture. The full code is available at https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/deep_residual_network/main-gpu.py . I am not very familiar with Python.

# Residual Block
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

# ResNet Module
class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes=10):
        super(ResNet, self).__init__()
        self.in_channels = 16
        self.conv = conv3x3(3, 16)
        self.bn = nn.BatchNorm2d(16)
        self.relu = nn.ReLU(inplace=True)
        self.layer1 = self.make_layer(block, 16, layers[0])
        self.layer2 = self.make_layer(block, 32, layers[0], 2)
        self.layer3 = self.make_layer(block, 64, layers[1], 2)
        self.avg_pool = nn.AvgPool2d(8)
        self.fc = nn.Linear(64, num_classes)

    def make_layer(self, block, out_channels, blocks, stride=1):
        downsample = None
        if (stride != 1) or (self.in_channels != out_channels):
            downsample = nn.Sequential(
                conv3x3(self.in_channels, out_channels, stride=stride),
                nn.BatchNorm2d(out_channels))
        layers = []
        layers.append(block(self.in_channels, out_channels, stride, downsample))
        self.in_channels = out_channels
        for i in range(1, blocks):
            layers.append(block(out_channels, out_channels))
        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.conv(x)
        out = self.bn(out)
        out = self.relu(out)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.avg_pool(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

resnet = ResNet(ResidualBlock, [3, 3, 3])

My main question is why should we pass 'block' every time? In the function

def make_layer(self, block, out_channels, blocks, stride=1):

instead of passing 'block' why cant we create an instance of 'ResidualBlock' and append it with layers as follows?

   block = ResidualBlock(self.in_channels, out_channels, stride, downsample)
   layers.append(block)

Output:
The ResNet module is designed to be generic, so that it can create networks with arbitrary blocks. So, if you do not pass the block which you want to create you'll have to write the name of the block explicitly like below.

# Residual Block
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

# ResNet Module
class ResNet(nn.Module):
    def __init__(self, layers, num_classes=10):
        super(ResNet, self).__init__()
        self.in_channels = 16
        self.conv = conv3x3(3, 16)
        self.bn = nn.BatchNorm2d(16)
        self.relu = nn.ReLU(inplace=True)
        self.layer1 = self.make_layer(16, layers[0])
        self.layer2 = self.make_layer(32, layers[0], 2)
        self.layer3 = self.make_layer(64, layers[1], 2)
        self.avg_pool = nn.AvgPool2d(8)
        self.fc = nn.Linear(64, num_classes)

    def make_layer(self, out_channels, blocks, stride=1):
        downsample = None
        if (stride != 1) or (self.in_channels != out_channels):
            downsample = nn.Sequential(
                conv3x3(self.in_channels, out_channels, stride=stride),
                nn.BatchNorm2d(out_channels))
        layers = []
        layers.append(ResidualBlock(self.in_channels, out_channels, stride, downsample))   # Major change here
        self.in_channels = out_channels
        for i in range(1, blocks):
            layers.append(ResidualBlock(out_channels, out_channels))    # Major change here
        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.conv(x)
        out = self.bn(out)
        out = self.relu(out)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.avg_pool(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

resnet = ResNet([3, 3, 3])

This reduces the capability of your ResNet module and binds it with only the ResidualBlock. Now, if you create some other type of block (say ResidualBlock2), you will need to create another Resnet2 module specifically for that. So, it's better to create a generic ResNet module which takes in the block parameter, so that it can be used with different types of blocks.

A trivial python example to clarify

Suppose you want to create a function that can apply a mathematical operation on a list and returns its output. So, you might create something like below

def exp(inp_list):
    out_list = []
    for num in inp_list:
        out_list.append(math.exp(num))
    return out_list

def floor(inp_list):
    out_list = []
    for num in inp_list:
        out_list.append(math.floor(num))
    return out_list

Here, we are doing an exponent and a floor operation on some input list. But, we can do a better job by defining a generic function to do the same as

def apply_func(fn, inp_list):
    out_list = []
    for num in inp_list:
        out_list.append(fn(num))
    return out_list

and now call this apply_func as apply_func(math.exp, inp_list) for exponential and as apply_func(math.floor, inp_list) for floor function. Also this opens up possibility for any kind of operation.

  Note: It's not a practical example as you can always use map or list comprehension for achieving the same thing. But, it demonstrates the use clearly.


Instruction: How can I update the parameters of a neural network in PyTorch?

Input: Let's say I wanted to multiply all parameters of a neural network in PyTorch (an instance of a class inheriting from torch.nn.Module) by 0.9. How would I do that?

Output:
Let net be an instance of a neural network nn.Module.
Then, to multiply all parameters by 0.9:
state_dict = net.state_dict()

for name, param in state_dict.items():
    # Transform the parameter as required.
    transformed_param = param * 0.9

    # Update the parameter.
    param.copy_(transformed_param)

If you want to only update weights instead of every parameter:
state_dict = net.state_dict()

for name, param in state_dict.items():
    # Don't update if this is not a weight.
    if not "weight" in name:
        continue

    # Transform the parameter as required.
    transformed_param = param * 0.9

    # Update the parameter.
    param.copy_(transformed_param)


Instruction: PyTorch forward pass using weights trained by Theano

Input:
I've trained a small size CNN binary classifier in Theano. To have a simpler code, I wanted to port the trained weights to PyTorch or numpy forward pass for predictions. The predictions by original Theano program are satisfying but the PyTorch forward pass predicted all the examples to one class.

Here is how I save trained weights in Theano using h5py:

layer0_w = layer0.W.get_value(borrow=True)
layer0_b = layer0.b.get_value(borrow=True)
layer1_w = layer1.W.get_value(borrow=True)
layer1_b = layer1.b.get_value(borrow=True)
layer2_w = layer2.W.get_value(borrow=True)
layer2_b = layer2.b.get_value(borrow=True)
sm_w = layer_softmax.W.get_value(borrow=True)
sm_b = layer_softmax.b.get_value(borrow=True)

h5_l0w = h5py.File('./model/layer0_w.h5', 'w')
h5_l0w.create_dataset('layer0_w', data=layer0_w)
h5_l0b = h5py.File('./model/layer0_b.h5', 'w')
h5_l0b.create_dataset('layer0_b', data=layer0_b)
h5_l1w = h5py.File('./model/layer1_w.h5', 'w')
h5_l1w.create_dataset('layer1_w', data=layer1_w)
h5_l1b = h5py.File('./model/layer1_b.h5', 'w')
h5_l1b.create_dataset('layer1_b', data=layer1_b)
h5_l2w = h5py.File('./model/layer2_w.h5', 'w')
h5_l2w.create_dataset('layer2_w', data=layer2_w)
h5_l2b = h5py.File('./model/layer2_b.h5', 'w')
h5_l2b.create_dataset('layer2_b', data=layer2_b)
h5_smw = h5py.File('./model/softmax_w.h5', 'w')
h5_smw.create_dataset('softmax_w', data=sm_w)
h5_smb = h5py.File('./model/softmax_b.h5', 'w')
h5_smb.create_dataset('softmax_b', data=sm_b)

Then load the weights to build a forward pass using Pytorch and Numpy:

import torch
import numpy as np
import torch.nn.functional as F
def model(data):

    conv0_out = F.conv2d(input=np2var(data),
                         weight=np2var(layer0_w),
                         bias=np2var(layer0_b)
                        )
    layer0_out = relu(var2np(conv0_out))

    conv1_out = F.conv2d(input=np2var(layer0_out),
                         weight=np2var(layer1_w),
                         bias=np2var(layer1_b)
                        )
    layer1_out = np.max(relu(var2np(conv1_out)), axis=2)

    dense_out=relu(np.matmul(layer1_out, layer2_w) + layer2_b)

    softmax_out = softmax(np.matmul(dense_out, softmax_w) + softmax_b)

    return softmax_out

def relu(x):
    return x * (x > 0)
def np2var(x):
    return torch.autograd.Variable(torch.from_numpy(x))
def var2np(x):
    return x.data.numpy()
def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

The input and kernel shapes for conv2d functions are the same for Theano and PyTorch, and network structures in two frameworks are the same. I couldn't detect any errors step by step. What could go wrong here?

Output:
Theano uses convolutions (by default, filter_flip=True) while PyTorch uses cross-correlation. So, for every convolutional layer, you need to flip the weights before using them in PyTorch.

You can use convert_kernel function from Keras to achieve this result.