Instruction: Is it possible to use different L1 / L2 regularization parameters for different sets of weights in chainer or pytorch?

Input:
(As an example) When implementing a simple linear model for noutput target values as a neural network in pytorch:

l1=L.Linear(ninput, noutput)

(call)

y = self.l1(x)
return y

Adding this hook will do L2 regularization on all weights, imposing the same alpha=0.01 everywhere:

optimizer.add_hook(optimizer.WeightDecay(rate=0.01))

Is it possible to use a different alpha for each set of weights leading from all ninput input units to one of the noutput output units?

Output:
Since we are working in pytorch it is possible to add other scalars to loss function yourself. So assume loss from you classfier is L ( assume it is a cross entropy loss ) and you have a linear layer defined as:

l1 = nn.Linear(in,out)

Now if you want to have different regularization on each set of weights then all you have to do is gather weights using ( i.e select using index) and add to the final loss:

loss = L (crossentropy loss) + sum ( alpha * norm(l1.weight[k]))

alpha the hyper-parameters and norm is mostly L2 norm,in pytorch it is just torch.norm(l1.weight) where index k is a tensor of indices of weights you want to select. Finally, you don't need to do the global regularization as you have done in the code.


Instruction: PyTorch 3 reshaping error

Input:
When training a CNN using PyTorch in Python, I get the following error:

RuntimeError: invalid argument 2: size '[-3 x 3136]' is invalid for input with 160000 elements at /opt/conda/conda-bld/pytorch-cpu_1515613813020/work/torch/lib/TH/THStorage.c:41

This is related to the x.view line in the model below:

class Net(nn.Module):

    def __init__(self):
        super(Net,self).__init__()
        self.conv1  = nn.Conv2d(3,32,5,padding=2) # 1 input, 32 out, filter size = 5x5, 2 block outer padding
        self.conv2  = nn.Conv2d(32,64,5,padding=2) # 32 input, 64 out,  filter size = 5x5, 2 block padding
        self.fc1    = nn.Linear(64*7*7,1024) # Fully connected layer
        self.fc2    = nn.Linear(1024,2) #Fully connected layer 2 out.

    def forward(self,x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2) # Max pool over convolution with 2x2 pooling
        x = F.max_pool2d(F.relu(self.conv2(x)), 2) # Max pool over convolution with 2x2 pooling
        x = x.view(-1,64*7*7) # tensor.view() reshapes the tensor
        x = F.relu(self.fc1(x)) # Activation function after passing through fully connected layer
        x = F.dropout(x, training=True) #Dropout regularisation
        x = self.fc2(x) # Pass through final fully connected layer
        return F.log_softmax(x) # Give results using softmax

model = Net()
print(model)

I'm not sure if this is a result of the images having 3 channels or something else entirely. I understand that this command should reshape the images into single dimensional arrays ready for the fully connected layer so I'm not sure how to fix this issue when the error is claiming an input of 160000 elements.

Output:
I will assume your input images are probably of size 200x200px (by size I mean here height x width, not taking the number of channels into account).

While your nn.Conv2d layers are defined to output tensors of the same size (with 32 channels for conv1 and 64 channels for con2), the F.max_pool2d are defined in such a way they divide height and width by 2.

So after 2 max-pooling operations, your tensors are of size 200 / (2 * 2) x 200 / (2 * 2) = 50x50px. With the 64 channels from conv2, you get 64 * 50 * 50 = 160000 elements.

Now, you need to adapt your view() so that it converts those inputs of shape (batch_size, 64, 50, 50) into (batch_size, 64 * 50 * 50) (to preserve the number of elements). You need to similarly adapt your 1st fully-connected layer.

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class Net(nn.Module):

    def __init__(self):
        super(Net,self).__init__()
        self.conv1  = nn.Conv2d(3,32,5,padding=2) # 1 input, 32 out, filter size = 5x5, 2 block outer padding
        self.conv2  = nn.Conv2d(32,64,5,padding=2) # 32 input, 64 out,  filter size = 5x5, 2 block padding
        self.fc1    = nn.Linear(64*50*50,1024) # Fully connected layer
        self.fc2    = nn.Linear(1024,2) #Fully connected layer 10 out.

    def forward(self,x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2) # Max pool over convolution with 2x2 pooling
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2) # Max pool over convolution with 2x2 pooling
        x = x.view(-1,64*50*50) # tensor.view() reshapes the tensor
        x = F.relu(self.fc1(x)) # Activation function after passing through fully connected layer
        x = F.dropout(x, training=True) #Dropout regularisation
        x = self.fc2(x) # Pass through final fully connected layer
        return F.log_softmax(x) # Give results using softmax

model = Net()
print(model)

x = np.ones((1, 3, 200, 200))
x = torch.tensor(x)
x = model.forward(x)
print(x)


Instruction: Getting diagonal "stripe" from matrix in NumPy or PyTorch

Input:
I need to get the diagonal "stripe" of a matrix. Say I have a matrix of size KxN (K>N):
[[ 0  1  2]
 [ 3  4  5]
 [ 6  7  8]
 [ 9 10 11]]

From it I need to extract a diagonal stripe, in this case, a matrix MxV size that is created by truncating the original one:
[[ 0  x  x]
 [ 3  4  x]
 [ x  7  8]
 [ x  x  11]]

So the result matrix is:
[[ 0  4  8]
 [ 3  7  11]]

I could define a bolean mask like so:
import numpy as np

X=np.arange(12).reshape(4,3)
mask=np.asarray([
  [ True,  False,  False],
  [ True,  True,  False],
  [ False, True,  True],
  [ False, False,  True]
])

>>> X
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]])

>>> X.T[mask.T].reshape(3,2).T
array([[ 0,  4,  8],
       [ 3,  7, 11]])

But I don't see how such a mask could be automatically generated to an arbitrary KxN matrix (e.g. 39x9, 360x96)
Is there a function that does this automatically either in numpy, scipy or pytorch?

Additional question: is it possible to get a "reverse stripe" instead? i.e.
[[ x   x   2]
 [ x   4   5]
 [ 6   7   x]
 [ 9   x   x]]

Output:
stride_tricks do the trick:

>>> import numpy as np
>>>
>>> def stripe(a):
...    a = np.asanyarray(a)
...    *sh, i, j = a.shape
...    assert i >= j
...    *st, k, m = a.strides
...    return np.lib.stride_tricks.as_strided(a, (*sh, i-j+1, j), (*st, k, k+m))
...
>>> a = np.arange(24).reshape(6, 4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15],
       [16, 17, 18, 19],
       [20, 21, 22, 23]])
>>> stripe(a)
array([[ 0,  5, 10, 15],
       [ 4,  9, 14, 19],
       [ 8, 13, 18, 23]])

If a is an array this creates a writable view, meaning that if you feel so inclined you can do things like

>>> stripe(a)[...] *= 10
>>> a
array([[  0,   1,   2,   3],
       [ 40,  50,   6,   7],
       [ 80,  90, 100,  11],
       [ 12, 130, 140, 150],
       [ 16,  17, 180, 190],
       [ 20,  21,  22, 230]])

UPDATE: bottom-left to top-right stripes can be obtained in the same spirit. Only minor complication: It is not based at the same address as the original array.

>>> def reverse_stripe(a):
...     a = np.asanyarray(a)
...     *sh, i, j = a.shape
...     assert i >= j
...     *st, k, m = a.strides
...     return np.lib.stride_tricks.as_strided(a[..., j-1:, :], (*sh, i-j+1, j), (*st, k, m-k))
...
>>> a = np.arange(24).reshape(6, 4)
>>> reverse_stripe(a)
array([[12,  9,  6,  3],
       [16, 13, 10,  7],
       [20, 17, 14, 11]])


Instruction: Expected 4D tensor as input, got 2D tensor instead

Input:
I'm trying to build a neural network using the pre-trained network VGG16 on Pytorch.

I understand that I need to adjust the classifier part of the network, so I have frozen the parameters to prevent backpropagation through them.

Code:

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import numpy as np
import time

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torch.autograd import Variable
from torchvision import datasets, transforms
import torchvision.models as models
from collections import OrderedDict

data_dir = 'flowers'
train_dir = data_dir + '/train'
valid_dir = data_dir + '/valid'
test_dir = data_dir + '/test'

train_transforms = transforms.Compose([transforms.Resize(224),
                                       transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                            std=[0.229, 0.224, 0.225])])

validn_transforms = transforms.Compose([transforms.Resize(224),
                                        transforms.CenterCrop(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize((0.485, 0.456, 0.406),
                                                            (0.229, 0.224, 0.225))])

test_transforms = transforms.Compose([ transforms.Resize(224),
                                       transforms.RandomResizedCrop(224),
                                       transforms.ToTensor(),
                                       transforms.Normalize((0.485, 0.456, 0.406),
                                                            (0.229, 0.224, 0.225))])

train_data = datasets.ImageFolder(train_dir,
                                transform=train_transforms)

validn_data = datasets.ImageFolder(valid_dir,
                                transform=validn_transforms)

test_data = datasets.ImageFolder(test_dir,
                                transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=True)
validnloader = torch.utils.data.DataLoader(validn_data, batch_size=32, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32, shuffle=True)

model = models.vgg16(pretrained=True)
model

for param in model.parameters():
    param.requires_grad = False

classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(3*224*224, 10000)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(10000, 5000)),
                          ('relu', nn.ReLU()),
                          ('fc3', nn.Linear(5000, 102)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

model.classifier = classifier

classifier

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)
model.cuda()

epochs = 1
steps = 0
training_loss = 0
print_every = 300
for e in range(epochs):
    model.train()
    for images, labels in iter(trainloader):
        steps == 1

        images.resize_(32,3*224*224)

        inputs = Variable(images.cuda())
        targets = Variable(labels.cuda())
        optimizer.zero_grad()

        output = model.forward(inputs)
        loss = criterion(output, targets)
        loss.backward()
        optimizer.step()

        training_loss += loss.data[0]

        if steps % print_every == 0:
            print("Epoch: {}/{}... ".format(e+1, epochs),
                  "Loss: {:.4f}".format(training_loss/print_every))

            running_loss = 0

  Traceback

ValueError                                Traceback (most recent call last)
<ipython-input-17-30552f4b46e8> in <module>()
     15         optimizer.zero_grad()
     16
---> 17         output = model.forward(inputs)
     18         loss = criterion(output, targets)
     19         loss.backward()

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.0-py3.6.egg/torchvision/models/vgg.py in forward(self, x)
     39
     40     def forward(self, x):
---> 41         x = self.features(x)
     42         x = x.view(x.size(0), -1)
     43         x = self.classifier(x)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     65     def forward(self, input):
     66         for module in self._modules.values():
---> 67             input = module(input)
     68         return input
     69

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    275     def forward(self, input):
    276         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 277                         self.padding, self.dilation, self.groups)
    278
    279

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in conv2d(input, weight, bias, stride, padding, dilation, groups)
     83     """
     84     if input is not None and input.dim() != 4:
---> 85         raise ValueError("Expected 4D tensor as input, got {}D tensor instead.".format(input.dim()))
     86
     87     f = _ConvNd(_pair(stride), _pair(padding), _pair(dilation), False,

ValueError: Expected 4D tensor as input, got 2D tensor instead.

Could it be because I am using Linear operation in my layer definitions?

Output:
There are two problems with your network -

You created your own classifier whose first layer accepts input of size (3*224*224), but this is not the output size of the features part of vgg16. Features output a tensor of size (25088)

You are resizing your input to be a tensor of shape (3*224*224) (for each batch) but the features part of vgg16 expects an input of (3, 224, 224). Your custom classifier comes after the features, so you need to prepare your input for features not for classifier.

Solution

To solve the first you need to change your definition of classifier to -

classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(25088, 10000)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(10000, 5000)),
                          ('relu', nn.ReLU()),
                          ('fc3', nn.Linear(5000, 102)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

To solve the second problem, change images.resize_(32,3*224*224) to images.resize_(32, 3, 224, 224).

P.S. - A word of advice - Your classifier's first layer output of 10000 units is very large. You should try to keep it around 4000 as done in the original classifier (Even better if you use the original weights for the first layer only, as those have proven to be good features as well over the time)


Instruction: What are transforms in PyTorch used for?

Input:
I am new with Pytorch and not very expert in CNN.
I have done a successful classifier with the tutorial that they provide Tutorial Pytorch, but I don't really understand what I am doing when loading the data.

They do some data augmentation and normalisation for training, but when I try to modify the parameters, the code does not work.

# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Am I extending my training dataset? I don't see the data augmentation.

Why if I modify the value of transforms.RandomResizedCrop(224) the data loading stop working?

Do I need to transform as well the test dataset?

I am a bit confused with this data transformation that they do.

Output:
transforms.Compose just clubs all the transforms provided to it. So, all the transforms in the transforms.Compose are applied to the input one by one.

Train transforms

transforms.RandomResizedCrop(224): This will extract a patch of size (224, 224) from your input image randomly. So, it might pick this path from topleft, bottomright or anywhere in between. So, you are doing data augmentation in this part. Also, changing this value won't play nice with the fully-connected layers in your model, so not advised to change this.
transforms.RandomHorizontalFlip(): Once we have our image of size (224, 224), we can choose to flip it. This is another part of data augmentation.
transforms.ToTensor(): This just converts your input image to PyTorch tensor.
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]): This is just input data scaling and these values (mean and std) must have been precomputed for your dataset. Changing these values is also not advised.

Validation transforms

transforms.Resize(256): First your input image is resized to be of size (256, 256)
transforms.CentreCrop(224): Crops the center part of the image of shape (224, 224)

Rest are the same as train

P.S.: You can read more about these transformations in the official docs


Instruction: Quickly find indices that have values larger than a threshold in Numpy/PyTorch

Input:
Task

Given a numpy or pytorch matrix, find the indices of cells that have values that are larger than a given threshold.

My implementation

#abs_cosine is the matrix
#sim_vec is the wanted

sim_vec = []
for m in range(abs_cosine.shape[0]):
    for n in range(abs_cosine.shape[1]):
        # exclude diagonal cells
        if m != n and abs_cosine[m][n] >= threshold:
            sim_vec.append((m, n))

Concerns

Speed. All other computations are built on Pytorch, using numpy is already a compromise, because it has moved computations from GPU to CPU. Pure python for loops will make the whole process even worse (for small data set already 5 times slower). I was wondering if we can move the whole computation to Numpy (or pytorch) without invoking any for loops?

An improvement I can think of (but got stuck...)

  bool_cosine = abs_cosine > threshold

which returns a boolean matrix of True and False. But I cannot find a way to quick retrieve the indices of the True cells.

Output:
The following is for PyTorch (fully on GPU)

# abs_cosine should be a Tensor of shape (m, m)
mask = torch.ones(abs_cosine.size()[0])
mask = 1 - mask.diag()
sim_vec = torch.nonzero((abs_cosine >= threshold)*mask)

# sim_vec is a tensor of shape (?, 2) where the first column is the row index and second is the column index

The following works in numpy

mask = 1 - np.diag(np.ones(abs_cosine.shape[0]))
sim_vec = np.nonzero((abs_cosine >= 0.2)*mask)
# sim_vec is a 2-array tuple where the first array is the row index and the second array is column index


Instruction: How do you load MNIST images into Pytorch DataLoader?

Input:
The pytorch tutorial for data loading and processing is quite specific to one example, could someone help me with what the function should look like for a more generic simple loading of images?

Tutorial: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html

My Data:

I have the MINST dataset as jpg's in the following folder structure. (I know I can just use the dataset class, but this is purely to see how to load simple images into pytorch without csv's or complex features).

The folder name is the label and the images are 28x28 png's in greyscale, no transformations required.

data
    train
        0
            3.png
            5.png
            13.png
            23.png
            ...
        1
            3.png
            10.png
            11.png
            ...
        2
            4.png
            13.png
            ...
        3
            8.png
            ...
        4
            ...
        5
            ...
        6
            ...
        7
            ...
        8
            ...
        9
            ...

Output:
Here's what I did for pytorch 0.4.1  (should still work in 1.3)

def load_dataset():
    data_path = 'data/train/'
    train_dataset = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )
    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=64,
        num_workers=0,
        shuffle=True
    )
    return train_loader

for batch_idx, (data, target) in enumerate(load_dataset()):
    #train network


Instruction: Lack of Sparse Solution with L1 Regularization in Pytorch

Input:
I am trying to implement L1 regularization onto the first layer of a simple neural network (1 hidden layer). I looked into some other posts on StackOverflow that apply l1 regularization using Pytorch to figure out how it should be done (references: Adding L1/L2 regularization in PyTorch?, In Pytorch, how to add L1 regularizer to activations?). No matter how high I increase lambda (the l1 regularization strength parameter) I do not get true zeros in the first weight matrix. Why would this be? (Code is below)

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class Network(nn.Module):
    def __init__(self,nf,nh,nc):
        super(Network,self).__init__()
        self.lin1=nn.Linear(nf,nh)
        self.lin2=nn.Linear(nh,nc)

    def forward(self,x):
        l1out=F.relu(self.lin1(x))
        out=F.softmax(self.lin2(l1out))
        return out, l1out

def l1loss(layer):
    return torch.norm(layer.weight.data, p=1)

nf=10
nc=2
nh=6
learningrate=0.02
lmbda=10.
batchsize=50

net=Network(nf,nh,nc)

crit=nn.MSELoss()
optimizer=torch.optim.Adagrad(net.parameters(),lr=learningrate)

xtr=torch.Tensor(xtr)
ytr=torch.Tensor(ytr)
#ytr=torch.LongTensor(ytr)
xte=torch.Tensor(xte)
yte=torch.LongTensor(yte)
#cyte=torch.Tensor(yte)

it=200
for epoch in range(it):
    per=torch.randperm(len(xtr))
    for i in range(0,len(xtr),batchsize):
        ind=per[i:i+batchsize]
        bx,by=xtr[ind],ytr[ind]
        optimizer.zero_grad()
        output, l1out=net(bx)
#        l1reg=l1loss(net.lin1)
        loss=crit(output,by)+lmbda*l1loss(net.lin1)
        loss.backward()
        optimizer.step()
    print('Epoch [%i/%i], Loss: %.4f' %(epoch+1,it, np.float32(loss.data.numpy())))

corr=0
tot=0
for x,y in list(zip(xte,yte)):
    output,_=net(x)
    _,pred=torch.max(output,-1)
    tot+=1 #y.size(0)
    corr+=(pred==y).sum()
print(corr)

Note: The data has 10 features (2 classes and 800 training samples) and only the first 2 are relevant (by design) so one would assume true zeros should be easy enough to learn.

Output:
Your usage of layer.weight.data removes the parameter (which is a PyTorch variable) from its automatic differentiation context, making it a constant when the optimiser takes the gradients. This results in zero gradients and that the L1 loss is not computed.

If you remove the .data, the norm is computed of the PyTorch variable and the gradients should be correct.

For more information on PyTorch's automatic differentiation mechanics, see this docs article or this tutorial.


Instruction: Load a single layer weights from a pretrained model

Input:
I want to specifically add the pretrained model parameters of some layers to my new network .
For Linear Layer i just did :

model_enc.linear_3d.weight = model_trained.linear_3d.weight
model_enc.linear_3d.bias = model_trained.linear_3d.bias

Will this suffice or are there any other parameters that I need to load or is there a easy way than this . My module is already trained and I just want to load params for few layers .

Thank you

Output:
Your solution should work and seems easy enough to me.

From the source code on https://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear you can see that the nn.Linear module has  the attributes in_features, out_features, weight1 and bias:

def __init__(self, in_features, out_features, bias=True):
    super(Linear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.weight = Parameter(torch.Tensor(out_features, in_features))
    if bias:
        self.bias = Parameter(torch.Tensor(out_features))
    else:
        self.register_parameter('bias', None)
    self.reset_parameters()

Therefore, as long as your in_features and out_features are identical you can just replace the weights and bias as you did.

Alternatively, you can replace the entire Linear module in in one network with the module of the other if you stored it as an attribute.


Instruction: Load a single image in a pretrained pytorch net

Input:
Total newbie here, I'm using this pytorch SegNet implementation with a '.pth' file containing weights from a 50 epochs training.
How can I load a single test image and see the net prediction?
I know this may sound like a stupid question but I'm stuck.
What I've got is:

from segnet import SegNet
import torch

model = SegNet(2)
model.load_state_dict(torch.load('./model_segnet_epoch50.pth'))

How do I "use" the net on a single test picture?

Output:
output = model(image)
.

Note that the image should be a Variable object and that the output will be as well.
If your image is, for example, a Numpy array, you can convert it like so:

var_image = Variable(torch.Tensor(image))


Instruction: AttributeError: 'torch.FloatTensor' object has no attribute 'item'

Input:
Here are the codes：

    from __future__ import print_function
    from itertools import count

    import torch
    import torch.autograd
    import torch.nn.functional as F

    POLY_DEGREE = 4
    W_target = torch.randn(POLY_DEGREE, 1) * 5
    b_target = torch.randn(1) * 5

    def make_features(x):
        x = x.unsqueeze(1)
        return torch.cat([x ** i for i in range(1, POLY_DEGREE+1)], 1)

    def f(x):
        return x.mm(W_target) + b_target.item()

This resulted in the following error message:

AttributeError: 'torch.FloatTensor' object has no attribute 'item'

How can I solve this please?

Output:
The function item() is new from PyTorch 0.4.0. When using earlier versions of PyTorch you will get this error.
So you can upgrade your PyTorch version your to solve this.

Edit:

I got through your example again. What do you want archive with item()?
In your case item() should just give you the (python) float value in the tensor.
Why do you want to use this? You can just leave out item().

So:

def f(x):
    return x.mm(W_target) + b_target

instead of:

def f(x):
    return x.mm(W_target) + b_target.item()

This should work for you, in PyTorch 0.4.0 there is not difference. It is also more efficient to leave out item().


Instruction: How can I make a FloatTensor with requires_grad=True from a numpy array using PyTorch 0.4.0?

Input:
Pytorch 0.4.0 introduced the merging on the Tensor and Variable classes.

Before this version, when I wanted to create a Variable with autograd from a numpy array I would do the following (where x is a numpy array):

x = Variable(torch.from_numpy(x).float(), requires_grad=True)

With PyTorch version 0.4.0, the migration guide shows how we can create Tensors with autograd enabled, examples show you can do things such as

x = torch.ones(3, 4, requires_grad=True)

and also set requires_grad to existing tensors

existing_tensor.requires_grad_()

I have tried the following three things to try and create a Tensor with requires_grad=True which give errors (where x is a numpy array):

The first is

x = FloatTensor(x, requires_grad=True)

which gives the error

TypeError: new() received an invalid combination of arguments - got
(numpy.ndarray, requires_grad=bool), but expected one of:
 * (torch.device device)
 * (tuple of ints size, torch.device device)
      didn't match because some of the keywords were incorrect:
requires_grad
 * (torch.Storage storage)
 * (Tensor other)
 * (object data, torch.device device)
      didn't match because some of the keywords were incorrect:
requires_grad

The second is to do

x = FloatTensor(x)
x.requires_grad()

And the third is

x = torch.from_numpy(x).single()
x.requires_grad()

Which both throw the following error on the second line:

TypeError: 'bool' object is not callable

These errors give me little hint at what I'm doing wrong, and since the latest version is so new its hard to find content online to help. How can I make a FloatTensor with requires_grad=True from a numpy array using PyTorch 0.4.0, preferably in a single line?

Output:
How can I make a FloatTensor with requires_grad=True from a numpy array using PyTorch 0.4.0, preferably in a single line?

If x is your numpy array this line should do the trick:

torch.tensor(x, requires_grad=True)

Here is a full example tested with PyTorch 0.4.0:

import numpy as np
import torch

x = np.array([1.3, 0.5, 1.9, 2.45])
print('np.array:', x)
t = torch.tensor(x, requires_grad=True)
print('tensor:', t)
print('requires_grad:', t.requires_grad)

This gives the following output:

np.array: [1.3  0.5  1.9  2.45]
tensor: tensor([ 1.3000,  0.5000,  1.9000,  2.4500], dtype=torch.float64)
requires_grad: True

Edit: dtype should be determined by the given dtype of your numpy array x.

I hope this helps.


Instruction: How to apply a custom function to specific columns in a matrix in PyTorch

Input:
I have a tensor of size [150, 182, 91], the first part is just the batch size while the matrix I am interested in is the 182x91 one.

I need to run a function on the 182x91 matrix for each of the 50 dimensions separately.

I need to get a diagonal matrix stripe of the 182x91 matrix, and the function I am using is the following one (based on my previous question: Getting diagonal matrix stripe automatically in numpy or pytorch):

 def stripe(a):

    i, j = a.size()
    assert (i >= j)

    out = torch.zeros((i - j + 1, j))
    for diag in range(0, i - j + 1):
        out[diag] = torch.diag(a, -diag)
    return out

The stripe function expects a matrix of size IxJ and can't deal with the 3rd dimension.

So when I run this:

some_matrix = x # <class 'torch.autograd.variable.Variable'> torch.Size([150, 182, 91])
get_diag = stripe(some_matrix)

I get this Error: ValueError: too many values to unpack (expected 2)

If I just try to skip the first dimension by doing x, i, j = a.size(),
I get this: RuntimeError: invalid argument 1: expected a matrix or a vector at

I am still on PyTorch 0.3.1. Any help is appreciated!

Output:
You can map the stripe function over the first dimension of your tensor using torch.unbind as

In [1]: import torch

In [2]: def strip(a):
   ...:     i, j = a.size()
   ...:     assert(i >= j)
   ...:     out = torch.zeros((i - j + 1, j))
   ...:     for diag in range(0, i - j + 1):
   ...:         out[diag] = torch.diag(a, -diag)
   ...:     return out
   ...:
   ...:

In [3]: a = torch.randn((182, 91)).cuda()

In [5]: output = strip(a)

In [6]: output.size()
Out[6]: torch.Size([92, 91])

In [7]: a = torch.randn((150, 182, 91))

In [8]: output = list(map(strip, torch.unbind(a, 0)))

In [9]: output = torch.stack(output, 0)

In [10]: output.size()
Out[10]: torch.Size([150, 92, 91])


Instruction: training a RNN in Pytorch

Input:
I want to have an RNN model and teach it to learn generating "ihello" from "hihell". I am new in Pytorch and following the instruction in a video to write the code.
I have written two python files named train.py and model.py.
this is model.py:

#----------------- model for teach rnn hihell to ihello
#-----------------  OUR MODEL ---------------------
import torch
import torch.nn as nn
from torch import autograd

class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.rnn=nn.RNN(input_size=input_size,hidden_size=hidden_size,batch_first=True)
    def forward(self,x,hidden):
        #Reshape input in (batch_size,sequence_length,input_size)
        x=x.view(batch_size,sequence_length,input_size)
        #Propagate input through RNN
        #Input:(batch,seq+len,input_size)
        out,hidden=self.rnn(x,hidden)
        out=out.view(-1,num_classes)
        return hidden,out
    def init_hidden(self):
        #Initialize hidden and cell states
        #(num_layers*num_directions,batch,hidden_size)
        return autograd.Variable(torch.zeros(num_layers,batch_size,hidden_size))

and this is train.py:

"""----------------------train for teach rnn to hihell to ihello--------------------------"""
#-----------------  DATA PREPARATION ---------------------
#Import
import torch
import torch.nn as nn
from torch import autograd
from model import Model
import sys

idx2char=['h','i','e','l','o']
#Teach hihell->ihello
x_data=[0,1,0,2,3,3]#hihell
y_data=[1,0,2,3,3,4]#ihello
one_hot_lookup=[[1,0,0,0,0],#0
                [0,1,0,0,0],#1
                [0,0,1,0,0],#2
                [0,0,0,1,0],#3
                [0,0,0,0,1]]#4
x_one_hot=[one_hot_lookup[x] for x in x_data]
inputs=autograd.Variable(torch.Tensor(x_one_hot))
labels=autograd.Variable(torch.LongTensor(y_data))
""" ----------- Parameters Initialization------------"""
num_classes = 5
input_size = 5  # one hot size
hidden_size = 5  # output from LSTM to directly predict onr-hot
batch_size = 1  # one sequence
sequence_length = 1  # let's do one by one
num_layers = 1  # one layer RNN
"""-----------------  LOSS AND TRAINING ---------------------"""
#Instantiate RNN model
model=Model()
#Set loss and optimizer function
#CrossEntropyLoss=LogSoftmax+NLLLOSS
criterion=torch.nn.CrossEntropyLoss()
optimizer=torch.optim.Adam(model.parameters(),lr=0.1)

"""----------------Train the model-------------------"""
for epoch in range(100):
    optimizer.zero_grad()
    loss=0
    hidden=model.init_hidden()
    sys.stdout.write("Predicted String:")
    for input,label in zip(inputs,labels):
        #print(input.size(),label.size())
        hidden,output=model(input,hidden)
        val,idx=output.max(1)
        sys.stdout.write(idx2char[idx.data[0]])
        loss+=criterion(output,label)
    print(",epoch:%d,loss:%1.3f"%(epoch+1,loss.data[0]))
    loss.backward()
    optimizer.step()

when I run train.py, I receive this error:

  self.rnn=nn.RNN(input_size=input_size,hidden_size=hidden_size,batch_first=True)
      NameError: name 'input_size' is not defined

I don't know why I receive this error because I have input_size=5 in the above lines of my code. could anybody help me? thanks.

Output:
The scope of the variables defined in train.py (num_classes, input_size, ...) is the train.py itself. They are only visible in this file. The model.py is oblivious to these.
I suggest including these arguments in the constructor:

class Model(nn.Module):
  def __init__(self, hidden_size, input_size):
    # same

and then call the Model as:

model = Model(hidden_size, input_size)

Similarly, for other variables that you defined in train.py (and want to use them in model.py) you have to pass them as arguments to either their respective functions, or to the constructor and store them as attributes.


Instruction: How does GPU utilization work in the context of neural network training?

Input:
I am using an AWS p3.2xlarge instance with the Deep Learning AMI (DLAMI). This instance has a single Tesla V100 (640 Tensor Cores and 5,120 CUDA Cores). When I run the PyTorch Seq2Seq Jupyter Notebook, I noticed that only 25% of the GPU is used. I monitor the GPU usage with the following command watch -n 1 nvidia-smi.

My question is, what determines GPU usage? Or, why is the GPU usage not 100%? The reason behind this question is related not only to inefficiency that may be a result of code but also cost ($3.06/hour). I am wondering if there is anything more that I can do to maximize the GPU usage.

Of course, this is a deep learning model that is being learned, and the training code sends one sample at a time through the network for learning. I am thinking that mini-batch learning may not be appropriate (e.g. sending a couple of samples through before backpropagating). I am also wondering if the network architecture (the number of layers, their parameters, their input tensor dimensions, etc.) constrains how GPU is being used. For example, if I add more layers or add more hidden nodes, should I expect GPU usage to go up?

Output:
The power of GPUs over CPUs is to run many operations at the same time.  However archiving this high level of parallelization is not always easy. Frameworks like Tensorflow or PyTorch do its best to optimise everything for GPU and parallelisation, but this is not possible for every case.

Computations in LSTMs and RNNs in general can be only parallelized to a very limited degree. The problem lies in their sequential structure, LSTMs and RNNs process only one input at a time, and they need to process everything in chronological order (to compute n+1 you always need to compute n before) - otherwise it wouldn't make sense.

So the natural way of processing data in RNNs is completely the opposite of parallelization, using mini-batching does help a lot, but does not solve the fundamental problem of LSTMs.

If you wan't a high amount of parallelization you need to use architectures like the "Transformer" proposed in the paper "Attention is all you need" by Google.

Summary

The degree of parallelization resp. the GPU acceleration of your model depends to a large extent on the architecture of the model itself.
With some architectures like RNNs parallelization is only possible to a limited degree.

Edit:

For example, if I add more layers or add more hidden nodes, should I expect GPU usage to go up?

When increasing the number of units within you should expect the GPU usage going up, matrix operations like passing an input to a hidden layer are can be well parallelized.

Adding layers is different, there you have the same problem what causes RNNs to be slow on GPU. To compute the next layer you need to have already the result of the previous layer. So you need to compute one layer after another, it's not possible to compute all at the same time.

This is the theory - In practice you might see some minor differences in GPU usage, depending on the actual implementation of the framework.


Instruction: Does a clean and extendable LSTM implementation exists in PyTorch?

Input:
I would like to create an LSTM class by myself, however, I don't want to rewrite the classic LSTM functions from scratch again.

Digging in the code of PyTorch, I only find a dirty implementation involving at least 3-4 classes with inheritance:

https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/modules/rnn.py#L323
https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/modules/rnn.py#L12
https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/_functions/rnn.py#L297

Does a clean PyTorch implementation of an LSTM exist somewhere? Any links would help.

For example, I know that clean implementations of a LSTM exists in TensorFlow, but I would need to derive a PyTorch one.

For a clear example, what I'm searching for is an implementation as clean as this, but in PyTorch:

Output:
The best implementation I found is here
https://github.com/pytorch/benchmark/blob/master/rnns/benchmarks/lstm_variants/lstm.py

It even implements four different variants of recurrent dropout, which is very useful!
If you take the dropout parts away you get

import math
import torch as th
import torch.nn as nn

class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        super(LSTM, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias
        self.i2h = nn.Linear(input_size, 4 * hidden_size, bias=bias)
        self.h2h = nn.Linear(hidden_size, 4 * hidden_size, bias=bias)
        self.reset_parameters()

    def reset_parameters(self):
        std = 1.0 / math.sqrt(self.hidden_size)
        for w in self.parameters():
            w.data.uniform_(-std, std)

    def forward(self, x, hidden):
        h, c = hidden
        h = h.view(h.size(1), -1)
        c = c.view(c.size(1), -1)
        x = x.view(x.size(1), -1)

        # Linear mappings
        preact = self.i2h(x) + self.h2h(h)

        # activations
        gates = preact[:, :3 * self.hidden_size].sigmoid()
        g_t = preact[:, 3 * self.hidden_size:].tanh()
        i_t = gates[:, :self.hidden_size]
        f_t = gates[:, self.hidden_size:2 * self.hidden_size]
        o_t = gates[:, -self.hidden_size:]

        c_t = th.mul(c, f_t) + th.mul(i_t, g_t)

        h_t = th.mul(o_t, c_t.tanh())

        h_t = h_t.view(1, h_t.size(0), -1)
        c_t = c_t.view(1, c_t.size(0), -1)
        return h_t, (h_t, c_t)

PS: The repository contains many more variants of LSTM and other RNNs:
https://github.com/pytorch/benchmark/tree/master/rnns/benchmarks.
Check it out, maybe the extension you had in mind is already there!

EDIT:
As mentioned in the comments, you can wrap the LSTM cell above to process sequential output:

import math
import torch as th
import torch.nn as nn

class LSTMCell(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        # As before

    def reset_parameters(self):
        # As before

    def forward(self, x, hidden):

        if hidden is None:
            hidden = self._init_hidden(x)

        # Rest as before

    @staticmethod
    def _init_hidden(input_):
        h = th.zeros_like(input_.view(1, input_.size(1), -1))
        c = th.zeros_like(input_.view(1, input_.size(1), -1))
        return h, c

class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        super().__init__()
        self.lstm_cell = LSTMCell(input_size, hidden_size, bias)

    def forward(self, input_, hidden=None):
        # input_ is of dimensionalty (1, time, input_size, ...)

        outputs = []
        for x in torch.unbind(input_, dim=1):
            hidden = self.lstm_cell(x, hidden)
            outputs.append(hidden[0].clone())

        return torch.stack(outputs, dim=1)

I havn't tested the code since I'm working with a convLSTM implementation. Please let me know if something is wrong.

UPDATE: Fixed links.


Instruction: Adapting pytorch softmax function

Input:
I am currently looking into the softmax function and I would like to adapt the orignally implemented for ome small tests.

I have been to the docs but there wasn't that much of usefull information about the function. This is the pytorch python implementation:

def __init__(self, dim=None):
    super(Softmax, self).__init__()
    self.dim = dim

def __setstate__(self, state):
    self.__dict__.update(state)
    if not hasattr(self, 'dim'):
        self.dim = None

def forward(self, input):
    return F.softmax(input, self.dim, _stacklevel=5)

Where can I find the F.softmax impementation?

One off the things I want to try for instance is the soft-margin softmax described here: Soft-Margin Softmax for Deep Classification

Where would be the best place to start?
Thanks in advance!

Output:
Softmax Implementation in PyTorch and Numpy
A Softmax function is defined as follows:

A direct implementation of the above formula is as follows:
def softmax(x):
    return np.exp(x) / np.exp(x).sum(axis=0)

Above implementation can run into arithmetic overflow because of np.exp(x).
To avoid the overflow, we can divide the numerator and denominator in the softmax equation with a constant C. Then the softmax function becomes following:

The above approach is implemented in PyTorch and we take log(C) as -max(x). Below is the PyTorch implementation:
def softmax_torch(x): # Assuming x has atleast 2 dimensions
    maxes = torch.max(x, 1, keepdim=True)[0]
    x_exp = torch.exp(x-maxes)
    x_exp_sum = torch.sum(x_exp, 1, keepdim=True)
    probs = x_exp/x_exp_sum
    return probs

A corresponding Numpy equivalent is as follows:
def softmax_np(x):
    maxes = np.max(x, axis=1, keepdims=True)[0]
    x_exp = np.exp(x-maxes)
    x_exp_sum = np.sum(x_exp, 1, keepdims=True)
    probs = x_exp/x_exp_sum
    return probs

We can compare the results with PyTorch implementation - torch.nn.functional.softmax using below snippet:
import torch
import numpy as np
if __name__ == "__main__":
    x = torch.randn(1, 3, 5, 10)
    std_pytorch_softmax = torch.nn.functional.softmax(x)
    pytorch_impl = softmax_torch(x)
    numpy_impl = softmax_np(x.detach().cpu().numpy())
    print("Shapes: x --> {}, std --> {}, pytorch impl --> {}, numpy impl --> {}".format(x.shape, std_pytorch_softmax.shape, pytorch_impl.shape, numpy_impl.shape))
    print("Std and torch implementation are same?", torch.allclose(std_pytorch_softmax, pytorch_impl))
    print("Std and numpy implementation are same?", torch.allclose(std_pytorch_softmax, torch.from_numpy(numpy_impl)))

References:

Softmax discussion
Discussion on Softmax implementation at PyTorch forumn
An SO thread on implementation of Softmax in Python


Instruction: Implementing word dropout in pytorch

Input:
I want to add word dropout to my network so that I can have sufficient training examples for training the embedding of the "unk" token. As far as I'm aware, this is standard practice. Let's assume the index of the unk token is 0, and the index for padding is 1 (we can switch them if that's more convenient).

This is a simple CNN network which implements word dropout the way I would have expected it to work:

class Classifier(nn.Module):
    def __init__(self, params):
        super(Classifier, self).__init__()
        self.params = params
        self.word_dropout = nn.Dropout(params["word_dropout"])
        self.pad = torch.nn.ConstantPad1d(max(params["window_sizes"])-1, 1)
        self.embedding = nn.Embedding(params["vocab_size"], params["word_dim"], padding_idx=1)
        self.convs = nn.ModuleList([nn.Conv1d(1, params["feature_num"], params["word_dim"] * window_size, stride=params["word_dim"], bias=False) for window_size in params["window_sizes"]])
        self.dropout = nn.Dropout(params["dropout"])
        self.fc = nn.Linear(params["feature_num"] * len(params["window_sizes"]), params["num_classes"])

    def forward(self, x, l):
        x = self.word_dropout(x)
        x = self.pad(x)
        embedded_x = self.embedding(x)
        embedded_x = embedded_x.view(-1, 1, x.size()[1] * self.params["word_dim"]) # [batch_size, 1, seq_len * word_dim]
        features = [F.relu(conv(embedded_x)) for conv in self.convs]
        pooled = [F.max_pool1d(feat, feat.size()[2]).view(-1, params["feature_num"]) for feat in features]
        pooled = torch.cat(pooled, 1)
        pooled = self.dropout(pooled)
        logit = self.fc(pooled)
        return logit

Don't mind the padding - pytorch doesn't have an easy way of using non zero padding in CNNs, much less trainable non-zero padding, so I'm doing it manually. Dropout also doesn't allow me to use non zero dropout, and I want to separate the padding token from the unk token. I'm keeping it in my example because it's the reason for this question's existence.

This doesn't work because dropout wants Float Tensors so that it can scale them properly, while my input is Long Tensors that don't need to be scaled.

Is there an easy way of doing this in pytorch? I essentially want to use LongTensor-friendly dropout (bonus: better if it will let me specify a dropout constant that isn't 0, so that I could use zero padding).

Output:
Actually I would do it outside of your model, before converting your input into a LongTensor.

This would look like this:

import random

def add_unk(input_token_id, p):
    #random.random() gives you a value between 0 and 1
    #to avoid switching your padding to 0 we add 'input_token_id > 1'
    if random.random() < p and input_token_id > 1:
        return 0
    else:
        return input_token_id

#than you have your input token_id
#for this example I take just a random number, lets say 127
input_token_id = 127

#let p be your probability for UNK
p = 0.01

your_input_tensor = torch.LongTensor([add_unk(input_token_id, p)])

Edit:

So there are two options which come to my mind which are actually GPU-friendly. In general both solutions should be much more efficient.

Option one - Doing computation directly in forward():

If you're not using torch.utils and don't have plans using it later this is probably the way to go.

Instead of doing the computation before we just do it in the forward() method of main PyTorch class. However I see no (simple) way doing this in torch 0.3.1., so you would need to upgrade to version 0.4.0:

So imagine x is your input vector:

>>> x = torch.tensor(range(10))
>>> x
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9])

probs is a vector containing uniform probabilities for dropout so we can check later agains our probability for dropout:

>>> probs = torch.empty(10).uniform_(0, 1)
>>> probs
tensor([ 0.9793,  0.1742,  0.0904,  0.8735,  0.4774,  0.2329,  0.0074,
         0.5398,  0.4681,  0.5314])

Now we apply the dropout probabilities probs on our input x:

>>> torch.where(probs > 0.2, x, torch.zeros(10, dtype=torch.int64))
tensor([ 0,  0,  0,  3,  4,  5,  0,  7,  8,  9])

Note: To see some effect I chose a dropout probability of 0.2 here. I reality you probably want it to be smaller.

You can pick for this any token / id you like, here is an example with 42 as unknown token id:

>>> unk_token = 42
>>> torch.where(probs > 0.2, x, torch.empty(10, dtype=torch.int64).fill_(unk_token))
tensor([  0,  42,  42,   3,   4,   5,  42,   7,   8,   9])

torch.where comes with PyTorch 0.4.0:
https://pytorch.org/docs/master/torch.html#torch.where

I don't know about the shapes of your network, but your forward() should look something like this then (when using mini-batching you need to flatten the input before applying dropout):

def forward_train(self, x, l):
    # probabilities
    probs = torch.empty(x.size(0)).uniform_(0, 1)
    # applying word dropout
    x = torch.where(probs > 0.02, x, torch.zeros(x.size(0), dtype=torch.int64))

    # continue like before ...
    x = self.pad(x)
    embedded_x = self.embedding(x)
    embedded_x = embedded_x.view(-1, 1, x.size()[1] * self.params["word_dim"]) # [batch_size, 1, seq_len * word_dim]
    features = [F.relu(conv(embedded_x)) for conv in self.convs]
    pooled = [F.max_pool1d(feat, feat.size()[2]).view(-1, params["feature_num"]) for feat in features]
    pooled = torch.cat(pooled, 1)
    pooled = self.dropout(pooled)
    logit = self.fc(pooled)
    return logit

Note: I named the function forward_train() so you should use another forward() without dropout for evaluation / predicting. But you could also use some if conditions with train().

Option two: using torch.utils.data.Dataset:

If you're using Dataset provided by torch.utils it is very easy to do this kind of pre-processing efficiently. Dataset uses strong multi-processing acceleration by default so the the code sample above just has to be executed in the __getitem__ method of your Dataset class.

This could look like this:

def __getitem__(self, index):
    'Generates one sample of data'
    # Select sample
    ID = self.input_tokens[index]

    # Load data and get label
    # using add ink_unk function from code above
    X = torch.LongTensor(add_unk(ID, p=0.01))
    y = self.targets[index]

    return X, y

This is a bit out of context and doesn't look very elegant but I think you get the idea. According to this blog post of Shervine Amidi at Stanford it should be no problem to do more complex pre-processing steps in this function:

  Since our code [Dataset is meant] is designed to be multicore-friendly, note that you
  can do more complex operations instead (e.g. computations from source
  files) without worrying that data generation becomes a bottleneck in
  the training process.

The linked blog post - "A detailed example of how to generate your data in parallel with PyTorch" - provides also a good guide for implementing the data generation with Dataset and DataLoader.

I guess you'll prefer option one - only two lines and it should be very efficient. :)

Good luck!


Instruction: AttributeError: 'module' object has no attribute 'float32'

Input:
I am trying to use OpenNMT-py with python 2.7. OpenNMT-py requires torchtext, so I installed it but now when I am running my program, I am getting the following error message.

Traceback (most recent call last):
  File "examples/StackPointerParser.py", line 23, in <module>
    from neuronlp2.io import get_logger, conllx_stacked_data
  File "./neuronlp2/__init__.py", line 7, in <module>
    from . import models
  File "./neuronlp2/models/__init__.py", line 4, in <module>
    from .parsing import *
  File "./neuronlp2/models/parsing.py", line 15, in <module>
    from onmt.modules import LayerNorm, Transformer
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/onmt/__init__.py", line 1, in <module>
    import onmt.io
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/onmt/io/__init__.py", line 1, in <module>
    from onmt.io.IO import collect_feature_vocabs, make_features, \
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/onmt/io/IO.py", line 8, in <module>
    import torchtext.data
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/torchtext/__init__.py", line 1, in <module>
    from . import data
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/torchtext/data/__init__.py", line 4, in <module>
    from .field import RawField, Field, ReversibleField, SubwordField, NestedField, LabelField
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/torchtext/data/field.py", line 61, in <module>
    class Field(RawField):
  File "/home/wasiahmad/software/anaconda2/lib/python2.7/site-packages/torchtext/data/field.py", line 115, in Field
    torch.float32: float,
AttributeError: 'module' object has no attribute 'float32'

I tried to look for a solution to resolve this issue but couldn't find any. Any help would be appreciated.

Output:
This is more a guess, as you have not given information about your version. But it seems to me that your torchtext version is not compatible with your PyTorch version.

Probably when you installed torchtext you got the newer version already made for PyTorch 0.4.0. But your PyTorch version installed is still older than 0.4.0 (version 0.3.1 or so).

If that is the case you have two options. Downgrading torchtext to a version compatible to yours (probably the version before). Or upgrading PyTorch to version 0.4.0.

I hope this helps.


Instruction: What is the state of the art way of doing regression with probability in pytorch

Input:
All regression examples I find are examples where you predict a real number and unlike with classification you dont the the confidence the model had when predicting that number. I have done in reinforcement learning another way the output is instead the mean and std and then you sample from that distribution. Then you know how confident the model is at predicting every value. Now I cant find how to do this using supervised learning in pytorch. The problem is that I dont understand how to perform sample from the distribution the get the actual value while training or what sort of loss function I should use, not sure how for example MSE or L1Smooth would work.

Is there any example ot there where this is done in pytorch in a robust and state of the art way?

Output:
The key point is that you do not need to sample from the NN-produced distribution. All you need is to optimize the likelihood of the target value under the NN distribution.

There is an example in the official PyTorch example on VAE (https://github.com/pytorch/examples/tree/master/vae), though for multidimensional Bernoulli distribution.

Since PyTorch 0.4, you can use torch.distributions: instantiate distribution  distro with outputs of your NN and then optimize  -distro.log_prob(target).

EDIT: As requested in a comment, a complete example of using the torch.distributions module.

First, we create a heteroscedastic dataset:

import numpy as np
import torch
X = np.random.uniform(size=300)
Y = X + 0.25*X*np.random.normal(size=X.shape[0])

We build a trivial model, which is perfectly able to match the generative process of our data:

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.mean_coeff = torch.nn.Parameter(torch.Tensor([0]))
        self.var_coeff = torch.nn.Parameter(torch.Tensor([1]))

    def forward(self, x):
        return torch.distributions.Normal(self.mean_coeff * x, self.var_coeff * x)

mdl = Model()
optim = torch.optim.SGD(mdl.parameters(), lr=1e-3)

Initialization of the model makes it always produce a standard normal, which is a poor fit for our data, so we train (note it is a very stupid batch training, but demonstrates that you can output a set of distributions for your batch at once):

for _ in range(2000): # epochs
    dist = mdl(torch.from_numpy(X).float())
    obj = -dist.log_prob(torch.from_numpy(Y).float()).mean()
    optim.zero_grad()
    obj.backward()
    optim.step()

Eventually, the learned parameters should match the values we used to construct the Y.

print(mdl.mean_coeff, mdl.var_coeff)
# tensor(1.0150) tensor(0.2597)


Instruction: torch.nn.embedding has run time error

Input:
I want to use torch.nn.Embedding. I have followed the codes in the documentation of embedding command.
here is the code:

# an Embedding module containing 10 tensors of size 3
embedding = nn.Embedding(10, 3)
# a batch of 2 samples of 4 indices each
input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
embedding(input)

The documentation says that you will receive this output:

tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])

but I don't receive this output. instead I receive this error:

Traceback (most recent call last):
  File "/home/mahsa/PycharmProjects/PyTorch_env_project/PyTorchZeroToAll-master/temporary.py", line 12, in <module>
    embedding(input)
  File "/home/mahsa/anaconda3/envs/pytorch_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mahsa/anaconda3/envs/pytorch_env/lib/python3.5/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    self.scale_grad_by_freq, self.sparse
RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition

Can anybody guide me about this error? and about the work of torch.nn.Embedding?

Output:
if we change this line:

input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])

with this:

input = autograd.Variable(torch.LongTensor([[1,2,4,5],[4,3,2,9]]))

the problem is solved!


Instruction: How can I downgrade the version pytorch from 0.4 to 0.31 with anaconda?

Input:
I'm using anaconda now and due to some code error I need to downgrade the pytorch from version 0.4 to version 0.31.
However, as I use anaconda as my python package management tool, I used the instruction below to downgrade it and encountered the following error:
 conda install pytorch=0.31 cuda80 -c soumith

PackagesNotFoundError: The following packages are not available from current channels:

pytorch=0.31

And I also have tried some normal method to revert the package into previous verion in anaconda, but they failed either.
Could anyone tell me how to downgrade it? Thank you so much!!

Output: conda install pytorch=0.3.1.0 cuda80 -c soumith


Instruction: Loading FITS images with PyTorch

Input:
I'm trying to create a CNN using PyTorch but my images need importing from the FITS format rather than conventional .png or .jpeg etc.

Is there a way to accomplish this easily using torch.utils.data.DataLoader or is there a place in the source code where I can put in a clause which will handle FITS files while loading in?

I have looked in the documentation and the most relevant thing I've found is the ToPILImage transformer which converts a tensor or ndarray into a PIL Image.

Currently I'm using an image loading routine as follows:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision

batch_size = 4

transform = transforms.Compose(
                   [transforms.Resize((32,32)),
                    transforms.ToTensor(),
                    ])

trainset = dset.ImageFolder(root="Documents/Image_data",transform=transform)
train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,shuffle=True)

Astropy: http://www.astropy.org/

Pytorch: https://pytorch.org/

torch.utils: https://pytorch.org/docs/master/data.html

UPDATE: Perhaps using torchvision.datasets.DatasetFolder instead of DataLoader, an inserting in my own FITS handler would work?

When trying to use this class I get the following error:

AttributeError: module 'torchvision.datasets' has no attribute 'DatasetFolder'

Is DatasetFolder actually supported by torchvision at this point in time?

Output:
From reading some combination of the docs and the code, I don't think you necessarily want to be using ImageFolder since it doesn't know anything about FITS.

Instead you should try using the more generic DataSetFolder class (which in fact is the parent class of ImageFolder).  You would pass it a list of extensions it should handle (i.e. ['.fits'] and a "loader" function that takes a FITS file and, it seems, should return a PIL.Image.

You could even make your own subclass following the example of ImageFolder.  E.g.

class FitsFolder(DatasetFolder):

    EXTENSIONS = ['.fits']

    def __init__(self, root, transform=None, target_transform=None,
                 loader=None):
        if loader is None:
            loader = self.__fits_loader

        super(FitsFolder, self).__init__(root, loader, self.EXTENSIONS,
                                         transform=transform,
                                         target_transform=target_transform)

    @staticmethod
    def __fits_loader(filename):
        data = fits.getdata(filename)
        return Image.fromarray(data)

The exact details of __fits_loader may depend on the details of your FITS files.  This basic example just uses the high-level fits.getdata() function which returns the first image array in the FITS file (some FITS files may have many extensions with many images, or have tables etc.).  So that part would be up to you.


Instruction: Pytorch Forward Pass Changes Each Time?

Input:
I am learning pytorch and running a toy regression problem. I am baffled by the fact that it appears that each time I run a tensor through a model, the prediction changes. Clearly this is cant be the case but what am I missing?

Pytorch version: 0.4.0

I am running here without GPU to eliminate that potential issue.

Code:

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F
from torch.nn.init import xavier_uniform_, xavier_normal_,uniform_
from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error

cuda=False #set to true uses GPU

#load boston data from scikit
boston = load_boston()
x=boston.data
y=boston.target
y=y.reshape(y.shape[0],1)

#change to tensors
x = torch.from_numpy(x)
y = torch.from_numpy(y)

#create dataset and use data loader
training_samples = utils_data.TensorDataset(x, y)
data_loader = utils_data.DataLoader(training_samples, batch_size=64)

#simple model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        #all the layers
        self.fc1   = nn.Linear(x.shape[1], 20)
        xavier_uniform_(self.fc1.weight.data) #this is how you can change the weight init
        self.drop = nn.Dropout(p=0.5)
        self.fc2   = nn.Linear(20, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x=  self.drop(x)
        x = self.fc2(x)
        return x

net=Net()

if cuda:
    net.cuda()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss(size_average=True)

# run the main training loop
epochs =50
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = data.float(), target.float()
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch

        cum_loss += output.item() #accumulate the loss

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()
    hold_loss.append(cum_loss/len(training_samples))

#training loss
plt.plot(np.array(hold_loss))

This part, if re-ran will return different predictions each time, the actuals dont change so the order of the data is not changing!

#score the training set
for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = data.float(), target.float()
        if batch_idx ==0:
           hold_pred=net(tr_x).data.numpy()
           hold_actual=tr_y.data.numpy().reshape(tr_y.data.numpy().shape[0],1)
        else:
            hold_pred =np.row_stack([hold_pred,net(tr_x).data.numpy()])
hold_actual=np.row_stack([hold_actual,tr_y.data.numpy().reshape(tr_y.data.numpy().shape[0],1)])

#view the first few predictions
print(hold_pred[0:10])
print(hold_actual[0:10])

Output:
Your network has a Dropout layer, which has for purpose to randomly sample (with a probability p=0.5 here) the data it receives during training (net.train() set before inference). See the doc for more information (usage, purpose).

This layer can be short-circuited during testing (net.eval() set before inference).


Instruction: In pytorch, how to use the weight parameter in F.cross_entropy()?

Input:
I'm trying to write some code like below:

x = Variable(torch.Tensor([[1.0,2.0,3.0]]))
y = Variable(torch.LongTensor([1]))
w = torch.Tensor([1.0,1.0,1.0])
F.cross_entropy(x,y,w)
w = torch.Tensor([1.0,10.0,1.0])
F.cross_entropy(x,y,w)

However, the output of cross entropy loss is always 1.4076 whatever w is. What is behind the weight parameter for F.cross_entropy()? How to use it correctly?
I'm using pytorch 0.3

Output:
The weight parameter is used to compute a weighted result for all inputs based on their target class. If you have only one input or all inputs of the same target class, weight won't impact the loss.

See the difference however with 2 inputs of different target classes:

import torch
import torch.nn.functional as F
from torch.autograd import Variable

x = Variable(torch.Tensor([[1.0,2.0,3.0], [1.0,2.0,3.0]]))
y = Variable(torch.LongTensor([1, 2]))
w = torch.Tensor([1.0,1.0,1.0])
res = F.cross_entropy(x,y,w)
# 0.9076
w = torch.Tensor([1.0,10.0,1.0])
res = F.cross_entropy(x,y,w)
# 1.3167


Instruction: How to train a Pytorch net

Input:
I'm using this Pytorch implementation of Segnet with pretrained values I found for object segmentation, and it works fine.
Now I want to resume the training from the values I have, using a new dataset with similar images.
How can I do that?

I guess I have to use the "train.py" file found in the repository, but I don't know what to write in order to replace the "fill the batch" comment.
Here is that portion of the code:

def train(epoch):
    model.train()

    # update learning rate
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

    # define a weighted loss (0 weight for 0 label)
    weights_list = [0]+[1 for i in range(17)]
    weights = np.asarray(weights_list)
    weigthtorch = torch.Tensor(weights_list)
    if(USE_CUDA):
        loss = nn.CrossEntropyLoss(weight=weigthtorch).cuda()
    else:
        loss = nn.CrossEntropyLoss(weight=weigthtorch)

    total_loss = 0

    # iteration over the batches
    batches = []
    for batch_idx,batch_files in enumerate(tqdm(batches)):

        # containers
        batch = np.zeros((args.batch_size,input_nbr, imsize, imsize), dtype=float)
        batch_labels = np.zeros((args.batch_size,imsize, imsize), dtype=int)

        # fill the batch
        # ...
        # What should I write here?

        batch_th = Variable(torch.Tensor(batch))
        target_th = Variable(torch.LongTensor(batch_labels))

        if USE_CUDA:
            batch_th =batch_th.cuda()
            target_th = target_th.cuda()

        # initilize gradients
        optimizer.zero_grad()

        # predictions
        output = model(batch_th)

        # Loss
        output = output.view(output.size(0),output.size(1), -1)
        output = torch.transpose(output,1,2).contiguous()
        output = output.view(-1,output.size(2))
        target = target.view(-1)

        l_ = loss(output.cuda(), target)
        total_loss += l_.cpu().data.numpy()
        l_.cuda()
        l_.backward()
        optimizer.step()

return total_loss/len(files)

Output:
If I had to guess he probablly made some Dataloader feeder that extended the Pytorch Dataloader class. See
https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

Near the bottom of the page you can see an example in which they loop over their data loader

for i_batch, sample_batched in enumerate(dataloader):

What this would like like for images for example is:

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=False, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batchSize, shuffle=True, num_workers=2)

for batch_idx, (inputs, targets) in enumerate(trainloader):
    # Using the pytorch data loader the inputs and targets are given
    # automatically
    inputs, targets = inputs.cuda(), targets.cuda()
    optimizer.zero_grad()
    inputs, targets = Variable(inputs), Variable(targets)

How exactly the author loads his files I don't know. You could follow the procedure from: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html  to make your own Dataloader though.


Instruction: How do I convert a Pandas dataframe to a PyTorch tensor?

Input:
How do I train a simple neural network with PyTorch on a pandas dataframe df?
The column df["Target"] is the target (e.g. labels) of the network. This doesn't work:
import pandas as pd
import torch.utils.data as data_utils

target = pd.DataFrame(df['Target'])
train = data_utils.TensorDataset(df, target)
train_loader = data_utils.DataLoader(train, batch_size=10, shuffle=True)

Output:
I'm referring to the question in the title as you haven't really specified anything else in the text, so just converting the DataFrame into a PyTorch tensor.

Without information about your data, I'm just taking float values as example targets here.

Convert Pandas dataframe to PyTorch tensor?

import pandas as pd
import torch
import random

# creating dummy targets (float values)
targets_data = [random.random() for i in range(10)]

# creating DataFrame from targets_data
targets_df = pd.DataFrame(data=targets_data)
targets_df.columns = ['targets']

# creating tensor from targets_df
torch_tensor = torch.tensor(targets_df['targets'].values)

# printing out result
print(torch_tensor)

Output:

tensor([ 0.5827,  0.5881,  0.1543,  0.6815,  0.9400,  0.8683,  0.4289,
         0.5940,  0.6438,  0.7514], dtype=torch.float64)

Tested with Pytorch 0.4.0.

I hope this helps, if you have any further questions - just ask. :)


Instruction: Change values inside a Pytorch 3D tensor

Input:
I have a 224x224 binary image in a tensor (1, 224, 224), with 0-pixels representing background a 1-pixels representing foreground.
I want to reshape it in a tensor (2, 224, 224), such as the first "layer" gt[0] has 1-pixels where there were 0-pixels in the original image and viceversa. This way one layer should show 1s where there is background and the other one will have 1s on the foreground (basically I need to have two complementary binary images in this tensor).

This is my code:

# gt is a tensor (1, 224, 224)
gt = gt.expand((2, 224, 224))
backgr = gt[0]
foregr = gt[1]

backgr[backgr == 0] = 2 # swap all 0s in 1s and viceversa
backgr[backgr == 1] = 0
backgr[backgr == 2] = 1

gt[0] = backgr

print(gt[0])
print(gt[1])

The problem is both layers are modified with this code and I can't figure out how to keep one of the two constant and change only gt[0].

Output:
Found a solution!

gt = gt.repeat(2, 1, 1)


Instruction: PyTorch LSTM - using word embeddings instead of nn.Embedding()

Input:
Is the nn.Embedding() essential for learning for an LSTM?

I am using an LSTM in PyTorch to predict NER - example of a similar task is here - https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html

Code wise, I am using code almost identical to the code in the tutorial above.

The only detail is -
I am using word2Vec instead of nn.Embedding().

So I remove the nn.Embedding() layer and provide the forward function the features from the word2Vec directly. The RNN does not learn.

Hence, Is the nn.Embedding() essential for learning for an LSTM?

Output:
nn.Embedding provides an embedding layer for you.

This means that the layer takes your word token ids and converts these to word vectors.

You can learn the weights for your nn.Embedding layer during the training process, or you can alternatively load pre-trained embedding weights.

When you want to use a pre-trained word2vec (embedding) model, you just load the pre-trained weights into the nn.Embedding layer.

You can take a look here on how to load a word2vec embedding layer using gensim library.

I hope this helps.


Instruction: How to perform finetuning on a Pytorch net

Input:
I'm using this implementation of SegNet in Pytorch, and I want to finetune it.
I've read online and I've found this method (basically freezing all layers except the last one in your net). My problem is that SegNet has more than 100 layers and I'm looking for a simpler way to do it, rather than writing 100 lines of code.

Do you think this could work? Or is this utter nonsense?

import torch.optim as optim

model = SegNet()
for name, param in model.named_modules():
    if name != 'conv11d':    # the last layer should remain active
        param.requires_grad = False

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

def train():
    ...

How can I check if this is working as intended?

Output:
This process is called finetuning and setting requires_grad to False is a good way to do this. From the pytorch docs:

  Every Tensor has a flag: requires_grad that allows for fine grained exclusion of subgraphs from gradient computation and can increase efficiency.

...

  If there’s a single input to an operation that requires gradient, its output will also require gradient. Conversely, only if all inputs don’t require gradient, the output also won’t require it. Backward computation is never performed in the subgraphs, where all Tensors didn’t require gradients.

See this pytorch tutorial for a relevant example.

One simple way of checking to see this is working is looking at the initial error rates. Assuming the task is similar to the task the net was originally trained on, they should be much lower than for a randomly initialized net.


Instruction: How to use PyTorch to print out the prediction accuracy of every class?

Input:
I am trying to use PyTorch to print out the prediction accuracy of every class based on the official tutorial link

But things seem to go wrong. My code intends to do this work is as following:

    for epoch in range(num_epochs):

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
       ... (this is given by the tutorial)

    (my code)

    class_correct = list(0. for i in range(3))
    class_total = list(0. for i in range(3))

    for data in dataloaders['val']:
        images, labels = data
        outputs = model(inputs)
        _, predicted = torch.max(outputs.data, 1)
        c = (predicted == labels.data).squeeze()

        for i in range(4):
            label = labels.data[i]
            class_correct[label] += c[i]
            class_total[label] += 1

    for i in range(3):
        print('Accuracy of {} : {} / {} = {:.4f} %'.format(i,
class_correct[i], class_total[i], 100 * class_correct[i].item() /
class_total[i]))

    print(file = f)
    print()

For example, the output of epoch 1/1 is :

I think the following equation should be satisfied:

  running_corrects := 2 + 2

But things does not happen as I think.

What's wrong there?
Hope someone can point out my fault and teach me how to do this correctly.

Thx!

Output:
Finally, I solved this problem. First, I compared two models' parameters and found out they were the same. So I confirmed that the model is the same. And then, I checked out two inputs and surprisedly found out they were different.

So I reviewed two models' inputs carefully and the answer was that the arguments passed to the second model did not update.

Code:

for data in dataloaders['val']:
    images, labels = data
    outputs = model(inputs)

Change to:

for data in dataloaders['val']:
    inputs, labels = data
    outputs = model(inputs)

Done!


Instruction: DLL files error in using pytorch

Input:
I add pytorch via pip installation and now I'm trying to use it, but have this dll error:

Traceback (most recent call last):
  File "F:/Python/Projects/1.py", line 2, in <module>
    import torch
  File "C:\Users\Saeed\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\__init__.py", line 78, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

I install msvcp71 and msvcr71 dll files but it's not worked!

Output:
You can use Dependency Walker to find out which dependency of that DLL might be missing. Use it to open the Python extension file that's failing to load. The file name should be something like:

C:\Users\Saeed\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\_C.pyd

Another common cause is a DLL for Python 64-bit while using Python 32-bit or vice-versa. But you installed with pip so it should be OK. Nevertheless, it's a good idea to verify this is not the case.


Instruction: Pytorch What's the difference between define layer in __init__() and directly use in forward()?

Input:
What is the difference between the method that define layers in __init__() function, call layer in forward later and the method that directly use layer in forward() function ?
Should I define every layer in my compute graph in constructed function(eg. __init__) before I write my compute graph?
Could I direct define and use them in forward()?

Output:
Everything which contains weights which you want to be trained during the training process should be defined in your __init__ method.

You don't need do define activation functions like softmax, ReLU or sigmoid in your __init__, you can just call them in forward.

Dropout layers for example also don't need to be defined in __init__, they can just be called in forward too. [However defining them in your __init__ has the advantage that they can be switched off easier during evaluation (by calling eval() on your model). You can see an example of both versions here.

Hope this is clear. Just ask if you have any further questions.


Instruction: Reshaping Pytorch tensor

Input:
I have a tensor of size (24, 2, 224, 224) in Pytorch.

24 = batch size
2 = matrixes representing foreground and
background
224 = image height dimension
224 = image width
dimension

This is the output of a CNN that performs binary segmentation. In each cell of the 2 matrixes is stored the probability for that pixel to be foreground or background: [n][0][h][w] + [n][1][h][w] = 1 for every coordinate

I want to reshape it into a tensor of size (24, 1, 224, 224). The values in the new layer should be 0 or 1 according to the matrix in which the probability was higher.

How can I do that? Which function should I use?

Output:
Using torch.argmax() (for PyTorch +0.4):

prediction = torch.argmax(tensor, dim=1) # with 'dim' the considered dimension
prediction = prediction.unsqueeze(1) # to reshape from (24, 224, 224) to (24, 1, 224, 224)

If the PyTorch version is below 0.4.0, one can use tensor.max() which returns both the max values and their indices (but which isn't differentiable over the index values):

_, prediction = tensor.max(dim=1)
prediction = prediction.unsqueeze(1) # to reshape from (24, 224, 224) to (24, 1, 224, 224)


Instruction: Pytorch - Stack dimension must be exactly the same?

Input:
In pytorch, given the tensors a of shape (1X11) and  b of shape (1X11), torch.stack((a,b),0) would give me a tensor of shape (2X11)

However, when a is of shape (2X11) and b is of shape (1X11), torch.stack((a,b),0) will raise an error cf. "the two tensor size must exactly be the same".

Because the two tensor are the output of a model (gradient included), I can't convert them to numpy to use np.stack() or np.vstack().

Is there any possible solution for least GPU memory usage?

Output:
It seems you want to use torch.cat() (concatenate tensors along an existing dimension) and not torch.stack() (concatenate/stack tensors along a new dimension):

import torch

a = torch.randn(1, 42, 1, 1)
b = torch.randn(1, 42, 1, 1)

ab = torch.stack((a, b), 0)
print(ab.shape)
# torch.Size([2, 1, 42, 1, 1])

ab = torch.cat((a, b), 0)
print(ab.shape)
# torch.Size([2, 42, 1, 1])
aab = torch.cat((a, ab), 0)
print(aab.shape)
# torch.Size([3, 42, 1, 1])


Instruction: Use torch.eq() only for some value in Pytorch

Input:
Is there a way to use torch.eq() or a similar function to compute element-based equality but only for some elements?
Let's say I need to know how many 1s are equal in the two tensors but I don't care about other numbers.

Any idea how to do this?

Output:
Let's say we have 2 tensors A and B filled with random elements and eventually some 1s somewhere. The tensor C is the result of want you aim for:

A = torch.rand((2, 3, 3))
B = torch.rand((2, 3, 3))

# fill A and B with some 1s
...

C = (A == 1) * (B == 1)

Using the following tensors we get:

(A) [[[ 0.6151,  1.0000,  0.6515],
         [ 0.3337,  0.4262,  0.0731],
         [ 0.4571,  0.2380,  1.0000]],

        [[ 1.0000,  0.1114,  0.8183],
         [ 0.9178,  1.0000,  1.0000],
         [ 0.8180,  0.8112,  0.2972]]]

(B) [[[ 0.4305,  1.0000,  0.5378],
         [ 0.4171,  0.4365,  0.2805],
         [ 0.1076,  0.1259,  0.9695]],

        [[ 1.0000,  0.0911,  1.0000],
         [ 0.6757,  0.5095,  0.4499],
         [ 0.5787,  1.0000,  1.0000]]]

(C) [[[ 0,  1,  0],
         [ 0,  0,  0],
         [ 0,  0,  0]],

        [[ 1,  0,  0],
         [ 0,  0,  0],
         [ 0,  0,  0]]]


Instruction: How to compute the cosine_similarity in pytorch for all rows in a matrix with respect to all rows in another matrix

Input:
In pytorch, given that I have 2 matrixes how would I compute cosine similarity of all rows in each with all rows in the other.

For example

Given the input =

matrix_1 = [a b]
           [c d]
matrix_2 = [e f]
           [g h]

I would like the output to be

output =

 [cosine_sim([a b] [e f])  cosine_sim([a b] [g h])]
 [cosine_sim([c d] [e f])  cosine_sim([c d] [g h])]

At the moment I am using torch.nn.functional.cosine_similarity(matrix_1, matrix_2) which returns the cosine of the row with only that corresponding row in the other matrix.

In my example I have only 2 rows, but I would like a solution which works for many rows. I would even like to handle the case where the number of rows in the each matrix is different.

I realize that I could use the expand, however I want to do it without using such a large memory footprint.

Output:
By manually computing the similarity and playing with matrix multiplication + transposition:

import torch
from scipy import spatial
import numpy as np

a = torch.randn(2, 2)
b = torch.randn(3, 2) # different row number, for the fun

# Given that cos_sim(u, v) = dot(u, v) / (norm(u) * norm(v))
#                          = dot(u / norm(u), v / norm(v))
# We fist normalize the rows, before computing their dot products via transposition:
a_norm = a / a.norm(dim=1)[:, None]
b_norm = b / b.norm(dim=1)[:, None]
res = torch.mm(a_norm, b_norm.transpose(0,1))
print(res)
#  0.9978 -0.9986 -0.9985
# -0.8629  0.9172  0.9172

# -------
# Let's verify with numpy/scipy if our computations are correct:
a_n = a.numpy()
b_n = b.numpy()
res_n = np.zeros((2, 3))
for i in range(2):
    for j in range(3):
        # cos_sim(u, v) = 1 - cos_dist(u, v)
        res_n[i, j] = 1 - spatial.distance.cosine(a_n[i], b_n[j])
print(res_n)
# [[ 0.9978022  -0.99855876 -0.99854881]
#  [-0.86285472  0.91716063  0.9172349 ]]


Instruction: Reproducable Pytorch Results & Random Seeds

Input:
I have a simple toy NN with Pytorch. I am setting all the seeds I can find in the docs as well as numpy random.

If I run the code below from top to bottom, the results appear to be reproducible.

BUT, if I run block 1 only once and then each time run block 2, the result changes (sometimes dramatically). I am unsure why this happens since the network is being re-initialized and optimizer reset each time.

I am using version 0.4.0

BLOCK #1

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F
from torch.nn.init import xavier_uniform_, xavier_normal_,uniform_

torch.manual_seed(123)

import random
random.seed(123)

from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

%matplotlib inline

cuda=True #set to true uses GPU

if cuda:
    torch.cuda.manual_seed(123)

#load boston data from scikit
boston = load_boston()
x=boston.data
y=boston.target
y=y.reshape(y.shape[0],1)

#train and test
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.3, random_state=123, shuffle=False)

#change to tensors
x_train = torch.from_numpy(x_train)
y_train = torch.from_numpy(y_train)

#create dataset and use data loader
training_samples = utils_data.TensorDataset(x_train, y_train)
data_loader_trn = utils_data.DataLoader(training_samples, batch_size=64,drop_last=False)

#change to tensors
x_test = torch.from_numpy(x_test)
y_test = torch.from_numpy(y_test)

#create dataset and use data loader
testing_samples = utils_data.TensorDataset(x_test, y_test)
data_loader_test = utils_data.DataLoader(testing_samples, batch_size=64,drop_last=False)

#simple model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        #all the layers
        self.fc1   = nn.Linear(x.shape[1], 20)
        xavier_uniform_(self.fc1.weight.data) #this is how you can change the weight init
        self.drop = nn.Dropout(p=0.5)
        self.fc2   = nn.Linear(20, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x=  self.drop(x)
        x = self.fc2(x)
        return x

BLOCK #2

net=Net()

if cuda:
    net.cuda()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss(size_average=False)

# run the main training loop
epochs =20
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    cum_records_epoch =0

    for batch_idx, (data, target) in enumerate(data_loader_trn):
        tr_x, tr_y = data.float(), target.float()
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch

        cum_loss += output.item() #accumulate the loss

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()

        cum_records_epoch +=len(tr_x)
        if batch_idx % 1 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, cum_records_epoch, len(data_loader_trn.dataset),
            100. * (batch_idx+1) / len(data_loader_trn), output.item()))
    print('Epoch average loss: {:.6f}'.format(cum_loss/cum_records_epoch))

    hold_loss.append(cum_loss/cum_records_epoch)

#training loss
plt.plot(np.array(hold_loss))
plt.show()

Output:
Possible Reason

Not knowing what the "sometimes dramatic differences" are, it is hard to answer for sure; but having different results when running [block_1 x1; block_2 x1] xN (read "running block_1 then block_2 once; and repeat both operations N times)   and [block_1 x1; block_2 xN] x1 makes sense, given how pseudo-random number generators (PRNGs) and seeds work.

In the first case, you are re-initializing the PRNGs in block_1 after each block_2, so each of the N instances of block_2 will access the same sequence of pseudo-random numbers, seeded by each block_1 before.

In the second case, the PRNGs are initialized only once, by the single block_1 run. So each instance of block_2 will have different random values.

(For more on PRNGs and seeds, you could check: random.seed(): What does it do?)

Simplified Example

Let's suppose numpy/CUDA/pytorch are actually using a really poor PRNG, which only returns incremented values (i.e. PRNG(x_n) = PRNG(x_(n-1)) + 1, with x_0 = seed). If you seed this generator with 0, it will thus return 1 the first random() call, 2 the second call, etc.

Now let also simplifies your blocks for the sake of the example:

def block_1():
    seed = 0
    print("seed: {}".format(seed))
    prng.seed(seed)

--

def block_2():
    res = "random results:"
    for i in range(4):
         res  += " {}".format(prng.random())
    print(res)

Let's compare [block_1 x1; block_2 x1] xN and [block_1 x1; block_2 xN] x1 with N=3:

for i in range(3):
    block_1()
    block_2()
# > seed: 0
# > random results: 1 2 3 4
# > seed: 0
# > random results: 1 2 3 4
# > seed: 0
# > random results: 1 2 3 4

block_1()
for i in range(3):
    block_2()
# > seed: 0
# > random results: 1 2 3 4
# > random results: 4 5 6 7
# > random results: 8 9 10 11


Instruction: Concatenating two tensors with different dimensions in Pytorch

Input:
Is it possible to concatenate two tensors with different dimensions without using for loop.

e.g. Tensor 1 has dimensions (15, 200, 2048) and Tensor 2 has dimensions (1, 200, 2048). Is it possible to concatenate 2nd tensor with 1st tensor along all the 15 indices of 1st dimension in 1st Tensor (Broadcast 2nd tensor along 1st dimension of Tensor 1 while concatenating along 3rd dimension of 1st tensor)? The resulting tensor should have dimensions (15, 200, 4096).

Is it possible to accomplish this without for loop ?

Output:
You could do the broadcasting manually (using Tensor.expand()) before the concatenation (using torch.cat()):
import torch

a = torch.randn(15, 200, 2048)
b = torch.randn(1, 200, 2048)

repeat_vals = [a.shape[0] // b.shape[0]] + [-1] * (len(b.shape) - 1)
# or directly repeat_vals = (15, -1, -1) or (15, 200, 2048) if shapes are known and fixed...
res = torch.cat((a, b.expand(*repeat_vals)), dim=-1)
print(res.shape)
# torch.Size([15, 200, 4096])


Instruction: can't import torch mac

Input:
I'm trying to import torch and I'm getting the next problem:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/__init__.py", line 66, in <module>
    import torch._dl as _dl_flags
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/_dl.so, 2): no suitable image found.  Did find:
    /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/_dl.so: mach-o, but wrong architecture
    /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/torch/_dl.so: mach-o, but wrong architecture

someone knows how can I solve this?
Thanks:)

Output:
Try like that:

mkdir test_torch
cd test_torch
python3 -m venv .venv
source .venv/bin/activate
pip install torch torchvision
python3

>>> import torch

Works for me. MacOS 10.13.4, Python 3.6.4

Or like that:

mkdir test_torch
cd test_torch
virtualenv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch torchvision
python2

>>> import torch

Works for me. MacOS 10.13.4, Python 2.7.10

If you don't need to use torch then you can install only torchvision

pip install --no-deps torchvision


Instruction: Byte Embedding in mLSTM Conceptual Struggle

Input:
I am trying to follow the OpenAI "Sentiment Neuron" experiment by reading through the PyTorch code posted on Github for training the model from scratch.
One thing I am not understanding is the byte-level embedding used in the code. I understood that the LSTM outputs a probability distribution for the value of the next byte and I assumed the "embedding" would just be a one-hot encoding of the byte value.
Looking at the code, I see that the model's input goes through a (trainable) dense embedding before going into the model. Confusingly, the output of the loss is computed between the model output and the upcoming byte value, which is not embedded.
My questions are:
1. How is the cross entropy loss computed? Does nn.CrossEntropyLoss take the softmax of its input and expand the target into a one-hot vector "under the hood"?
2. If we want to generate byte strings from this LSTM, how do we embed the output to feed back into the model for the next step? Do we embed the highest likelihood or take a softmax of the output and use some sort of weighted embedding?
I'm new to LSTM and I'm trying to learn but I just don't get it! I appreciate any help!

Output:
Even though the same symbols are being used for input and output, it's perfectly acceptable to have different representations used at each end. Cross entropy is a function of two probability distributions. In this case, the two distributions are the softmax distribution given by the model, and a point mass on the "correct" byte.

For question 1, yes that is what is being done in terms of inputs and outputs (although the implementation might be optimized).

To answer question 2, the most common thing is to form the softmax distribution at each step, then sample from it.


Instruction: Pytorch, can't run backward() on even the most simple network without getting an error

Input:
I am new to pytorch and I can't run backward() on even the most simple network without generating an error.
For example:

(Linear(6, 6)(Variable(torch.zeros([10, 6]))) - Variable(torch.zeros([10, 6]))).backward()

Throws the following error

{RuntimeError}element 0 of variables does not require grad and does not have a grad_fn

What have I done wrong in the code to create this issue?

Output:
Try adding a grad_output of matching shape as a parameter to backward:

(Linear(6, 6)(Variable(torch.zeros([10, 6]))) - Variable(torch.zeros([10, 6]))).backward(torch.zeros([10, 6]))

The following answer has more details: Why should be the function backward be called only on 1 element tensor or with gradients w.r.t to Variable?


Instruction: RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2864

Input:
Im trying to build a dataloader, This is what it looks like

`class WhaleData(Dataset):
def __init__(self, data_file, root_dir , transform = None):
    self.csv_file = pd.read_csv(data_file)
    self.root_dir = root_dir
    self.transform = transforms.Resize(224)

def __len__(self):
    return len(os.listdir(self.root_dir))

def __getitem__(self, index):
    image = os.path.join(self.root_dir, self.csv_file['Image'][index])
    image = Image.open(image)
    image = self.transform(image)
    image = np.array(image)
    label  = self.csv_file['Image'][index]
    sample = {'image': image, 'label':label}
    return sample

trainset  = WhaleData(data_file = '/mnt/55-91e8-b2383e89165f/Ryan/1234/train.csv',
     root_dir = '/mnt/4d55-91e8-b2383e89165f/Ryan/1234/train')
train_loader = torch.utils.data.DataLoader(trainset , batch_size = 4, shuffle =True,num_workers= 2)
for i, batch in enumerate(train_loader):
      (i, batch)

When i try running this block of code, i get this error,I do get the nature of the error that all my images may not be of the same shape,and my images are not all of the same shape ,But if im not wrong the error should only arise when i feed them to the network because the images are all of different shapes,But why is it throwing an errror here?
Any suggestions on where i might have gone wrong will be extremly helpful,
I would be happy to provide any extra information if needed,

Thanks

RuntimeError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 116, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 116, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 105, in default_collate
return torch.stack([torch.from_numpy(b) for b in batch], 0)
  File "/usr/local/lib/python3.5/dist-packages/torch/functional.py", line 64, in stack
    return torch.cat(inputs, dim)
RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic    /THTensorMath.c:2864

Output:
The error appears when PyTorch tries to stack together the images into a single batch tensor (cf. torch.stack([torch.from_numpy(b) for b in batch], 0) from your trace). As you mentioned, since the images have different shape, the stacking fails (i.e. a tensor (B, H, W) can only be created by stacking B tensors if all these tensors have for shape (H, W)).

Note: I'm not fully sure, but setting batch_size=1 for torch.utils.data.DataLoader(...) may remove this particular error, as it probably won't need calling torch.stack() anymore).