Instruction: Torch code producing CUDA Runtime Error

Input:
a friend of mine implemented a sparse version of torch.bmm that actually works, but when I try a test, I have a runtime error (that has nothing to do with this implementation), that I don't understand. I have seen a few topics about if but couldn't find a solution. Here is the code, and the error:

if __name__ == "__main__":
     tmp = torch.zeros(1).cuda()
     batch_csr = BatchCSR()
     sparse_bmm = SparseBMM()

     i=torch.LongTensor([[0,5,8], [1,5,8], [2,5,8]])
     v=torch.FloatTensor([4,3,8])
     s=torch.Size([3,500,500])

     indices, values, size = i,v,s

     a_ = torch.sparse.FloatTensor(indices, values, size).cuda().transpose(2, 1)
     batch_size, num_nodes, num_faces = a_.size()

     a = a_.to_dense()

     for _ in range(10):
        b = torch.randn(batch_size, num_faces, 16).cuda()
        torch.cuda.synchronize()
        time1 = time.time()
        result = torch.bmm(a, b)
        torch.cuda.synchronize()
        time2 = time.time()
        print("{} CuBlas dense bmm".format(time2 - time1))

        torch.cuda.synchronize()
        time1 = time.time()
        col_ind, col_ptr = batch_csr(a_.indices(), a_.size())
        my_result = sparse_bmm(a_.values(), col_ind, col_ptr, a_.size(), b)
        torch.cuda.synchronize()
        time2 = time.time()
        print("{} My sparse bmm".format(time2 - time1))

        print("{} Diff".format((result-my_result).abs().max()))

And the error:

Traceback (most recent call last):
  File "sparse_bmm.py", line 72, in <module>
    b = torch.randn(3, 500, 16).cuda()
  File "/home/bizeul/virtual_env/lib/python2.7/site-packages/torch/_utils.py", line 65, in _cuda
    return new_type(self.size()).copy_(self, async)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorCopy.c:18

When running with the command CUDA_LAUNCH_BLOCKING=1, I get the error :

/b/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:121: void indexAddSmallIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 1, SrcDim = 1, IdxDim = -2]: block: [0,0,0], thread: [0,0,0] Assertion `dstIndex < dstAddDimSize` failed.
THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THCS/generic/THCSTensorMath.cu line=292 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "sparse_bmm.py", line 69, in <module>
    a = a_.to_dense()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THCS/generic/THCSTensorMath.cu:292

Output:
The indices that you are passing to create the sparse tensor are incorrect.

here is how it should be:

i = torch.LongTensor([[0, 1, 2], [5, 5, 5], [8, 8, 8]])

How to create a sparse tensor:

Lets take a simpler example. Lets say we want the following tensor:

  0   0   0   2   0
  0   0   0   0   0
  0   0   0   0  20
[torch.cuda.FloatTensor of size 3x5 (GPU 0)]

As you can see, the number (2) needs to be in the (0, 3) location of the sparse tensor. And the number (20) needs to be in the (2, 4) location.

In order to create this, our index tensor should look like this

[[0 , 2],
 [3 , 4]]

And, now for the code to create the above sparse tensor:

i=torch.LongTensor([[0, 2], [3, 4]])
v=torch.FloatTensor([2, 20])
s=torch.Size([3, 5])
a_ = torch.sparse.FloatTensor(indices, values, size).cuda()

More comments regarding the assert error by cuda:

Assertion 'dstIndex < dstAddDimSize' failed. tells us that, its highly likely, you've got an index out of bounds. So whenever you notice that, look for places where you might have supplied the wrong indices to any of the tensors.


Instruction: Torch: How to shuffle a tensor by its rows?

Input:
I am currently working in torch to implement a random shuffle (on the rows, the first dimension in this case) on some input data. I am new to torch, so I have some troubles figuring out how permutation works..

The following is supposed to shuffle the data:

if argshuffle then
    local perm = torch.randperm(sids:size(1)):long()
    print("\n\n\nSize of X and y before")
    print(X:view(-1, 1000, 128):size())
    print(y:size())
    print(sids:size())
    print("\nPerm size is: ")
    print(perm:size())
    X = X:view(-1, 1000, 128)[{{perm},{},{}}]
    y = y[{{perm},{}}]
    print(sids[{{1}, {}}])
    sids = sids[{{perm},{}}]
    print(sids[{{1}, {}}])
    print(X:size())
    print(y:size())
    print(sids:size())
    os.exit(69)
end

This prints out

Size of X and y before
99
1000
128
[torch.LongStorage of size 3]

99
1
[torch.LongStorage of size 2]

99
1
[torch.LongStorage of size 2]

Perm size is:
99
[torch.LongStorage of size 1]
5
[torch.LongStorage of size 1x1]
5
[torch.LongStorage of size 1x1]

99
1000
128
[torch.LongStorage of size 3]

99
1
[torch.LongStorage of size 2]

99
1
[torch.LongStorage of size 2]

Out of the value, I can imply that the function did not shuffle the data. How can I make it shuffle correctly, and what is the common solution in lua/torch?

Output:
I also faced a similar issue. In the documentation, there is no shuffle function for tensors (there are for dataset loaders). I found a workaround to the problem using torch.randperm.

>>> a=torch.rand(3,5)
>>> print(a)
tensor([[0.4896, 0.3708, 0.2183, 0.8157, 0.7861],
        [0.0845, 0.7596, 0.5231, 0.4861, 0.9237],
        [0.4496, 0.5980, 0.7473, 0.2005, 0.8990]])
>>> # Row shuffling
...
>>> a=a[torch.randperm(a.size()[0])]
>>> print(a)
tensor([[0.4496, 0.5980, 0.7473, 0.2005, 0.8990],
        [0.0845, 0.7596, 0.5231, 0.4861, 0.9237],
        [0.4896, 0.3708, 0.2183, 0.8157, 0.7861]])
>>> # column shuffling
...
>>> a=a[:,torch.randperm(a.size()[1])]
>>> print(a)
tensor([[0.2005, 0.7473, 0.5980, 0.8990, 0.4496],
        [0.4861, 0.5231, 0.7596, 0.9237, 0.0845],
        [0.8157, 0.2183, 0.3708, 0.7861, 0.4896]])

I hope it answers the question!


Instruction: necessity of transposed convolution when feature maps are not downsampled

Input:
I was reading a paper here. The authors in the paper have proposed a symmetric generator network which contains a stack of convolution layers followed by a stack of de-convolution (transposed convolution) layers. It is also mentioned that a stride of 1 with appropriate padding is used to ensure that feature map size is same as input image size.

My question is if there is no downsampling, then why transposed conolution layers are used? Can't the generator be constructed only with convolution layers? Am I missing something about transposed convolution layers here (is it being used for some other purpose)? Please help.

Update: I am re-opening this question, as I came across this paper which states in section 2.1.1 that "deconvolution is used to compensate the details". However, I am not able to appreciate this because there is no downsampling of feature maps in the proposed model. Can somebody explain why deconvolution is preferred over convolution here? What makes deconvolution layer perform better than convolution in this case?

Output:
In theory spatial convolution can be used as a replacement for fractionally-strided convolution. Typically this is avoided because, even without some type of pooling, convolutional layers can produce outputs that are smaller than their corresponding inputs (see the formulae for owidth and oheight in the docs here). Using nn.SpatialConvolution to produce outputs that are larger than inputs would require a great deal of inefficient zero-padding to reach the original input size. To make the reverse process easier, torch functionality was added for fractionally-strided convolution.

That being said, this case is a bit different since the size at each layer remains constant. So it is quite possible that using nn.SpatialConvolution for the entire generator will work. You'll still want to mirror the encoder's nInputPlane and nOutputPlane pattern to successfully move from feature space back to input space.

Likely the authors referred to the decoder process as using transpose convolution just for clarity and generality.

This article discusses convolution and fractionally-strided convolution, and provides nice graphics that I do not wish to copy here.


Instruction: PyTorch Linear Algebra Gradients

Input:
I'm looking to back-propagate gradients through a singular value decomposition for regularisation purposes. PyTorch currently does not support backpropagation through a singular value decomposition.
I know that I could write my own custom function that operates on a Variable; takes its .data tensor, applies the torch.svd to it, wraps a Variable around its singular values and returns it in the forward pass, and in the backward pass applies the appropriate Jacobian matrix to the incoming gradients.
However, I was wondering whether there was a more elegant (and potentially faster) solution, where I could overwrite the "Type Variable doesn't implement stateless method svd" Error directly, call Lapack, etc. ?
If someone could guide me through the appropriate steps and source files I need to look at, I'd be very grateful. I suppose these steps would similarly apply to other linear algebra operations which have no associated backward method currently.

Output:
torch.svd with forward and backward pass is now available in the Pytorch master:

http://pytorch.org/docs/master/torch.html#torch.svd

You need to install Pytorch from source:
https://github.com/pytorch/pytorch/#from-source


Instruction: Failed to compute dot product of torch.cuda.FloatTensor

Input:
I used a GPU to compute the dot product of the output of neural networks and a torch.cuda.FloatTensor (both of them are stored in GPU) but got an error saying:

TypeError: dot received an invalid combination of arguments - got (torch.cuda.FloatTensor) but expected (torch.FloatTensor tensor).

the codes are like

p = torch.exp(vector.dot(ht))

here vector is a torch FloatTensor and ht is the output of neural networks.

I've struggled with these things for days but still got no idea. Thanks in advance for any possible solution!

Output:
What does the following error message mean?

  TypeError: dot received an invalid combination of arguments - got (torch.cuda.FloatTensor) but expected (torch.FloatTensor tensor).

It means dot function expected cpu tensor but you are providing a gpu (cuda) tensor.

So, how to solve the problem of your code?

p = torch.exp(vector.dot(ht))

As you mentioned vector is a FloatTensor, so ht should be a FloatTensor as well but ht is a cuda.FloatTensor (because, your neural network model is in gpu memory).

So, you should convert vector to cuda.FloatTensor by doing the following.

vector = vector.cuda()

OR, you can convert the cuda.FloatTensor to cpu tensor by doing the following. Please note, .cpu() method is not applicable for Variable. In that case, you can use .data.cpu().

ht = ht.cpu()

It should solve your problem.


Instruction: What would be equivalent of pytorch's torch.nn.CosineEmbeddingLoss in tensorflow?

Input: CosineEmbeddingLoss in Pytorch is the perfect function I am looking for in tensorflow, but I can only find tf.losses.cosine_distance. Is there a way or code that writes CosineEmbeddingLoss in tensorflow?

Output:
A TensorFlow version of CosineEmbeddingLoss:

import tensorflow as tf

def CosineEmbeddingLoss(margin=0.):
  def _cosine_similarity(x1, x2):
    """Cosine similarity between two batches of vectors."""
    return tf.reduce_sum(x1 * x2, axis=-1) / (
        tf.norm(x1, axis=-1) * tf.norm(x2, axis=-1))
  def _cosine_embedding_loss_fn(input_one, input_two, target):
    similarity = _cosine_similarity(input_one, input_two)
    return tf.reduce_mean(tf.where(
        tf.equal(target, 1),
        1. - similarity,
        tf.maximum(tf.zeros_like(similarity), similarity - margin)))
  return _cosine_embedding_loss_fn

Running it alongside Torch's version:

import tensorflow as tf
import numpy
import torch
from torch.autograd import Variable

first_values = numpy.random.normal(size=[100, 3])
second_values = numpy.random.normal(size=[100, 3])
labels = numpy.random.randint(2, size=[100]) * 2 - 1
torch_result = torch.nn.CosineEmbeddingLoss(margin=0.5)(
    Variable(torch.FloatTensor(first_values)),
    Variable(torch.FloatTensor(second_values)),
    Variable(torch.IntTensor(labels))).data.numpy()
with tf.Graph().as_default():
  with tf.Session():
    tf_result = CosineEmbeddingLoss(margin=0.5)(
        first_values, second_values, labels).eval()
print(torch_result, tf_result)

Seems to match to within reasonable precision:

array([ 0.35702518], dtype=float32)  0.35702516587462357


Instruction: Unique values in PyTorch tensor

Input: I'm tying to find distinct values in a PyTorch tensor. Is there an efficient analogue of Tensorflow's unique op?

Output:
There is a torch.unique() method in 0.4.0

In torch <= 0.3.1 you can try:

import torch
import numpy as np

x = torch.rand((3,3)) * 10
np.unique(x.round().numpy())


Instruction: Confused about tensor dimensions and batch sizes in pytorch

Input:
So I'm very new to PyTorch and Neural Networks in general, and I'm having some problems creating a Neural Network that classifies names by gender.
I based this off of the PyTorch tutorial for RNNs that classify names by nationality, but I decided not to go with a recurrent approach... Stop me right here if this was the wrong idea!
However, whenever I try to run an input through the network it tells me:

RuntimeError: matrices expected, got 3D, 2D tensors at /py/conda-bld/pytorch_1493681908901/work/torch/lib/TH/generic/THTensorMath.c:1232

I know this has something to do with how PyTorch always expects there to be a batch size or something, and I have my tensor set up that way, but you can probably tell by this point that I have no idea what I'm talking about.
Here's my code:

from future import unicode_literals, print_function, division
from io import open
import glob
import unicodedata
import string
import torch
import torchvision
import torch.nn as nn
import torch.optim as optim
import random
from torch.autograd import Variable
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

"""------GLOBAL VARIABLES------"""

all_letters = string.ascii_letters + " .,;'"
num_letters = len(all_letters)
all_names = {}
genders = ["Female", "Male"]

"""-------DATA EXTRACTION------"""

def findFiles(path):
    return glob.glob(path)

def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in all_letters
    )

# Read a file and split into lines
def readLines(filename):
    lines = open(filename, encoding='utf-8').read().strip().split('\n')
    return [unicodeToAscii(line) for line in lines]

for file in findFiles("/home/andrew/PyCharm/PycharmProjects/CantStop/data/names/*.txt"):
    gender = file.split("/")[-1].split(".")[0]
    names = readLines(file)
    all_names[gender] = names

"""-----DATA INTERPRETATION-----"""

def nameToTensor(name):
    tensor = torch.zeros(len(name), 1, num_letters)
    for index, letter in enumerate(name):
        tensor[index][0][all_letters.find(letter)] = 1
    return tensor

def outputToGender(output):
    gender, gender_index = output.data.topk(1)
    if gender_index[0][0] == 0:
        return "Female"
    return "Male"

"""------NETWORK SETUP------"""

class Net(nn.Module):
    def __init__(self, input_size, output_size):
        super(Net, self).__init__()
        #Layer 1
        self.Lin1 = nn.Linear(input_size, int(input_size/2))
        self.ReLu1 = nn.ReLU()
        self.Batch1 = nn.BatchNorm1d(int(input_size/2))
        #Layer 2
        self.Lin2 = nn.Linear(int(input_size/2), output_size)
        self.ReLu2 = nn.ReLU()
        self.Batch2 = nn.BatchNorm1d(output_size)
        self.softMax = nn.LogSoftmax()

    def forward(self, input):
        output1 = self.Batch1(self.ReLu1(self.Lin1(input)))
        output2 = self.softMax(self.Batch2(self.ReLu2(self.Lin2(output1))))
        return output2

NN = Net(num_letters, 2)

"""------TRAINING------"""

def getRandomTrainingEx():
    gender = genders[random.randint(0, 1)]
    name = all_names[gender][random.randint(0, len(all_names[gender])-1)]
    gender_tensor = Variable(torch.LongTensor([genders.index(gender)]))
    name_tensor = Variable(nameToTensor(name))
    return gender_tensor, name_tensor, gender

def train(input, target):
    loss_func = nn.NLLLoss()

    optimizer = optim.SGD(NN.parameters(), lr=0.0001, momentum=0.9)

    optimizer.zero_grad()

    output = NN(input)

    loss = loss_func(output, target)
    loss.backward()
    optimizer.step()

    return output, loss

all_losses = []
current_loss = 0

for i in range(100000):
    gender_tensor, name_tensor, gender = getRandomTrainingEx()
    output, loss = train(name_tensor, gender_tensor)
    current_loss += loss

    if i%1000 == 0:
        print("Guess: %s, Correct: %s, Loss: %s" % (outputToGender(output), gender, loss.data[0]))

    if i%100 == 0:
        all_losses.append(current_loss/10)
        current_loss = 0

# plt.figure()
# plt.plot(all_losses)
# plt.show()

Please help a newbie out!

Output:
Debugging your bug out:

Pycharm is a helpful python debugger that let you set breakpoint and views dimension of your tensor.
For easier debug, do not stack forward thing up like that

output1 = self.Batch1(self.ReLu1(self.Lin1(input)))

Instead,

h1 = self.ReLu1(self.Lin1(input))
h2 = self.Batch1(h1)

For the stacktrace, Pytorch also provide Pythonic error stacktrack. I believe that before

RuntimeError: matrices expected, got 3D, 2D tensors at /py/conda-bld/pytorch_1493681908901/work/torch/lib/TH/generic/THTensorMath.c:1232

There are some python error stacktrace that point right into your code. For easier debug, as I said, don't stack forward.

You use Pycharm to create break point before crash point. In debugger watcher Then use Variable(torch.rand(dim1, dim2)) to test out forward pass input, output dimension, and if a dimension is incorrect. Comparing with dimension of input. Call input.size() in debugger watcher.

For example,  self.ReLu1(self.Lin1(Variable(torch.rand(10, 20)))).size() . If it show read text (error), then the input dimension is incorrect. Else, it show the size of the output.

Read the docs

In Pytorch Docs, it specify input/output dimension.  It also have a example code snip

>>> rnn = nn.RNN(10, 20, 2)
>>> input = Variable(torch.randn(5, 3, 10))
>>> h0 = Variable(torch.randn(2, 3, 20))
>>> output, hn = rnn(input, h0)

You may use the code snip in PyCharm Debugger to explore dimension of input, output of specific layer of your interest (RNN, Linear, BatchNorm1d).


Instruction: How to get mini-batches in pytorch in a clean and efficient way?

Input:
I was trying to do a simple thing which was train a linear model with Stochastic Gradient Descent (SGD) using torch:

import numpy as np

import torch
from torch.autograd import Variable

import pdb

def get_batch2(X,Y,M,dtype):
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    batch_ys = torch.FloatTensor(Y[batch_indices]).type(dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

def poly_kernel_matrix( x,D ):
    N = len(x)
    Kern = np.zeros( (N,D+1) )
    for n in range(N):
        for d in range(D+1):
            Kern[n,d] = x[n]**d;
    return Kern

## data params
N=5 # data set size
Degree=4 # number dimensions/features
D_sgd = Degree+1
##
x_true = np.linspace(0,1,N) # the real data points
y = np.sin(2*np.pi*x_true)
y.shape = (N,1)
## TORCH
dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
X_mdl = poly_kernel_matrix( x_true,Degree )
X_mdl = Variable(torch.FloatTensor(X_mdl).type(dtype), requires_grad=False)
y = Variable(torch.FloatTensor(y).type(dtype), requires_grad=False)
## SGD mdl
w_init = torch.zeros(D_sgd,1).type(dtype)
W = Variable(w_init, requires_grad=True)
M = 5 # mini-batch size
eta = 0.1 # step size
for i in range(500):
    batch_xs, batch_ys = get_batch2(X_mdl,y,M,dtype)
    # Forward pass: compute predicted y using operations on Variables
    y_pred = batch_xs.mm(W)
    # Compute and print loss using operations on Variables. Now loss is a Variable of shape (1,) and loss.data is a Tensor of shape (1,); loss.data[0] is a scalar value holding the loss.
    loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
    # Use autograd to compute the backward pass. Now w will have gradients
    loss.backward()
    # Update weights using gradient descent; w1.data are Tensors,
    # w.grad are Variables and w.grad.data are Tensors.
    W.data -= eta * W.grad.data
    # Manually zero the gradients after updating weights
    W.grad.data.zero_()

#
c_sgd = W.data.numpy()
X_mdl = X_mdl.data.numpy()
y = y.data.numpy()
#
Xc_pinv = np.dot(X_mdl,c_sgd)
print('J(c_sgd) = ', (1/N)*(np.linalg.norm(y-Xc_pinv)**2) )
print('loss = ',loss.data[0])

the code runs fine and all though my get_batch2 method seems really dum/naive, its probably because I am new to pytorch but I have not found a good place where they discuss how to retrieve data batches. I went through their tutorials (http://pytorch.org/tutorials/beginner/pytorch_with_examples.html) and through the data set (http://pytorch.org/tutorials/beginner/data_loading_tutorial.html) with no luck. The tutorials all seem to assume that one already has the batch and batch-size at the beginning and then proceeds to train with that data without changing it (specifically look at http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-variables-and-autograd).

So my question is do I really need to turn my data back into numpy so that I can fetch some random sample of it and then turn it back to pytorch with Variable to be able to train in memory? Is there no way to get mini-batches with torch?

I looked at a few functions torch provides but with no luck:

#pdb.set_trace()
#valid_indices = torch.arange(0,N).numpy()
#valid_indices = np.array( range(N) )
#batch_indices = np.random.choice(valid_indices,size=M,replace=False)
#indices = torch.LongTensor(batch_indices)
#batch_xs, batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
#batch_xs,batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)

even though the code I provided works fine I am worried that its not an efficient implementation AND that if I were to use GPUs that there would be a considerable further slow down (because my guess it putting things in memory and then fetching them back to put them GPU like that is silly).

I implemented a new one based on the answer that suggested to use torch.index_select():

def get_batch2(X,Y,M):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    #X,Y = X.data.numpy(), Y.data.numpy()
    X,Y = X, Y
    N = X.size()[0]
    batch_indices = torch.LongTensor( np.random.randint(0,N+1,size=M) )
    pdb.set_trace()
    batch_xs = torch.index_select(X,0,batch_indices)
    batch_ys = torch.index_select(Y,0,batch_indices)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

however, this seems to have issues because it does not work if X,Y are NOT variables...which is really odd. I added this to the pytorch forum: https://discuss.pytorch.org/t/how-to-get-mini-batches-in-pytorch-in-a-clean-and-efficient-way/10322

Right now what I am struggling with is making this work for gpu. My most current version:

def get_batch2(X,Y,M,dtype):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    #X,Y = X.data.numpy(), Y.data.numpy()
    X,Y = X, Y
    N = X.size()[0]
    if dtype ==  torch.cuda.FloatTensor:
        batch_indices = torch.cuda.LongTensor( np.random.randint(0,N,size=M) )# without replacement
    else:
        batch_indices = torch.LongTensor( np.random.randint(0,N,size=M) ).type(dtype)  # without replacement
    pdb.set_trace()
    batch_xs = torch.index_select(X,0,batch_indices)
    batch_ys = torch.index_select(Y,0,batch_indices)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

the error:

RuntimeError: tried to construct a tensor from a int sequence, but found an item of type numpy.int64 at index (0)

I don't get it, do I really have to do:

ints = [ random.randint(0,N) for i i range(M)]

to get the integers?

It would also be ideal if the data could be a variable. It seems that it torch.index_select does not work for Variable type data.

this list of integers thing still doesn't work:

TypeError: torch.addmm received an invalid combination of arguments - got (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor), but expected one of:
 * (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)

Output:
If I'm understanding your code correctly, your get_batch2 function appears to be taking random mini-batches from your dataset without tracking which indices you've used already in an epoch. The issue with this implementation is that it likely will not make use of all of your data.

The way I usually do batching is creating a random permutation of all the possible vertices using torch.randperm(N) and loop through them in batches. For example:

n_epochs = 100 # or whatever
batch_size = 128 # or whatever

for epoch in range(n_epochs):

    # X is a torch Variable
    permutation = torch.randperm(X.size()[0])

    for i in range(0,X.size()[0], batch_size):
        optimizer.zero_grad()

        indices = permutation[i:i+batch_size]
        batch_x, batch_y = X[indices], Y[indices]

        # in case you wanted a semi-full example
        outputs = model.forward(batch_x)
        loss = lossfunction(outputs,batch_y)

        loss.backward()
        optimizer.step()

If you like to copy and paste, make sure you define your optimizer, model, and lossfunction somewhere before the start of the epoch loop.

With regards to your error, try using torch.from_numpy(np.random.randint(0,N,size=M)).long() instead of torch.LongTensor(np.random.randint(0,N,size=M)). I'm not sure if this will solve the error you are getting, but it will solve a future error.


Instruction: BCELoss for binary pixel-wise segmentation pytorch

Input:
I'm implementing a UNet for binary segmentation while using Sigmoid and BCELoss. The problem is that after several iterations the network tries to predict very small values per pixel  while for some regions it should predict values close to one (for ground truth mask region). Does it give any intuition about the wrong behavior?

Besides, there exist NLLLoss2d which is used for pixel-wise loss. Currently, I'm simply ignoring this and I'm using MSELoss() directly. Should I use NLLLoss2d with Sigmoid activation layer?

Thanks

Output:
You might want to use torch.nn.BCEWithLogitsLoss(), replacing the Sigmoid and the BCELoss function.

An excerpt from the docs tells you why its always better to use this loss function implementation.

  This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.


Instruction: How does one make sure that everything is running on GPU automatically in Pytorch?

Input:
I wanted a way with minimal amount of code such that everything in my script runs automatically in GPU (or the standard way pytorch did it). Something like:

torch.everything_to_gpu()

and then it "just works". I don't care about manually putting things in GPU etc. I just want it to do its stuff automatically (sort of the way tensorflow does it?). I did see a related question in the pytorch forum but it doesn't seem that they address my issue directly.

Right now it seems to me (from the examples I've been through) that one can do something like what I want by specifying a simple type to every torch Variable/tensor as follows:

dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

so as long as every variable/tensor is takes dtype somehow e.g.

Variable(torch.FloatTensor(x).type(dtype), requires_grad=False)

then we can use that single variable to control what is in GPU and not. The issue that I am encountering that makes things ambiguous to me if such a single command exists is when using torch.nn.Module package. For example when using

l = torch.nn.Linear(D_in,D_out)

or costume NN classes (that inherit from it). Such cases seems that the best way to deal with it is to use the:

torch.nn.Module.cuda(device_id=device_id) # device_id = None is the default

function/method. However this seems to suggest to me that there might be other hidden functions that I might not be aware of to make sure that everything does indeed run in GPU.

Thus: Is there a centralized way to make sure everything runs in some (ideally automatically) assigned GPU?

In reflection I think one thing that is confusing me is that I don't understand the model of how pytorch carriers on computations on GPU. For example, I am fairly certain tht the way MATLAB works is that if at least one thing is on GPU then all further computations will be on GPU. So I guess, I am wondering, is this how pytorch works? If possible, how does it compare to TensorFlow?

Output:
I think that there is no such thing.

From what I've seen, people usually create classes that:
i) Inherit from nn.Module.
ii) Have an attribute describing model parameters (e.g. self.opt);
iii) Set each variable/parameters as attributes (e.g. self.my_var)
iv) Then call .cuda() on it if a kind of -use_gpu parameter is set.

I also use a maybe_cuda(variable) function inside my classes in order to create variables easier (pass a Variable, return variable.cuda() if opt.cuda is True.

In fact, I did something like this (may not be perfect, but found it practical):

class MyModule(nn.Module):
    def __init__(self, opt):
        super(MyModule, self).__init__()
        self.opt = opt

    def maybe_cuda(self, variable):
        if self.opt.cuda:
            return variable.cuda()
        return variable

class Model(MyModule):
    def __init__(self, opt, other_arg):
        super(Model, self).__init__(opt)

        self.linear = nn.Linear(opt.size1, opt.size2)
        self.W_out = nn.Parameter(_____)

    def forward(self, ____):
        # create a variable, put it on GPU if possible
        my_var = self.maybe_cuda(Variable(torch.zeros(___)))


Instruction: How to install pytorch on windows subsystem for linux

Input: my windows10 has subsystem for linux of 14.04. I tried  to install pytorch on the preinstalled python2 but couldn't work.The error is: torch-0.2.0.post1-cp27-cp27m-manylinux1_x86_64.whl is not a supported wheel on this platform. I tried to install python3.6 then install pytorch with it, but still couldn't work.The error is missing module 'apt_pkg'. Anyone has idea on this?

Output: According to this it should be working now via anaconda's package manager conda.


Instruction: Pytorch vectors with empty dimensions

Input:
I noticed that some vectors in pytorch have empty dimensions like a=torch.randn(1) has size (1L,). What are these types of vectors called and why are they missing the second dimension? I'm on OSX and Ubuntu - not sure if that makes a difference.

Example:

>>> import torch
>>> a=torch.randn(1)
>>> a

 2.1358
[torch.FloatTensor of size 1]

>>> a.size()
(1L,)
>>> a=torch.randn(1).size()
>>> a
(1L,)

Output:
It is not a pytorch specific. This convention is followed in python as well. Let a vector of size (5,) and a matrix of size (5,1) is different. Although they have the same number of elements.

>>> import numpy as np
>>> a=np.random.rand(5)
>>> a.shape
>>> (5,)

>>> b=np.random.rand(5,1)
>>> (5,1)


Instruction: Matching PyTorch w/ CNTK (VGG on CIFAR)

Input:
I am trying to understand how PyTorch works and want to replicate a simple CNN training on CIFAR. The CNTK script gets to 0.76 accuracy after 168 seconds of training (10 epochs), which is similar to my MXNet script (0.75 accuracy after 153 seconds).

However, my PyTorch script is lagging behind a lot at 0.71 accuracy and 354 seconds. I appreciate I will get differences in accuracy due to stochastic weight initialisation, etc. However the difference across frameworks is much greater than difference within a framework, initialising randomly between runs.

The reasons I can think of:

MXNet and CNTK are initialized to xavier/glorot uniform; not sure how to do this in PyTorch and so perhaps the weights are initialised to 0
CNTK does gradient-clipping by default; not sure if PyTorch has the equivalent
Perhaps the bias is dropped in PyTorch by default
I use SGD with momentum; perhaps the PyTorch implementation of momentum is a bit different

Edit:

I have tried specifying the weight initialisation, however it seems to have no big effect:

self.conv1 = nn.Conv2d(3, 50, kernel_size=3, padding=1)
init.xavier_uniform(self.conv1.weight, gain=np.sqrt(2.0))
init.constant(self.conv1.bias, 0)

Output:
I try to answer your first two questions:

weight initialization: different kinds of layers have their own method, you can find the default weight initialization of all these layers in the following link: https://github.com/pytorch/pytorch/tree/master/torch/nn/modules
gradient-clipping: you might want to use torch.nn.utils.clip_grad_norm

In addition, I am curious why you don't use torchvision.transforms torch.utils.data.DataLoader and torchvision.datasets.CIFAR10 to load and preprocess your data?

There is a similar image classification tutorial of cifar for Pytorch
http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py
Hope this can help you.


Instruction: How to correctly use Numpy's FFT function in PyTorch?

Input:
I was recently introduced to PyTorch and began running through the library's documentation and tutorials.
In the "Creating extensions using numpy and scipy" tutorial, under "Parameter-less example", a sample function is created using numpy called BadFFTFunction.
The description for the function states:

"This layer doesn’t particularly do anything useful or mathematically
correct.
It is aptly named BadFFTFunction"

The function and its usage are given as:
from numpy.fft import rfft2, irfft2

class BadFFTFunction(Function):

    def forward(self, input):
        numpy_input = input.numpy()
        result = abs(rfft2(numpy_input))
        return torch.FloatTensor(result)

    def backward(self, grad_output):
        numpy_go = grad_output.numpy()
        result = irfft2(numpy_go)
        return torch.FloatTensor(result)

def incorrect_fft(input):
    return BadFFTFunction()(input)

input = Variable(torch.randn(8, 8), requires_grad=True)
result = incorrect_fft(input)
print(result.data)
result.backward(torch.randn(result.size()))
print(input.grad)

Unfortunately, I was only recently introduced to signal processing as well, and am unsure of where the (likely obvious) error is in this function.
I am wondering, how might one go about fixing this function so that its forward and backward outputs are correct?
How can BadFFTFunction be fixed so that a differentiable FFT function can be used in PyTorch?

Output:
I think the errors are: First, the function, despite having FFT in its name, only returns the amplitudes/absolute values of the FFT output, not the full complex coefficients. Also, just using the inverse FFT to compute the gradient of the amplitudes probably doesn't make much sense mathematically (?).

There is a package called pytorch-fft that tries to make an FFT-function available in pytorch. You can see some experimental code for autograd functionality here. Also note discussion in this issue.


Instruction: PyTorch RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed

Input:
I’m trying to create a basic binary classifier in Pytorch that classifies whether my player plays on the right or the left side in the game Pong. The input is an 1x42x42 image and the label is my player's side (right = 1 or left = 2). The code:

class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

net = Net(42 * 42, 100, 2)

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer_net = torch.optim.Adam(net.parameters(), 0.001)
net.train()

while True:
    state = get_game_img()
    state = torch.from_numpy(state)

    # right = 1, left = 2
    current_side = get_player_side()
    target = torch.LongTensor(current_side)
    x = Variable(state.view(-1, 42 * 42))
    y = Variable(target)
    optimizer_net.zero_grad()
    y_pred = net(x)
    loss = criterion(y_pred, y)
    loss.backward()
    optimizer.step()

The error I get:

  File "train.py", line 109, in train
    loss = criterion(y_pred, y)
  File "/home/shani/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shani/anaconda2/lib/python2.7/site-packages/torch/nn/modules/loss.py", line 321, in forward
    self.weight, self.size_average)
  File "/home/shani/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 533, in cross_entropy
    return nll_loss(log_softmax(input), target, weight, size_average)
  File "/home/shani/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 501, in nll_loss
    return f(input, target)
  File "/home/shani/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/thnn/auto.py", line 41, in forward
    output, *self.additional_args)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /py/conda-bld/pytorch_1493676237139/work/torch/lib/THNN/generic/ClassNLLCriterion.c:57

Output:
For most of deeplearning library, target(or label) should start from 0.

It means that your target should be in the range of  [0,n)  with n-classes.


Instruction: RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

Input:
On ubuntu14.04,I use pytorch with cudnn.This problem happened:

Traceback (most recent call last):
  File "main.py", line 58, in <module>
    test_detect(test_loader, nod_net, get_pbb, bbox_result_path,config1,n_gpu=config_submit['n_gpu'])
  File "/home/ubuntu/nndl/DSB2017/test_detect.py", line 52, in test_detect
    output = net(input,inputcoord)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 58, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/nndl/DSB2017/net_detector.py", line 102, in forward
    out = self.preBlock(x)#16
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 351, in forward
    self.padding, self.dilation, self.groups)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 119, in conv3d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

I have google it for severial hours an am really confused.What made this happen?

Output:
I just encountered this problem on ubuntu16.04 and solved it. My solution was to run

sudo rm -rf ~/.nv

and then reboot.


Instruction: Is there a function to extract image patches in PyTorch?

Input:
Given a batch of images, I'd like to extract all possible image patches, similar to a convolution. In TensorFlow, we can use tf.extract_image_patches to achieve this. Is there an equivalent function in PyTorch?

Thank you.

Output:
Unfortunately, there might not be a direct way to achieve your goal.
But Tensor.unfold function might be a solution.
https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837/2
This website might help you.


Instruction: PyTorch giving cuda runtime error

Input:
I have made a slight modification in my code so that it does not use DataParallel and DistributedDataParallel. The code is as follows:

import argparse
import os
import shutil
import time

import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.distributed as dist
import torch.optim
import torch.utils.data
import torch.utils.data.distributed
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torchvision.models as models

model_names = sorted(name for name in models.__dict__
    if name.islower() and not name.startswith("__")
    and callable(models.__dict__[name]))

parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')
parser.add_argument('data', metavar='DIR',
                    help='path to dataset')
parser.add_argument('--arch', '-a', metavar='ARCH', default='resnet18',
                    choices=model_names,
                    help='model architecture: ' +
                        ' | '.join(model_names) +
                        ' (default: resnet18)')
parser.add_argument('-j', '--workers', default=4, type=int, metavar='N',
                    help='number of data loading workers (default: 4)')
parser.add_argument('--epochs', default=90, type=int, metavar='N',
                    help='number of total epochs to run')
parser.add_argument('--start-epoch', default=0, type=int, metavar='N',
                    help='manual epoch number (useful on restarts)')
parser.add_argument('-b', '--batch-size', default=256, type=int,
                    metavar='N', help='mini-batch size (default: 256)')
parser.add_argument('--lr', '--learning-rate', default=0.1, type=float,
                    metavar='LR', help='initial learning rate')
parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
                    help='momentum')
parser.add_argument('--weight-decay', '--wd', default=1e-4, type=float,
                    metavar='W', help='weight decay (default: 1e-4)')
parser.add_argument('--print-freq', '-p', default=10, type=int,
                    metavar='N', help='print frequency (default: 10)')
parser.add_argument('--resume', default='', type=str, metavar='PATH',
                    help='path to latest checkpoint (default: none)')
parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
                    help='evaluate model on validation set')
parser.add_argument('--pretrained', dest='pretrained', action='store_true',
                    help='use pre-trained model')
parser.add_argument('--world-size', default=1, type=int,
                    help='number of distributed processes')
parser.add_argument('--dist-url', default='tcp://224.66.41.62:23456', type=str,
                    help='url used to set up distributed training')
parser.add_argument('--dist-backend', default='gloo', type=str,
                    help='distributed backend')

best_prec1 = 0

def main():
    global args, best_prec1
    args = parser.parse_args()

    args.distributed = args.world_size > 1

    if args.distributed:
        dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,
                                world_size=args.world_size)

    # create model
    if args.pretrained:
        print("=> using pre-trained model '{}'".format(args.arch))
        model = models.__dict__[args.arch](pretrained=True)
    else:
        print("=> creating model '{}'".format(args.arch))
        model = models.__dict__[args.arch]()

    if not args.distributed:
        if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
            #model.features = torch.nn.DataParallel(model.features)
            model.cuda()
        #else:
            #model = torch.nn.DataParallel(model).cuda()
    else:
        model.cuda()
        #model = torch.nn.parallel.DistributedDataParallel(model)

    # define loss function (criterion) and optimizer
    criterion = nn.CrossEntropyLoss().cuda()

    optimizer = torch.optim.SGD(model.parameters(), args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

    # optionally resume from a checkpoint
    if args.resume:
        if os.path.isfile(args.resume):
            print("=> loading checkpoint '{}'".format(args.resume))
            checkpoint = torch.load(args.resume)
            args.start_epoch = checkpoint['epoch']
            best_prec1 = checkpoint['best_prec1']
            model.load_state_dict(checkpoint['state_dict'])
            optimizer.load_state_dict(checkpoint['optimizer'])
            print("=> loaded checkpoint '{}' (epoch {})"
                  .format(args.resume, checkpoint['epoch']))
        else:
            print("=> no checkpoint found at '{}'".format(args.resume))

    cudnn.benchmark = True

    # Data loading code
    traindir = os.path.join(args.data, 'train')
    valdir = os.path.join(args.data, 'val')
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])

    train_dataset = datasets.ImageFolder(
        traindir,
        transforms.Compose([
            transforms.RandomSizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize,
        ]))

    if args.distributed:
        train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
    else:
        train_sampler = None

    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler)

    val_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transforms.Compose([
            transforms.Scale(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True)

    if args.evaluate:
        validate(val_loader, model, criterion)
        return

    for epoch in range(args.start_epoch, args.epochs):
        if args.distributed:
            train_sampler.set_epoch(epoch)
        adjust_learning_rate(optimizer, epoch)

        # train for one epoch
        train(train_loader, model, criterion, optimizer, epoch)

        # evaluate on validation set
        prec1 = validate(val_loader, model, criterion)

        # remember best prec@1 and save checkpoint
        is_best = prec1 > best_prec1
        best_prec1 = max(prec1, best_prec1)
        save_checkpoint({
            'epoch': epoch + 1,
            'arch': args.arch,
            'state_dict': model.state_dict(),
            'best_prec1': best_prec1,
            'optimizer' : optimizer.state_dict(),
        }, is_best)

def train(train_loader, model, criterion, optimizer, epoch):
    batch_time = AverageMeter()
    data_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    # switch to train mode
    model.train()

    end = time.time()
    for i, (input, target) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)

        target = target.cuda(async=True)
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)

        # compute output
        output = model(input_var)
        loss = criterion(output, target_var)

        # measure accuracy and record loss
        prec1, prec5 = accuracy(output.data, target, topk=(1, 5))
        losses.update(loss.data[0], input.size(0))
        top1.update(prec1[0], input.size(0))
        top5.update(prec5[0], input.size(0))

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        if i % args.print_freq == 0:
            print('Epoch: [{0}][{1}/{2}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})\t'
                  'Prec@5 {top5.val:.3f} ({top5.avg:.3f})'.format(
                   epoch, i, len(train_loader), batch_time=batch_time,
                   data_time=data_time, loss=losses, top1=top1, top5=top5))

def validate(val_loader, model, criterion):
    batch_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    # switch to evaluate mode
    model.eval()

    end = time.time()
    for i, (input, target) in enumerate(val_loader):
        target = target.cuda(async=True)
        input_var = torch.autograd.Variable(input, volatile=True)
        target_var = torch.autograd.Variable(target, volatile=True)

        # compute output
        output = model(input_var)
        loss = criterion(output, target_var)

        # measure accuracy and record loss
        prec1, prec5 = accuracy(output.data, target, topk=(1, 5))
        losses.update(loss.data[0], input.size(0))
        top1.update(prec1[0], input.size(0))
        top5.update(prec5[0], input.size(0))

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        if i % args.print_freq == 0:
            print('Test: [{0}/{1}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})\t'
                  'Prec@5 {top5.val:.3f} ({top5.avg:.3f})'.format(
                   i, len(val_loader), batch_time=batch_time, loss=losses,
                   top1=top1, top5=top5))

    print(' * Prec@1 {top1.avg:.3f} Prec@5 {top5.avg:.3f}'
          .format(top1=top1, top5=top5))

    return top1.avg

def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'):
    torch.save(state, filename)
    if is_best:
        shutil.copyfile(filename, 'model_best.pth.tar')

class AverageMeter(object):
    """Computes and stores the average and current value"""
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

def accuracy(output, target, topk=(1,)):
    """Computes the precision@k for the specified values of k"""
    maxk = max(topk)
    batch_size = target.size(0)

    _, pred = output.topk(maxk, 1, True, True)
    pred = pred.t()
    correct = pred.eq(target.view(1, -1).expand_as(pred))

    res = []
    for k in topk:
        correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
        res.append(correct_k.mul_(100.0 / batch_size))
    return res

if __name__ == '__main__':
    main()

And, when I run this code on a set of images with the alexnet neuralnet architecture, it gives a weird cuda error, which is as follows:

=> creating model 'alexnet'
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=70 error=30 : unknown error
Traceback (most recent call last):
  File "imagenet2.py", line 319, in <module>
    main()
  File "imagenet2.py", line 87, in main
    model.cuda()
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 147, in cuda
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 124, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 147, in <lambda>
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python2.7/dist-packages/torch/_utils.py", line 66, in _cuda
    return new_type(self.size()).copy_(self, async)
  File "/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py", line 266, in _lazy_new
    _lazy_init()
  File "/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py", line 85, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/torch/lib/THC/THCGeneral.c:70

Command used for running the code: python imagenet.py --world-size 1 --arch 'alexnet' <image_folder>

Where did I go wrong?

PS: Running on an AWS g2.2xlarge Ubuntu instance.

The CUDA version is as follows:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

Output:
CUDNN gives useless error messages. For debug, test your net on CPU using net.cpu() or just simple remove the net.cuda(). You will have to do the same with training, validation and output variables.
It seams the problem is that you used pre-trained AlexNet on a images of size different than 224x224. According to the documentation it should work as long as the image size is at least 224x224.
This is probably a tensor shaping problem due to a hard-coded parameter in pytorch's implementation of AlexNet. In vision/torchvision/models/alexnet.py at line 44 it says

x = x.view(x.size(0), 256 * 6 * 6)

change it to

x = x.view(x.size(0), -1)

This should allow it to work with different images sizes.

I subbmitted this modification to the github repository, but I guess it has not been updated yet.


Instruction: How to compute cross entropy loss for binary classification in Pytorch ？

Input:
For binary classification, my output and label is like this

output = [0.7, 0.3, 0.1, 0.9 ... ]
label = [1, 0, 0, 1 ... ]

where the output is the probability for precited label = 1

And I want a cross entropy like this:

def cross_entropy(output, label):
    return sum(-label * log(output) - (1 - label) * log(1 - output))

However, this gives me a NaN error because that in log(output) the output might be zero.

I know there is torch.nn.CrossEntropyLoss however it does not apply for my data format here.

Output:
import torch
import torch.nn.functional as F
def my_binary_cross_entrophy(output,label):
    label = label.float()
    #print(label)
    loss = 0
    for i in range(len(label)):
        loss += -(label[i]*math.log(output[i])+(1-label[i])*math.log(1-output[i]))
        #print(loss)
    return loss/len(label)

label1 = torch.randint(0,2,(3,)).float()
output = torch.rand(3)
my_binary_cross_entrophy(output,label1)

The value it returned is the same as F.binary_cross_entropy value.

F.binary_cross_entropy(output,label1)


Instruction: Seq2seq pytorch Inference slow

Input:
I tried the seq2seq pytorch implementation available here seq2seq . After profiling the evaluation(evaluate.py) code, the piece of code taking longer time was the decode_minibatch method

def decode_minibatch(
    config,
    model,
    input_lines_src,
    input_lines_trg,
    output_lines_trg_gold
):
    """Decode a minibatch."""
    for i in xrange(config['data']['max_trg_length']):

        decoder_logit = model(input_lines_src, input_lines_trg)
        word_probs = model.decode(decoder_logit)
        decoder_argmax = word_probs.data.cpu().numpy().argmax(axis=-1)
        next_preds = Variable(
            torch.from_numpy(decoder_argmax[:, -1])
        ).cuda()

        input_lines_trg = torch.cat(
            (input_lines_trg, next_preds.unsqueeze(1)),
            1
        )

return input_lines_trg

Trained the model on GPU and loaded the model in CPU mode to make inference. But unfortunately, every sentence seems to take ~10sec. Is slow prediction expected on pytorch?

Any fixes, suggestions to speed up would be much appreciated. Thanks.

Output:
One solution for slow performance may be to use a toolkit optimized for the inference, such as OpenVINO. OpenVINO is optimized for Intel hardware but it should work with any CPU. It optimizes the inference performance by e.g. graph pruning or fusing some operations together.
You can find a full tutorial on how to convert the PyTorch model here (FastSeg) and here (BERT). Some snippets below.
Install OpenVINO
The easiest way to do it is using PIP. Alternatively, you can use this tool to find the best way in your case.
pip install openvino-dev[pytorch,onnx]

Save your model to ONNX
OpenVINO cannot convert PyTorch model directly for now but it can do it with ONNX model. This sample code assumes the model is for computer vision.
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(model, dummy_input, "model.onnx", opset_version=11)

Use Model Optimizer to convert ONNX model
The Model Optimizer is a command line tool which comes from OpenVINO Development Package so be sure you have installed it. It converts the ONNX model to OV format (aka IR), which is a default format for OpenVINO. It also changes the precision to FP16 (to further increase performance). Run in command line:
mo --input_model "model.onnx" --input_shape "[1, 3, 224, 224]" --mean_values="[123.675, 116.28 , 103.53]" --scale_values="[58.395, 57.12 , 57.375]" --data_type FP16 --output_dir "model_ir"

Run the inference on the CPU
The converted model can be loaded by the runtime and compiled for a specific device e.g. CPU or GPU (integrated into your CPU like Intel HD Graphics). If you don't know what is the best choice for you, just use AUTO.
# Load the network
ie = Core()
model_ir = ie.read_model(model="model_ir/model.xml")
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")

# Get output layer
output_layer_ir = compiled_model_ir.output(0)

# Run inference on the input image
result = compiled_model_ir([input_image])[output_layer_ir]

Disclaimer: I work on OpenVINO.


Instruction: "expected CPU tensor(got CUDA tensor)" error for PyTorch

Input:
In my inference code with Trained PyTorch model, What's wrong ?

There is an runtime error message: "expected CPU tensor(got CUDA tensor)"

import torch
import torch.nn as nn
#from __future__ import print_function
import argparse
from PIL import Image
import torchvision.models as models
import skimage.io
from torch.autograd import Variable as V
from torch.nn import functional as f
from torchvision import transforms as trn

# define image transformation
centre_crop = trn.Compose([
trn.ToPILImage(),
trn.Scale(256),
trn.CenterCrop(224),
trn.ToTensor(),
trn.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

filename=r'ed91.png'
img = skimage.io.imread(filename)

x = V(centre_crop(img).unsqueeze(0), volatile=True)

model = models.__dict__['resnet18']()
model = torch.nn.DataParallel(model).cuda()

model = torch.load('mw_model0831.pth')
#model.load_state_dict(checkpoint['state_dict'])

#best_prec1 = checkpoint['best_prec1']
logit = model(x)

print(logit)
print(len(logit))
h_x = f.softmax(logit).data.squeeze()

How can I fix this ?

Output:
The error is because the model is on GPU while your input image x is on CPU. You have to make sure that they are both on GPU or CPU.

Also, model.cuda() and x.cuda() behaves slightly differently: model.cuda() will put the model on GPU, but x.cuda() just return a new Variable on GPU, leaving the original x untouched. You have to assign the return value to x explicitly.

You can find a detailed discussion here.


Instruction: How to combine multiple models together?

Input:
I am trying to "parallelize" Neural Network models to speed up training. One idea I had was to run two models on two computers and combine the results somehow.

Is this possible? If not, what are the options to parallelize model training on two computers?

I am open to use any neural network framework.

Output:
I think you mean distributed tensorflow?

See official document: https://www.tensorflow.org/deploy/distributed


Instruction: PyTorch equivalence for softmax_cross_entropy_with_logits

Input: I was wondering is there an equivalent PyTorch loss function for TensorFlow's softmax_cross_entropy_with_logits?

Output:
is there an equivalent PyTorch loss function for TensorFlow's softmax_cross_entropy_with_logits?

torch.nn.functional.cross_entropy
This takes logits as inputs (performing log_softmax internally). Here "logits" are just some values that are not probabilities (i.e. not necessarily in the interval [0,1]).
But, logits are also the values that will be converted to probabilities.
If you consider the name of the tensorflow function you will understand it is pleonasm (since the with_logits part assumes softmax will be called).
In the PyTorch implementation looks like this:
loss = F.cross_entropy(x, target)

Which is equivalent to :
lp = F.log_softmax(x, dim=-1)
loss = F.nll_loss(lp, target)

It is not F.binary_cross_entropy_with_logits because this function assumes multi label classification:
F.sigmoid + F.binary_cross_entropy = F.binary_cross_entropy_with_logits

It is not torch.nn.functional.nll_loss either because this function takes log-probabilities (after log_softmax()) not logits.


Instruction: Resized copy of Pytorch Tensor/Dataset

Input:
I have a homemade dataset with a few million rows.   I am trying to make truncated copies.  So I clip the tensors that I'm using to make the original dataset and create a new dataset.   However, when I save the new dataset, which is only 20K rows, it's the same size on disk as the original dataset.  Otherwise everything seems kosher, including, when I check, the size of the new tensors.   What am I doing wrong?

#original dataset - 2+million rows
dataset = D.TensorDataset(training_data, labels)
torch.save(dataset, filename)

#20k dataset for experiments
d = torch.Tensor(training_data[0:20000])
l = torch.Tensor(labels[0:20000])
ds_small = D.TensorDataset(d,l)
#this is the same size as the one above on disk... approx 1.45GB
torch.save(ds_small, filename_small)

Thanks

Output:
In your code d and training_data share the same memory, even if you use slicing during the creation of d. I don't know why this is the case, but answer anyway to give you a solution:

d = x[0:10000].clone()
l = y[0:10000].clone()

clonewill give you Tensors with a memory independent from the old Tensor's and the file size will be much smaller.

Note that using torch.Tensor() is not necessary when creating d and l since training_data and labels are already tensors.


Instruction: Pytorch load model

Input: I used tensorflow some days ago. To build conv layers with fixed weights is easy, just pass the weight kernel to conv2d(). And it is convenient to load pretrained models such as VGG19. But I found it did't work that way using pytorch, because conv2d() doesn't accept an explicit kernel but a kernel size. So I wonder if there is any possibility that we can reuse the weights in VGG19 by simply passing it to a method like conv2d(). Any reply will be appreciated.

Output:
I can see that you have 2 questions . How to use a pre-trained model like VGG in pyTorch and how to set the weights for a particular layer like nn.conv2d().

For creating a pretrained Vgg model you can use the below code.

from torchvision import models
model_vgg = models.vgg16(pretrained=True)
for param in model_vgg.parameters():
    param.requires_grad = False

In PyTorch you implement neural network subclassing nn.Module which contains the parameters() function which returns all the weights associated with the network.

Setting the weights of a particular layer .

decoder = nn.Linear(10, 100)
decoder.weight = #Do anything which is valid.

You can check my code here to know more on how to use a trained model.


Instruction: How to get the inner module of Unet?

Input:
I created a UNet with the the UnetGenerator. You can find the resulting structure here.

How do I get the module Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))?
How do I get the module (5): ConvTranspose2d(256, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))?

I want to get the inner modules to change the attributes of certain layers.
I tried something like net.modules(i).modules(i), but it doesn't work. I refer the docs, I haven't get a good idea to do it.

My initial intention is to change the attributes of certain layers when training. I may add a custom layer myLayer, in which self.mode='normal'. When training, I hope I can change its attribute myLayer.mode = 'capture' to make it change its behaviour in training.

Output:
All the subclasses of nn.Module has an attribute called children. Which you will be able to access using the code below.

unet = UnetGenerator(512,512,4)
layers = list(unet.children())
len(layers)

For the network I created using the above code , I can access one of the layers inside the network and change the properties like below.

l = layers[0]
conv = list(l.children())[0][0]
conv.kernel_size = (2,2)

If you are training the network without using any pre-trained weights then you could make the changes in the source code before you create the network object.


Instruction: How to correctly implement a batch-input LSTM network in PyTorch?

Input:
This release of PyTorch seems provide the PackedSequence for variable lengths of input for recurrent neural network. However, I found it's a bit hard to use it correctly.

Using pad_packed_sequence to recover an output of a RNN layer which were fed by pack_padded_sequence, we got a T x B x N tensor outputs where T is the max time steps, B is the batch size and N is the hidden size. I found that for short sequences in the batch, the subsequent output will be all zeros.

Here are my questions.

For a single output task where the one would need the last output of all the sequences, simple outputs[-1] will give a wrong result since this tensor contains lots of zeros for short sequences. One will need to construct indices by sequence lengths to fetch the individual last output for all the sequences. Is there more simple way to do that?
For a multiple output task (e.g. seq2seq), usually one will add a linear layer N x O and reshape the batch outputs T x B x O into TB x O and compute the cross entropy loss with the true targets TB (usually integers in language model). In this situation, do these zeros in batch output matters?

Output:
Question 1 - Last Timestep

This is the code that i use to get the output of the last timestep. I don't know if there is a simpler solution. If it is, i'd like to know it. I followed this discussion and grabbed the relative code snippet for my last_timestep method. This is my forward.

class BaselineRNN(nn.Module):
    def __init__(self, **kwargs):
        ...

    def last_timestep(self, unpacked, lengths):
        # Index of the last output for each sequence.
        idx = (lengths - 1).view(-1, 1).expand(unpacked.size(0),
                                               unpacked.size(2)).unsqueeze(1)
        return unpacked.gather(1, idx).squeeze()

    def forward(self, x, lengths):
        embs = self.embedding(x)

        # pack the batch
        packed = pack_padded_sequence(embs, list(lengths.data),
                                      batch_first=True)

        out_packed, (h, c) = self.rnn(packed)

        out_unpacked, _ = pad_packed_sequence(out_packed, batch_first=True)

        # get the outputs from the last *non-masked* timestep for each sentence
        last_outputs = self.last_timestep(out_unpacked, lengths)

        # project to the classes using a linear layer
        logits = self.linear(last_outputs)

        return logits

Question 2 - Masked Cross Entropy Loss

Yes, by default the zero padded timesteps (targets) matter. However, it is very easy to mask them. You have two options, depending on the version of PyTorch that you use.

PyTorch 0.2.0: Now pytorch supports masking directly in the CrossEntropyLoss, with the ignore_index argument. For example, in language modeling or seq2seq, where i add zero padding,  i mask the zero padded words (target) simply like this:

loss_function = nn.CrossEntropyLoss(ignore_index=0)
PyTorch 0.1.12 and older: In the older versions of PyTorch, masking was not supported, so you had to implement your own workaround. I solution that i used, was masked_cross_entropy.py, by jihunchoi. You may be also interested in this discussion.


Instruction: Do I need to define the backward() in custom loss function?

Input:
I have already to define my own loss function. It does work. The feedforward may not have problem. But I am not sure whether it is correct because I don't define the backward().

class _Loss(nn.Module):
    def __init__(self, size_average=True):
        super(_Loss, self).__init__()
        self.size_average = size_average
class MyLoss(_Loss):
    def forward(self, input, target):
        loss = 0
        weight = np.zeros((BATCH_SIZE,BATCH_SIZE))
        for a in range(BATCH_SIZE):
            for b in range(BATCH_SIZE):
                weight[a][b] = get_weight(target.data[a][0])
        for i in range(BATCH_SIZE):
            for j in range(BATCH_SIZE):
                a_ij= (input[i]-input[j]-target[i]+target[j])*weight[i,j]
                loss += F.relu(a_ij)
        return loss

The question I want to ask is that

1) Do I need to define the backward() to loss function?

2) How to define the backward()?

3) Is there are any way to do the index of the data while doing SGD in torch?

Output:
You can write a loss function like below.

def mse_loss(input, target):
            return ((input - target) ** 2).sum() / input.data.nelement()

You do not need to implement backward function. All the above parameters of the loss functions should be PyTorch variables and the rest is taken care by torch.autograd function.


Instruction: Pytorch: define custom function

Input:
I wanted to write my own activation function, but I got a problem. Saying the matrix multiplication will call .data. I searched but got little useful information. Any help will be appreciated. The error information is

 Traceback (most recent call last):
      File "defineAutogradFuncion.py", line 126, in <module>
        test = gradcheck(argmin, input, eps=1e-6, atol=1e-4)
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/autograd/gradcheck.py", line 154, in gradcheck
        output = func(*inputs)
      File "defineAutogradFuncion.py", line 86, in forward
        output = output.mm(dismap).squeeze(-1)
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/autograd/variable.py", line 578, in mm
        output = Variable(self.data.new(self.data.size(0), matrix.data.size(1)))
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/tensor.py", line 374, in data
        raise RuntimeError('cannot call .data on a torch.Tensor: did you intend to use autograd.Variable?')
    RuntimeError: cannot call .data on a torch.Tensor: did you intend to use autograd.Variable?

    class Softargmin(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """
    @staticmethod
    def forward(self, input):
        """
        In the forward pass we receive a Tensor containing the input and return a
        Tensor containing the output. You can cache arbitrary Tensors for use in the
        backward pass using the save_for_backward method.
        """
        #P = Fun.softmax(-input)
        inputSqueeze = input.squeeze(-1)
        P = Fun.softmax(-inputSqueeze)
        self.save_for_backward(P)

        output = P.permute(0,2,3,1)
        dismap = torch.arange(0,output.size(-1)+1).unsqueeze(1)
        output = output.mm(dismap).squeeze(-1)
       return output
    @staticmethod
    def backward(self, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        P, = self.saved_tensors
        P = P.unsqueeze(-1)
        Pk = torch.squeeze(P,-1).permute(0,2,3,1)
        k = torch.arange(0,Pk.size(-1)+1).unsqueeze(1)
        sumkPk = Pk.mm(k)
        sumkPk = sumkPk.unsqueeze(1).expand(P.size())
        i = torch.arange(0,Pk.size(-1)+1).view(1,-1,1,1,1).expand(P.size())
        grad_output_expand =grad_output.unsqueeze(-1).unsqueeze(1).expand(P.size())
        grad_input = grad_output_expand*P*(sumkPk-i)
        return grad_input

Output:
The most basic element in PyTorch is a Tensor, which is the equivalent of numpy.ndarray with the only difference being that a Tensor can be put onto a GPU for any computation.

A Variable is a wrapper around Tensor that contains three attributes: data, grad and grad_fn. data contains the original Tensor; grad contains the derivative/gradient of some value with respect to this Variable; and grad_fn is a pointer to the Function object that created this Variable. The grad_fn attribute is actually the key for autograd to work properly since PyTorch uses those pointers to build the computation graph at each iteration and carry out the differentiations for all Variables in your graph accordingly. This is not only about differentiating correctly through this custom Function object you are creating.

Hence whenever you create some Tensor in your computation that requires differentiation, wrap it as a Variable. First, this would enable the Tensor to be able to save the resulting derivative/gradient value after you call backward(). Second, this helps autograd build a correct computation graph.

Another thing to notice is that whenever you send a Variable into your computation graph, any value that is computed using this Variable will automatically be a Variable. So you don't have to manually wrap all Tensors in your computation graph.

You might want to take a look at this.

Going back to your error, it's a little difficult to figure out what is really causing the trouble because you are not showing all of your code (information like how you are using this custom Function in your computation graph), but I suspect that what most likely has happened is that you used this Function in a subgraph that required to be differentiated through, when PyTorch used numerical gradient check on your model to see if the differentiation is correct, it assumed that every node in that subgraph was a Variable because that is necessary for differentiation through that subgraph to happen, then it tried to call the data attribute of that Variable, most likely because that value is used somewhere in the differentiation, and failed because that node was in fact a Tensor and did not have a data attribute.


Instruction: Adding a linear layer to an existing model on Pytorch

Input:
I'm trying to add a new layer to an existing network (as the first layer) and train it on the original input. When I add a convolutional layer everything works perfectly but when I change it to linear it doesn't seem to train. Any ideas why?
Here is the whole network:

class ActorCritic(torch.nn.Module): #original model
    def __init__(self, num_inputs, action_space):
        super(ActorCritic, self).__init__()
        self.conv1 = nn.Conv2d(num_inputs, 32, 3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv4 = nn.Conv2d(32, 32, 3, stride=2, padding=1)

        self.lstm = nn.LSTMCell(32 * 3 * 3, 256)

        num_outputs = action_space.n
        self.critic_linear = nn.Linear(256, 1)
        self.actor_linear = nn.Linear(256, num_outputs)

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.conv1(inputs))
        x = F.elu(self.conv2(x))
        x = F.elu(self.conv3(x))
        x = F.elu(self.conv4(x))
        x = x.view(-1, 32 * 3 * 3)
        hx, cx = self.lstm(x, (hx, cx))
        x = hx
        return self.critic_linear(x), self.actor_linear(x), (hx, cx)

class TLModel(torch.nn.Module): #new model
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Linear(1*1*42*42, 1*1*42*42)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs.view(-1, 1*1*42*42)))
        return self.pretrained_model((x.view(1,1,42,42), (hx, cx)))

I tried different activation functions (not just elu). it works with conv:

 class TLModel(torch.nn.Module):
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Conv2d(num_inputs, num_inputs, 1)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs))
        return self.pretrained_model((x, (hx, cx)))

The number of inputs is 1 and the size of an input is 1x1x42x42

Output:
It would be useful if you had supplied the error message. From what you have written, I can only guess that you have forgotten to squeeze your input. You write your input is of size 1x1x42x42, i.e. it is 4-dimensional. nn.Conv2D expects a 4-dimensional input. nn.Linear instead expects a 2-dimensional input.

Therefore, try to call input = input.squeeze() before feeding it to your model. This removes singleton dimensions, and hence will make your input 2-dimensional as there are two singleton dimensions.

As a side note, nn.Linear expects input of dimension  batch_size x feat_dim. Does a linear layer really make sense on your data?
As another side note, when people usually add layers to networks they put them in the end not the beginning, but I trust you have good reasons to do so and know what you're doing :)

Good Luck!


Instruction: Pytorch maxpooling over channels dimension

Input: I was trying to build a cnn to with Pytorch, and had difficulty in maxpooling. I have taken the cs231n held by Stanford. As I recalled, maxpooling can be used as a dimensional deduction step, for example, I have this (1, 20, height, width) input ot max_pool2d (assuming my batch_size is 1). And if I use (1, 1) kernel, I want to get output like this: (1, 1, height, width), which means the kernel should be slide over the channel dimension. However, after checking the pytorch docs, it says the kernel slides over height and width. And thanks to @ImgPrcSng on Pytorch forum who told me to use max_pool3d, and it turned out worked well. But there is still a reshape operation between the output of the conv2d layer and the input of the max_pool3d layer. So it is hard to be aggregated into a nn.Sequential, so I wonder is there another way to do this?

Output:
Would something like this work?
from torch.nn import MaxPool1d
import torch.nn.functional as F

class ChannelPool(MaxPool1d):
    def forward(self, input):
        n, c, w, h = input.size()
        input = input.view(n, c, w * h).permute(0, 2, 1)
        pooled = F.max_pool1d(
            input,
            self.kernel_size,
            self.stride,
            self.padding,
            self.dilation,
            self.ceil_mode,
            self.return_indices,
        )
        _, _, c = pooled.size()
        pooled = pooled.permute(0, 2, 1)
        return pooled.view(n, c, w, h)

Or, using einops
from torch.nn import MaxPool1d
import torch.nn.functional as F
from einops import rearrange

class ChannelPool(MaxPool1d):
    def forward(self, input):
        n, c, w, h = input.size()
        pool = lambda x: F.max_pool1d(
            x,
            self.kernel_size,
            self.stride,
            self.padding,
            self.dilation,
            self.ceil_mode,
            self.return_indices,
        )
        return rearrange(
            pool(rearrange(input, "n c w h -> n (w h) c")),
            "n (w h) c -> n c w h",
            n=n,
            w=w,
            h=h,
        )


Instruction: Pytorch: Trying to apply the transform to a numpy array... fails with an error

Input:
Any help will be much appreciated. The code in transforms.py says that the transformation should/would apply to PIL images as well as ndarrays.
Given the transforms:

data_transforms = {
    'train': transforms.Compose([
        transforms.Scale(256),
        transforms.Pad(4,0),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Scale(256),
        transforms.Pad(4,0),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

I wish to apply the transform on ndarrays that I obtained from some other code. Let's say it is x_data, whose shape is (1000,120,160,3) where the dimensions are (total rows, width, height, channels)

doing the following fails (All I'm trying to do is apply a transformation) :

foo = data_transforms['train']
bar = foo(x_data[0])

with the following message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-93-a703e3b9c76d> in <module>()
----> 1 foo(x_data[1])

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torchvision-0.1.9-py3.5.egg/torchvision/transforms.py in __call__(self, img)
     32     def __call__(self, img):
     33         for t in self.transforms:
---> 34             img = t(img)
     35         return img
     36

~/anaconda3/envs/pytorch/lib/python3.5/site-packages/torchvision-0.1.9-py3.5.egg/torchvision/transforms.py in __call__(self, img)
    185         """
    186         if isinstance(self.size, int):
--> 187             w, h = img.size
    188             if (w <= h and w == self.size) or (h <= w and h == self.size):
    189                 return img

TypeError: 'int' object is not iterable

Output:
Most transforms method take only PIL objects as input. But you can add another transform called transforms.ToPILImage() which takes an nd-array as input, to convert an nd-array to PIL object. So in your case, dictionary variable should become:

data_transforms = {
'train': transforms.Compose([
    transforms.ToPILImage()
    transforms.Scale(256),
    transforms.Pad(4,0),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
'val': transforms.Compose([
    transforms.Scale(256),
    transforms.Pad(4,0),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
}

Note that these transformations work sequentially. So it's necessary you add the toPILImage transform as the first transformation. Hence your nd-array is first converted to PIL object, and then other transformations are applied.


Instruction: Loss is not decreasing for convolutional autoencoder

Input:
I'm trying to train a convolutional autoencoder to encode and decode a piano roll representation of monophonic midi clips. I reduced the note range to 3 octaves, divide songs into 100 time step pieces (where 1 time step = 1/100th of a second), and train the net in batches of 3 pieces.

I'm using Adagrad as my optimizer, and MSE as my loss function. The loss is huge, and I see no decrease in average loss even after hundreds of training examples are fed in.

Here's my code:

"""
Most absolutely simple assumptions:
  - not changing the key of any of the files
  - not changing the tempo of any of the files

- take blocks of 36 by 100
- divide up all songs by this amount, cutting off any excess from the
end, train
"""
from __future__ import print_function
import cPickle as pickle
import numpy as np
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from reverse_pianoroll import piano_roll_to_pretty_midi as pr2pm

N = 1000
# load a NxMxC dataset
    # N: Number of clips
    # M: Piano roll size, the number of midi notes that could possibly be 'on'
    # C: Clip length, in 100ths of a second
dataset = pickle.load(open('mh-midi-data.pickle', 'rb'))
######## take a subset of the data for training ######
# based on the mean and standard deviation of non zero entries in the data, I've
# found that the most populous, and thus best range of notes to take is from
# 48 to 84 (C2 - C5); this is 3 octaves, which is much less than the original
# 10 and a half. Additionally, we're going to take a subsample of 1000 because
# i'm training on my macbook and the network is pretty simple
######################################################
dataset = dataset[:, :, 48:84, :]
dataset = dataset[:N]
######################################################

midi_dim, clip_len = dataset.shape[2:]

class Autoencoder(nn.Module):
    def __init__(self, **kwargs):
        super(Autoencoder, self).__init__(**kwargs)
        # input is 3 x 1 x 36 x 100
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=14, kernel_size=(midi_dim, 2))
        # now transformed to 3 x 14 x 1 x 99
        self.conv2 = nn.Conv2d(in_channels=14, out_channels=77, kernel_size=(1, 4))
        # now transformed to 3 x 77 x 1 x 96
        input_size = 3*77*1*96
        self.fc1 = nn.Linear(input_size, input_size/2)
        self.fc2 = nn.Linear(input_size/2, input_size/4)
        self.fc3 = nn.Linear(input_size/4, input_size/2)
        self.fc4 = nn.Linear(input_size/2, input_size)
        self.tconv2 = nn.ConvTranspose2d(in_channels=77, out_channels=14, kernel_size=(1, 4))
        self.tconv1 = nn.ConvTranspose2d(in_channels=14, out_channels=1, kernel_size=(midi_dim, 2))
        self.sigmoid = nn.Sigmoid()
        return

    def forward(self, x):
        # print("1: {}".format(x.size()))
        x = F.relu(self.conv1(x))
        # print("2: {}".format(x.size()))
        x = F.relu(self.conv2(x))
        # print("3: {}".format(x.size()))
        x = x.view(-1, np.prod(x.size()[:]))
        # print("4: {}".format(x.size()))
        x = F.relu(self.fc1(x))
        # print("5: {}".format(x.size()))
        h = F.relu(self.fc2(x))
        # print("6: {}".format(h.size()))
        d = F.relu(self.fc3(h))
        # print("7: {}".format(d.size()))
        d = F.relu(self.fc4(d))
        # print("8: {}".format(d.size()))
        d = d.view(3, 77, 1, 96)
        # print("9: {}".format(d.size()))
        d = F.relu(self.tconv2(d))
        # print("10: {}".format(d.size()))
        d = self.tconv1(d)
        d = self.sigmoid(d)
        # print("11: {}".format(d.size()))
        return d

net = Autoencoder()
loss_fn = nn.MSELoss()
# optimizer = optim.SGD(net.parameters(), lr=1e-3, momentum=0.9)
optimizer = optim.Adagrad(net.parameters(), lr=1e-3)

batch_count = 0
avg_loss = 0.0
print_every = 3
print("Beginning Training")
for epoch in xrange(2):
    # for i, clip in enumerate(dataset):
    for i in xrange(len(dataset)/3):
        batch = dataset[(3*i):(3*i + 3), :, :]
        # get the input, wrap it in a Variable
        inpt = Variable(torch.from_numpy(batch).type(torch.FloatTensor))

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outpt = net(inpt)
        loss = loss_fn(outpt, inpt)
        loss.backward()
        optimizer.step()

        # print stats out
        avg_loss += loss.data[0]
        if batch_count % print_every == print_every - 1:
           print('epoch: %d, batch_count: %d, loss: %.3f'%(
                epoch + 1, batch_count + 1, avg_loss / print_every))
           avg_loss = 0.0
        batch_count += 1

print('Finished Training')

I'm really a beginner with this stuff, so any advice would be greatly appreciated.

Output: Double check that you normalize your inpt to be in the range of 0 to 1. For instance, if you are working with images you could just divide inpt variable by 255.


Instruction: How to apply the torch.inverse() function of PyTorch to every sample in the batch?

Input:
This may seem like a basic question, but I am unable to work it through.

In the forward pass of my neural network, I have an output tensor of shape 8x3x3, where 8 is my batch size. We can assume each 3x3 tensor to be a non-singular matrix. I need to find the inverse of these matrices.
The PyTorch inverse() function only works on square matrices. Since I now have 8x3x3, how do I apply this function to every matrix in the batch in a differentiable manner?

If I iterate through the samples and append the inverses to a python list, which I then convert to a PyTorch tensor, should it be a problem during backprop? (I am asking since converting PyTorch tensors to numpy to perform some operations and then back to a tensor won't compute gradients during backprop for such operations)

I also get the following error when I try to do something like that.

a = torch.arange(0,8).view(-1,2,2)
b = [m.inverse() for m in a]
c = torch.FloatTensor(b)

  TypeError: 'torch.FloatTensor' object does not support indexing

Output:
EDIT:

As of Pytorch version 1.0, torch.inverse now supports batches of tensors. See here. So you can simply use the built-in function torch.inverse

OLD ANSWER

There are plans to implement batched inverse soon. For discussion, see for example issue 7500 or issue 9102. However, as of the time of writing, the current stable version (0.4.1), no batch inverse operation is available.

Having said that, recently batch support for torch.gesv was added. This can be (ab)used to define your own batched inverse operation along the following lines:

def b_inv(b_mat):
    eye = b_mat.new_ones(b_mat.size(-1)).diag().expand_as(b_mat)
    b_inv, _ = torch.gesv(eye, b_mat)
    return b_inv

I found that this gives good speed-ups over a for loop when running on GPU.


Instruction: Implementing a loss function (MSVE) in Reinforcement learning

Input:
I am trying to build a temporal difference learning agent for Othello. While the rest of my implementation seems to run as intended I am wondering about the loss function used to train my network. In Sutton's book "Reinforcement learning: An Introduction", the Mean Squared Value Error (MSVE is presented as the standard loss function. It is basically a Mean Square Error multiplied with the on policy distribution. (Sum over all states s ( onPolicyDistribution(s) * [V(s) - V'(s,w)]² ) )

My question is now: How do I obtain this on policy distribution when my policy is an e-greedy function of a learned value function? Is it even necessary and what's the issue if I just use an MSELoss instead?

I'm implementing all of this in pytorch, so bonus points for an easy implementation there :)

Output: As you mentioned, in your case, it sounds like you are doing Q-learning, so you do not need to do policy gradient as described in Sutton's book. That is need when you are learning a policy. You are not learning a policy, you are learning a value function and using that to act.


Instruction: How can I extract the feature vector of the last hidden layer of the Alex net in pytorch?

Input:
I’ve trained an Alex net in pytorch and I want to extract feature vectors from the layers
What is the function which I should use?

Output:
In my opinion, you can define a function in the model class which takes input and output the features you like, for example:

class model(nn.Module):
    def __init__(self):
        # init codes
    def forward(self, input):
        # forward codes
    def yourfunc(self, input):
        # codes
        #return feature1, feature2

This yourfunc only takes your input and outputs features you need, it does no backward computation. You just need to call it wherever you need. And I don't think there are built-in functions in PyTorch can do that since it is easy to implement by yourself.


Instruction: How to implement LSTM layer with multiple cells in Pytorch?

Input:
I intend to implement an LSTM with 2 layers and 256 cells in each layer. I am trying to understand the PyTorch LSTM framework for the same. The variables in torch.nn.LSTM that I can edit are input_size, hidden_size, num_layers, bias, batch_first, dropout and bidirectional.

However, how do I have multiple cells in a single layer?

Output:
These cells will be automatically unrolled based on your sequence size in the input. Please check out this code:

# One cell RNN input_dim (4) -> output_dim (2). sequence: 5, batch 3
# 3 batches 'hello', 'eolll', 'lleel'
# rank = (3, 5, 4)
inputs = Variable(torch.Tensor([[h, e, l, l, o],
                                [e, o, l, l, l],
                                [l, l, e, e, l]]))
print("input size", inputs.size())  # input size torch.Size([3, 5, 4])

# Propagate input through RNN
# Input: (batch, seq_len, input_size) when batch_first=True
# B x S x I
out, hidden = cell(inputs, hidden)
print("out size", out.size())  # out size torch.Size([3, 5, 2])

You can find more examples at https://github.com/hunkim/PyTorchZeroToAll/.


Instruction: Is it possible to train pytorch and tensorflow model together on one GPU?

Input:
I have a pytorch model and a tensorflow model, I want to train them together  on one GPU, following the process bellow: input --> pytorch model--> output_pytorch --> tensorflow model --> output_tensorflow --> pytorch model.

Is is possible to do this? If answer is yes, is there any problem which I will encounter?

Thanks in advance.

Output:
I haven't done this but it is possible but implementing is can be a little bit.
You can consider each network as a function, you want to - in some sense - compose these function to form your network, to do this you can compute the final function by just giving result of one network to the other and then use chain-rule to compute the derivatives(using symbolic differentiation from both packages).

I think a good way for implementing this you might be to wrap TF models as a PyTorch Function and use tf.gradients for computing the backward pass.
Doing gradient updates can really get hard (because some variables exist in TF's computation graph) you can turn TF variables to PyTorch Variable turn them into placeholdes in TF computation graph, feed them in feed_dict and update them using PyTorch mechanisms, but I think it would be really hard to do, instead if you do your updates inside backward method of the function you might be able to do the job(it is really ugly but might do the job).


Instruction: L1 norm as regularizer in Pytorch

Input:
I need to add an L1 norm as a regularizer to create a sparsity condition in my neural network. I would like to train my network for classification. I tried to construct an L1 norm by myself, like here, but it didn't work.
I need to add the regularizer after ConvTranspose2d, something like this Keras example:
model.add(Dense(64, input_dim=64,
            kernel_regularizer=regularizers.l2(0.01),
            activity_regularizer=regularizers.l1(0.01)))

But my network was created in PyTorch like so:
upconv = nn.ConvTranspose2d(inner_nc, outer_nc,
                            kernel_size=4, stride=2,
                            padding=1, bias=use_bias)
down = [downrelu, downconv]
up = [uprelu, upconv, upnorm]
model = down + up

Output:
You're overthinking this. As I see from your Keras code, you're trying to impose a L1 penalty on the activations of your layer. The simplest way would just be to do something like the following:
activations_to_regularise = upconv(input)
output = remaining_netowrk(activations_to_regularise)

Then have your normal loss function to assess the output against a target and also incorporate the L1 loss into the objective, such that you get
total_loss = criterion(output, target) + 0.01 * activations_to_regularise.abs()