Instruction: Convert 'int' to pytorch 'Variable' makes problems

Input:
First project with pytorch and I got stuck trying to convert an MNIST label 'int' into a torch 'Variable'. Debugger says it has no dimension?!

# numpy mnist data
X_train, Y_train = read_data("training")
X_test , Y_test  = read_data("testing")

arr = np.zeros(5)
for i in range(5):
    # in your training loop:
    costs_ = 0
    for k in range(10000):
        optimizer.zero_grad()                            # zero the gradient buffers
        a = torch.from_numpy(np.expand_dims(X_train[k].flatten(), axis=0)).float()
        b = torch.from_numpy(np.array(Y_train[k], dtype=np.float)).float()
        input = Variable(a)
        output = net(input)
        target = Variable(b)                             # PROBLEM!!
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()                                 # Does the update

        costs_ += loss.data.numpy()
    arr[i] = costs_
    print(i)

Thrown error is: "RuntimeError: input and target have different number of elements: input[1 x 1] has 1 elements, while target[] has 0 elements at /b/wheel/pytorch-src/torch/lib/THNN/generic/MSECriterion.c:12"

Output:
The error is telling you exactly what is happening. Your target variable is empty.

Edit (after the comment below):

if Y_train[k] = 5, then np.array(Y_train[k], dtype=np.float).shape = (), and in turn Variable(b) becomes a tensor with no dimension.

In order to fix this you will need to pass a list to np.array() and not a integer or a float.

Like this:

b = torch.from_numpy(np.array([Y_train[k]], dtype=np.float)).float()


Instruction: Pytorch: how to add L1 regularizer to activations?

Input:
I would like to add the L1 regularizer to the activations output from a ReLU.
More generally, how does one add a regularizer only to a particular layer in the network?

Related material:

This similar post refers to adding L2 regularization, but it appears to add the regularization penalty to all layers of the network.

nn.modules.loss.L1Loss() seems relevant, but I do not yet understand how to use this.

The legacy module L1Penalty seems relevant also, but why has it been deprecated?

Output:
Here is how you do this:

In your Module's forward return final output and layers' output for which you want to apply L1 regularization
loss variable will be sum of cross entropy loss of output w.r.t. targets and L1 penalties.

Here's an example code

import torch
from torch.autograd import Variable
from torch.nn import functional as F

class MLP(torch.nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.linear1 = torch.nn.Linear(128, 32)
        self.linear2 = torch.nn.Linear(32, 16)
        self.linear3 = torch.nn.Linear(16, 2)

    def forward(self, x):
        layer1_out = F.relu(self.linear1(x))
        layer2_out = F.relu(self.linear2(layer1_out))
        out = self.linear3(layer2_out)
        return out, layer1_out, layer2_out

batchsize = 4
lambda1, lambda2 = 0.5, 0.01

model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)

# usually following code is looped over all batches
# but let's just do a dummy batch for brevity

inputs = Variable(torch.rand(batchsize, 128))
targets = Variable(torch.ones(batchsize).long())

optimizer.zero_grad()
outputs, layer1_out, layer2_out = model(inputs)
cross_entropy_loss = F.cross_entropy(outputs, targets)

all_linear1_params = torch.cat([x.view(-1) for x in model.linear1.parameters()])
all_linear2_params = torch.cat([x.view(-1) for x in model.linear2.parameters()])
l1_regularization = lambda1 * torch.norm(all_linear1_params, 1)
l2_regularization = lambda2 * torch.norm(all_linear2_params, 2)

loss = cross_entropy_loss + l1_regularization + l2_regularization
loss.backward()
optimizer.step()


Instruction: How do you implement variable-length recurrent neural networks?

Input:
What is a full working example (not snippets) of variable-length sequence inputs into recurrent neural networks (RNNs)?

For example PyTorch supposedly can implement variable-length sequences as input into RNNs, but there do not seem to be examples of full working code.

Relevant:
https://github.com/pytorch/pytorch/releases/tag/v0.1.10
https://discuss.pytorch.org/t/about-the-variable-length-input-in-rnn-scenario/345

Output:
Sadly, there is no such thing as 'variable length' neural networks. This is because there is no way a network can 'know' which weights to use for extra input nodes that it wasn't trained for.

However, the reason you are seeing a 'variable length' on that page, is because they process:

a b c d e
a b c d e f g h
a b c d
a b

as

a b c d e 0 0 0
a b c d e f g h
a b c d 0 0 0 0
a b 0 0 0 0 0 0

They convert all 'empty' variables to 0. Which makes sense, as 0 does not add anything tot he networks hidden layers regardless of the weights, as anything*0 = 0.

So basically, you can have 'variable length' inputs, but you have to define some kind of maximum size; all inputs that are smaller than that size should be padded with zeros.

If you are classifying sentences on the other hand, you could use LSTM/GRU networks to handle the input sequentially.


Instruction: How do you use PyTorch PackedSequence in code?

Input:
Can someone give a full working code (not a snippet, but something that runs on a variable-length recurrent neural network) on how would you use the PackedSequence method in PyTorch?

There do not seem to be any examples of this in the documentation, github, or the internet.

https://github.com/pytorch/pytorch/releases/tag/v0.1.10

Output:
Not the most beautiful piece of code, but this is what I gathered for my personal use after going through PyTorch forums and docs. There can be certainly better ways to handle the sorting - restoring part, but I chose it to be in the network itself
EDIT: See answer from @tusonggao which makes torch utils take care of sorting parts
class Encoder(nn.Module):
    def __init__(self, vocab_size, embedding_size, embedding_vectors=None, tune_embeddings=True, use_gru=True,
                 hidden_size=128, num_layers=1, bidrectional=True, dropout=0.6):
        super(Encoder, self).__init__()
        self.embed = nn.Embedding(vocab_size, embedding_size, padding_idx=0)
        self.embed.weight.requires_grad = tune_embeddings
        if embedding_vectors is not None:
            assert embedding_vectors.shape[0] == vocab_size and embedding_vectors.shape[1] == embedding_size
            self.embed.weight = nn.Parameter(torch.FloatTensor(embedding_vectors))
        cell = nn.GRU if use_gru else nn.LSTM
        self.rnn = cell(input_size=embedding_size, hidden_size=hidden_size, num_layers=num_layers,
                        batch_first=True, bidirectional=True, dropout=dropout)

    def forward(self, x, x_lengths):
        sorted_seq_lens, original_ordering = torch.sort(torch.LongTensor(x_lengths), dim=0, descending=True)
        ex = self.embed(x[original_ordering])
        pack = torch.nn.utils.rnn.pack_padded_sequence(ex, sorted_seq_lens.tolist(), batch_first=True)
        out, _ = self.rnn(pack)
        unpacked, unpacked_len = torch.nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
        indices = Variable(torch.LongTensor(np.array(unpacked_len) - 1).view(-1, 1)
                                                                       .expand(unpacked.size(0), unpacked.size(2))
                                                                       .unsqueeze(1))
        last_encoded_states = unpacked.gather(dim=1, index=indices).squeeze(dim=1)
        scatter_indices = Variable(original_ordering.view(-1, 1).expand_as(last_encoded_states))
        encoded_reordered = last_encoded_states.clone().scatter_(dim=0, index=scatter_indices, src=last_encoded_states)
        return encoded_reordered


Instruction: Pytorch: Convert FloatTensor into DoubleTensor

Input:
I have 2 numpy arrays, which I convert into tensors to use the TensorDataset object.

import torch.utils.data as data_utils

X = np.zeros((100,30))
Y = np.zeros((100,30))

train = data_utils.TensorDataset(torch.from_numpy(X).double(), torch.from_numpy(Y))
train_loader = data_utils.DataLoader(train, batch_size=50, shuffle=True)

when I do:

for batch_idx, (data, target) in enumerate(train_loader):
    data, target = Variable(data), Variable(target)
    optimizer.zero_grad()
    output = model(data)               # error occurs here

I get the fallowing error:

  TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.DoubleTensor, torch.FloatTensor), but expected one of:
   [...]
   * (float beta, float alpha, torch.DoubleTensor mat1, torch.DoubleTensor mat2)
        didn't match because some of the arguments have invalid types: (int, int, torch.DoubleTensor, torch.FloatTensor)
   * (float beta, float alpha, torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
        didn't match because some of the arguments have invalid types: (int, int, torch.DoubleTensor, torch.FloatTensor)

The last error comes from:

  output.addmm_(0, 1, input, weight.t())

As you see in my code I tried converting the tensor by using .double() - but this did not work. Why is he casting one array into a FloatTensor object and the other into a DoubleTensor?
Any ideas?

Output:
Your numpy arrays are 64-bit floating point and will be converted to torch.DoubleTensor standardly. Now, if you use them with your model, you'll need to make sure that your model parameters are also Double. Or you need to make sure, that your numpy arrays are cast as Float, because model parameters are standardly cast as float.

Hence, do either of the following:

data_utils.TensorDataset(torch.from_numpy(X).float(), torch.from_numpy(Y).float())

or do:

model.double()

Depeding, if you want to cast your model parameters, inputs and targets as Float or as Double.


Instruction: Why do we need to explicitly call zero_grad()?

Input: Why do we need to explicitly zero the gradients in PyTorch? Why can't gradients be zeroed when loss.backward() is called? What scenario is served by keeping the gradients on the graph and asking the user to explicitly zero the gradients?

Output:
We explicitly need to call zero_grad() because, after loss.backward() (when gradients are computed), we need to use optimizer.step() to proceed gradient descent. More specifically, the gradients are not automatically zeroed because these two operations, loss.backward() and optimizer.step(), are separated, and optimizer.step() requires the just computed gradients.

In addition, sometimes, we need to accumulate gradient among some batches; to do that, we can simply call backward multiple times and optimize once.


Instruction: PyTorch: Extract learned weights correctly

Input:
I am trying to extract the weights from a linear layer, but they do not appear to change, although error is dropping monotonously (i.e. training is happening). Printing the weights' sum, nothing happens because it stays constant:

  np.sum(model.fc2.weight.data.numpy())

Here are the code snippets:

def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # Train on train set
        print(np.sum(model.fc2.weight.data.numpy()))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data), Variable(data)
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()

and

# Define model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(100, 80, bias=False)
        init.normal(self.fc1.weight, mean=0, std=1)
        self.fc2 = nn.Linear(80, 87)
        self.fc3 = nn.Linear(87, 94)
        self.fc4 = nn.Linear(94, 100)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        return x

Maybe I am looking on the wrong parameters, although I checked the docs. Thanks for your help!

Output:
Use model.parameters() to get trainable weight for any model or layer. Remember to put it inside  list(), or you cannot print it out.

The following code snip worked

>>> import torch
>>> import torch.nn as nn
>>> l = nn.Linear(3,5)
>>> w = list(l.parameters())
>>> w


Instruction: Re-weight the input to a neural network

Input:
For example, I feed a set of images into a CNN. And the default weight of these images is 1. How can I re-weight some of these images so that they have different weights? Can 'DataLoader' achieve this goal in pytorch?

I learned two other possibilities:

Defining a custom loss function, providing weights for each sample as I require.
Repeating samples in the training set, which will result in more frequent samples having a higher weight in the final loss.

Is there any other way, we can achieve that? Any suggestion would be appreciated.

Output:
I can think of two ways to achieve this.

Pass on the weight explicitly, when you backpropagate the gradients.
After you computed loss, and when you're about to backpropagate, you can pass a Tensor to backward() and all the subsequent gradients will be scaled by the corresponding element, i.e. do something like
loss = torch.pow(out-target,2)
loss.backward(my_weights) # is a Tensor of same dimension as loss

However, if you use want to assign individual weights in a batch, you can't use the custom loss functions from the nn.module which aggregates the loss over all samples in a batch.

Use torch.utils.data.sampler.WeightedRandomSampler
If you use PyTorch's data.utils anyway, this is simpler than multiplying your training set. However it doesn't assign exact weights, since it's stochastic. But if you're iterating over your training set a sufficient number of times, it's probably close enough.


Instruction: Torch sum a tensor along an axis

Input:
How do I sum over the columns of a tensor?
torch.Size([10, 100])    --->    torch.Size([10])

Output:
The simplest and best solution is to use torch.sum().
To sum all elements of a tensor:
torch.sum(x) # gives back a scalar

To sum over all rows (i.e. for each column):
torch.sum(x, dim=0) # size = [ncol]

To sum over all columns (i.e. for each row):
torch.sum(x, dim=1) # size = [nrow]

It should be noted that the dimension summed over is eliminated from the resulting tensor.


Instruction: Apply json config file using parse_args() in pycharm

Input:
I'm running sequence to sequence code in git, but I got error about parse_args().
my code is like this:

parser = argparse.ArgumentParser()
parser.add_argument(
    "--config",
    help="path to json config",
    required=True)
args = parser.parse_args()
config_file_path = args.config
config = read_config(config_file_path)
experiment_name = hyperparam_string(config)

my config file is like this :

{
  "training": {
    "optimizer": "adam",
    "clip_c": 1,
    "lrate": 0.0002,
  },
  "management": {
    "monitor_loss": 1000,
    "print_samples": 20000
  }

When I run

args = parser.parse_args()

Pycharm raises error

pydevconsole.py: error: argument --config is required

I'd like to know that run this code through pycharm applying json config file.

I have searched in google since yesterday, but I can't find it.

Please help..

Output:
args = parser.parse_args() parses the sys.argv[1:] list, which is provided to the interpreter from the operating system shell - ie. from the commandline.

$:python prog.py --config afilename

You can also do

args = parser.parse_args(['--config', 'afilename'])

this handy during testing.

It also helps to:

import sys
print(sys.argv)


Instruction: how to append a singleton numpy array item into a list?

Input:
I am attaching the code first which will give you a better idea

`   prediction = prediction.data.max(1)[1] #gives a tensor value
    prediction = (prediction.cpu().numpy().item()) #converts that tensor into a numpy array
    result.append(int_to_word[prediction])`

I am using pytorch for word generation. The line prediction = prediction.data.max(1)[1] gives us the class label with the maximum probability, which turns out to be a  tensor value.

prediction = (prediction.cpu().numpy().item()) this statement converts the tensor into a numpy array and the funtion item() extracts the value from the array.

Now when I try to append this value using pattern.append(prediction)
I am getting the following error
'numpy.ndarray' object has no attribute 'append'

I am not able to understand as to why I am getting this error. I have already converted the numpy array into a scalr value, didn't I?? Still, why am I getting that error?

Can anybody please explain and provide a solution to this. I would be most grateful.

Output:
Numpy arrays are immutable with respect to their dimensions. They do not support the append operation. You'll have to declare results as a list, then append your values to your list, and then convert it to a numpy array:

result = []
...

result.append(prediction) # inside some loop

...

result = np.array(result)


Instruction: No N-dimensional tranpose in PyTorch

Input:
PyTorch's torch.transpose function only transposes 2D inputs. Documentation is here.

On the other hand, Tensorflow's tf.transpose function allows you to transpose a tensor of N arbitrary dimensions.

Can someone please explain why PyTorch does not/cannot have N-dimension transpose functionality? Is this due to the dynamic nature of the computation graph construction in PyTorch versus Tensorflow's Define-then-Run paradigm?

Output:
It's simply called differently in pytorch. torch.Tensor.permute will allow you to swap dimensions in pytorch like tf.transpose does in TensorFlow.

As an example of how you'd convert a 4D image tensor from NHWC to NCHW (not tested, so might contain bugs):

>>> img_nhwc = torch.randn(10, 480, 640, 3)
>>> img_nhwc.size()
torch.Size([10, 480, 640, 3])
>>> img_nchw = img_nhwc.permute(0, 3, 1, 2)
>>> img_nchw.size()
torch.Size([10, 3, 480, 640])


Instruction: How to debug(monitoring value of object in other class' function) in Pycharm

Input:
I'm running seq2seq code in pycharm in order to study pytorch.

The code has many classes and these classes have many function.

I'd like to monitor value of objects in other function, so I'm running code in console one by one.

Is there any good way to this using debug?

I haven't done debug before.

Please help me..

Output:
I'm not familiar with these tools specifically, but here is how I would approach it. It's also kinda hard to express how to properly use a gui interactively through text, so if you are new to a debugger in general it might be good to start with some tutorials. Jetbrains has some PyCharm debugger tutorials online.

PyCharm debugger tutorial 1

PyCharm debugger tutorial 2

When you are running the debugger, set breakpoints and you can see all of the local variables in the scope to your current object. If you are wanting to monitor 2 places, you could set 2 breakpoints. Or you could stop at one and move forward (look at Step Over, F8 and Step Into, F7 until the second object is available. I think specifically for you I would look at the Debugger, Frames. Essentially you can jump backwards in time from your current breakpoint to where your current function was called, and so on and so forth for ~10 calls. This might get you what you are looking for, but it is somewhat project dependent unfortunately.


Instruction: How do you alter the size of a Pytorch Dataset?

Input:
Say I am loading MNIST from torchvision.datasets.MNIST, but I only want to load in 10000 images total, how would I slice the data to limit it to only some number of data points?  I understand that the DataLoader is a generator yielding data in the size of the specified batch size, but how do you slice datasets?
tr = datasets.MNIST('../data', train=True, download=True, transform=transform)
te = datasets.MNIST('../data', train=False, transform=transform)
train_loader = DataLoader(tr, batch_size=args.batch_size, shuffle=True, num_workers=4, **kwargs)
test_loader = DataLoader(te, batch_size=args.batch_size, shuffle=True, num_workers=4, **kwargs)

Output:
It is important to note that when you create the DataLoader object, it doesnt immediately load all of your data (its impractical for large datasets). It provides you an iterator that you can use to access each sample.

Unfortunately, DataLoader doesnt provide you with any way to control the number of samples you wish to extract. You will have to use the typical ways of slicing iterators.

Simplest thing to do (without any libraries) would be to stop after the required number of samples is reached.

nsamples = 10000
for i, image, label in enumerate(train_loader):
    if i > nsamples:
        break

    # Your training code here.

Or, you could use itertools.islice to get the first 10k samples. Like so.

for image, label in itertools.islice(train_loader, stop=10000):

    # your training code here.


Instruction: Find number of non-zero elements in a tensor along an aixs

Input:
I want to find the number of non-zero elements in a tensor along a particular axis. Is there any PyTorch function which can do this?

I tried to use the nonzero() method in PyTorch.

torch.nonzero(losses).size(0)

Here, lossess is a tensor of shape 64 x 1. When I run the above statement, it gives me the following error.

TypeError: Type Variable doesn't implement stateless method nonzero

But if I run, torch.nonzero(losses.data).size(0), then it works fine. Any clue, why this is happening or what the error means?

Output: Meaning of the error message - TypeError: Type Variable doesn't implement stateless method nonzero is, we cannot use torch.nonzero() on autograd.Variable but only on simple tensors. Also it should be noted that, tensors are stateless while the Variables are stateful.


Instruction: PyTorch - Element-wise multiplication between a variable and a tensor?

Input:
As of PyTorch 0.4 this question is no longer valid. In 0.4 Tensors and Variables were merged.

How can I perform element-wise multiplication with a variable and a tensor in PyTorch? With two tensors works fine. With a variable and a scalar works fine. But when attempting to perform element-wise multiplication with a variable and tensor I get:

XXXXXXXXXXX in mul
    assert not torch.is_tensor(other)
AssertionError

For example, when running the following:

import torch

x_tensor = torch.Tensor([[1, 2], [3, 4]])
y_tensor = torch.Tensor([[5, 6], [7, 8]])

x_variable = torch.autograd.Variable(x_tensor)

print(x_tensor * y_tensor)
print(x_variable * 2)
print(x_variable * y_tensor)

I would expect the first and last print statements to show similar results. The first two multiplications work as expected, with the error coming up in the third. I have attempted the aliases of * in PyTorch (i.e. x_variable.mul(y_tensor), torch.mul(y_tensor, x_variable), etc.).

It seems that element-wise multiplication between a tensor and a variable is not supported given the error and the code which produces it. Is this correct? Or is there something I'm missing? Thank you!

Output:
Yes, you are correct. Elementwise multiplication (like most other operations) is only supported for Tensor * Tensor or Variable * Variable, but not for Tensor * Variable.

To perform your multiplication above, wrap your Tensor as a Variable which doesn't require gradients. The additional overhead is insignificant.

y_variable = torch.autograd.Variable(y_tensor, requires_grad=False)
x_variable * y_variable # returns Variable

But obviously, only use Variables though, if you actually require automatic differentiation through a graph. Else you can just perform the operation on the Tensors directly as you did in your question.


Instruction: Pytorch nn.functional.batch_norm for 2D input

Input:
I am currently implementing a model on which I need to change the running mean and standard deviation during test time. As such, I assume the nn.functional.batch_norm would be a better choice than the nn.BatchNorm2d

However, I have batches of images as input, and am currently not sure how to take in the images. How would I apply nn.functional.batch_norm on batches of 2D images?

The current code I have is this, I post this even though this is not correct:

mu = torch.mean(inp[0])
stddev = torch.std(inp[0])
x = nn.functional.batch_norm(inp[0], mu, stddev, training=True, momentum=0.9)

Output:
The key is that 2D batchnorm performs the same normalization for each channel. i.e. if you have a batch of data with shape (N, C, H, W) then your mu and stddev should be shape (C,). If your images do not have a channel dimension, then add one using view.

Warning: if you set training=True then batch_norm computes and uses the appropriate normalization statistics for the argued batch (this means we don't need to calculate the mean and std ourselves). Your argued mu and stddev are supposed to be the running mean and running std for all training batches. These tensors are updated with the new batch statistics in the batch_norm function.

# inp is shape (N, C, H, W)
n_chans = inp.shape[1]
running_mu = torch.zeros(n_chans) # zeros are fine for first training iter
running_std = torch.ones(n_chans) # ones are fine for first training iter
x = nn.functional.batch_norm(inp, running_mu, running_std, training=True, momentum=0.9)
# running_mu and running_std now have new values

If you want to just use your own batch statistics, try this:

# inp is shape (N, C, H, W)
n_chans = inp.shape[1]
reshaped_inp = inp.permute(1,0,2,3).contiguous().view(n_chans, -1) # shape (C, N*W*H)
mu = reshaped_inp.mean(-1)
stddev = reshaped_inp.std(-1)
x = nn.functional.batch_norm(inp, mu, stddev, training=False)


Instruction: requires_grad relation to leaf nodes

Input:
From the docs:

  requires_grad – Boolean indicating whether the Variable has been
  created by a subgraph containing any Variable, that requires it.  Can
  be changed only on leaf Variables

What does it mean by leaf nodes here? Are leaf nodes only the input nodes?
If it can be only changed at the leaf nodes, how can I freeze layers then?

Output:
Leaf nodes of a graph are those nodes (i.e. Variables) that were not computed directly from other nodes in the graph. For example:

import torch
from torch.autograd import Variable

A = Variable(torch.randn(10,10)) # this is a leaf node
B = 2 * A # this is not a leaf node
w = Variable(torch.randn(10,10)) # this is a leaf node
C = A.mm(w) # this is not a leaf node

If a leaf node requires_grad, all subsequent nodes computed from it will automatically also require_grad. Else, you could not apply the chain rule to calculate the gradient of the leaf node which requires_grad. This is the reason why requires_grad can only be set for leaf nodes: For all others, it can be smartly inferred and is in fact determined by the settings of the leaf nodes used for computing these other variables.
Note that in a typical neural network, all parameters are leaf nodes. They are not computed from any other Variables in the network. Hence, freezing layers using requires_gradis simple. Here, is an example taken from the PyTorch docs:

model = torchvision.models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

# Replace the last fully-connected layer
# Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 100)

# Optimize only the classifier
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)

Even though, what you really do is freezing the entire gradient computation (which is what you should be doing as it avoids unnecessary computation). Technically, you could leave the requires_grad flag on, and only define your optimizer for a subset of the parameters that you would like to learn.


Instruction: How two rows can be swapped in a torch tensor?

Input:
var = [[0, 1, -4, 8],
       [2, -3, 2, 1],
       [5, -8, 7, 1]]

var = torch.Tensor(var)

Here, var is a 3 x 4 (2d) tensor. How the first and second row can be swapped to get the following 2d tensor?

2, -3, 2, 1
0, 1, -4, 8
5, -8, 7, 1

Output:
The other answer does not work, as some dimensions get overwritten before they are copied:
>>> var = [[0, 1, -4, 8],
       [2, -3, 2, 1],
       [5, -8, 7, 1]]
>>> x = torch.tensor(var)
>>> index = torch.LongTensor([1, 0, 2])
>>> x[index] = x
>>> x
tensor([[ 0,  1, -4,  8],
        [ 0,  1, -4,  8],
        [ 5, -8,  7,  1]])

For me, it suffices to create a new tensor (with separate underlying storage) to hold the result:
>>> x = torch.tensor(var)
>>> index = torch.LongTensor([1, 0, 2])
>>> y = torch.zeros_like(x)
>>> y[index] = x

Alternatively, you can use index_copy_ (following this explanation in discuss.pytorch.org), although I don't see an advantage for either way at the moment.


Instruction: How padded sequences given as packed sequences are dealt by RNN in pytorch?

Input:
In pytorch, we can give a packed sequence as an input to the RNN. From official doc, input of an RNN can be as follows.

  input (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence.

Example

packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
outputs, hidden = self.rnn(packed, hidden)
outputs, output_lengths = torch.nn.utils.rnn.pad_packed_sequence(outputs)

Here, embedded is the embedded representation of a batch input.

My question is, how the computation is carried out for packed sequences in RNN? How the hidden states are computed for padded sequences in a batch through packed representation?

Output:
For the second question: hidden states at padded sequences will not be computed.

To answer how is that happening, let's first see what pack_padded_sequence does for us:

from torch.nn.utils.rnn import pad_sequence, pad_packed_sequence, pack_padded_sequence

raw = [ torch.ones(25, 300) / 2,
        torch.ones(22, 300) / 2.3,
        torch.ones(15, 300) / 3.2 ]
padded = pad_sequence(raw)  # size: [25, 3, 300]

lengths = torch.as_tensor([25, 22, 15], dtype=torch.int64)
packed = pack_padded_sequence(padded, lengths)

so far we randomly created a three tensor with different length (timestep in the context of RNN) , and we first pad them to the same length, then packed it. Now if we run

print(padded.size())
print(packed.data.size()) # packed.data refers to the "packed" tensor

we will see:

torch.Size([25, 3, 300])
torch.Size([62, 300])

Obviously 62 does not come from 25 * 3. So what pack_padded_sequence does is only keep the meaningful timestep of each batch entry according to the lengths tensor we passed to pack_padded_sequence (i.e. if we passed [25, 25, 25] to it, the size of packed.data would still be [75, 300] even though the raw tensor does not change). In short, rnn would no even see the pad timestep with pack_padded_sequence

And now let's see what's the difference after we pass padded and packed to rnn

rnn = torch.nn.RNN(input_size=300, hidden_size=2)
padded_outp, padded_hn = rnn(padded) # size: [25, 3, 2] / [1, 3, 2]
packed_outp, packed_hn = rnn(packed) # 'PackedSequence' Obj / [1, 3, 2]
undo_packed_outp, _ = pad_packed_sequence(packed_outp)

# return "h_n"
print(padded_hn) # tensor([[[-0.2329, -0.6179], [-0.1158, -0.5430],[ 0.0998, -0.3768]]])
print(packed_hn) # tensor([[[-0.2329, -0.6179], [ 0.5622,  0.1288], [ 0.5683,  0.1327]]]

# the output of last timestep (the 25-th timestep)
print(padded_outp[-1]) # tensor([[[-0.2329, -0.6179], [-0.1158, -0.5430],[ 0.0998, -0.3768]]])
print(undo_packed_outp.data[-1]) # tensor([[-0.2329, -0.6179], [ 0.0000,  0.0000], [ 0.0000,  0.0000]]

The values of padded_hn and packed_hn are different since rnn DOES compute the pad for padded yet not for the 'packed' (PackedSequence object), which also can be observed from the last hidden state: all three batch entry in padded got non-zero last hidden state even if its length is less than 25. But for packed, the last hidden state for shorter data is not computed (i.e. 0)

p.s. another observation:

print([(undo_packed_outp[:, i, :].sum(-1) != 0).sum() for i in range(3)])

would give us [tensor(25), tensor(22), tensor(15)], which align to the actual length of our input.


Instruction: Backpropagation without updating weights (DCGAN)

Input:
The basic idea is, that I load a trained model (DCGAN) and make a anomaly detection with it on images. For the anomaly detection I have to do some iterations on the test phase to evaluate it, if it is a anomaly or not.

For that I have two Loss-Functions in the test setup, which should be calculating a backpropagation to the generator input and update the latent vector. But it only should update the latent vector, not the weights on the graph.

Is this possible?

Probably, if I use only a pytorch-variable of my latent vector and set the variable output of the generator to
"requires_grad=False"
Like in the docs --> Pytorch

Output:
Yes, you're on the right track there.
You can individually set the requires_grad attribute of your model parameters (more precisely of all leaf nodes in your computational graph). I am not familiar with DCGAN, but I assume the latent vector is a trainable parameter, too (else a back-propagation update makes little sense to me).

The following code snippet adapted from the PyTorch docs might be useful for your purposes:

# Load your model
model = torchvision.models.resnet18(pretrained=True)

# Disable gradient computation for all parameters
for param in model.parameters():
    param.requires_grad = False

# Enable gradient computation for a specific layer (here the last fc layer)
for param in model.fc.parameters():
    param.requires_grad = True

# Optimize only the the desired parameters (for you latent vectors)
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)


Instruction: Why does an attribute has a method?

Input: I use pytorch define a Variable x, compute and get its gradient x.grad, meaning grad is an attribute of instance x. But, I can use x.grad.data.zero_() to set x.grad to zero meaning that data.zero_() is method of x.grad. Why does an attribute has a method? Thanks a lot.

Output: It will be just an object attribute, for example we can have an attribute of type String. And we all now String have his own methods including the method that prints out the value of itself


Instruction: Why RNN need two bias vectors?

Input:
In pytorch RNN implementation, there are two biases, b_ih and b_hh.
Why is this? Is it different from using one bias? If yes, how? Will it affect performance or efficiency?

Output:
The formular in Pytorch Document in RNN is self-explained. That is b_ih and b_hh in the equation.

You may think that b_ih is bias for input (which pair with w_ih, weight for input) and b_hh is bias for hidden (pair with w_hh, weight for hidden)


Instruction: Include .whl installation in requirements.txt

Input:
How do I include this in the requirements.txt file?
For Linux:

pip install http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl
pip install torchvision

FOR MacOS:

pip install http://download.pytorch.org/whl/torch-0.1.12.post2-cp27-none-macosx_10_7_x86_64.whl
pip install torchvision

Output:
You can use environment markers:

http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl ; sys_platform == "linux"
http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl ; sys_platform == "linux2"
http://download.pytorch.org/whl/torch-0.1.12.post2-cp27-none-macosx_10_7_x86_64.whl ; sys_platform == "darwin"
torchvision

(Double Linux entries: linux2 for Python 2, linux for Python 3.)


Instruction: Understanding a simple LSTM pytorch

Input:
import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
input = Variable(torch.randn(5, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
c0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input, (h0, c0))

This is the LSTM example from the docs. I don't know understand the following things:

What is output-size and why is it not specified anywhere?
Why does the input have 3 dimensions. What does 5 and 3 represent?
What are 2 and 3 in h0 and c0, what do those represent?

Edit:

import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F

num_layers=3
num_hyperparams=4
batch = 1
hidden_size = 20
rnn = nn.LSTM(input_size=num_hyperparams, hidden_size=hidden_size, num_layers=num_layers)

input = Variable(torch.randn(1, batch, num_hyperparams)) # (seq_len, batch, input_size)
h0 = Variable(torch.randn(num_layers, batch, hidden_size)) # (num_layers, batch, hidden_size)
c0 = Variable(torch.randn(num_layers, batch, hidden_size))
output, hn = rnn(input, (h0, c0))
affine1 = nn.Linear(hidden_size, num_hyperparams)

ipdb.set_trace()
print output.size()
print h0.size()

  *** RuntimeError: matrices expected, got 3D, 2D tensors at

Output:
The output for the LSTM is the output for all the hidden nodes on the final layer.
hidden_size - the number of LSTM blocks per layer.
input_size - the number of input features per time-step.
num_layers - the number of hidden layers.
In total there are hidden_size * num_layers LSTM blocks.

The input dimensions are (seq_len, batch, input_size).
seq_len - the number of time steps in each input stream.
batch - the size of each batch of input sequences.

The hidden and cell dimensions are: (num_layers, batch, hidden_size)

  output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t.

So there will be hidden_size * num_directions outputs. You didn't initialise the RNN to be bidirectional so num_directions is 1. So output_size = hidden_size.

Edit: You can change the number of outputs by using a linear layer:

out_rnn, hn = rnn(input, (h0, c0))
lin = nn.Linear(hidden_size, output_size)
v1 = nn.View(seq_len*batch, hidden_size)
v2 = nn.View(seq_len, batch, output_size)
output = v2(lin(v1(out_rnn)))

Note: for this answer I assumed that we're only talking about non-bidirectional LSTMs.

Source: PyTorch docs.


Instruction: How to simplify DataLoader for Autoencoder in Pytorch

Input:
Is there any easier way to set up the dataloader, because input and target data is the same in case of an autoencoder and to load the data during training? The DataLoader always requires two inputs.

Currently I define my dataloader like this:

X_train     = rnd.random((300,100))
X_val       = rnd.random((75,100))
train       = data_utils.TensorDataset(torch.from_numpy(X_train).float(), torch.from_numpy(X_train).float())
val         = data_utils.TensorDataset(torch.from_numpy(X_val).float(), torch.from_numpy(X_val).float())
train_loader= data_utils.DataLoader(train, batch_size=1)
val_loader  = data_utils.DataLoader(val, batch_size=1)

and train like this:

for epoch in range(50):
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data), Variable(target).detach()
        optimizer.zero_grad()
        output = model(data, x)
        loss = criterion(output, target)

Output:
Why not subclassing TensorDataset to make it compatible with unlabeled data ?

class UnlabeledTensorDataset(TensorDataset):
    """Dataset wrapping unlabeled data tensors.

    Each sample will be retrieved by indexing tensors along the first
    dimension.

    Arguments:
        data_tensor (Tensor): contains sample data.
    """
    def __init__(self, data_tensor):
        self.data_tensor = data_tensor

    def __getitem__(self, index):
        return self.data_tensor[index]

And something along these lines for training your autoencoder

X_train     = rnd.random((300,100))
train       = UnlabeledTensorDataset(torch.from_numpy(X_train).float())
train_loader= data_utils.DataLoader(train, batch_size=1)

for epoch in range(50):
    for batch in train_loader:
        data = Variable(batch)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, data)


Instruction: Indexing second dimension of Tensor using indices

Input:
I selected element in my tensor using a tensor of indices. Here the code below I use list of indices 0, 3, 2, 1 to select 11, 15, 2, 5

>>> import torch
>>> a = torch.Tensor([5,2,11, 15])
>>> torch.randperm(4)

 0
 3
 2
 1
[torch.LongTensor of size 4]

>>> i = torch.randperm(4)
>>> a[i]

 11
 15
  2
  5
[torch.FloatTensor of size 4]

Now, I have

>>> b = torch.Tensor([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> b

  5   2  11  15
  5   2  11  15
  5   2  11  15
[torch.FloatTensor of size 3x4]

Now, I want to use indices to select column 0, 3, 2, 1. In others word, I want a tensor like this

>>> b

 11  15   2   5
 11  15   2   5
 11  15   2   5
[torch.FloatTensor of size 3x4]

Output:
If using pytorch version v0.1.12

For this version there isnt an easy way to do this. Even though pytorch promises tensor manipulation to be exactly like numpy's, there are some capabilities that are still lacking. This is one of them.

Typically you would be able to do this relatively easily if you were working with numpy arrays. Like so.

>>> i = [2, 1, 0, 3]
>>> a = np.array([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> a[:, i]

array([[11,  2,  5, 15],
       [11,  2,  5, 15],
       [11,  2,  5, 15]])

But the same thing with Tensors will give you an error:

>>> i = torch.LongTensor([2, 1, 0, 3])
>>> a = torch.Tensor([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> a[:,i]

The error:

  TypeError: indexing a tensor with an object of type torch.LongTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument.

What that TypeError is telling you is, if you plan to use a LongTensor or a ByteTensor for indexing, then the only valid syntax is a[<LongTensor>] or a[<ByteTensor>]. Anything other than that will not work.

Because of this limitation, you have two options:

Option 1: Convert to numpy, permute, then back to Tensor

>>> i = [2, 1, 0, 3]
>>> a = torch.Tensor([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> np_a = a.numpy()
>>> np_a = np_a[:,i]
>>> a = torch.from_numpy(np_a)
>>> a

 11   2   5  15
 11   2   5  15
 11   2   5  15
[torch.FloatTensor of size 3x4]

Option 2: Move the dim you want to permute to 0 and then do it

you will move the dim that you are looking to permute, (in your case dim=1) to 0, perform the permutation, and move it back. Its a bit hacky, but it gets the job done.

def hacky_permute(a, i, dim):
    a = torch.transpose(a, 0, dim)
    a = a[i]
    a = torch.transpose(a, 0, dim)
    return a

And use it like so:

>>> i = torch.LongTensor([2, 1, 0, 3])
>>> a = torch.Tensor([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> a = hacky_permute(a, i, dim=1)
>>> a

 11   2   5  15
 11   2   5  15
 11   2   5  15
[torch.FloatTensor of size 3x4]

If using pytorch version v0.2.0

Direct indexing using a tensor now works in this version. ie.

>>> i = torch.LongTensor([2, 1, 0, 3])
>>> a = torch.Tensor([[5, 2, 11, 15],[5, 2, 11, 15], [5, 2, 11, 15]])
>>> a[:,i]

 11   2   5  15
 11   2   5  15
 11   2   5  15
[torch.FloatTensor of size 3x4]


Instruction: PyTorch RuntimeError : Gradients are not CUDA tensors

Input:
I am getting the following error while doing seq to seq on characters and feeding to LSTM, and decoding to words using attention. The forward propagation is fine but while computing loss.backward() I am getting the following error.

  RuntimeError: Gradients aren't CUDA tensors

My train() function is as followed.

def train(input_batch, input_batch_length, target_batch, target_batch_length, batch_size):

    # Zero gradients of both optimizers
    encoderchar_optimizer.zero_grad()
    encoder_optimizer.zero_grad()
    decoder_optimizer.zero_grad()

    encoder_input = Variable(torch.FloatTensor(len(input_batch), batch_size, 500))

    for ix , w in enumerate(input_batch):
        w = w.contiguous().view(15, batch_size)
        reshaped_input_length = [x[ix] for x in input_batch_length] # [15 ,.. 30 times] * 128
        if USE_CUDA:
            w = w.cuda()
            #reshaped_input_length =  Variable(torch.LongTensor(reshaped_input_length)).cuda()
        hidden_all , output = encoderchar(w, reshaped_input_length)
        encoder_input[ix] = output.transpose(0,1).contiguous().view(batch_size, -1)
        if USE_CUDA:
            encoder_input = encoder_input.cuda()

    temporary_target_batch_length = [15] * batch_size

    encoder_hidden_all, encoder_output = encoder(encoder_input, target_batch_length)
    decoder_input = Variable(torch.LongTensor([SOS_token] * batch_size))
    decoder_hidden = encoder_output

    max_target_length = max(temporary_target_batch_length)
    all_decoder_outputs = Variable(torch.zeros(max_target_length, batch_size, decoder.output_size))

    # Move new Variables to CUDA
    if USE_CUDA:
        decoder_input = decoder_input.cuda()
        all_decoder_outputs = all_decoder_outputs.cuda()
        target_batch =  target_batch.cuda()

    # Run through decoder one time step at a time
    for t in range(max_target_length):
        decoder_output, decoder_hidden, decoder_attn = decoder(
            decoder_input, decoder_hidden, encoder_hidden_all
        )

        all_decoder_outputs[t] = decoder_output
        decoder_input = target_batch[t] # Next input is current target
        if USE_CUDA:
            decoder_input = decoder_input.cuda()

    # Loss calculation and backpropagation
    loss = masked_cross_entropy(
        all_decoder_outputs.transpose(0, 1).contiguous(), # -> batch x seq
        target_batch.transpose(0, 1).contiguous(), # -> batch x seq
        target_batch_length
    )
    loss.backward()

    # Clip gradient norms
    ecc = torch.nn.utils.clip_grad_norm(encoderchar.parameters(), clip)
    ec = torch.nn.utils.clip_grad_norm(encoder.parameters(), clip)
    dc = torch.nn.utils.clip_grad_norm(decoder.parameters(), clip)

    # Update parameters with optimizers
    encoderchar_optimizer.step()
    encoder_optimizer.step()
    decoder_optimizer.step()

    return loss.data[0], ec, dc

Full Stack Trace is here.

RuntimeError                              Traceback (most recent call last)
<ipython-input-10-9778e12ded02> in <module>()
     11         data_target_batch_index= Variable(torch.LongTensor(data_target_batch_index)).transpose(0,1)
     12         # Send the data for training
---> 13         loss, ar1, ar2 = train(data_input_batch_index, data_input_batch_length, data_target_batch_index, data_target_batch_length, batch_size)
     14
     15         # Keep track of loss

<ipython-input-8-9c71c385f8cd> in train(input_batch, input_batch_length, target_batch, target_batch_length, batch_size)
     54         target_batch_length
     55     )
---> 56     loss.backward()
     57
     58     # Clip gradient norms

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/variable.py in backward(self, gradient, retain_variables)
    144                     'or with gradient w.r.t. the variable')
    145             gradient = self.data.new().resize_as_(self.data).fill_(1)
--> 146         self._execution_engine.run_backward((self,), (gradient,), retain_variables)
    147
    148     def register_hook(self, hook):

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/function.py in _do_backward(self, gradients, retain_variables)
    207     def _do_backward(self, gradients, retain_variables):
    208         self.retain_variables = retain_variables
--> 209         result = super(NestedIOFunction, self)._do_backward(gradients, retain_variables)
    210         if not retain_variables:
    211             del self._nested_output

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/function.py in backward(self, *gradients)
    215     def backward(self, *gradients):
    216         nested_gradients = _unflatten(gradients, self._nested_output)
--> 217         result = self.backward_extended(*nested_gradients)
    218         return tuple(_iter_None_tensors(result))
    219

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/_functions/rnn.py in backward_extended(self, grad_output, grad_hy)
    314             grad_hy,
    315             grad_input,
--> 316             grad_hx)
    317
    318         if any(self.needs_input_grad[1:]):

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/backends/cudnn/rnn.py in backward_grad(fn, input, hx, weight, output, grad_output, grad_hy, grad_input, grad_hx)
    371                 hidden_size, dcy.size()))
    372         if not dhy.is_cuda or not dy.is_cuda or (dcy is not None and not dcy.is_cuda):
--> 373             raise RuntimeError('Gradients aren\'t CUDA tensors')
    374
    375         check_error(cudnn.lib.cudnnRNNBackwardData(

RuntimeError: Gradients aren't CUDA tensors

any suggestions about why I am doing wrong?

Output:
Make sure that all the objects that inherit nn.Module also call their .cuda(). Make sure to call before you pass any tensor to them. (essentially before training)
For example, (and I am guessing your encoder and decoder are such objects), do this right before you call train().
encoder = encoder.cuda()
decoder = decoder.cuda()

This ensures that all of the model's parameters are initialized in cuda memory.
Edit
In general, whenever you have this kind of error,

RuntimeError: Gradients aren't CUDA tensors

somewhere, (from your model creation, to defining inputs, to finally supplying the outputs to the loss function) you missed specifying a Variable object to be in GPU memory. You will have go through every step in your model, verifying all Variable objects to be in GPU memory.
Additionally, you dont have to call .cuda() on the outputs. Given that the inputs are in gpu's memory, all operations also takes place in gpu's memory, and so are your outputs.


Instruction: Is torchvision.datasets.cifar.CIFAR10 a list or not?

Input:
When I try to figure out what is inside torchvision.datasets.cifar.CIFAR10, I did some simple code

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                    download=True, transform=transform)
print(trainset[1])
print(trainset[:10])
print(type(trainset))

However, I got some error when I try

print(trainset[:10])

The error info is

TypeError: Cannot handle this data type

I am wondering why I can use trainset[1], but not trainset[:10]?

Output:
Slicing isnt supported by CIFAR10, which is why you are getting that error. If you want the first 10 you will have to do this instead:
print([trainset[i] for i in range(10)])

More Info
The main reason why you can index an instance of CIFAR10 class is because the class implements __getitem__() function.
So, when you call trainset[i] you are essentially calling trainset.__getitem__(i)
Now, in python3, slicing expressions is also handled through __getitem__() where the slicing expression is passed to __getitem__() as a slice object.
So, trainset[2:10] is equivalent to trainset.__getitem__(slice(2, 10))
And since the two different types of objects being passed to __getitem__
are expected to do entirely different things, you have to handle them explicitly.
Unfortunately it isnt, as you can see from the __getitem__ method implementation of CIFAR10 class:
def __getitem__(self, index):
    if self.train:
        img, target = self.train_data[index], self.train_labels[index]
    else:
        img, target = self.test_data[index], self.test_labels[index]

    # doing this so that it is consistent with all other datasets
    # to return a PIL Image
    img = Image.fromarray(img)

    if self.transform is not None:
        img = self.transform(img)

    if self.target_transform is not None:
        target = self.target_transform(target)

    return img, target


Instruction: Is there a way to use an external loss function in pytorch?

Input:
A typical skeleton of pytorch neural network has a forward() method, then we compute loss based on outputs of forward pass, and call backward() on that loss to update the gradients. What if my loss is determined externally (e.g. by running simulation in some RL environment). Can I still leverage this typical structure this way?

This might be a bit dumb as we no longer know exactly how much each element of output influences the loss, but perhaps there is some trickery I'm not aware of. Otherwise I'm not sure how neural nets can be used in combination with other RL algorithms.

Thank you!

Output:
In this case it appears easiest to me abstract the forward pass (your policy?) from the loss computation. This is because (as you note) in most scenarios, you will need to obtain a state (from your environment), then compute an action (essentially the forward pass), then feed that action back to the environment to obtain a reward/ loss from your environment.

Of course, you could probably call your environment within the forward pass once you computed an action to then calculate the resultant loss. But why bother? It will get even more complicated (though possible) once you are are taking several steps in your environment until you obtain a reward/ loss.

I would suggest you take a look at the following RL example for an application of policy gradients within openAI gym: https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py#L43

The essential ideas are:

Create a policy (as an nn.module) that takes in a state and returns
a stochastic policy
Wrap the computation of a policy and the sampling of an action from the policy into one function.
Call this function repeatedly to take steps through your environment, record actions and rewards.
Once an episode is finished, register rewards and perform only now the back-propagation and gradient updates.

While this example is specific to REINFORCE, the general idea of structuring your code is applicable to other RL algorithms. Besides, you'll find two other examples in the same repo.

Hope this helps.


Instruction: Numpy/PyTorch method for partial tiling

Input:
I seem to recall encountering a Numpy or PyTorch method similar to numpy.tile, except that it allowed partial tiling to reach specified dimensions. So if I had

a = np.array([[5, 6, 7, 8],
              [1, 2, 3, 4]])

(or, correspondingly, t = torch.Tensor(a) if this is a PyTorch method),

then

a.mystery_method((3,4))

would produce output

array([[5, 6, 7, 8],
       [1, 2, 3, 4],
       [5, 6, 7, 8]])

Unfortunately, I now can't find this method in the Numpy or Pytorch docs. Does it actually exist, and what is it if so?

Output:
You can use np.resize -

M = 3 # number of rows for output
np.resize(a,(M,a.shape[1]))

Another way with np.take or simply indexing along the first axis for performance -

np.take(a,np.arange(M)%a.shape[0],axis=0) # with np.take
a[np.arange(M)%a.shape[0]]                # with indexing

Runtime test -

In [91]: a = np.random.randint(0,9,(2000,4000))

In [92]: M = 3000

In [93]: %timeit np.resize(a,(M,a.shape[1]))
10 loops, best of 3: 24.1 ms per loop

In [94]: %timeit np.take(a,np.arange(M)%a.shape[0],axis=0)
100 loops, best of 3: 16.2 ms per loop

In [95]: %timeit a[np.arange(M)%a.shape[0]]
100 loops, best of 3: 15.2 ms per loop


Instruction: pytorch: how to directly find gradient w.r.t. loss

Input:
In theano, it was very easy to get the gradient of some variable w.r.t. a given loss:

loss = f(x, w)
dl_dw = tt.grad(loss, wrt=w)

I get that pytorch goes by a different paradigm, where you'd do something like:

loss = f(x, w)
loss.backwards()
dl_dw = w.grad

The thing is I might not want to do a full backwards propagation through the graph - just along the path needed to get to w.

I know you can define Variables with requires_grad=False if you don't want to backpropagate through them.  But then you have to decide that at the time of variable-creation (and the requires_grad=False property is attached to the variable, rather than the call which gets the gradient, which seems odd).

My Question is is there some way to backpropagate on demand (i.e. only backpropagate along the path needed to compute dl_dw, as you would in theano)?

Output:
It turns out that this is reallyy easy.  Just use torch.autograd.grad

Example:

import torch
import numpy as np
from torch.autograd import grad

x = torch.autograd.Variable(torch.from_numpy(np.random.randn(5, 4)))
w = torch.autograd.Variable(torch.from_numpy(np.random.randn(4, 3)), requires_grad=True)
y = torch.autograd.Variable(torch.from_numpy(np.random.randn(5, 3)))
loss = ((x.mm(w) - y)**2).sum()
(d_loss_d_w, ) = grad(loss, w)

assert np.allclose(d_loss_d_w.data.numpy(), (x.transpose(0, 1).mm(x.mm(w)-y)*2).data.numpy())

Thanks to JerryLin for answering the question here.


Instruction: Running conv2d on tensor [batch, channel, sequence, H,W] in Pytorch

Input:
I am working on a video frame data where I am getting input data as tensor of the form [batch,channel,frame_sequence,height, weight] (let denote it by [B,C,S,H,W] for clarity. So each batch basically consists of a consecutive sequence of frame. What I basically want to do is run an encoder (consisting of several conv2d) on each frame ie each [C,H,W] and get it back as [B,C_output,S,H_output,W_output]. Now conv2d expects input as (N,C_in,H_in,W_in) form. I am wondering what's the best way to do this without messing up the order within the 5D tensor.
So far I am considering following way of thoughts:

>>> # B,C,seq,h,w
# 4,2, 5,  3,3

>>> x = Variable(torch.rand(4,2,5,3,3))
>>> x.size()
#torch.Size([4, 2, 5, 3, 3])
>>> x = x.permute(0,2,1,3,4)
>>> x.size() #expected = 4,5,2,3,3 B,seq,C,h,w
#torch.Size([4, 5, 2, 3, 3])
>>> x = x.contiguous().view(-1,2,3,3)
>>> x.size()
#torch.Size([20, 2, 3, 3])

And then run conv2d (encoder) on the updated x and reshape it. But I think It wouldn't preserve the original order of tensor. So, how can I achieve the goal?

Output:
What you are doing is completely fine. It will preserve the order. You can verify this by visualizing them.

I quickly built this for displaying the images stored in a 4d tensor (where dim=0 is batch) or a 5d tensor (where dim=0 is batch and dim=1 is sequence):

def custom_imshow(tensor):
    if tensor.dim() == 4:
        count = 1
        for i in range(tensor.size(0)):
            img = tensor[i].numpy()
            plt.subplot(1, tensor.size(0), count)
            img = img / 2 + 0.5     # unnormalize
            img = np.transpose(img, (1, 2, 0))
            count += 1
            plt.imshow(img)
            plt.axis('off')

    if tensor.dim() == 5:
        count = 1
        for i in range(tensor.size(0)):
            for j in range(tensor.size(1)):
                img = tensor[i][j].numpy()
                plt.subplot(tensor.size(0), tensor.size(1), count)
                img = img / 2 + 0.5  # unnormalize
                img = np.transpose(img, (1, 2, 0))

                plt.imshow(img)
                plt.axis('off')
                count +=1

Lets say we use the CIFAR-10 dataset (consisting of 32x32x3 size images).

For a tensor x:

>>> x.size()
torch.Size([4, 5, 3, 32, 32])
>>> custom_imshow(x)

After doing x.view(-1, 3, 32, 32):

 # x.size() -> torch.Size([4, 5, 3, 32, 32])
 >>> x = x.view(-1, 3, 32, 32)
 >>> x.size()
 torch.Size([20, 3, 32, 32])
 >>> custom_imshow(x)

And if you go back to a 5d tensor view:

# x.size() -> torch.Size([20, 3, 32, 32])
>>> x.view(4, 5, 3, 32, 32)
>>> x.size()
torch.Size([4, 5, 3, 32, 32])
>>> custom_imshow(x)


Instruction: Replace all nonzero values by zero and all zero values by a specific value

Input: I have a 3d tensor which contains some zero and nonzero values. I want to replace all nonzero values by zero and zero values by a specific value. How can I do that?

Output:
Pretty much exactly how you would do it using numpy, like so:

tensor[tensor!=0] = 0

In order to replace zeros and non-zeros, you can just chain them together. Just be sure to use a copy of the tensor, since they get modified:

def custom_replace(tensor, on_zero, on_non_zero):
    # we create a copy of the original tensor,
    # because of the way we are replacing them.
    res = tensor.clone()
    res[tensor==0] = on_zero
    res[tensor!=0] = on_non_zero
    return res

And use it like so:

>>>z
(0 ,.,.) =
  0  1
  1  3

(1 ,.,.) =
  0  1
  1  0
[torch.LongTensor of size 2x2x2]

>>>out = custom_replace(z, on_zero=5, on_non_zero=0)
>>>out
(0 ,.,.) =
  5  0
  0  0

(1 ,.,.) =
  5  0
  0  5
[torch.LongTensor of size 2x2x2]


Instruction: Tensor type mismatch when moving to GPU

Input:
I'm getting the following error when trying to move my network and tensors to GPU. I've checked that the network parameters are moved to the GPU and check each batch's tensor and move them if they're not already on the GPU. But I'm still getting this issue say that there's a mismatch in the tensor types - one is a torch.cuda.FloatTensor and the other is a torch.FloatTensor? Could someone tell me what I'm doing wrong? Thanks.

My code:

class Train():
  def __init__(self, network, training, address):
    self.network    = network
    self.address    = address
    self.batch_size = training['batch_size']
    self.iterations = training['iterations']
    self.samples    = training['samples']
    self.data       = training['data']
    self.lr         = training['lr']
    self.noisy_lr   = training['nlr']
    self.cuda       = training['cuda']
    self.save       = training['save']
    self.scale      = training['scale']
    self.limit      = training['limit']
    self.replace    = training['strategy']
    self.optimizer  = torch.optim.Adam(self.network.parameters(), lr=self.lr)

  def tensor_to_Variable(self, t):
    if next(self.network.parameters()).is_cuda and not t.is_cuda:
        t = t.cuda()

    return Variable(t)

  def train(self):
    if self.cuda:
        self.network.cuda()
    dh = DataHandler(self.data)
    loss_fn = torch.nn.MSELoss()
    losses    = []
    validate  = []
    val_size  = 100
    val_diff  = 1
    total_val = float(val_size * self.batch_size)
    hypos     = []
    labels    = []

    # training loop
    for i in range(self.iterations):
        x, y = dh.get_batch(self.batch_size)
        x = self.tensor_to_Variable(x)
        y = self.tensor_to_Variable(y)

        self.optimizer.zero_grad()
        hypo = self.network(x)
        loss = loss_fn(hypo, y)
        loss.backward()
        self.optimizer.step()

class Feedforward(nn.Module):
   def __init__(self, topology):
    super(Feedforward, self).__init__()
    self.input_dim     = topology['features']
    self.num_hidden    = topology['hidden_layers']
    self.hidden_dim    = topology['hidden_dim']
    self.output_dim    = topology['output_dim']
    self.input_layer   = nn.Linear(self.input_dim, self.hidden_dim)
    self.hidden_layer  = nn.Linear(self.hidden_dim, self.hidden_dim)
    self.output_layer  = nn.Linear(self.hidden_dim, self.output_dim)
    self.dropout_layer = nn.Dropout(p=0.2)

def forward(self, x):
    batch_size = x.size()[0]
    feat_size  = x.size()[1]
    input_size = batch_size * feat_size

    self.input_layer = nn.Linear(input_size, self.hidden_dim)
    hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)

    for _ in range(self.num_hidden):
        hidden = self.dropout_layer(F.relu(self.hidden_layer(hidden)))

    output_size = batch_size * self.output_dim
    self.output_layer = nn.Linear(self.hidden_dim, output_size)
    return self.output_layer(hidden).view(output_size)

The error:

Traceback (most recent call last):
  File "/media/project/train.py", line 78, in train
    hypo = self.network(x)

 * (torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (float beta, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (float beta, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
 * (float beta, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
      didn't match because some of the arguments have invalid types: (int, int, torch.cuda.FloatTensor, torch.FloatTensor)
 * (float beta, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
      didn't match because some of the arguments have invalid types: (int, int, torch.cuda.FloatTensor, torch.FloatTensor)

Stacktrace:

Traceback (most recent call last):
File "smpl.py", line 90, in <module>
main()
File "smpl.py", line 80, in main
trainer.train()
File "/media/mpl/temp/train.py", line 82, in train
hypo = self.network(x)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(input, **kwargs)
File "model/network.py", line 35, in forward
hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(input, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/linear.py", line 54, in forward
return self.backend.Linear()(input, self.weight, self.bias)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/linear.py", line 10, in forward
output.addmm(0, 1, input, weight.t())
TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.cuda.FloatTensor, torch.FloatTensor), but expected one of: (torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
(torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2) (float beta, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
(float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2) (float beta, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
(float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2) (float beta, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
didn't match because some of the arguments have invalid types: (int, int, torch.cuda.FloatTensor, torch.FloatTensor)
* (float beta, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
didn't match because some of the arguments have invalid types: (int, int, torch.cuda.FloatTensor, torch.FloatTensor

Output:
This is happening because you are re-initializing self.input_layer in your forward() function.

The call self.network.cuda() moves all of the model parameters into cuda. Which means any and all the layers you initialize at the creation of your FeedForward object will be moved to cuda memory. But when you reinitialize self.input_layer in your forward() function, you initialize that layer's parameters in cpu and not gpu. Same goes for self.output_layer.


Instruction: TypeError: a float is required

Input:
l have a pytorch variable :

preds[4,4]
Out[305]:
Variable containing:
-96.7809
[torch.cuda.FloatTensor of size 1 (GPU 0)]

l want to do the following :

 import math
 x=preds[4,4]
 y=maths.exp(x)
 z= y / (y+1)

However when l do :

y=maths.exp(x)

l get the following error :

   math.exp(preds[4,4])
TypeError: a float is required

How can l transform a torch variable to a float in order to be able to do these operations ?

Thanks

Output:
Indexing a Variable object doesnt convert it into a scalar. Its still a Variable object. However indexing a numpy array does. So converting the Variable object into a numpy and then indexing the way you want it should do the trick.
But there are some small pitfalls when converting a Variable to numpy.
If preds is a Variable stored in cpu memory, you can simply do this.
nparr = preds.data.numpy()
x = nparr[4, 4]

However, if preds is in gpu memory, you will have to first transfer the Variable into cpu memory before you convert it into a numpy object, like so:
preds = preds.cpu()

and then do the same as above.
nparr = preds.data.numpy()
x = nparr[4, 4]

In both cases x is a scalar (a float in your case) and you can use it in any math operations you choose.
Edit:
yes, @mexmex is right, you could also directly index the tensor that is wrapped in the Variable to extract the scalar value at any given index.
Like so:
 x = preds.data[4, 4]


Instruction: Regression loss functions incorrect

Input:
I'm trying a basic averaging example, but the validation and loss don't match and the network fails to converge if I increase the training time. I'm training a network with 2 hidden layers, each 500 units wide on three integers from the range [0,9] with a learning rate of 1e-1, Adam, batch size of 1, and dropout for 3000 iterations and validate every 100 iterations. If the absolute difference between the label and the hypothesis is less than a threshold, here I set the threshold to 1, I consider that correct. Could someone let me know if this is an issue with the choice of loss function, something wrong with Pytorch, or something I'm doing. Below are some plots:

val_diff = 1
acc_diff = torch.FloatTensor([val_diff]).expand(self.batch_size)

Loop 100 times to during validation:

num_correct += torch.sum(torch.abs(val_h - val_y) < acc_diff)

Append after each validation phase:

validate.append(num_correct / total_val)

Here are some examples of the (hypothesis, and labels):

[...(-0.7043088674545288, 6.0), (-0.15691305696964264, 2.6666667461395264),
 (0.2827358841896057, 3.3333332538604736)]

I tried six of the loss functions in the API that are typically used for regression:

                                                    torch.nn.L1Loss(size_average=False)

                                                                      torch.nn.L1Loss()

                                                     torch.nn.MSELoss(size_average=False)

                                                                   torch.nn.MSELoss()

                                             torch.nn.SmoothL1Loss(size_average=False)

                                                               torch.nn.SmoothL1Loss()

Thanks.

Network code:

class Feedforward(nn.Module):
    def __init__(self, topology):
        super(Feedforward, self).__init__()
        self.input_dim     = topology['features']
        self.num_hidden    = topology['hidden_layers']
        self.hidden_dim    = topology['hidden_dim']
        self.output_dim    = topology['output_dim']
        self.input_layer   = nn.Linear(self.input_dim, self.hidden_dim)
        self.hidden_layer  = nn.Linear(self.hidden_dim, self.hidden_dim)
        self.output_layer  = nn.Linear(self.hidden_dim, self.output_dim)
        self.dropout_layer = nn.Dropout(p=0.2)

    def forward(self, x):
        batch_size = x.size()[0]
        feat_size  = x.size()[1]
        input_size = batch_size * feat_size

        self.input_layer = nn.Linear(input_size, self.hidden_dim).cuda()
        hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)

        for _ in range(self.num_hidden):
            hidden = self.dropout_layer(F.relu(self.hidden_layer(hidden)))

        output_size = batch_size * self.output_dim
        self.output_layer = nn.Linear(self.hidden_dim, output_size).cuda()
        return self.output_layer(hidden).view(output_size)

Training code:

def train(self):
    if self.cuda:
        self.network.cuda()

    dh        = DataHandler(self.data)
    # loss_fn = nn.L1Loss(size_average=False)
    # loss_fn = nn.L1Loss()
    # loss_fn = nn.SmoothL1Loss(size_average=False)
    # loss_fn = nn.SmoothL1Loss()
    # loss_fn = nn.MSELoss(size_average=False)
    loss_fn   = torch.nn.MSELoss()
    losses    = []
    validate  = []
    hypos     = []
    labels    = []
    val_size  = 100
    val_diff  = 1
    total_val = float(val_size * self.batch_size)

    for i in range(self.iterations):
        x, y = dh.get_batch(self.batch_size)
        x = self.tensor_to_Variable(x)
        y = self.tensor_to_Variable(y)

        self.optimizer.zero_grad()
        loss = loss_fn(self.network(x), y)
        loss.backward()
        self.optimizer.step()

Output:
It looks like you've misunderstood how layers in pytorch works, here are a few tips:

In your forward when you do nn.Linear(...) you are definining new layers instead of using those you pre-defined in your network __init__. Therefore, it cannot learn anything as weights are constantly reinitalized.
You shouldn't need to call .cuda() inside net.forward(...) since you've already copied the network on gpu in your train by calling self.network.cuda()
Ideally the net.forward(...) input should directly have the shape of the first layer so you won't have to modify it. Here you should have x.size() <=> Linear -- > (Batch_size,  Features).

Your forward should look close to this:

def forward(self, x):
    x = F.relu(self.input_layer(x))
    x = F.dropout(F.relu(self.hidden_layer(x)),training=self.training)
    x = self.output_layer(x)
    return x


Instruction: Any pytorch tools to monitor neural network's training?

Input: Are there any tools to monitor network's training in PyTorch? Like tensorboard in tensorflow.

Output:
I am using tensorboardX. It supports most (if not all) of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. I haven't tried the rest, like audio and graph, but the repo also contains examples for those use cases. The installation can be done easily with pip. It's all explained in the README file of the repo.

There are also other github repos which implement a wrapper for PyTorch (and other languages/frameworks) to tensorboard. As far as I know they support fewer functionalities. But have a look at:

Crayon
Tensorboard-Logger


Instruction: Flatten layer of PyTorch build by sequential container

Input:
I am trying to build a cnn by sequential container of PyTorch, my problem is I cannot figure out how to flatten the layer.

main = nn.Sequential()
self._conv_block(main, 'conv_0', 3, 6, 5)
main.add_module('max_pool_0_2_2', nn.MaxPool2d(2,2))
self._conv_block(main, 'conv_1', 6, 16, 3)
main.add_module('max_pool_1_2_2', nn.MaxPool2d(2,2))
main.add_module('flatten', make_it_flatten)

What should I put in the "make_it_flatten"?
I tried to flatten the main but it do not work, main do not exist something call view

main = main.view(-1, 16*3*3)

Output:
This might not be exactly what you are looking for, but you can simply create your own nn.Module that flattens any input, which you can then add to the nn.Sequential() object:

class Flatten(nn.Module):
    def forward(self, x):
        return x.view(x.size()[0], -1)

The x.size()[0] will select the batch dim, and -1 will compute all remaining dims to fit the number of elements, thereby flattening any tensor/Variable.

And using it in nn.Sequential:

main = nn.Sequential()
self._conv_block(main, 'conv_0', 3, 6, 5)
main.add_module('max_pool_0_2_2', nn.MaxPool2d(2,2))
self._conv_block(main, 'conv_1', 6, 16, 3)
main.add_module('max_pool_1_2_2', nn.MaxPool2d(2,2))
main.add_module('flatten', Flatten())


Instruction: Which deep learning library support the compression of the deep learning models to be used on the phones?

Input:
I want to build an advanced deep learning model (for example: a model that uses attention) and use it on android phones (without training of course) i will only use it for inference.
and i want a library that can do that and can compress the model size to be used on the phone or android.
and do you know any projects or apps similar to my goal?

Output:
There is a Caffe fork called Ristretto. It allows compressing neural nets for lower numerical precision (less than 32 bits per parameter), while keeping high accuracy. MXNet and Tensorflow also have this feature now. Pytorch doesn't have it yet.
These tools allow to reduce the memory required for storing the neural net parameters, but they are not specific to Android.


Instruction: KeyError when trying to modify pytorch-example

Input:
I'm trying to modify this pytorch-example (https://github.com/pytorch/examples/blob/master/mnist/main.py) to work with my own dataset.

I tried to feed my data into a dataloader. I encapsulated the data in two different ways: one time as an extension of torch.utils.data.Dataset and one time as torch.utils.data.TensorDataset. Unfortunately I always get the same error that I don't understand:

Traceback (most recent call last):
  File "main.py", line 142, in <module>
    train(epoch)
  File "main.py", line 112, in train
    output = model(data)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 210, in __call__
    result = self.forward(*input, **kwargs)
  File "main.py", line 90, in forward
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 210, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 235, in forward
    self.padding, self.dilation, self.groups)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/functional.py", line 54, in conv2d
    return f(input, weight, bias) if bias is not None else f(input, weight)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 33, in forward
    output = self._update_output(input, weight, bias)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 88, in _update_output
    return self._thnn('update_output', input, weight, bias)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 147, in _thnn
    return impl[fn_name](self, self._bufs[0], input, weight, *args)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 213, in call_update_output
    backend = type2backend[type(input)]
  File "/usr/local/lib/python2.7/dist-packages/torch/_thnn/__init__.py", line 13, in __getitem__
    return self.backends[name].load()
KeyError: <class 'torch.cuda.ByteTensor'>

Here's my main.py, which is basically this example: https://github.com/pytorch/examples/blob/master/mnist/main.py

from __future__ import print_function
import argparse
import os
import glob
import numpy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data_utils
from PIL import Image
from torchvision import datasets, transforms
from torch.autograd import Variable
from InputData import InputData

# Training settings
parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                    help='input batch size for training (default: 64)')
parser.add_argument('--test-batch-size', type=int, default=10, metavar='N',
                    help='input batch size for testing (default: 1000)')
parser.add_argument('--epochs', type=int, default=1, metavar='N',
                    help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                    help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                    help='SGD momentum (default: 0.5)')
parser.add_argument('--no-cuda', action='store_true', default=False,
                    help='disables CUDA training')
parser.add_argument('--seed', type=int, default=1, metavar='S',
                    help='random seed (default: 1)')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                    help='how many batches to wait before logging training status')
args = parser.parse_args()
args.cuda = not args.no_cuda and torch.cuda.is_available()

torch.manual_seed(args.seed)
if args.cuda:
    torch.cuda.manual_seed(args.seed)

kwargs = {'num_workers': 1, 'pin_memory': True} if args.cuda else {}

# Original DataLoader - WORKS:
'''
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
'''

# DataLoader as extension of data.Dataset:

train_loader = torch.utils.data.DataLoader(InputData('~/bakk-arbeit/data', train=True),
                                           batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(InputData('~/bakk-arbeit/data', train=False),
                                          batch_size=args.batch_size, shuffle=True, **kwargs)

# DataLoader as data.TensorDataset:
'''
data_folder = os.path.expanduser('~/bakk-arbeit/data')
InputData = InputData()
train = data_utils.TensorDataset(InputData.read_image_files(os.path.join(data_folder, 'training')),InputData.read_label_files(os.path.join(data_folder, 'training')))
test = data_utils.TensorDataset(InputData.read_image_files(os.path.join(data_folder, 'test')),InputData.read_label_files(os.path.join(data_folder, 'test')))
train_loader = data_utils.DataLoader(train, batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = data_utils.DataLoader(test, batch_size=args.batch_size, shuffle=True, **kwargs)
'''

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5) # change to 3 input channels for InputData!
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50) # change 320 to 500 for InputData to match 32x32
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320) # # change 320 to 500 for InputData to match 32x32
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

model = Net()
if args.cuda:
    model.cuda()

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # data = data.numpy()
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data[0] # sum up batch loss
        pred = output.data.max(1)[1] # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

for epoch in range(1, args.epochs + 1):
    train(epoch)
    test()

...and here's my InputData.py that extends data.Dataset:

import torch
import numpy
import torch.utils.data as data
import glob
import os
from PIL import Image

class InputData(data.Dataset):
    train_folder = 'training'
    test_folder = 'test'

    def __init__(self, root='', train=True):
        self.root = os.path.expanduser(root)
        self.train = train  # training set or test set
        if root:
            if self.train:
                self.training_labels = self.read_label_files(os.path.join(self.root, self.train_folder))
                #with open(os.path.join(self.root, 'training_labels.pt'), 'wb') as f:
                    # torch.save(self.read_label_files(os.path.join(self.root, self.train_folder)), f)
                # with open(os.path.join(self.root, 'training_images.pt'), 'wb') as f:
                    #torch.save(self.read_image_files(os.path.join(self.root, self.train_folder)), f)
                self.training_images = self.read_image_files(os.path.join(self.root, self.train_folder))
            else:
                self.test_images = self.read_image_files(os.path.join(self.root, self.test_folder))
                self.test_labels = self.read_label_files(os.path.join(self.root, self.test_folder))
        print('initialized')

    def read_image_files(self, path):
        print('reading image files...')
        image_list = []
        # ten = torch.ByteTensor(3,32,32)
        for filename in glob.glob(path + '/*.png'):
            im = Image.open(filename)
            data = numpy.asarray(im)
            data = numpy.swapaxes(data,0,2)
            image_list.append(data)
        image_list = numpy.asarray(image_list)
        t = torch.from_numpy(image_list)
        # ten = torch.stack([ten, t])
        print('done!')
        return t

    def read_label_files(self, path):
        print('reading labels...')
        labels = []
        for filename in glob.glob(path + '/*.png'):
            base = os.path.basename(filename)
            im_class = int(base[:1])
            labels.append(im_class)
        print('done!')
        return torch.LongTensor(labels)

    def __getitem__(self, index):
        """
        Args:
            index (int): Index

        Returns:
            tuple: (image, target) where target is index of the target class.
        """
        if self.train:
            img, target = self.training_images[Index], self.training_labels[Index]
        else:
            img, target = self.test_images[Index], self.test_labels[Index]

        # img = Image.fromarray(img.numpy(), mode='RGB')
        # -> won't work for me??? returns TypeError: batch must contain tensors, numbers, or lists; found <class 'PIL.Image.Image'>

        return img, target

    def __len__(self):
        if self.train:
            return len(self.training_images)
        else:
            return len(self.test_images)

What am I doing wrong?

Output:
It seems that most operation are defined on FloatTensor and DoubleTensor (source), and your model gets a ByteTensor in model(data).

I would go ahead an make sure that my dataset object outputs FloatTensors. Debug the line before model(data) and see the tensor type of data. I would guess it's a ByteTensor, that would be a good place to start.


Instruction: Axes don't match array error in pytorch

Input:
I was trying to run the vnet implementation in pytorch
(https://github.com/mattmacy/vnet.pytorch) and after normalising the scans with

x_max = 512
y_max = 512
z_max = 500
voxspacing = 0.7

when I call the tran function on the line where the for loop is enumerating through the data loader I get a

  axes don't match array error.

I don't understand how to fix this. Any help would be really helpful. I am a newbie in machine learning

Output:
comment the lines 417 and 418. the issue will get fixed

the issue is because of these 2 lines

if self.transform is not None:
     img = self.transform(img)


Instruction: Error importing PyTorch - Python

Input:
I have tried to install PyTorch using the following command in terminal:

pip install http://download.pytorch.org/whl/torch-0.2.0.post1-cp27-none-macosx_10_7_x86_64.whl

I then run the following code in python:

import torch
torch.__file__

and get the following error:

 File "/Users/brian/anaconda/lib/python2.7/site-packages/torch/__init__.py", line 48, in <module>
import torch._dl as _dl_flags

AttributeError: 'module' object has no attribute '_dl'

I have attempted to research the issue but cannot find a clear answer. any help would be appreciated.

EDIT:
These are the results from trying to re-install PyTorch using the following commands:

pip install http://download.pytorch.org/whl/torch-0.2.0.post1-cp27-none-macosx_10_7_x86_64.whl
pip install torchvision

Results:

Requirement already satisfied: pyyaml in ./anaconda/lib/python2.7/site-packages (from torch==0.2.0.post1)
Requirement already satisfied: numpy in ./anaconda/lib/python2.7/site-packages (from torch==0.2.0.post1)

Requirement already satisfied: torchvision in ./anaconda/lib/python2.7/site-packages
Requirement already satisfied: pillow in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: torch in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: numpy in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: six in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: olefile in ./anaconda/lib/python2.7/site-packages (from pillow->torchvision)
Requirement already satisfied: pyyaml in ./anaconda/lib/python2.7/site-packages (from torch->torchvision)

EDIT II

The following command:

conda install pytorch torchvision cuda80 -c soumith

yields this result:

Fetching package metadata ...........

PackageNotFoundError: Packages missing in current channels:

   - cuda80

We have searched for the packages in the following channels:

  - https://conda.anaconda.org/soumith/osx-64
  - https://conda.anaconda.org/soumith/noarch
  - https://repo.continuum.io/pkgs/free/osx-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/osx-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/osx-64
  - https://repo.continuum.io/pkgs/pro/noarch

Output:
On http://pytorch.org/ With:

OS: OSX
Package Manager: pip
Python: 2.7
CUDA: None

I've got:

pip install http://download.pytorch.org/whl/torch-0.2.0.post1-cp27-none-macosx_10_7_x86_64.whl
pip install torchvision
# OSX Binaries dont support CUDA, install from source if CUDA is needed

Are you sure you select all correct options?
Have you try the second command, pip install torchvision?

Edit

It seems in your log that you have anaconda installed on your mac, that mean you should have select Package manager: conda rather than pip.

In that case, you should remove the Package installed using pip:

pip uninstall torch

And use the command using anaconda instead:

OS: OSX
Package Manager: conda
Python: 2.7
CUDA: None

run the command:

conda install pytorch torchvision -c soumith


Instruction: PyTorch: How to get around the RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables

Input:
With PyTorch I'm having a problem doing an operation with two Variables:

sub_patch  : [torch.FloatTensor of size 9x9x32]

pred_patch : [torch.FloatTensor of size 5x5x32]

sub_patch is a Variable made by torch.zeros
pred_patch is a Variable of which I index each of the 25 nodes with a nested for-loop, and that I multiply with its corresponding unique filter (sub_filt_patch) of size [5,5,32]. The result is added to its respective place in sub_patch.

This is a piece of my code:

for i in range(filter_sz):
    for j in range(filter_sz):

        # index correct filter from filter tensor
        sub_filt_col = (patch_col + j) * filter_sz
        sub_filt_row = (patch_row + i) * filter_sz

        sub_filt_patch = sub_filt[sub_filt_row:(sub_filt_row + filter_sz), sub_filt_col:(sub_filt_col+filter_sz), :]

        # multiply filter and pred_patch and sum onto sub patch
        sub_patch[i:(i + filter_sz), j:(j + filter_sz), :] += (sub_filt_patch * pred_patch[i,j]).sum(dim=3)

The error I get from the bottom line of the piece of code here is

RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables, but detected that there are 2 objects sharing it

I get why it happens, since sub_patch is a Variable, and pred_patch is a Variable too, but how can I get around this error? Any help would be greatly appreciated!

Thank you!

Output:
I've found the problem to be in

        sub_patch[i:(i + filter_sz), j:(j + filter_sz), :] += (sub_filt_patch * pred_patch[i,j]).sum(dim=3)

When separating this line into this:

sub_patch[i:(i + filter_sz), j:(j + filter_sz), :] = sub_patch[i:(i + filter_sz), j:(j + filter_sz), :] + (sub_filt_patch * pred_patch[i,j]).sum(dim=3)

Then it worked!

The difference between a += b and a = a + b is that in the first case, b is added in a inplace (so the content of a is changed to now contain a+b). In the second case, a brand new tensor is created that contains a+b and then you assign this new tensor to the name a.
To be able to compute gradients, you sometimes need to keep the original value of a, and so we prevent inplace operation from being done because otherwise, we won't be able to compute gradients.


Instruction: What is the first parameter (gradients) of the backward method, in pytorch?

Input:
We have the following code from the pytorch documentation:

x = torch.randn(3)
x = Variable(x, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)

What exactly is the gradients parameter that we pass into the backward method and based on what do we initialize it?

Output:
To fully answer your question, it'd require a somewhat longer explanation that evolves around the details of how Backprop or, more fundamentally, the chain rule works.

The short programmatic answer is that the backwards function of a Variable computes the gradient of all variables in the computation graph attached to that Variable. (To clarify: if you have a = b + c, then the computation graph (recursively) points first to b, then to c, then to how those are computed, etc.) and cumulatively stores (sums) these gradients in the .grad attribute of these Variables. When you then call opt.step(), i.e. a step of your optimizer, it adds a fraction of that gradient to the value of these Variables.

That said, there are two answers when you look at it conceptually: If you want to train a Machine Learning model, you normally want to have the gradient with respect to some loss function. In this case, the gradients computed will be such that the overall loss (a scalar value) will decrease when applying the step function. In this special case, we want to compute the gradient to a specific value, namely the unit length step (so that the learning rate will compute the fraction of the gradients that we want). This means that if you have a loss function, and you call loss.backward(), this will compute the same as loss.backward(torch.FloatTensor([1.])).

While this is the common use case for backprop in DNNs, it is only a special case of general differentiation of functions. More generally, the symbolic differentiation packages (autograd in this case, as part of pytorch) can be used to compute gradients of earlier parts of the computation graph with respect to any gradient at a root of whatever subgraph you choose. This is when the key-word argument gradient comes in useful, since you can provide this "root-level" gradient there, even for non-scalar functions!

To illustrate, here's a small example:

a = nn.Parameter(torch.FloatTensor([[1, 1], [2, 2]]))
b = nn.Parameter(torch.FloatTensor([[1, 2], [1, 2]]))
c = torch.sum(a - b)
c.backward(None)  # could be c.backward(torch.FloatTensor([1.])) for the same result
print(a.grad, b.grad)

prints:

Variable containing:
 1  1
 1  1
[torch.FloatTensor of size 2x2]
 Variable containing:
-1 -1
-1 -1
[torch.FloatTensor of size 2x2]

While

a = nn.Parameter(torch.FloatTensor([[1, 1], [2, 2]]))
b = nn.Parameter(torch.FloatTensor([[1, 2], [1, 2]]))
c = torch.sum(a - b)
c.backward(torch.FloatTensor([[1, 2], [3, 4]]))
print(a.grad, b.grad)

prints:

Variable containing:
 1  2
 3  4
[torch.FloatTensor of size 2x2]
 Variable containing:
-1 -2
-3 -4
[torch.FloatTensor of size 2x2]

and

a = nn.Parameter(torch.FloatTensor([[0, 0], [2, 2]]))
b = nn.Parameter(torch.FloatTensor([[1, 2], [1, 2]]))
c = torch.matmul(a, b)
c.backward(torch.FloatTensor([[1, 1], [1, 1]]))  # we compute w.r.t. a non-scalar variable, so the gradient supplied cannot be scalar, either!
print(a.grad, b.grad)

prints

Variable containing:
 3  3
 3  3
[torch.FloatTensor of size 2x2]
 Variable containing:
 2  2
 2  2
[torch.FloatTensor of size 2x2]

and

a = nn.Parameter(torch.FloatTensor([[0, 0], [2, 2]]))
b = nn.Parameter(torch.FloatTensor([[1, 2], [1, 2]]))
c = torch.matmul(a, b)
c.backward(torch.FloatTensor([[1, 2], [3, 4]]))  # we compute w.r.t. a non-scalar variable, so the gradient supplied cannot be scalar, either!
print(a.grad, b.grad)

prints:

Variable containing:
  5   5
 11  11
[torch.FloatTensor of size 2x2]
 Variable containing:
 6  8
 6  8
[torch.FloatTensor of size 2x2]


Instruction: How to run Pytorch model in normal non-parallel way?

Input:
I am going through this script, and there is a code block which takes 2 options into account, DataParallel and DistributedDataParallel here:

if not args.distributed:
    if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
        model.features = torch.nn.DataParallel(model.features)
        model.cuda()
    else:
        model = torch.nn.DataParallel(model).cuda()
else:
    model.cuda()
    model = torch.nn.parallel.DistributedDataParallel(model)

What if I don't want either of these options, and I want to run it without even DataParallel. How do I do it?

How do I define my model so that it runs as a plain nn and not parallelizing anything?

Output:
DataParallel is a wrapper object to parallelize the computation on multiple GPUs of the same machine, see here.
DistributedDataParallel is also a wrapper object that lets you distribute the data on multiple devices, see here.

If you don't want it, you can simply remove the wrapper and use the model as it is:

if not args.distributed:
    if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
        model.features = model.features
        model.cuda()
    else:
        model = model.cuda()
else:
    model.cuda()
    model = model

This is to keep code modification to a minimum. Of course, since parallelization is of no interest to you, you could drop this whole if statement to something along the lines of:

if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
    model.features = model.features
model = model.cuda()

Note that this code assumes you are running on the GPU.