Instruction: torch.pow does not work

Input:
I'm trying to create a custom loss function using PyTorch, and am running into a simple error.

When I try to use torch.pow to take the exponent of a PyTorch Variable, I get the following error message:

AttributeError: 'torch.LongTensor' object has no attribute 'pow'

In the python terminal, I created a simple Variable, and attempted to do the same, and received the same error. Here's a snippet that should recreate the problem:

import torch
from torch.autograd import Variable
import numpy as np

v = Variable(torch.from_numpy(np.array([1, 2, 3, 4])))
torch.pow(v, 2)

I can't find any information on this issue, and nothing is showing up in search results. Help?

EDIT: this problem also occurs when I try to use torch.sqrt()
EDIT: same problem also happens if I try to do

v.pow(2)

pow is definitely a method of v, and the docs clearly state that pow is a method that exists and takes a tensor as it's argument. I really don't see how this is happening, and it seems to me that the docs are just flat out wrong and these methods don't actually work.

Output:
You need to initialize the tensor as floats, because pow always returns a Float.

import torch
from torch.autograd import Variable
import numpy as np

v = Variable(torch.from_numpy(np.array([1, 2, 3, 4], dtype="float32")))
torch.pow(v, 2)

You can cast it back to integers afterwards

torch.pow(v, 2).type(torch.LongTensor)

yields

Variable containing:
  1
  4
  9
 16
[torch.LongTensor of size 4]


Instruction: Print exact value of PyTorch tensor (floating point precision)

Input:
I'm trying to print torch.FloatTensor like:
a = torch.FloatTensor(3,3)
print(a)

This way I can get a value like:
0.0000e+00  0.0000e+00  3.2286e-41
1.2412e-40  1.2313e+00  1.6751e-37
2.6801e-36  3.5873e-41  9.4463e+21

But I want to get more accurate value, like 10 decimal point:
0.1234567891+01

With other python numerical objects, I could get it with:
print('{:.10f}'.format(a))

but in the case of a tensor, I get this error:
TypeError: unsupported format string passed to torch.FloatTensor.__format__

How can I print more precise values of tensors?

Output:
You can set the precision options:
torch.set_printoptions(precision=10)

There are more formatting options on the documentation page, it is very similar to numpy's.


Instruction: AttributeError: 'CrossEntropyLoss' object has no attribute 'backward'

Input:
i am trying to train a very basic CNN on CIFAR10 data set and getting the following error :
AttributeError: 'CrossEntropyLoss' object has no attribute 'backward'

criterion =nn.CrossEntropyLoss
optimizer=optim.SGD(net.parameters(),lr=0.001,momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            # get the inputs
            inputs, labels = data
             # wrap them in Variable
            inputs, labels = Variable(inputs), Variable(labels)

            # zero the parameter gradients
            optimizer.zero_grad()
            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            # print statistics
            running_loss += loss.data[0]
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

Output:
Issue resolved. My mistake, I was missing the parenthesis

criterion = nn.CrossEntropyLoss()


Instruction: binarize input for pytorch

Input:
may I ask how to make data loaded in pytorch become binarized once it is loaded?
Like Tensorflow can done this through:

train_data  = mnist.input_data.read_data_sets(data_directory, one_hot=True)

How can pytorch achieve the one_hot=True effect.

The data_loader I have now is:

torch.set_default_tensor_type('torch.FloatTensor')
train_loader = torch.utils.data.DataLoader(
        datasets.MNIST('data/', train=True, download=True,
                       transform=transforms.Compose([
                         # transforms.RandomHorizontalFlip(),
                           transforms.ToTensor()])),
        batch_size=batch_size, shuffle=False)

I want to make data in train_loader be binarized.
Now what I am doing is: After loading the data,

for data,_ in train_loader:
    torch.round(data)
    data = Variable(data)

Use the torch.round() function. Is this correct?

Output:
The one-hot encoding idea is used for classification. It sounds like you are trying to create an autoencoder perhaps.

If you are creating an autoencoder then there is no need to round as BCELoss can handle values between 0 and 1. Note when training it is better not to apply the sigmoid and instead to use BCELossWithLogits as it provides numerical stability.

Here is an example of an autoencoder with MNIST

If instead you are attempting to do classifcation then there is no need for a one hot vector, you simply output the number of neurons equal to the number of classes i.e for MNIST output 10 neurons and then pass it to CrossEntropyLoss along with a LongTensor with corresponding expected class values

Here is an example of classification on MNIST


Instruction: Implementing a tutorial for Tensorflow and SaaK Transform: Handwritten Digits Recognition

Input:
I am learning to work with Tensorflow. I have installed most of the libraries.

# load libs
import torch
import argparse
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import numpy as np
from data.datasets import MNIST
import torch.utils.data as data_utils
from sklearn.decomposition import PCA
import torch.nn.functional as F
from torch.autograd import Variable

But I am facing an error like this:

ImportError Traceback (most recent call last)
<ipython-input-8-257b7af364e1> in <module>()
  5 import matplotlib.pyplot as plt
  6 import numpy as np
----> 7 from data.datasets import MNIST
  8 import torch.utils.data as data_utils
  9 from sklearn.decomposition import PCA

ImportError: No module named data.datasets

I have installed the "dataset" library using the following command:

sudo -H pip install dataset

But I am still seeing this error. I am using python 2.7.
I am pretty new to python. Can anyone help me in identifying the missing thing. Thanks in advance.

Output:
The dataset directory is placed next to main.py, so probably when you tried sudo -H pip install dataset you actually installed a irrelevant package.

So uninstall the wrong package and it would work:

sudo -H pip uninstall dataset


Instruction: Load pytorch model from different working directory

Input:
I want to load and access a pretrained model from the directory outside where all files are saved. Here is how directory structure is:

-MyProject
----Model_checkpoint_and_scripts
------access_model.py
--run_model.py
--other files

When run model calls access_model.py it looks for model.py in current working directory and does not find it. As suggested here, I could use

the_model = TheModelClass(*args, **kwargs)
the_model.load_state_dict(torch.load(PATH))

But in this case what is a good way to save all the arguments to initialize the model? I am thinking to pickle the command line args but there are some arguments like vocab size which are calculated.

Thanks

Output:
I was able to load it by adding the following lines:

here = os.path.dirname(os.path.abspath(__file__))
sys.path.append(here)


Instruction: Why should the training label for Generator in GAN be always True?

Input:
I am currently learning deep learning especially GAN.
I found a simple code of GAN from a web site below.
https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f

However, in the code, I don't understand why we always need to give true label to Generator as below.

        for g_index in range(g_steps):
            # 2. Train G on D's response (but DO NOT train D on these labels)
            G.zero_grad()

            gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            g_fake_data = G(gen_input)
            dg_fake_decision = D(preprocess(g_fake_data.t()))
            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

            g_error.backward()
            g_optimizer.step()  # Only optimizes G's parameters

Specifically, on this line.

            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

Input data for Generator is fake data(includes noise), so if we assign True labels on those input data, I think Generator ends up creating data which is similar to fake data(doesn't look like genuine). Is my understanding wrong? Sorry for the silly question, but if you have knowledge, plz help me out.
I'll put a whole code below.

    #!/usr/bin/env python

    # Generative Adversarial Networks (GAN) example in PyTorch.
    # See related blog post at https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f#.sch4xgsa9
    import numpy as np
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim
    from torch.autograd import Variable

    # Data params
    data_mean = 4
    data_stddev = 1.25

    # Model params
    g_input_size = 1     # Random noise dimension coming into generator, per output vector
    g_hidden_size = 50   # Generator complexity
    g_output_size = 1    # size of generated output vector
    d_input_size = 100   # Minibatch size - cardinality of distributions
    d_hidden_size = 50   # Discriminator complexity
    d_output_size = 1    # Single dimension for 'real' vs. 'fake'
    minibatch_size = d_input_size

    d_learning_rate = 2e-4  # 2e-4
    g_learning_rate = 2e-4
    optim_betas = (0.9, 0.999)
    num_epochs = 30000
    print_interval = 200
    d_steps = 1  # 'k' steps in the original GAN paper. Can put the discriminator on higher training freq than generator
    g_steps = 1

    # ### Uncomment only one of these
    #(name, preprocess, d_input_func) = ("Raw data", lambda data: data, lambda x: x)
    (name, preprocess, d_input_func) = ("Data and variances", lambda data: decorate_with_diffs(data, 2.0), lambda x: x * 2)

    print("Using data [%s]" % (name))

    # ##### DATA: Target data and generator input data

    def get_distribution_sampler(mu, sigma):
        return lambda n: torch.Tensor(np.random.normal(mu, sigma, (1, n)))  # Gaussian

    def get_generator_input_sampler():
        return lambda m, n: torch.rand(m, n)  # Uniform-dist data into generator, _NOT_ Gaussian

    # ##### MODELS: Generator model and discriminator model

    class Generator(nn.Module):
        def __init__(self, input_size, hidden_size, output_size):
            super(Generator, self).__init__()
            self.map1 = nn.Linear(input_size, hidden_size)
            self.map2 = nn.Linear(hidden_size, hidden_size)
            self.map3 = nn.Linear(hidden_size, output_size)

        def forward(self, x):
            x = F.elu(self.map1(x))
            x = F.sigmoid(self.map2(x))
            return self.map3(x)

    class Discriminator(nn.Module):
        def __init__(self, input_size, hidden_size, output_size):
            super(Discriminator, self).__init__()
            self.map1 = nn.Linear(input_size, hidden_size)
            self.map2 = nn.Linear(hidden_size, hidden_size)
            self.map3 = nn.Linear(hidden_size, output_size)

        def forward(self, x):
            x = F.elu(self.map1(x))
            x = F.elu(self.map2(x))
            return F.sigmoid(self.map3(x))

    def extract(v):
        return v.data.storage().tolist()

    def stats(d):
        return [np.mean(d), np.std(d)]

    def decorate_with_diffs(data, exponent):
        mean = torch.mean(data.data, 1, keepdim=True)
        mean_broadcast = torch.mul(torch.ones(data.size()), mean.tolist()[0][0])
        diffs = torch.pow(data - Variable(mean_broadcast), exponent)
        return torch.cat([data, diffs], 1)

    d_sampler = get_distribution_sampler(data_mean, data_stddev)
    gi_sampler = get_generator_input_sampler()
    G = Generator(input_size=g_input_size, hidden_size=g_hidden_size, output_size=g_output_size)
    D = Discriminator(input_size=d_input_func(d_input_size), hidden_size=d_hidden_size, output_size=d_output_size)
    criterion = nn.BCELoss()  # Binary cross entropy: http://pytorch.org/docs/nn.html#bceloss
    d_optimizer = optim.Adam(D.parameters(), lr=d_learning_rate, betas=optim_betas)
    g_optimizer = optim.Adam(G.parameters(), lr=g_learning_rate, betas=optim_betas)

    for epoch in range(num_epochs):
        for d_index in range(d_steps):
            # 1. Train D on real+fake
            D.zero_grad()

            #  1A: Train D on real
            d_real_data = Variable(d_sampler(d_input_size))
            d_real_decision = D(preprocess(d_real_data))
            d_real_error = criterion(d_real_decision, Variable(torch.ones(1)))  # ones = true
            d_real_error.backward() # compute/store gradients, but don't change params

            #  1B: Train D on fake
            d_gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            d_fake_data = G(d_gen_input).detach()  # detach to avoid training G on these labels
            d_fake_decision = D(preprocess(d_fake_data.t()))
            d_fake_error = criterion(d_fake_decision, Variable(torch.zeros(1)))  # zeros = fake
            d_fake_error.backward()
            d_optimizer.step()     # Only optimizes D's parameters; changes based on stored gradients from backward()

        for g_index in range(g_steps):
            # 2. Train G on D's response (but DO NOT train D on these labels)
            G.zero_grad()

            gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            g_fake_data = G(gen_input)
            dg_fake_decision = D(preprocess(g_fake_data.t()))
            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

            g_error.backward()
            g_optimizer.step()  # Only optimizes G's parameters

        if epoch % print_interval == 0:
            print("%s: D: %s/%s G: %s (Real: %s, Fake: %s) " % (epoch,
                                                                extract(d_real_error)[0],
                                                                extract(d_fake_error)[0],
                                                                extract(g_error)[0],
                                                                stats(extract(d_real_data)),
                                                                stats(extract(d_fake_data))))

Output:
In this part of the code you are training G to fool D, so G generates fake data and asks D whether it thinks it's real (true labels), D's gradients are then propogated all the way to G (this is possible as D's input was G's output) so that it will learn to better fool D in the next iteration.

The inputs of G are not trainable and G only tries to transform them into real data (data similar to what d_sampler generates)


Instruction: Best way in Pytorch to upsample a Tensor and transform it to rgb?

Input:
For a nice output in Tensorboard I want to show a batch of input images, corresponding target masks and output masks in a grid.
Input images have different size then the masks. Furthermore the images are obviously RGB.
From a batch of e.g. 32 or 64 I only want to show the first 4 images.

After some fiddling around I came up with the following example code. Good thing: It works.
But I am really not sure if I missed something in Pytorch. It just looks much longer then I expected. Especially the upsampling and transformation to RGB seems wild. But the other transformations I found would not work for a whole batch.

import torch
from torch.autograd import Variable
import torch.nn.functional as FN
import torchvision.utils as vutils
from tensorboardX import SummaryWriter
import time
batch = 32
i_size = 192
o_size = 112
nr_imgs = 4

# Tensorboard init
writer = SummaryWriter('runs/' + time.strftime('%Y%m%d_%H%M%S'))

input_image=Variable(torch.rand(batch,3,i_size,i_size))
target_mask=Variable(torch.rand(batch,o_size,o_size))
output_mask=Variable(torch.rand(batch,o_size,o_size))

# upsample target_mask, add dim to have gray2rgb
tm = FN.upsample(target_mask[:nr_imgs,None], size=[i_size, i_size], mode='bilinear')
tm = torch.cat( (tm,tm,tm), dim=1)  # grayscale plane to rgb

# upsample target_mask, add dim to have gray2rgb
om = FN.upsample(output_mask[:nr_imgs,None], size=[i_size, i_size], mode='bilinear')
om = torch.cat( (om,om,om), dim=1)  # grayscale plane to rgb

# add up all images and make grid
imgs = torch.cat( ( input_image[:nr_imgs].data, tm.data, om.data ) )
x = vutils.make_grid(imgs, nrow=nr_imgs, normalize=True, scale_each=True)

# Tensorboard img output
writer.add_image('Image', x, 0)

EDIT: Found this on Pytorchs Issues list. Its about Batch support for Transform. Seems there are no plans to add batch transforms in the future. So my current code might be the best solution for the time being, anyway?

Output: Maybe you can just convert your Tensors to the numpy array (.data.cpu().numpy() ) and use opencv to do upsampling? OpenCV implementation should be quite fast.


Instruction: How to find and understand the autograd source code in PyTorch

Input: I have a good understanding of the autograd algorithm, and I think I should learn about the source code in PyTorch. However, when I see the project on GitHub, I am confused by the structure, cuz so many files include autograd. So which part is the most important core code of autograd?

Output:
Try to understand the autograd variable is probably the first thing, what you can do. From my understanding is autograd only a naming for the modules, which containing classes with enhancement of gradients and backward functions.
Be aware a lot of the algorithm, e.g. back-prop through the graph, is hidden in compiled code.
If you look into the __init__.py, you can get a glimpse about all important functions (backward & grad)


Instruction: How can I give a name to each module in ModuleList?

Input:
I have the following component in my model:

feedfnn = []
for task_name, num_class in self.tasks:
    if self.config.nonlinear_fc:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    else:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    feedfnn.append(ffnn)
self.ffnn = nn.ModuleList(feedfnn)

When I print my model, I get the description of the above component as:

(ffnn): ModuleList (
(0): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 2)
)
(1): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
(2): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
)

Can I put a specific name like (task1): Sequential, (task2): Sequential instead of (0): Sequential, (1): Sequential?

Output:
That's straightforward.

Simply start with an empty ModuleListand then use add_module for it. For example,

import torch.nn as nn
from collections import OrderedDict

final_module_list = nn.ModuleList()

a_sequential_module_with_names = nn.Sequential(OrderedDict([
        ('dropout1', nn.Dropout(0.1)),
        ('dense1', nn.Linear(10, 10)),
        ('tanh', nn.Tanh()),
        ('dropout2', nn.Dropout(0.1)),
        ('dense2', nn.Linear(10, 10)),
        ('tanh', nn.Tanh()),
        ('dropout3', nn.Dropout(0.1)),
        ('dense3', nn.Linear(10, 10))]))

final_module_list.add_module('Stage 1', a_sequential_module_with_names)
final_module_list.add_module('Stage 2', a_sequential_module_with_names)
etc.


Instruction: When should I use nn.ModuleList and when should I use nn.Sequential?

Input: I am new to Pytorch and one thing that I don't quite understand is the usage of nn.ModuleList and nn.Sequential. Can I know when I should use one over the other? Thanks.

Output:
nn.ModuleList does not have a forward method, but nn.Sequential does have one. So you can wrap several modules in nn.Sequential and run it on the input.
nn.ModuleList is just a Python list (though it's useful since the parameters can be discovered and trained via an optimizer). While nn.Sequential is a module that sequentially runs the component on the input.


Instruction: Calculating Distance in Parameter Space for PyTorch Network

Input:
I'm new to PyTorch. I want to keep track of the distance in parameter-space my model travels through its optimization. This is the code I'm using.

class ParameterDiffer(object):
    def __init__(self, network):
        network_params = []
        for p in network.parameters():
            network_params.append(p.data.numpy())
        self.network_params = network_params

    def get_difference(self, network):
        total_diff = 0.0
        for i, p in enumerate(network.parameters()):
            p_np = p.data.numpy()
            diff = self.network_params[i] - p_np
            # print(diff)
            scalar_diff = np.sum(diff ** 2)
            total_diff += scalar_diff
        return total_diff

Will this work? I keep track of total_diff through time, and am logging it, but it seems to be zero ALWAYS. Even though the model's performance is improving, which confuses me greatly.

Output:
The cause

This is because the way PyTorch treat conversion between numpy array and torch Tensor. If the underlying data type between numpy array and torch Tensor are the same, they will share the memory. Change the value of one will also change the value of the other. I will show a concrete example here,

x = Variable(torch.rand(2, 2))
y = x.data.numpy()
x
Out[39]:
Variable containing:
 0.8442  0.9968
 0.7366  0.4701
[torch.FloatTensor of size 2x2]
y
Out[40]:
array([[ 0.84422851,  0.996831  ],
       [ 0.73656738,  0.47010136]], dtype=float32)

Then if you change x in-place and see the value in x and y, you will find they are still the same.

x += 2
x
Out[42]:
Variable containing:
 2.8442  2.9968
 2.7366  2.4701
[torch.FloatTensor of size 2x2]
y
Out[43]:
array([[ 2.84422851,  2.99683094],
       [ 2.7365675 ,  2.47010136]], dtype=float32)

So during your model update, the parameter in your model and in the class ParameterDiffer will always be the same. That is why you are seeing zeros.

How to work around this?

If the numpy array and torch Tensor's underlying data type are not compatible, it will force a copy of original data in torch Tensor, which will make the numpy array and torch Tensor have separate memory.

A simple way is just to convert numpy array to type np.float64. Instead of

network_params.append(p.data.numpy())

You can use

network_params.append(p.data.numpy().astype(np.float64))

Important references

http://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#numpy-bridge
https://github.com/pytorch/pytorch/issues/2246


Instruction: Pytorch: Can’t load images using ImageFolder

Input:
I’m trying to load images using “ImageFolder”.

data_dir = './train_dog'          # directory structure is
train_dog/image
dset = datasets.ImageFolder(data_dir, transform)
train_loader = torch.utils.data.DataLoader(dset, batch_size=128, shuffle=True)

However, it seems not working. So I checked the stored data as below

print dset[0][0]

Then it shows only 3 tensors(size 64x64).

[torch.FloatTensor of size 3x64x64]

There are more than 10,000 images in the folder. How come it can’t store all data?

Output:
You should try this:

print len(dset)

which represents the size of the dataset, aka the number of image files.

dset[0] means the (shuffled) first index of the dataset, where dset[0][0] contains the input image tensor and dset[0][1] contains the corresponding label or target.


Instruction: When will the computation graph be freed if I only do forward for some samples?

Input:
I have a use case where I do forward for each sample in a batch and only accumulate loss for some of the samples based on some condition on the model output of the sample. Here is an illustrating code,

for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    total_loss = 0

    loss_count_local = 0
    for i in range(len(target)):
        im = Variable(data[i].unsqueeze(0).cuda())
        y = Variable(torch.FloatTensor([target[i]]).cuda())

        out = model(im)

        # if out satisfy some condtion, we will calculate loss
        # for this sample, else proceed to next sample
        if some_condition(out):
            loss = criterion(out, y)
        else:
            continue

        total_loss += loss
        loss_count_local += 1

        if loss_count_local == 32 or i == (len(target)-1):
            total_loss /= loss_count_local
            total_loss.backward()
            total_loss = 0
            loss_count_local = 0

    optimizer.step()

My question is, as I do forward for all samples but only do backward for some of the samples. When will the graph for those samples which do not contribute to the loss be freed? Will these graphs be freed only after the for loop has ended or immediately after I do forward for the next sample? I am a little confused here.

Also for those samples that do contribute to the total_loss, their graph will be freed immediately after we do total_loss.backward(). Is that right?

Output:
Let's start with a general discussion of how PyTorch frees memory:

First, we should emphasize that PyTorch uses an implicitly declared graph that is stored in Python object attributes. (Remember, it's Python, so everything is an object). More specifically, torch.autograd.Variables have a .grad_fn attribute. This attribute's type defines what kind of computation node we have (e.g. an addition), and the input to that node.

This is important because Pytorch frees memory simply by using the standard python garbage collector (if fairly aggressively). In this context, this means that the (implicitly declared) computation graphs will be kept alive as long as there are references to the objects holding them in the current scope!

This means that if you e.g. do some kind of batching on samples s_1 ... s_k, compute the loss for each and add the loss at the end, that cumulative loss will hold references to each individual loss, which in turn holds references to each of the computation nodes that computed it.

So your question applied to your code is more about how Python (or, more specifically its garbage collector) handles references than about Pytorch does. Since you accumulate the loss in one object (total_loss), you keep pointers alive, and thereby do not free the memory until you re-initialize that object in the outer loop.

Applied to your example, this means that the computation graph you create in the forward pass (at out = model(im)) is only referenced by the out object and any future computations thereof. So if you compute the loss and sum it, you will keep references to out alive, and thereby to the computation graph. If you do not use it, however, the garbage collector should recursively collect out, and its computation graph.


Instruction: How to cast a 1-d IntTensor to int in Pytorch

Input:
How do I convert a 1-D IntTensor to an integer? This:
IntTensor.int()

Gives an error:
KeyError: Variable containing:
 423
[torch.IntTensor of size 1]

Output:
You can use:

print(dictionary[IntTensor.data[0]])

The key you're using is an object of type autograd.Variable.
.data gives the tensor and the index 0 can be used to access the element.


Instruction: torch.nn has no attribute named upsample

Input:
Following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-perform-neural-style-transfer-with-python-3-and-pytorch#step-2-%E2%80%94-running-your-first-style-transfer-experiment

When I run the example in Jupyter notebook, I get the following:

So, I've tried troubleshooting, which eventually got me to running it as per the github example (https://github.com/zhanghang1989/PyTorch-Multi-Style-Transfer) says to via command line:

python main.py eval --content-image images/content/venice-boat.jpg --style-image images/21styles/candy.jpg --model models/21styles.model --content-size 1024 --cuda=0

However, this gives the following error:

Traceback (most recent call last):
File "main.py", line 287, in <module>
main()
File "main.py", line 44, in main
evaluate(args)
File "main.py", line 242, in evaluate
stylemodel = Net(ngf=args.ngf)
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 284, in init
model += [upblock(ngf*expansion, 32, 2, normlayer),
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 127, in init
kernelsize=1, stride=1, upsample=stride)
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 167, in init
self.upsamplelayer = torch.nn.Upsample(scalefactor=upsample)
AttributeError: module 'torch.nn' has no attribute 'Upsample'

I can't figure out how to get past this?

Output:
I think the reason maybe that you have an older version of PyTorch on your system. On my system, the pytorch version is 0.2.0, torch.nn has a module called Upsample.

You can uninstall your current version of pytorch and reinstall it.


Instruction: Reshaping 2 numpy dimensions into one with zipping?

Input:
I tried searching for this question but I couldn't find anything relevant.

The quickest way to describe the problem is with a simple example:
lets say I have a 2D numpy arrayl like this:

[[0, 1, 2, 3],
[10, 11, 12, 13],
[20, 21, 22, 23]]

So it has the shape [3,6]
I want to reshape it into a 1D array that looks like this:

[0, 10 ,20 ,1 ,11 ,21 ,2 ,12 ,22 ,3 ,13 ,23 ]

Unlike the array we get with reshape:

[ 0,  1,  2,  3, 10, 11, 12, 13, 20, 21, 22, 23]

Now, to the actual problem I have...
I actually have a 3D array and I want to reshape it into 2D array, and I want to do so with the method described above.
And another example would be:

import numpy
a = numpy.array([[[0,1,2],[10,11,12],[20,21,22]],
[[100,101,102],[110,111,112],[120,121,122]],
 [[200,201,202],[210,211,212],[220,221,222]]])
a.shape
a.reshape(3,9)
OUTPUT: array([[  0,   1,   2,  10,  11,  12,  20,  21,  22],
       [100, 101, 102, 110, 111, 112, 120, 121, 122],
       [200, 201, 202, 210, 211, 212, 220, 221, 222]])

And again, I want my output to look like this instead:

[[  0,  10,  20,   1,  11,  21,   2,  12,  22],
 [100, 110, 120, 101, 111, ..................],
 [...........................................]]

EDIT: just for the sake of people who google this problem, I add some search terms that some people might search for:

Interweave dimensions numpy array

Zip dimensions numpy array

Numpy reshape ordered dimensions

Tensor reshape dimensions timesteps

Output:
We need to swap the last two axes with np.swapaxes or np.transpose and then reshape.

For 2D input case, it would be -

a.swapaxes(-2,-1).ravel()

For 3D input case, only the reshape part changes -

a.swapaxes(-2,-1).reshape(a.shape[0],-1)

Generic way : To make it generic that would cover all n-dim array cases -

a.swapaxes(-2,-1).reshape(a.shape[:-2] + (-1,)).squeeze()

Sample runs

2D case :

In [186]: a
Out[186]:
array([[ 0,  1,  2,  3],
       [10, 11, 12, 13],
       [20, 21, 22, 23]])

In [187]: a.swapaxes(-2,-1).reshape(a.shape[:-2] + (-1,)).squeeze()
Out[187]: array([ 0, 10, 20,  1, 11, 21,  2, 12, 22,  3, 13, 23])

3D case :

In [189]: a
Out[189]:
array([[[  0,   1,   2],
        [ 10,  11,  12],
        [ 20,  21,  22]],

       [[100, 101, 102],
        [110, 111, 112],
        [120, 121, 122]],

       [[200, 201, 202],
        [210, 211, 212],
        [220, 221, 222]]])

In [190]: a.swapaxes(-2,-1).reshape(a.shape[:-2] + (-1,)).squeeze()
Out[190]:
array([[  0,  10,  20,   1,  11,  21,   2,  12,  22],
       [100, 110, 120, 101, 111, 121, 102, 112, 122],
       [200, 210, 220, 201, 211, 221, 202, 212, 222]])

Runtime test -

In [14]: a = np.random.rand(3,3,3)

# @mahdi n75's soln
In [15]: %timeit np.reshape(a,(3,9), order='F')
1000000 loops, best of 3: 1.22 µs per loop

In [16]: %timeit a.swapaxes(-2,-1).reshape(a.shape[0],-1)
1000000 loops, best of 3: 1.01 µs per loop

In [20]: a = np.random.rand(30,30,30)

# @mahdi n75's soln
In [21]: %timeit np.reshape(a,(30,900), order='F')
10000 loops, best of 3: 28.4 µs per loop

In [22]: %timeit a.swapaxes(-2,-1).reshape(a.shape[0],-1)
10000 loops, best of 3: 18.9 µs per loop

In [17]: a = np.random.rand(300,300,300)

# @mahdi n75's soln
In [18]: %timeit np.reshape(a,(300,90000), order='F')
1 loop, best of 3: 333 ms per loop

In [19]: %timeit a.swapaxes(-2,-1).reshape(a.shape[0],-1)
10 loops, best of 3: 52.4 ms per loop


Instruction: Automatically set cuda variables in pytorch

Input:
I'm trying to automatically set Variables to use cuda if a flag use_gpu is activated.

Usually I would do:

import torch
from torch import autograd
use_gpu = torch.cuda.is_available()

foo = torch.randn(5)

if use_gpu:
    fooV = autograd.Variable(foo.cuda())
else:
    fooV = autograd.Variable(foo)

However, to do things faster when I have to initialise variables in different parts of the code I wanted to do something like this:

import torch
from torch import autograd
use_gpu = torch.cuda.is_available()

class Variable(autograd.Variable):
    def __init__(self, data, *args, **kwargs):
        if use_gpu:
            data = data.cuda()
        super(Variable, self).__init__(data, *args, **kwargs)

foo = torch.randn(5)
fooV = Variable(foo)

Unfortunately, this doesn't seem to be working and fooV doesn't contain a cuda Tensor when the use_gpu is True.

Output:
An alternative is to use type method on CPU Tensor to convert it to GPU Tensor,

if use_cuda:
    dtype = torch.cuda.FloatTensor
else:
    dtype = torch.FloatTensor

x = torch.rand(2,2).type(dtype)

Also, you can find a discussion here.


Instruction: tf.concat operation in PyTorch

Input:
torch.stack is not what I am searching.

I am looking for the Tensorflow's concat operation for Pytorch.

I have searched the doc http://pytorch.org/docs/0.3.0/

Output:
I belive torch.cat is what you are looking for.

http://pytorch.org/docs/master/torch.html#torch.cat


Instruction: Create a nnModule that's just the identity

Input:
I'm trying to debug a pretty complex interaction between different nnModules. It would be very helpful for me to be able to replace one of them with just an identity network for debugging purposes. For example:

net_a = NetworkA()
net_b = NetworkB()
net_c = NetworkC()

input = Autograd.Variable(torch.rand(10,2))

out = net_a(input)
out = net_b(out)
out = net_c(out)

I would like to be able to just change the second line to net_b = IdentityNet(), instead of having to go through and reconnect all my As to Cs. But, when I make a completely empty nnModule, the optimizer throws ValueError: optimizer got an empty parameter list.

Is there any workaround this?

A minimum non-working example:

import torch.optim as optim

class IdentityModule(nnModule):
    def forward(self, inputs):
        return inputs

identity = IdentityModule()
opt = optim.Adam(identity, lr=0.001)
out = identity(any_tensor)
error = torch.mean(out)
error.backward()
opt.step()

Output:
The problem you encounter here is a logical one. Look at what it means when you do:

error.backward()
opt.step()

.backward() will recursively compute the gradients from your output to any input you pass into the network. In terms of the computation graph, there are two noteworthy kinds of inputs: the input you pass in, and the nn.Parameters that model the network's behavior. When you then do opt.step(), PyTorch will look for any input that it can update to change the output of the network, namely the nn.Parameters().

However, your Pseudo-code does not have a single nn.Parameter!, since the identity module does not contain one. So when you call these functions, the opt.step() has no targets, explaining the error message.

This does not extend to the case you describe earlier. If you chain a module with no parameters into a larger chain with some that do have parameters, there are parameters to train in the computation graph.

You need to make sure, though, that the optimizer indeed gets all of these parameters passed upon initialization. A simple trick is to print these:

net_a = SomeNetwork()
net_b = IdentityNetwork()  # has no parameters
net_c = SomeNetwork()

print(list(net_a.parameters()))  # will contain whatever parameters in net_a
print(list(net_b.parameters()))  # will be []
print(list(net_c.parameters()))  # will contain whatever parameters in net_c
# to train all of them, you can do one of two things:
# 1. create new module. This works, since `.parameters()` collects params recursively from all submodules.
class NewNet(nn.Module):
    def __init__(self):
        nn.Module.__init__(self)
        self.net_a = net_a
        self.net_b = identity
        self.net_c = net_c

    def forward(self, input):
        return self.net_c(self.net_b(self.net_a(input)))
all_parameters = list(NewNet().parameters())
print(all_parameters)  # will contain a list of all parameters of net_a and net_c

# 2. simply merge the lists
all_parameters = list(net_a.parameters()) + list(net_b.parameters()) + list(net_c.parameters())
print(all_parameters)  # will contain a list of all parameters of net_a and net_c

opt = optim.SGD(all_parameters)


Instruction: Error when upgrading PyTorch to 0.3.0

Input:
I tried to update PyTorch to the recently released 0.3.0,

conda install pytorch=0.3.0 torchvision -c pytorch

But I'm getting the following error:

NoPackagesFoundError: Dependency missing in current linux-64 channels:
 - pytorch 0.3.0* -> mkl >=2018

Output:
You can also update the condo itself and then try again:

conda update conda
conda update pytorch -c pytorch

It will use the updated channels and download the latest mil version.


Instruction: "Installing From Source" Within Anaconda Environment

Input:
What I would like to do:

I am using macOS and Anaconda 2.
I would like to install a Python package (specifically PyTorch) from source.
I would like to install all the dependencies and the package itself within an Anaconda environment.
I don't want this Anaconda environment to be the default/ root Anaconda environment, but an environment I particularly created for installing this package and its dependencies from source.

What I have done:

First, I created the environment as follows

conda create --name my_env python=3.5

Now, the instructions for installing PyTorch from source are as follows:

export CMAKE_PREFIX_PATH=[anaconda root directory]
conda install numpy pyyaml setuptools cmake cffi
git clone --recursive https://github.com/pytorch/pytorch
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

Now, my questions are:

Following this instructions, requires me to specify anaconda root directory for the CMAKE_PREFIX_PATH. What should that directory be given that I want everything set-up in my_env?
Is it reasonable to create an extra environment for a package installed from source and its dependencies? Why would one do or not do it? My motivation is mainly fear that one day I  may screw my system up big time and hence want things to be cleanly separated.

If you can only answer one of the two questions, that is already greatly appreciated. Thanks!

Output:
I received this answer from the Anaconda Google discussion group and re-post it here in case anyone else is interested.

It is the path to my_env.  If you created it with -n my_env and you haven't otherwise changed your envs dir, it'll be in <anaconda root>/envs/my_env
Yes, this is definitely good practice.  The cleanest way to use conda is to install miniconda, not anaconda, and to install as little as possible into the root environment.


Instruction: pytorch loss value not change

Input:
I wrote a module based on this article: http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/

The idea is pass the input into multiple streams then concat together and connect to a FC layer. I divided my source code into 3 custom modules: TextClassifyCnnNet >> FlatCnnLayer >> FilterLayer

FilterLayer:

class FilterLayer(nn.Module):
    def __init__(self, filter_size, embedding_size, sequence_length, out_channels=128):
        super(FilterLayer, self).__init__()

        self.model = nn.Sequential(
            nn.Conv2d(1, out_channels, (filter_size, embedding_size)),
            nn.ReLU(inplace=True),
            nn.MaxPool2d((sequence_length - filter_size + 1, 1), stride=1)
        )

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))

    def forward(self, x):
        return self.model(x)

FlatCnnLayer:

class FlatCnnLayer(nn.Module):
    def __init__(self, embedding_size, sequence_length, filter_sizes=[3, 4, 5], out_channels=128):
        super(FlatCnnLayer, self).__init__()

        self.filter_layers = nn.ModuleList(
            [FilterLayer(filter_size, embedding_size, sequence_length, out_channels=out_channels) for
             filter_size in filter_sizes])

    def forward(self, x):
        pools = []
        for filter_layer in self.filter_layers:
            out_filter = filter_layer(x)
            # reshape from (batch_size, out_channels, h, w) to (batch_size, h, w, out_channels)
            pools.append(out_filter.view(out_filter.size()[0], 1, 1, -1))
        x = torch.cat(pools, dim=3)

        x = x.view(x.size()[0], -1)
        x = F.dropout(x, p=dropout_prob, training=True)

        return x

TextClassifyCnnNet (main module):

class TextClassifyCnnNet(nn.Module):
    def __init__(self, embedding_size, sequence_length, num_classes, filter_sizes=[3, 4, 5], out_channels=128):
        super(TextClassifyCnnNet, self).__init__()

        self.flat_layer = FlatCnnLayer(embedding_size, sequence_length, filter_sizes=filter_sizes,
                                       out_channels=out_channels)

        self.model = nn.Sequential(
            self.flat_layer,
            nn.Linear(out_channels * len(filter_sizes), num_classes)
        )

    def forward(self, x):
        x = self.model(x)

        return x

def fit(net, data, save_path):
    if torch.cuda.is_available():
        net = net.cuda()

    for param in list(net.parameters()):
        print(type(param.data), param.size())

    optimizer = optim.Adam(net.parameters(), lr=0.01, weight_decay=0.1)

    X_train, X_test = data['X_train'], data['X_test']
    Y_train, Y_test = data['Y_train'], data['Y_test']

    X_valid, Y_valid = data['X_valid'], data['Y_valid']

    n_batch = len(X_train) // batch_size

    for epoch in range(1, n_epochs + 1):  # loop over the dataset multiple times
        net.train()
        start = 0
        end = batch_size

        for batch_idx in range(1, n_batch + 1):
            # get the inputs
            x, y = X_train[start:end], Y_train[start:end]
            start = end
            end = start + batch_size

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            predicts = _get_predict(net, x)
            loss = _get_loss(predicts, y)
            loss.backward()
            optimizer.step()

            if batch_idx % display_step == 0:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, batch_idx * len(x), len(X_train), 100. * batch_idx / (n_batch + 1), loss.data[0]))

        # print statistics
        if epoch % display_step == 0 or epoch == 1:
            net.eval()
            valid_predicts = _get_predict(net, X_valid)
            valid_loss = _get_loss(valid_predicts, Y_valid)
            valid_accuracy = _get_accuracy(valid_predicts, Y_valid)
            print('\r[%d] loss: %.3f - accuracy: %.2f' % (epoch, valid_loss.data[0], valid_accuracy * 100))

    print('\rFinished Training\n')

    net.eval()

    test_predicts = _get_predict(net, X_test)
    test_loss = _get_loss(test_predicts, Y_test).data[0]
    test_accuracy = _get_accuracy(test_predicts, Y_test)
    print('Test loss: %.3f - Test accuracy: %.2f' % (test_loss, test_accuracy * 100))

    torch.save(net.flat_layer.state_dict(), save_path)

def _get_accuracy(predicts, labels):
    predicts = torch.max(predicts, 1)[1].data[0]
    return np.mean(predicts == labels)

def _get_predict(net, x):
    # wrap them in Variable
    inputs = torch.from_numpy(x).float()
    # convert to cuda tensors if cuda flag is true
    if torch.cuda.is_available:
        inputs = inputs.cuda()
    inputs = Variable(inputs)
    return net(inputs)

def _get_loss(predicts, labels):
    labels = torch.from_numpy(labels).long()
    # convert to cuda tensors if cuda flag is true
    if torch.cuda.is_available:
        labels = labels.cuda()
    labels = Variable(labels)
    return F.cross_entropy(predicts, labels)

It seems that parameters 're just updated slightly each epoch, the accuracy remains for all the process. While with the same implementation and the same params in Tensorflow, it runs correctly.

I'm new to Pytorch, so maybe my instructions has something wrong, please help me to find out. Thank you!

P.s: I try to use F.nll_loss + F.log_softmax instead of F.cross_entropy. Theoretically, it should return the same, but in fact another result is printed out (but it still be a wrong loss value)

Output:
I realised that L2_loss in Adam Optimizer make loss value remain unchanged (I haven't tried in other Optimizer yet). It works when I remove L2_loss:

# optimizer = optim.Adam(net.parameters(), lr=0.01, weight_decay=0.1)
optimizer = optim.Adam(model.parameters(), lr=0.001)

=== UPDATE (See above answer for more detail!) ===

self.features = nn.Sequential(self.flat_layer)
self.classifier = nn.Linear(out_channels * len(filter_sizes), num_classes)

...

optimizer = optim.Adam([
    {'params': model.features.parameters()},
    {'params': model.classifier.parameters(), 'weight_decay': 0.1}
], lr=0.001)


Instruction: How Pytorch Tensor get the index of specific value

Input:
With python lists, we can do:
a = [1, 2, 3]
assert a.index(2) == 1

How can a pytorch tensor find the .index() directly?

Output:
I think there is no direct translation from list.index() to a pytorch function. However, you can achieve similar results using tensor==number and then the nonzero() function. For example:
t = torch.Tensor([1, 2, 3])
print ((t == 2).nonzero(as_tuple=True)[0])

This piece of code returns

1
[torch.LongTensor of size 1x1]


Instruction: Why are Embeddings in PyTorch implemented as Sparse Layers?

Input:
Embedding Layers in PyTorch are listed under "Sparse Layers" with the limitation:

  Keep in mind that only a limited number of optimizers support sparse gradients: currently it’s optim.SGD (cuda and cpu), and optim.Adagrad (cpu)

What is the reason for this? For example in Keras I can train an architecture with an Embedding Layer using any optimizer.

Output:
Upon closer inspection sparse gradients on Embeddings are optional and can be turned on or off with the sparse parameter:

  class torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2, scale_grad_by_freq=False, sparse=False)

Where:

  sparse (boolean, optional) – if True, gradient w.r.t. weight matrix
  will be a sparse tensor. See Notes for more details regarding sparse
  gradients.

And the "Notes" mentioned are what I quoted in the question about a limited number of optimizers being supported for sparse gradients.

Update:

It is theoretically possible but technically difficult to implement some optimization methods on sparse gradients. There is an open issue in the PyTorch repo to add support for all optimizers.

Regarding the original question, I believe Embeddings can be treated as sparse because it is possible to operate on the input indices directly rather than converting them to one-hot encodings for input into a dense layer. This is explained in @Maxim's answer to my related question.


Instruction: How to multiply a matrix by a vector in PyTorch

Input:
I'm playing around with PyTorch with the aim of learning it, and I have a very dumb question: how can I multiply a matrix by a single vector?

Here's what I've tried:

>>> import torch
>>> a = torch.rand(4,4)
>>> a

 0.3162  0.4434  0.9318  0.8752
 0.0129  0.8609  0.6402  0.2396
 0.5720  0.7262  0.7443  0.0425
 0.4561  0.1725  0.4390  0.8770
[torch.FloatTensor of size 4x4]

>>> b = torch.rand(4)
>>> b

 0.1813
 0.7090
 0.0329
 0.7591
[torch.FloatTensor of size 4]

>>> a.mm(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: invalid argument 2: dimension 1 out of range of 1D tensor at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:24
>>> a.mm(b.t())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: t() expects a 2D tensor, but self is 1D
>>> b.mm(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: matrices expected, got 1D, 2D tensors at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:1288
>>> b.t().mm(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: t() expects a 2D tensor, but self is 1D

On the other hand, if I do

>>> b = torch.rand(4,2)

then my first attempt, a.mm(b), works fine. So the problem is just that I'm multiplying a vector rather than a matrix --- but how can I do this?

Output:
You're looking for

torch.mv(a,b)

Note that for the future, you may also find torch.matmul() useful. torch.matmul() infers the dimensionality of your arguments and accordingly performs either dot products between vectors, matrix-vector or vector-matrix multiplication, matrix multiplication or batch matrix multiplication for higher order tensors.


Instruction: Strange behaviour in PyTorch

Input:
I started learning pyTorch recently.I moved from Torch community as I like Python.

I encountered this strange behaviour in pyTorch.Any insights as why this happened would be appreciated.

x=torch.Tensor(10,1).fill_(1)
y=torch.Tensor(10).fill_(2)
print(x.size())     #output is torch.Size([10, 1])
print(y.size())     #output is  torch.Size([10])
z=x+y
z.size()
z.size()            #output is torch.Size([10, 10])

Output of z is 10 x 10 with value 3 which means it is clearly adding the Tensors(I also checked with other numbers) and then constructing  a 10 x 10 Tensor with the values after addition.

Can someone explain me why is this happening . I tried it in Torch(Lua) it did a cumulative addition and returned the Tensor I expected.

Output:
When you're doing the sum between the torch Tensors,  broadcasting is happening in the background. It's the same behaviour that you'd also see when you do the addition using NumPy. And, PyTorch simply follows the same broadcasting rules that is followed in NumPy.

You can read and understand broadcasting here: NumPy Broadcasting


Instruction: PyTorch Slow Batch matrix multiplication on GPU

Input:
I am using Batch Matrix Multiplication on 2 3d tensors of sizes (100 , 128 , 128 ) each.

import torch
a = torch.randn(100,128,128)
b = torch.randn(100,128,128)

import time

t0 = time.time()
torch.bmm(a,b)
print(time.time() - t0)

  0.03233695030212402

Now if i do the same thing on GPU it takes a lot longer

a = a.cuda()
b = b.cuda()
t0 = time.time()
torch.bmm(a,b)
print(time.time() - t0)

  30.574532985687256

Why does it take so long to solve on GPU?

I have a GTX 1050 Ti 4GB

And processor core i3-6100 3.7Ghz

Output:
GPU:

30.57 secs is the total time taken by following steps:

CPU launches kernels* on the device (GPU)
CPU allocates memory on GPU
CPU copies input data to GPU
CPU launches kernels on GPU to process the input data
CPU copies output results to itself

*Kernel is a serial code which is a small part of the original code.

CPU

Whereas, 0.0323 secs is the total time taken by either:

Intra-CPU communications
CPU and main memory communications

These are very fast and they also have to execute fewer commands to get work done.

Conclusion

Hence it's a trade-off between the complexity of the commands (CPU commands being more complex) and speed-up due to parellelism and simpler nature of commands (GPU can fire more parallel threads and they are much simpler in nature).

If there were more computations to be done, then GPU speed-up would overwhelm the extra time taken for CPU-GPU communications.


Instruction: pytorch PIP and CONDA error?

Input:
Guys I am new to python and deeplearning world

I tried to install pytorch using conda

I get this Error...

(base) C:\WINDOWS\system32>conda install pytorch

  `Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

pytorch

Current channels:

https://repo.continuum.io/pkgs/main/win-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/win-64

Couldnt post all the channels due to reputation issue on stackoverflow...

Trying Pip for installing Pytorch It just opens pytorch site after this error:

  (base) C:\WINDOWS\system32>pip install pytorch
  Collecting pytorch
    Using cached pytorch-0.1.2.tar.gz
  Building wheels for collected packages: pytorch
    Running setup.py bdist_wheel for pytorch ... error
    Complete output from command C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d C:\Users\micha\AppData\Local\Temp\tmpqmo4j08upip-wheel- --python-tag cp36:
    Traceback (most recent call last):
      File "", line 1, in
      File "C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py", line 17, in
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
  Running setup.py install for pytorch ... error
    Complete output from command C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\micha\AppData\Local\Temp\pip-vms7q49e-record\install-record.txt --single-version-externally-managed --compile:
    Traceback (most recent call last):
      File "", line 1, in
      File "C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py", line 13, in
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

----------------------------------------

Exception:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\micha\AppData\Local\Temp\pip-vms7q49e-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
    requirement_set.cleanup_files()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 729, in cleanup_files
    req.remove_temporary_source()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 977, in remove_temporary_source
    rmtree(self.source_dir)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\ProgramData\Anaconda3\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\ProgramData\Anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\ProgramData\Anaconda3\lib\shutil.py", line 393, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch'

I also tried conda install -c peterjc123 pytorch=0.1.12 and soumith but I get the same error not found

Any Idea where I am going wrong

Tried other forum tips and post also reinstalled Anaconda but still the same issue

Output:
However I found how to use Pytorch on windows here:
[https://www.superdatascience.com/pytorch/]

conda install -c peterjc123 pytorch

Did the trick for me ...


Instruction: How to use pytorch DataLoader with a 3-D matrix for LSTM input?

Input:
I have a dataset of 3-D(time_stepinputsizetotal_num) matrix which is a .mat file. I want to use DataLoader to get a input dataset for LSTM which batch_size is 5. My code is as following:

file_path = "…/database/frameLength100/notOverlap/a.mat"
mat_data = s.loadmat(file_path)
tensor_data = torch.from_numpy(mat_data[‘a’]) #Tensor

class CustomDataset(Dataset):

def __init__(self, tensor_data):
    self.tensor_data = tensor_data

def __getitem__(self, index):
    data = self.tensor_data[index]
    label = 1;
    return data, label

def __len__(self):
    return len(self.tensor_data)
custom_dataset = CustomDataset(tensor_data=tensor_data)
train_loader = DataLoader(dataset=custom_dataset, batch_size=5, shuffle=True)

I think the code is wrong but I have no idea how to correct it. What makes me confused is how how can I make DataLoader know which dimension is ‘total_num’ so that I get the dataset which batch size is 5.

Output:
If I understand correctly, you want the batching to happen along the total_num dimension, i. e. dimension 2.

You could simply use that the dimension to index your dataset, i.e. change __getitem__ to data = self.tensor_data[:, :, index], and accordingly in __len__, return self.tensor_data.size(2) instead of len(self.tensor_data). Each batch will then have size [time_step, inputsize, 5].


Instruction: Issues installing pytorch for OS X with conda

Input:
I used to have pytorch working for python 3 on OS X but now I can't get it to install automatically for some reason (I don't want to do from source).

I did:

conda install pytorch torchvision -c pytorch

as the website suggested... then I got a mkl error so I installed it but it still complains about it:

(FTIR_py3) brandomiranda~/home_simulation_research/FTIR/FTIR_proj $ conda install pytorch torchvision -c pytorch
Fetching package metadata ...........
Solving package specifications:

PackageNotFoundError: Package not found: '' Dependencies missing in current osx-64 channels:
  - pytorch -> mkl >=2018
  - torchvision -> pytorch >=0.3 -> mkl >=2018

You can search for packages on anaconda.org with

    anaconda search -t conda mkl

You may need to install the anaconda-client command line client with

    conda install anaconda-client

but I do have mkl:

(FTIR_py3) brandomiranda~/home_simulation_research/FTIR/FTIR_proj $ conda install mkl
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /Users/brandomiranda/miniconda3/envs/FTIR_py3:
#
mkl                       2017.0.3                      0

anyone know whats going on? it used to work a few days ago...

longer thread on pytorch forum:

https://discuss.pytorch.org/t/issues-installing-pytorch-for-os-x-with-conda/11496

Output:
As suggested in PyTorch forum, I think you should first install MKL. Your error trace also says that MKL is missing in your system. You can install MKL by doing:

$ conda install -c anaconda mkl

After this, install pytorch and torchvision by

$ conda install -c pytorch pytorch torchvision


Instruction: Pytorch: CNN don't learn anything after torch.cat()?

Input:
I try to concatenate Variable in the network with code like this

    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)

    x = x.view(x.size(0), -1)
    x= torch.cat((x,angle),1) # from here I concat it.
    x = self.dropout1(self.relu1(self.bn1(self.fc1(x))))
    x = self.dropout2(self.relu2(self.bn2(self.fc2(x))))
    x = self.fc3(x)

And then I find my network learn nothing and give acc always around 50%. So I print param.grad and as I expected, they are all nan. Does anyone encounter this thing before?

I ran the code without concatenation before and it works out well. So I suppose this is where the rub is and the system doesn't throw any error or exception. if any other backup info is needed, please let me know.

Thank you.

Output: Probably the error is somewhere outside of the code that you provided. Try to check if there are nan's in your input and check if the loss function is not resulting in nan.


Instruction: Why do we need to call zero_grad() in PyTorch?

Input:
Why does zero_grad() need to be called during training?
|  zero_grad(self)
|      Sets gradients of all model parameters to zero.

Output:
In PyTorch, for every mini-batch during the training phase, we typically want to explicitly set the gradients to zero before starting to do backpropragation (i.e., updating the Weights and biases) because PyTorch accumulates the gradients on subsequent backward passes. This accumulating behaviour is convenient while training RNNs or when we want to compute the gradient of the loss summed over multiple mini-batches. So, the default action has been set to accumulate (i.e. sum) the gradients on every loss.backward() call.
Because of this, when you start your training loop, ideally you should zero out the gradients so that you do the parameter update correctly. Otherwise, the gradient would be a combination of the old gradient, which you have already used to update your model parameters, and the newly-computed gradient. It would therefore point in some other direction than the intended direction towards the minimum (or maximum, in case of maximization objectives).
Here is a simple example:
import torch
from torch.autograd import Variable
import torch.optim as optim

def linear_model(x, W, b):
    return torch.matmul(x, W) + b

data, targets = ...

W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

optimizer = optim.Adam([W, b])

for sample, target in zip(data, targets):
    # clear out the gradients of all Variables
    # in this optimizer (i.e. W, b)
    optimizer.zero_grad()
    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()
    optimizer.step()

Alternatively, if you're doing a vanilla gradient descent, then:
W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

for sample, target in zip(data, targets):
    # clear out the gradients of Variables
    # (i.e. W, b)
    W.grad.data.zero_()
    b.grad.data.zero_()

    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()

    W -= learning_rate * W.grad.data
    b -= learning_rate * b.grad.data

Note:

The accumulation (i.e., sum) of gradients happens when .backward() is called on the loss tensor.
As of v1.7.0, Pytorch offers the option to reset the gradients to None optimizer.zero_grad(set_to_none=True) instead of filling them with a tensor of zeroes. The docs claim that this setting reduces memory requirements and slightly improves performance, but might be error-prone if not handled carefully.


Instruction: pytorch: Does variable.long() guarantee 64-bits?

Input: In pytorch, I have a variable that might be IntTensor or cuda.IntTensor. It needs to be changed to 64-bits retaining cpu/gpu. Does variable.long() guarantee 64-bits on all implementations of pytorch? If not, how can variable be converted to 64-bits retaining cpu/gpu for all implementations?

Output:
From PyTorch documentation here, torch.LongTensor or torch.cuda.LongTensor can be used to ensure a 64-bit signed integer.

Similarly, for 64-bit floating points, you can use torch.DoubleTensor or torch.cuda.DoubleTensor.

You can convert them into variables by using Variable(tensor) method where tensor represents the tensor created above.


Instruction: Computing gradients of intermediate nodes in PyTorch

Input:
I'm trying to learn how autograd works in PyTorch. In the simple program below, I don't understand why gradients of loss w.r.t W1 and W2 are None.  As far as I understand from the documentation, W1 and W2 are volatile, therefore gradients cannot be computed.  Is that it? I mean, how I cannot take derivative of the loss w.r.t intermediate nodes? Can anyone explain me what I am missing here?

import torch
import torch.autograd as tau

W = tau.Variable(torch.FloatTensor([[0, 1]]), requires_grad=True)
a = tau.Variable(torch.FloatTensor([[2, 2]]), requires_grad=False)
b = tau.Variable(torch.FloatTensor([[3, 3]]), requires_grad=False)

W1 = W  + a * a
W2 = W1 - b * b * b
Z = W2 * W2

print 'W:', W
print 'W1:', W1
print 'W2:', W2
print 'Z:', Z

loss = torch.sum((Z - 3) * (Z - 3))
print 'loss:', loss

# free W gradient buffer in case you are running this cell more than 2 times
if W.grad is not None: W.grad.data.zero_()

loss.backward()
print 'W.grad:', W.grad

# all of them are None
print 'W1.grad:', W1.grad
print 'W2.grad:', W2.grad
print 'a.grad:', a.grad
print 'b.grad:', b.grad
print 'Z.grad:', Z.grad

Output:
When required, intermediate gradients are accumulated in a C++ buffer but in order to save memory they are not retained by default (exposed in python object).
Only gradients of leaf Variables set with requires_grad=True will be retained (so Win your example)

One way to retain intermediate gradients is to register a hook. One hook for this job is retain_grad() (see PR)
In your example, if you write W2.retain_grad(), intermediate gradient of W2 will be exposed in W2.grad

W1 and W2 are not volatile (you can check by accessing their volatile attribute (ie: W1.volatile)) and cannot be because they are not leaf variables (such as W, a and b). On the contrary, the computation of their gradients is required, see their requires_grad attribute.
If only one leaf variable is volatile, the whole backward graph is not constructed (You can check by making a volatile and look at the loss gradient function)

a = tau.Variable(torch.FloatTensor([[2, 2]]), volatile=True)
# ...
assert loss.grad_fn is None

To sum up

Volatility implies no gradient computation: Useful in inference mode

Only one leaf variable set volatile disable gradient computation

Requiring gradients implies gradient computation. Intermediate ones are exposed or not

Only one leaf variable requiring grad enable gradient computation


Instruction: Unknown Python syntax in PyTorch: a instance can directly receive a parameter

Input:
I have a question about Python syntax when I'm learning PyTorch. The following codes are an example from the PyTorch document.

m = nn.Linear(20, 30)
input = autograd.Variable(torch.randn(128, 20))
output = m(input)
print(output.size())

This first line is to create an instance m, but why this instance m can directly receive a parameter like the line 3? I think it should use method to deal with parameter like m.method(input).

Output:
In python, any object can define a __call__ method that allows it to be used as a function like in your example.

Reference: https://docs.python.org/2/reference/datamodel.html#object.call


Instruction: Installing a CPU-based library version on a GPU enabled machine

Input: I want to install a CPU version of PyTorch on a server which is equipped with a nVIDIA Tesla GPU. Will it work or can I only install a GPU version (with CUDA) on this server for PyTorch to function properly?

Output: The version of PyTorch with GPU support also works with CPU (but the cpu training of neural networks is really slow). So you can install the GPU version. Make sure you install PyTorch compiled with the correct cuda version (cuda 7.5, cuda 8.0 or cuda 9.0).


Instruction: PyTorch Softmax Dimensions error

Input:
I'm attempting to write a simple NN module, with 2 layers, first layer ReLU activation, output softmax with 3 classes (one-hot encoded). It seems theres something wrong with the way I'm using the softmax function, but I'm not sure what's going on.

X is 178x13
Y is 178x3

Dataset I'm using is fairly simple, and can be found here.

I keep getting the error:

RuntimeError: dimension out of range (expected to be in range of [-2, 1], but got 3) .

.

import pandas as pd
import numpy as np
import torch
from torch.autograd import Variable
from sklearn.preprocessing import LabelBinarizer

# Read in dataset, specifying that this set didn't come with column headers
x = pd.read_csv('Datasets/wine.data', header=None)

# Rename columns
x.columns = ['Class', 'A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 'A9', 'A10', 'A11', 'A12', 'A13']

y = x[['Class']].values

#turn class labels into one-hot encoding
one_hot = LabelBinarizer()
y = Variable(torch.from_numpy(one_hot.fit_transform(y)), )

x = Variable(torch.from_numpy(x.iloc[:, 1:14].values).float())

N, D_in, H, D_out = y.shape[0], x.shape[1], 20, 3

# Implement neural net with nn module

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    torch.nn.LogSoftmax(dim=3)
)

loss_fn = torch.nn.NLLLoss

learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for t in range(500):
    y_pred = model(x)

    loss = loss_fn(y_pred, y)
    print("Iteration: %d | Loss: %.3f" % (t, loss))

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

Output:
This was a problem because for NLLLoss:

The target that this loss expects is a class index (0 to N-1, where N = number of classes)

And I had been trying to give it the one-hot encoded vector. I solved my issue by doing:

loss = loss_fn(y_pred, torch.max(y, 1)[1])

Where torch.max found the maximum values and their respective index.


Instruction: Is it possible to implement a multilayered LSTM with LSTMCells modules in PyTorch?

Input:
In PyTorch there is a LSTM module which in addition to input sequence, hidden states, and  cell states accepts a num_layers argument which specifies how many layers will our LSTM have.

There is however another module LSTMCell which has just input size and number of hidden states as parameters, there is no num_layers since this is a single cell in a multi-layered LSTM.

My question is what is the proper way to connect together the LSTMCell modules to achieve a same effect as a multi layered LSTM with num_layers > 1

Output:
LSTMCell is the basic building block of an LSTM network. You should use the LSTM module (which uses LSTMCell internally). If you want to do this yourself, the best way is to read the source code (https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py).

Basically you want to use one LSTMCell for each layer, and you should be careful on how to go from input to output, layer by layer taking into account the hidden states.
I also have basic implementation of a convolutional LSTM but the idea is the same. You can check it here:
https://github.com/rogertrullo/pytorch_convlstm/


Instruction: Pytorch: Create an boolean tensor (type: torch.ByteTensor)?

Input:
I want to create a tensor only containing boolean values.
In Matlab that would be

a = false(10,1)

Output:
Already found it:

a = torch.zeros(10)
b = a.type(torch.ByteTensor)


Instruction: How do I check if PyTorch is using the GPU?

Input: How do I check if PyTorch is using the GPU? The nvidia-smi command can detect GPU activity, but I want to check it directly from inside a Python script.

Output:
These functions should help:
>>> import torch

>>> torch.cuda.is_available()
True

>>> torch.cuda.device_count()
1

>>> torch.cuda.current_device()
0

>>> torch.cuda.device(0)
<torch.cuda.device at 0x7efce0b03be0>

>>> torch.cuda.get_device_name(0)
'GeForce GTX 950M'

This tells us:

CUDA is available and can be used by one device.
Device 0 refers to the GPU GeForce GTX 950M, and it is currently chosen by PyTorch.


Instruction: Pytorch Operation to detect NaNs

Input:
Is there a Pytorch-internal procedure to detect NaNs in Tensors? Tensorflow has the tf.is_nan and the tf.check_numerics operations ... Does Pytorch have something similar, somewhere? I could not find something like this in the docs...

I am looking specifically for a Pytorch internal routine, since I would like this to happen on the GPU as well as on the CPU. This excludes numpy - based solutions (like np.isnan(sometensor.numpy()).any()) ...

Output:
You can always leverage the fact that nan != nan:

>>> x = torch.tensor([1, 2, np.nan])
tensor([  1.,   2., nan.])
>>> x != x
tensor([ 0,  0,  1], dtype=torch.uint8)

With pytorch 0.4 there is also torch.isnan:

>>> torch.isnan(x)
tensor([ 0,  0,  1], dtype=torch.uint8)


Instruction: Where can I see the source code for pytorch's MSELoss?

Input:
I use the U-NET network to train my data.
But I need to modify its loss function to reduce the loss of pixels below 1 to reduce the impact of negative cases on network weights. But I opened the source code in pycharm MSELOSS, see this:

class MSELoss(_Loss):
    r"""Creates a criterion that measures the mean squared error between
    `n` elements in the input `x` and target `y`:

    :math:`{loss}(x, y)  = 1/n \sum |x_i - y_i|^2`

    `x` and `y` arbitrary shapes with a total of `n` elements each.

    The sum operation still operates over all the elements, and divides by `n`.

    The division by `n` can be avoided if one sets the internal variable
    `size_average` to `False`.

    """
    pass

I can't get anything useful.

Output:
There you go: https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L1423 However, it calls the C api

def mse_loss(input, target, size_average=True, reduce=True):
    """
    mse_loss(input, target, size_average=True, reduce=True) -> Variable
    Measures the element-wise mean squared error.
    See :class:`~torch.nn.MSELoss` for details.
    """
    return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss,
input, target, size_average, reduce)

def own_mse_loss(input, target, size_average=True):
    L = (input - target) ** 2
    return torch.mean(L) if size_average else torch.sum(L)


Instruction: What is the function in TensorFlow that is equivalent to expand() in PyTorch?

Input:
Let's say I have a 2 x 3 matrix and I want to create a 6 x 2 x 3 matrix where each element in the first dimension is the original 2 x 3 matrix.

In PyTorch, I can do this:

import torch
from torch.autograd import Variable
import numpy as np

x = np.array([[1, 2, 3], [4, 5, 6]])
x = Variable(torch.from_numpy(x))

# y is the desired result
y = x.unsqueeze(0).expand(6, 2, 3)

What is the equivalent way to do this in TensorFlow? I know unsqueeze() is equivalent to tf.expand_dims() but I don't TensorFlow has anything equivalent to expand(). I'm thinking of using tf.concat on a list of the 1 x 2 x 3 tensors but am not sure if this is the best way to do it.

Output:
Tensorflow automatically broadcasts, so in general you don't need to do any of this. Suppose you have a y' of shape 6x2x3 and your x is of shape 2x3, then you can already do y'*x or y'+x will already behave as if you had expanded it. But if for some other reason you really need to do it, then the command in tensorflow is tile:

y = tf.tile(tf.reshape(x, (1,2,3)), multiples=(6,1,1))

Docs: https://www.tensorflow.org/api_docs/python/tf/tile


Instruction: Pytorch: Intermediate testing during training

Input:
How can I test my pytorch model on validation data during training?
I know that there is the function myNet.eval() which apparantly switches of any dropout layers, but is it also preventing the gradients from being accumulated?
Also how would I undo the myNet.eval() command in order to continue with the training?

If anyone has some code snippet / toy example I would be grateful!

Output:
  How can I test my pytorch model on validation data during training?

There are plenty examples where there are train and test steps for every epoch during training. An easy one would be the official MNIST example. Since pytorch does not offer any high-level training, validation or scoring framework you have to write it yourself. Commonly this consists of

a data loader (commonly based on torch.utils.dataloader.Dataloader)
a main loop over the total number of epochs
a train() function that uses training data to optimize the model
a test() or valid() function to measure the effectiveness of the model given validation data and a metric

This is also what you will find in the linked example.

Alternatively you can use a framework that provides basic looping and validation facilities so you don't have to implement everything by yourself all the time.

tnt is torchnet for pytorch, supplying you with different metrics (such as accuracy) and abstraction of the train loop. See this MNIST example.
inferno and torchsample attempt to model things very similar to Keras and provide some tools for validation
skorch is a scikit-learn wrapper for pytorch that lets you use all the tools and metrics from sklearn

  Also how would I undo the myNet.eval() command in order to continue with the training?

myNet.train() or, alternatively, supply a boolean to switch between eval and training: myNet.train(True) for train mode.


Instruction: Advantages and Disadvantages of MXNet compared to other Deep Learning APIs

Input:
Recently I decided to learn MXNet, as some code I need to use, is written using this API.

However, I would like to know which are the advantages and disadvantages of MXNet compared to the other Deep Learning Libraries out there.

Output:
Perhaps the biggest reason for considering MXNet is its high-performance imperative API. This is one of the most important advantages of MXNet to other platforms. Imperative API with autograd makes it much easier and more intuitive to compose and debug a network. PyTorch also supports imperative API, but MXNet is the only platform AFAIK that supports hybridization, which effectively allows your imperative model to be converted to a symbol for similar performance to symbolic API. Here is a link to tutorials on Gluon, MXNet's imperative API: http://gluon.mxnet.io/

Given that you're using an example code, it is possible that the example was written using symbolic API. You may notice MXNet's advantage in symbolic API when training on many GPUs. Otherwise you won't notice much of a difference (except perhaps in some memory usage).

Tensorflow does have a one year head-start to MXNet and as a result it has a larger user base, but it only supports symbolic API (imperative API is very new and is only meant for experimentation), which is significantly harder to debug a network when you run into issues. However MXNet has quickly caught up in features and with 1.0 release, I don't think there is anything in TF that MXNet doesn't support.


Instruction: PyTorch: Relation between Dynamic Computational Graphs - Padding - DataLoader

Input:
As far as I understand, the strength of PyTorch is supposed to be that it works with dynamic computational graphs. In the context of NLP, that means that sequences with variable lengths do not necessarily need to be padded to the same length. But, if I want to use PyTorch DataLoader, I need to pad my sequences anyway because the DataLoader only takes tensors - given that me as a total beginner does not want to build some customized collate_fn.

Now this makes me wonder - doesn’t this wash away the whole advantage of dynamic computational graphs in this context?
Also, if I pad my sequences to feed it into the DataLoader as a tensor with many zeros as padding tokens at the end (in the case of word ids), will it have any negative effect on my training since PyTorch may not be optimized for computations with padded sequences (since the whole premise is that it can work with variable sequence lengths in the dynamic graphs), or does it simply not make any difference?

I will also post this question in the PyTorch Forum...

Thanks!

Output:
  In the context of NLP, that means that sequences with variable lengths do not necessarily need to be padded to the same length.

This means that you don't need to pad sequences unless you are doing data batching which is currently the only way to add parallelism in PyTorch. DyNet has a method called autobatching (which is described in detail in this paper) that does batching on the graph operations instead of the data, so this might be what you want to look into.

  But, if I want to use PyTorch DataLoader, I need to pad my sequences anyway because the DataLoader only takes tensors - given that me as a total beginner does not want to build some customized collate_fn.

You can use the DataLoader given you write your own Dataset class and you are using batch_size=1. The twist is to use numpy arrays for your variable length sequences (otherwise default_collate will give you a hard time):

from torch.utils.data import Dataset
from torch.utils.data.dataloader import DataLoader

class FooDataset(Dataset):
    def __init__(self, data, target):
        assert len(data) == len(target)
        self.data = data
        self.target = target
    def __getitem__(self, index):
        return self.data[index], self.target[index]
    def __len__(self):
        return len(self.data)

data = [[1,2,3], [4,5,6,7,8]]
data = [np.array(n) for n in data]
targets = ['a', 'b']

ds = FooDataset(data, targets)
dl = DataLoader(ds, batch_size=1)

print(list(enumerate(dl)))
# [(0, [
#  1  2  3
# [torch.LongTensor of size 1x3]
# , ('a',)]), (1, [
#  4  5  6  7  8
# [torch.LongTensor of size 1x5]
# , ('b',)])]

  Now this makes me wonder - doesn’t this wash away the whole advantage of dynamic computational graphs in this context?

Fair point but the main strength of dynamic computational graphs are (at least currently) mainly the possibility of using debugging tools like pdb which rapidly decrease your development time. Debugging is way harder with static computation graphs. There is also no reason why PyTorch would not implement further just-in-time optimizations or a concept similar to DyNet's auto-batching in the future.

  Also, if I pad my sequences to feed it into the DataLoader as a tensor with many zeros as padding tokens at the end [...], will it have any negative effect on my training [...]?

Yes, both in runtime and for the gradients. The RNN will iterate over the padding just like normal data which means that you have to deal with it in some way. PyTorch supplies you with tools for dealing with padded sequences and RNNs, namely pad_packed_sequence and pack_padded_sequence. These will let you ignore the padded elements during RNN execution, but beware: this does not work with RNNs that you implement yourself (or at least not if you don't add support for it manually).


Instruction: PyTorch's dataloader "too many open files" error when no files should be open

Input:
So this is a minimal code which illustrates the issue:

This is the Dataset:

class IceShipDataset(Dataset):
    BAND1='band_1'
    BAND2='band_2'
    IMAGE='image'

    @staticmethod
    def get_band_img(sample,band):
        pic_size=75
        img=np.array(sample[band])
        img.resize(pic_size,pic_size)
        return img

    def __init__(self,data,transform=None):
        self.data=data
        self.transform=transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):

        sample=self.data[idx]
        band1_img=IceShipDataset.get_band_img(sample,self.BAND1)
        band2_img=IceShipDataset.get_band_img(sample,self.BAND2)
        img=np.stack([band1_img,band2_img],2)
        sample[self.IMAGE]=img
        if self.transform is not None:
                sample=self.transform(sample)
        return sample

And this is the code which fails:

PLAY_BATCH_SIZE=4
#load data. There are 1604 examples.
with open('train.json','r') as f:
        data=f.read()
data=json.loads(data)

ds=IceShipDataset(data)
playloader = torch.utils.data.DataLoader(ds, batch_size=PLAY_BATCH_SIZE,
                                          shuffle=False, num_workers=4)
for i,data in enumerate(playloader):
        print(i)

It gives that weird open files error in the for loop…
My torch version is 0.3.0.post4

If you want the json file, it is available at Kaggle (https://www.kaggle.com/c/statoil-iceberg-classifier-challenge)

I should mention that the error has nothing to do with the state of my laptop:

yoni@yoni-Lenovo-Z710:~$ lsof | wc -l
89114
yoni@yoni-Lenovo-Z710:~$ cat /proc/sys/fs/file-max
791958

What am I doing wrong here?

Output:
I know how to fix the error, but I don't have a complete explanation for why it happens.

First, the solution: you need to make sure that the image data is stored as numpy.arrays, when you call json.loads it loads them as python lists of floats. This causes the torch.utils.data.DataLoader to individually transform each float in the list into a torch.DoubleTensor.

Have a look at default_collate in torch.utils.data.DataLoader - your __getitem__ returns a dict which is a mapping, so default_collate gets called again on each element of the dict. The first couple are ints, but then you get to the image data which is a list, i.e. a collections.Sequence - this is where things get funky as default_collate is called on each element of the list. This is clearly not what you intended. I don't know what the assumption in torch is about the contents of a list versus a numpy.array, but given the error it would appear that that assumption is being violated.

The fix is pretty trivial, just make sure the two image bands are numpy.arrays, for instance in __init__

def __init__(self,data,transform=None):
    self.data=[]
    for d in data:
        d[self.BAND1] = np.asarray(d[self.BAND1])
        d[self.BAND2] = np.asarray(d[self.BAND2])
        self.data.append(d)
    self.transform=transform

or after you load the json, what ever - doesn't really matter where you do it, as long as you do it.

Why does the above results in too many open files?

I don't know, but as the comments pointed out, it is likely to do with interprocess communication and lock files on the two queues data is taken from and added to.

Footnote: the train.json was not available for download from Kaggle due to the competition still being open (??). I made a dummy json file that should have the same structure and tested the fix on that dummy file.


Instruction: Pytorch: How to compute IoU (Jaccard Index) for semantic segmentation

Input: Can someone provide a toy example of how to compute IoU (intersection over union) for semantic segmentation in pytorch?

Output:
I found this somewhere and adapted it for me. I'll post the link if I can find it again. Sorry in case this was a dublicate.
The key function here is the function called iou. The wrapping function evaluate_performance is not universal, but it shows that one needs to iterate over all results before computing IoU.

import torch
import pandas as pd  # For filelist reading
import myPytorchDatasetClass  # Custom dataset class, inherited from torch.utils.data.dataset

def iou(pred, target, n_classes = 12):
  ious = []
  pred = pred.view(-1)
  target = target.view(-1)

  # Ignore IoU for background class ("0")
  for cls in xrange(1, n_classes):  # This goes from 1:n_classes-1 -> class "0" is ignored
    pred_inds = pred == cls
    target_inds = target == cls
    intersection = (pred_inds[target_inds]).long().sum().data.cpu()[0]  # Cast to long to prevent overflows
    union = pred_inds.long().sum().data.cpu()[0] + target_inds.long().sum().data.cpu()[0] - intersection
    if union == 0:
      ious.append(float('nan'))  # If there is no ground truth, do not include in evaluation
    else:
      ious.append(float(intersection) / float(max(union, 1)))
  return np.array(ious)

def evaluate_performance(net):
    # Dataloader for test data
    batch_size = 1
    filelist_name_test = '/path/to/my/test/filelist.txt'
    data_root_test = '/path/to/my/data/'
    dset_test = myPytorchDatasetClass.CustomDataset(filelist_name_test, data_root_test)
    test_loader = torch.utils.data.DataLoader(dataset=dset_test,
                                              batch_size=batch_size,
                                              shuffle=False,
                                              pin_memory=True)
    data_info = pd.read_csv(filelist_name_test, header=None)
    num_test_files = data_info.shape[0]
    sample_size = num_test_files

    # Containers for results
    preds = Variable(torch.zeros((sample_size, 60, 36, 60)))
    gts = Variable(torch.zeros((sample_size, 60, 36, 60)))

    dataiter = iter(test_loader)
    for i in xrange(sample_size):
        images, labels, filename = dataiter.next()
        images = Variable(images).cuda()
        labels = Variable(labels)
        gts[i:i+batch_size, :, :, :] = labels
        outputs = net(images)
        outputs = outputs.permute(0, 2, 3, 4, 1).contiguous()
        val, pred = torch.max(outputs, 4)
        preds[i:i+batch_size, :, :, :] = pred.cpu()
    acc = iou(preds, gts)
    return acc


Instruction: Pytorch - RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed

Input:
I keep running into this error:

  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I had searched in Pytorch forum, but still can’t find out what I have done wrong in my custom loss function. My model is nn.GRU, and here is my custom loss function:

def _loss(outputs, session, items):  # `items` is a dict() contains embedding of all items
    def f(output, target):
        pos = torch.from_numpy(np.array([items[target["click"]]])).float()
        neg = torch.from_numpy(np.array([items[idx] for idx in target["suggest_list"] if idx != target["click"]])).float()
        if USE_CUDA:
            pos, neg = pos.cuda(), neg.cuda()
        pos, neg = Variable(pos), Variable(neg)

        pos = F.cosine_similarity(output, pos)
        if neg.size()[0] == 0:
            return torch.mean(F.logsigmoid(pos))
        neg = F.cosine_similarity(output.expand_as(neg), neg)

        return torch.mean(F.logsigmoid(pos - neg))

    loss = map(f, outputs, session)
return -torch.mean(torch.cat(loss))

Training code:

    # zero the parameter gradients
    model.zero_grad()

    # forward + backward + optimize
    outputs, hidden = model(inputs, hidden)
    loss = _loss(outputs, session, items)
    acc_loss += loss.data[0]

    loss.backward()
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in model.parameters():
        p.data.add_(-learning_rate, p.grad.data)

Output:
The problem is from my training loop: it doesn’t detach or repackage the hidden state in between batches? If so, then loss.backward() is trying to back-propagate all the way through to the start of time, which works for the first batch but not for the second because the graph for the first batch has been discarded.

there are two possible solutions.

1) detach/repackage the hidden state in between batches. There are (at
    least) three ways to do this (and I chose this solution):

 hidden.detach_()
 hidden = hidden.detach()

2) replace loss.backward() with loss.backward(retain_graph=True) but know that each successive batch will take more time than the previous one because it will have to back-propagate all the way through to the start of the first batch.

Example


Instruction: What's the difference between "hidden" and "output" in PyTorch LSTM?

Input:
I'm having trouble understanding the documentation for PyTorch's LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says:

  Outputs: output, (h_n, c_n)

  output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
  h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
  c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len

It seems that the variables output and h_n both give the values of the hidden state. Does h_n just redundantly provide the last time step that's already included in output, or is there something more to it than that?

Output:
I made a diagram. The names follow the PyTorch docs, although I renamed num_layers to w.

output comprises all the hidden states in the last layer ("last" depth-wise, not time-wise). (h_n, c_n) comprises the hidden states after the last timestep, t = n, so you could potentially feed them into another LSTM.

The batch dimension is not included.


Instruction: PyTorch: How to change the learning rate of an optimizer at any given moment (no LR schedule)

Input:
Is it possible in PyTorch to change the learning rate of the optimizer in the middle of training dynamically (I don't want to define a learning rate schedule beforehand)?

So let's say I have an optimizer:

optim = torch.optim.SGD(model.parameters(), lr=0.01)

Now due to some tests which I perform during training, I realize my learning rate is too high so I want to change it to say 0.001. There doesn't seem to be a method optim.set_lr(0.001) but is there some way to do this?

Output:
So the learning rate is stored in optim.param_groups[i]['lr'].
optim.param_groups is a list of the different weight groups which can have different learning rates. Thus, simply doing:
for g in optim.param_groups:
    g['lr'] = 0.001

will do the trick.

**Alternatively,**
as mentionned in the comments, if your learning rate only depends on the epoch number, you can use a learning rate scheduler.
For example (modified example from the doc):
torch.optim.lr_scheduler import LambdaLR
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
# Assuming optimizer has two groups.
lambda_group1 = lambda epoch: epoch // 30
lambda_group2 = lambda epoch: 0.95 ** epoch
scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

Also, there is a prebuilt learning rate scheduler to reduce on plateaus.


Instruction: Is location-dependent convolution filter possible in PyTorch or TensorFlow?

Input:
Let's pretend that in plus of having an image, I also have a gradient from left to right on the X axis of an image, and another gradient from top to bottom on the Y axis. Those two gradients are of the same size of the image, and could both range from -0.5 to 0.5.

Now, I'd like to make the convolution kernel (a.k.a. convolution filter, or convolution weights) depend on the (x, y) location in the gradient. So the kernel is a function of the gradient as if the kernel was the output of a nested mini-neural net. This would make the weights of the filter to be different in every position, but slightly similar to their neighbors. How do I do that within PyTorch or TensorFlow?

Sure, I could compute a Toeplitz matrix (a.k.a. diagonal-constant matrix)  by myself, but the matrix multiplication would take O(n^3) operations if pretending x==y==n, whereas convolutions can be implemented in O(n^2) normally. Or I could maybe iterate on every element myself and do the multiplications in an unvectorized fashion.

Any better ideas? I'd like to see creativity here, thinking about how could this be implemented neatly. I believe coding that would be an interesting way to build a network layer capable of doing things similar to a simplified version of a Spatial Transformer Networks, but which's spatial transformation would be independent of the image.

Output:
Here is a solution I thought for a simplified version of this problem where a linear combination of weights would be used rather than truly using a nested mini neural network:

It may be possible to do 4 different convolutions passes so as to have 4 feature maps, then to multiply those 4 maps with the gradients (2 vertical and 2 horizontal gradients), and add them together so that only 1 map remains. However, that would be a linear combination of the different maps which is simpler than truly using a nested neural network which would alter the kernel in the first place.