Instruction: Extracting the top-k value-indices from a 1-D Tensor

Input:
Given a 1-D tensor in Torch (torch.Tensor), containing values which can be compared (say floating point), how can we extract the indices of the top-k values in that tensor?
Apart from the brute-force method, I am looking for some API call, that Torch/lua provides, which can perform this task efficiently.

Output:
As of pull request #496 Torch now includes a built-in API named torch.topk. Example:

> t = torch.Tensor{9, 1, 8, 2, 7, 3, 6, 4, 5}

-- obtain the 3 smallest elements
> res = t:topk(3)
> print(res)
 1
 2
 3
[torch.DoubleTensor of size 3]

-- you can also get the indices in addition
> res, ind = t:topk(3)
> print(ind)
 2
 4
 6
[torch.LongTensor of size 3]

-- alternatively you can obtain the k largest elements as follow
-- (see the API documentation for more details)
> res = t:topk(3, true)
> print(res)
 9
 8
 7
[torch.DoubleTensor of size 3]

At the time of writing the CPU implementation follows a sort and narrow approach (there are plans to improve it in the future). That being said an optimized GPU implementation for cutorch is currently being reviewed.


Instruction: How to Display Custom Images in Tensorboard (e.g. Matplotlib Plots)?

Input:
The Image Dashboard section of the Tensorboard ReadMe says:

  Since the image dashboard supports arbitrary pngs, you can use this to embed custom visualizations (e.g. matplotlib scatterplots) into TensorBoard.

I see how a pyplot image could be written to file, read back in as a tensor, and then used with tf.image_summary() to write it to TensorBoard, but this statement from the readme suggests there is a more direct way. Is there? If so, is there any further documentation and/or examples of how to do this efficiently?

Output:
It is quite easy to do if you have the image in a memory buffer. Below, I show an example, where a pyplot is saved to a buffer and then converted to a TF image representation which is then sent to an image summary.

import io
import matplotlib.pyplot as plt
import tensorflow as tf

def gen_plot():
    """Create a pyplot plot and save to buffer."""
    plt.figure()
    plt.plot([1, 2])
    plt.title("test")
    buf = io.BytesIO()
    plt.savefig(buf, format='png')
    buf.seek(0)
    return buf

# Prepare the plot
plot_buf = gen_plot()

# Convert PNG buffer to TF image
image = tf.image.decode_png(plot_buf.getvalue(), channels=4)

# Add the batch dimension
image = tf.expand_dims(image, 0)

# Add image summary
summary_op = tf.summary.image("plot", image)

# Session
with tf.Session() as sess:
    # Run
    summary = sess.run(summary_op)
    # Write summary
    writer = tf.train.SummaryWriter('./logs')
    writer.add_summary(summary)
    writer.close()

This gives the following TensorBoard visualization:


Instruction: Python wheels: cp27mu not supported

Input:
I'm trying to install pytorch (http://pytorch.org/) on Linux, and according to my machine configuration, the command I should run is:

pip install https://s3.amazonaws.com/pytorch/whl/torch-0.1.6.post17-cp27-cp27mu-linux_x86_64.whl

On one machine (Linux distribution Slackware 14.1) the installation fails with error: torch-0.1.6.post17-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform., while on another one (Ubuntu 15.10) it succeeds. From what I understood, the problem seems to be the cp27mu in the wheel name.

Using the command import pip; print(pip.pep425tags.get_supported()) from the Python shell, I get this from the Slackware machine:

[('cp27', 'cp27m', 'manylinux1_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any'), ('py26', 'none', 'any'), ('py25', 'none', 'any'), ('py24', 'none', 'any'), ('py23', 'none', 'any'), ('py22', 'none', 'any'), ('py21', 'none', 'any'), ('py20', 'none', 'any')]

and this from the Ubuntu machine:

[('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any'), ('py26', 'none', 'any'), ('py25', 'none', 'any'), ('py24', 'none', 'any'), ('py23', 'none', 'any'), ('py22', 'none', 'any'), ('py21', 'none', 'any'), ('py20', 'none', 'any')]

From https://www.python.org/dev/peps/pep-0513/, it seems to me that supporting cp27m or cp27mu depends on an option passed at compile time, --enable-unicode.

Now, maybe at this point I shouldn't even be asking the question, but just to be sure, does this mean that I have to compile Python with the --enable-unicode=ucs4 on the Slackware machine in order to install that wheel?

Output:
This is exactly that.
Recompile python under slack with --enable-unicode=ucs4 and you can then install the whl.


Instruction: Loading Torch7 trained models (.t7) in PyTorch

Input:
I am using Torch7 library for implementing neural networks.  Mostly, I rely on pre-trained models. In Lua I use torch.load function to load a model saved as torch .t7 file. I am curious about switching to PyTorch( http://pytorch.org) and I read the documents. I couldn't find any information regarding the mechanisms to load a pre-trained model. The only relevant information I was able to find is this page:http://pytorch.org/docs/torch.html

But the function torch.load described in the page seems to load a file saved with pickle. If someone has additional information on loading .t7 models in PyTorch, please share it here.

Output: As of PyTorch 1.0 torch.utils.serialization is completely removed. Hence no one can import models from Lua Torch into PyTorch anymore. Instead, I would suggest installing PyTorch 0.4.1 through pip in a conda environment (so that you can remove it after this) and use this repo to convert your Lua Torch model to PyTorch model, not just the torch.nn.legacy model that you cannot use for training. Then use PyTorch 1.xx to do whatever with it. You can also train your converted Lua Torch models in PyTorch this way :)


Instruction: PyTorch: How to use DataLoaders for custom Datasets

Input:
How to make use of the torch.utils.data.Dataset and torch.utils.data.DataLoader on your own data (not just the torchvision.datasets)?

Is there a way to use the inbuilt DataLoaders which they use on TorchVisionDatasets to be used on any dataset?

Output:
Yes, that is possible. Just create the objects by yourself, e.g.

import torch.utils.data as data_utils

train = data_utils.TensorDataset(features, targets)
train_loader = data_utils.DataLoader(train, batch_size=50, shuffle=True)

where features and targets are tensors. features has to be 2-D, i.e. a matrix where each line represents one training sample, and targets may be 1-D or 2-D, depending on whether you are trying to predict a scalar or a vector.

Hope that helps!

EDIT: response to @sarthak's question

Basically yes. If you create an object of type TensorData, then the constructor investigates whether the first dimensions of the feature tensor (which is actually called data_tensor) and the target tensor (called target_tensor) have the same length:

assert data_tensor.size(0) == target_tensor.size(0)

However, if you want to feed these data into a neural network subsequently, then you need to be careful. While convolution layers work on data like yours, (I think) all of the other types of layers expect the data to be given in matrix form. So, if you run into an issue like this, then an easy solution would be to convert your 4D-dataset (given as some kind of tensor, e.g. FloatTensor) into a matrix by using the method view. For your 5000xnxnx3 dataset, this would look like this:

2d_dataset = 4d_dataset.view(5000, -1)

(The value -1 tells PyTorch to figure out the length of the second dimension automatically.)


Instruction: What does .view() do in PyTorch?

Input:
What does .view() do to a tensor x? What do negative values mean?
x = x.view(-1, 16 * 5 * 5)

Output:
view() reshapes the tensor without copying memory, similar to numpy's reshape().
Given a tensor a with 16 elements:
import torch
a = torch.range(1, 16)

To reshape this tensor to make it a 4 x 4 tensor, use:
a = a.view(4, 4)

Now a will be a 4 x 4 tensor. Note that after the reshape the total number of elements need to remain the same. Reshaping the tensor a to a 3 x 5 tensor would not be appropriate.
What is the meaning of parameter -1?
If there is any situation that you don't know how many rows you want but are sure of the number of columns, then you can specify this with a -1. (Note that you can extend this to tensors with more dimensions. Only one of the axis value can be -1). This is a way of telling the library: "give me a tensor that has these many columns and you compute the appropriate number of rows that is necessary to make this happen".
This can be seen in this model definition code. After the line x = self.pool(F.relu(self.conv2(x))) in the forward function, you will have a 16 depth feature map. You have to flatten this to give it to the fully connected layer. So you tell PyTorch to reshape the tensor you obtained to have specific number of columns and tell it to decide the number of rows by itself.


Instruction: How do I print the model summary in PyTorch?

Input:
How do I print the summary of a model in PyTorch like what model.summary() does in Keras:
Model Summary:
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_1 (InputLayer)             (None, 1, 15, 27)     0
____________________________________________________________________________________________________
convolution2d_1 (Convolution2D)  (None, 8, 15, 27)     872         input_1[0][0]
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 8, 7, 27)      0           convolution2d_1[0][0]
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 1512)          0           maxpooling2d_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 1)             1513        flatten_1[0][0]
====================================================================================================
Total params: 2,385
Trainable params: 2,385
Non-trainable params: 0

Output:
While you will not get as detailed information about the model as in Keras' model.summary, simply printing the model will give you some idea about the different layers involved and their specifications.

For instance:

from torchvision import models
model = models.vgg16()
print(model)

The output in this case would be something as follows:

VGG (
  (features): Sequential (
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU (inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU (inplace)
    (4): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU (inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU (inplace)
    (9): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU (inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU (inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU (inplace)
    (16): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU (inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU (inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU (inplace)
    (23): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU (inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU (inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU (inplace)
    (30): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
  )
  (classifier): Sequential (
    (0): Dropout (p = 0.5)
    (1): Linear (25088 -> 4096)
    (2): ReLU (inplace)
    (3): Dropout (p = 0.5)
    (4): Linear (4096 -> 4096)
    (5): ReLU (inplace)
    (6): Linear (4096 -> 1000)
  )
)

Now you could, as mentioned by Kashyap, use the state_dict method to get the weights of the different layers. But using this listing of the layers would perhaps provide more direction is creating a helper function to get that Keras like model summary! Hope this helps!


Instruction: How do I save a trained model in PyTorch?

Input:
How do I save a trained model in PyTorch? I have read that:

torch.save()/torch.load() is for saving/loading a serializable object.
model.state_dict()/model.load_state_dict() is for saving/loading model state.

Output:
Found this page on their github repo:

Recommended approach for saving a model
There are two main approaches for serializing and restoring a model.
The first (recommended) saves and loads only the model parameters:
torch.save(the_model.state_dict(), PATH)

Then later:
the_model = TheModelClass(*args, **kwargs)
the_model.load_state_dict(torch.load(PATH))

The second saves and loads the entire model:
torch.save(the_model, PATH)

Then later:
the_model = torch.load(PATH)

However in this case, the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors.

See also: Save and Load the Model section from the official PyTorch tutorials.


Instruction: L1/L2 regularization in PyTorch

Input: How do I add L1/L2 regularization in PyTorch without manually computing it?

Output: See the documentation. Add a weight_decay parameter to the optimizer for L2 regularization.


Instruction: How can I install torchtext?

Input:
I have PyTorch installed in my machine but whenever I try to do the following-

from torchtext import data
from torchtext import datasets

I get the following error.

ImportError: No module named 'torchtext'

How can I install torchtext?

Output: The package was released with setuptools support. You can clone the repository and run python setup.py install. Unfortunately, I don't think that they have released it on pip.


Instruction: What is the difference between view() and unsqueeze() in Torch?

Input:
Use of unsqueeze():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.unsqueeze(0).size()) # prints - torch.size([1, 2, 4, 3])

Use of view():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.view(1, -1, -1, -1).size()) # prints - torch.size([1, 2, 4, 3])

According to documentation, unsqueeze() inserts singleton dim at position given as parameter and view() creates a view with different dimensions of the storage associated with tensor.
What view() does is clear to me, but I am unable to distinguish it from unsqueeze(). Moreover, I don't understand when to use view() and when to use unsqueeze()?
Any help with good explanation would be appreciated!

Output:
view() can only take a single -1 argument.
So, if you want to add a singleton dimension, you would need to provide all the dimensions as arguments. For e.g., if A is a 2x3x4 tensor, to add a singleton dimension, you would need to do A:view(2, 1, 3, 4).
However, sometimes, the dimensionality of the input is unknown when the operation is being used. Thus, we dont know that A is 2x3x4, but we would still like to insert a singleton dimension. This happens a lot when using minibatches of tensors, where the last dimension is usually unknown. In these cases, the nn.Unsqueeze is useful and lets us insert the dimension without explicitly being aware of the other dimensions when writing the code.


Instruction: Why tensor.view() is not working in pytorch?

Input:
I have the following piece of code.

embedded = self.embedding(input).view(1, 1, -1)
embedded = self.drop(embedded)
print(embedded[0].size(), hidden[0].size())
concatenated_output = torch.cat((embedded[0], hidden[0]), 1)

The last line of the code is giving me the following error.

  RuntimeError: inconsistent tensor sizes at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.9_1487344852722/work/torch/lib/THC/generic/THCTensorMath.cu:141

Please note, when I am printing the tensor shapes at line no. 3, I am getting the following output.

torch.size([1, 300]) torch.size([1, 1, 300])

Why I am getting [1, 300] shape for embedded tensor even though I have used the view method as view(1, 1, -1)?

Any help would be appreciated!

Output:
embedded was a 3d-tensor and hidden was a tuple of two elements (hidden states and cell states) where each element is a 3d-tensor. hidden was the output from LSTM layer. In PyTorch, LSTM returns hidden states [h] and cell states [c] as a tuple which made me confused about the error.

So, I updated the last line of the code as follows and it solved the problem.

concatenated_output = torch.cat((embedded, hidden[0]), 1)


Instruction: How to convert a list or numpy array to a 1d torch tensor?

Input:
I have a list (or, a numpy array) of float values. I want to create a 1d torch tensor that will contain all those values. I can create the torch tensor and run a loop to store the values.

But I want to know is there any way, I can create a torch tensor with initial values from a list or array? Also suggest me if there is any pythonic way to achieve this as I am working in pytorch.

Output:
These are general operations in pytorch and available in the documentation. PyTorch allows easy interfacing with numpy. There is a method called from_numpy and the documentation is available here

import numpy as np
import torch
array = np.arange(1, 11)
tensor = torch.from_numpy(array)


Instruction: AttributeError: module 'torch' has no attribute 'cmul'

Input:
I was trying to do element-wise multiplication of two tensors using the example provided here.

My code:

import torch

x = torch.Tensor([2, 3])
y = torch.Tensor([2, 1])
z = torch.cmul(x, y)
print(z)

It is giving me the following error.

AttributeError: module 'torch' has no attribute 'cmul'

Can anyone tell me why I am getting this error?

Output:
I got the solution. Instead of using cmul, I need to use mul. The following code worked for me!

import torch

x = torch.Tensor([2, 3])
y = torch.Tensor([2, 1])
z = torch.mul(x, y)
print(z)

PS: I was using pytorch, not lua.


Instruction: Performing Convolution (NOT cross-correlation) in pytorch

Input:
I have a network that I am trying to implement in pytorch, and I cannot seem to figure out how to implement "pure" convolution. In tensorflow it could be accomplished like this:
def conv2d_flipkernel(x, k, name=None):
    return tf.nn.conv2d(x, flipkernel(k), name=name,
                        strides=(1, 1, 1, 1), padding='SAME')

With the flipkernel function being:
def flipkernel(kern):
      return kern[(slice(None, None, -1),) * 2 + (slice(None), slice(None))]

How can something similar be done in pytorch?

Output:
TLDR Use the convolution from the functional toolbox, torch.nn.fuctional.conv2d, not torch.nn.Conv2d, and flip your filter around the vertical and horizontal axis.

torch.nn.Conv2d is a convolutional layer for a network. Because weights are learned, it does not matter if it is implemented using cross-correlation, because the network will simply learn a mirrored version of the kernel (Thanks @etarion for this clarification).
torch.nn.fuctional.conv2d performs convolution with the inputs and weights provided as arguments, similar to the tensorflow function in your example. I wrote a simple test to determine whether, like the tensorflow function, it is actually performing cross-correlation and it is necessary to flip the filter for correct convolutional results.
import torch
import torch.nn.functional as F
import torch.autograd as autograd
import numpy as np

#A vertical edge detection filter.
#Because this filter is not symmetric, for correct convolution the filter must be flipped before element-wise multiplication
filters = autograd.Variable(torch.FloatTensor([[[[-1, 1]]]]))

#A test image of a square
inputs = autograd.Variable(torch.FloatTensor([[[[0,0,0,0,0,0,0], [0, 0, 1, 1, 1, 0, 0],
                                             [0, 0, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0, 0],
                                            [0,0,0,0,0,0,0]]]]))
print(F.conv2d(inputs, filters))

This outputs
Variable containing:
(0 ,0 ,.,.) =
  0  0  0  0  0  0
  0  1  0  0 -1  0
  0  1  0  0 -1  0
  0  1  0  0 -1  0
  0  0  0  0  0  0
[torch.FloatTensor of size 1x1x5x6]

This output is the result for cross-correlation. Therefore, we need to flip the filter
def flip_tensor(t):
    flipped = t.numpy().copy()

    for i in range(len(filters.size())):
        flipped = np.flip(flipped,i) #Reverse given tensor on dimention i
    return torch.from_numpy(flipped.copy())

print(F.conv2d(inputs, autograd.Variable(flip_tensor(filters.data))))

The new output is the correct result for convolution.
Variable containing:
(0 ,0 ,.,.) =
  0  0  0  0  0  0
  0 -1  0  0  1  0
  0 -1  0  0  1  0
  0 -1  0  0  1  0
  0  0  0  0  0  0
[torch.FloatTensor of size 1x1x5x6]


Instruction: AttributeError: cannot assign module before Module.__init__() call

Input:
I am getting the following error.

Traceback (most recent call last):
  File "main.py", line 63, in <module>
    question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args)
  File "/net/if5/wua4nw/wasi/academic/research_with_prof_chang/projects/question_answering/duplicate_question_detection/source/question_classifier.py", line 26, in __init__
    self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 255, in __setattr__
    "cannot assign module before Module.__init__() call")
AttributeError: cannot assign module before Module.__init__() call

I have a class as follows.

class QuestionClassifier(nn.Module):

     def __init__(self, dictionary, embeddings_index, max_seq_length, args):
         self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
         self.encoder = EncoderRNN(args.emsize, args.nhid, args.model, args.bidirection, args.nlayers, args.dropout)
         self.drop = nn.Dropout(args.dropout)

So, when I run the following line:

question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args)

I get the above mentioned error. Here, EmbeddingLayer and EncoderRNN is a class written by me which inherits nn.module like the QuestionClassifier class.

What I am doing wrong here?

Output:
Looking at the pytorch source code for Module, we see in the docstring an example of deriving from Module includes:

 class Model(nn.Module):
        def __init__(self):
            super(Model, self).__init__()
            self.conv1 = nn.Conv2d(1, 20, 5)
            self.conv2 = nn.Conv2d(20, 20, 5)

So you probably want to call Module's init the same way in your derived class:

super(QuestionClassifier, self).__init__()


Instruction: How to count the amount of layers in a CNN?

Input:
The Pytorch implementation of ResNet-18.
has the following structure, which appears to be 54 layers, not 18.
So why is it called "18"? How many layers does it actually have?

ResNet (
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
  (relu): ReLU (inplace)
  (maxpool): MaxPool2d (size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1))
  (layer1): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
    (1): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer2): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer3): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer4): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (avgpool): AvgPool2d (
  )
  (fc): Linear (512 -> 1000)
)

Output:
From your output, we can know that there are 20 convolution layers (one 7x7 conv, 16 3x3 conv, and plus 3 1x1 conv for downsample). Basically, if you ignore the 1x1 conv, and counting the FC (linear) layer, the number of layers are 18.
And I've also made an example on how to visualize your architecture in pytorch via graphviz, hope it will help you understand your architecture.


Instruction: IDE autocomplete for pytorch

Input:
I'm using Visual Studio Code. Recently tried out Kite. Both of which doesn't seem to have autocomplete for pytorch.

Is is possible with these tools ? If not, can someone suggest an editor that does ?

Thank you!

Output:
Use Pycharm https://www.jetbrains.com/pycharm/
Get Community version, because it is free with debugger and autocomplete. (more than enough for student)

To get autocomplete and run/debug the code on Pycharm IDE, you have to set correct project interpreter path to your environment (or virtual environment) which you install pytorch

The screenshot shows that I set Project Interpreter path to conda environment named 'pytorch', which I install Pytorch.


Instruction: PyTorch reshape tensor dimension

Input:
I want to reshape a vector of shape (5,) into a matrix of shape (1, 5).
With numpy, I can do:
>>> import numpy as np
>>> a = np.array([1, 2, 3, 4, 5])
>>> a.shape
(5,)
>>> a = np.reshape(a, (1, 5))
>>> a.shape
(1, 5)
>>> a
array([[1, 2, 3, 4, 5]])

But how do I do this with PyTorch?

Output:
Use  torch.unsqueeze(input, dim, out=None):
>>> import torch
>>> a = torch.Tensor([1, 2, 3, 4, 5])
>>> a

 1
 2
 3
 4
 5
[torch.FloatTensor of size 5]

>>> a = a.unsqueeze(0)
>>> a

 1  2  3  4  5
[torch.FloatTensor of size 1x5]


Instruction: Pytorch, what are the gradient arguments

Input:
I am reading through the documentation of PyTorch and found an example where they write

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)

where x was an initial variable, from which y was constructed (a 3-vector). The question is, what are the 0.1, 1.0 and 0.0001 arguments of the gradients tensor ? The documentation is not very clear on that.

Output:
The original code I haven't found on PyTorch website anymore.

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)

The problem with the code above is there is no function based on how to calculate the gradients. This means we don't know how many parameters (arguments the function takes) and the dimension of parameters.
To fully understand this I created an example close to the original:

Example 1:

a = torch.tensor([1.0, 2.0, 3.0], requires_grad = True)
b = torch.tensor([3.0, 4.0, 5.0], requires_grad = True)
c = torch.tensor([6.0, 7.0, 8.0], requires_grad = True)

y=3*a + 2*b*b + torch.log(c)
gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients,retain_graph=True)

print(a.grad) # tensor([3.0000e-01, 3.0000e+00, 3.0000e-04])
print(b.grad) # tensor([1.2000e+00, 1.6000e+01, 2.0000e-03])
print(c.grad) # tensor([1.6667e-02, 1.4286e-01, 1.2500e-05])

I assumed our function is y=3*a + 2*b*b + torch.log(c) and the parameters are tensors with three elements inside.
You can think of the gradients = torch.FloatTensor([0.1, 1.0, 0.0001]) like this is the accumulator.
As you may hear, PyTorch autograd system calculation is equivalent to Jacobian product.

In case you have a function, like we did:
y=3*a + 2*b*b + torch.log(c)

Jacobian would be [3, 4*b, 1/c]. However, this Jacobian is not how PyTorch is doing things to calculate the gradients at a certain point.
PyTorch uses forward pass and backward mode automatic differentiation (AD) in tandem.
There is no symbolic math involved and no numerical differentiation.

Numerical differentiation would be to calculate δy/δb, for b=1 and b=1+ε where ε is small.

If you don't use gradients in y.backward():

Example 2

a = torch.tensor(0.1, requires_grad = True)
b = torch.tensor(1.0, requires_grad = True)
c = torch.tensor(0.1, requires_grad = True)
y=3*a + 2*b*b + torch.log(c)

y.backward()

print(a.grad) # tensor(3.)
print(b.grad) # tensor(4.)
print(c.grad) # tensor(10.)

You will simply get the result at a point, based on how you set your a, b, c tensors initially.
Be careful how you initialize your a, b, c:

Example 3:

a = torch.empty(1, requires_grad = True, pin_memory=True)
b = torch.empty(1, requires_grad = True, pin_memory=True)
c = torch.empty(1, requires_grad = True, pin_memory=True)

y=3*a + 2*b*b + torch.log(c)

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)

print(a.grad) # tensor([3.3003])
print(b.grad) # tensor([0.])
print(c.grad) # tensor([inf])

If you use torch.empty() and don't use pin_memory=True you may have different results each time.
Also, note gradients are like accumulators so zero them when needed.

Example 4:

a = torch.tensor(1.0, requires_grad = True)
b = torch.tensor(1.0, requires_grad = True)
c = torch.tensor(1.0, requires_grad = True)
y=3*a + 2*b*b + torch.log(c)

y.backward(retain_graph=True)
y.backward()

print(a.grad) # tensor(6.)
print(b.grad) # tensor(8.)
print(c.grad) # tensor(2.)

Lastly few tips on terms PyTorch uses:
PyTorch creates a dynamic computational graph when calculating the gradients in forward pass. This looks much like a tree.
So you will often hear the leaves of this tree are input tensors and the root is output tensor.
Gradients are calculated by tracing the graph from the root to the leaf and multiplying every gradient in the way using the chain rule. This multiplying occurs in the backward pass.
Back some time I created PyTorch Automatic Differentiation tutorial that you may check interesting explaining all the tiny details about AD.


Instruction: Error when compiling pytorch: 'cstdint' file not found

Input:
I was attempting to compile pytorch using NO_CUDA=1 python setup.py install on Mac OS X, but I got these errors:

In file included from /Users/ezyang/Dev/pytorch-tmp/torch/lib/tmp_install/include/THPP/Tensor.hpp:3:
/Users/ezyang/Dev/pytorch-tmp/torch/lib/tmp_install/include/THPP/Storage.hpp:6:10: fatal error:
      'cstdint' file not found
#include <cstdint>
         ^
1 error generated.
In file included from torch/csrc/autograd/functions/init.cpp:2:
In file included from torch/csrc/autograd/functions/batch_normalization.h:4:
In file included from /Users/ezyang/Dev/pytorch-tmp/torch/lib/tmp_install/include/THPP/THPP.h:4:
/Users/ezyang/Dev/pytorch-tmp/torch/lib/tmp_install/include/THPP/Storage.hpp:6:10: fatal error:
      'cstdint' file not found
#include <cstdint>
         ^
1 error generated.
1 error generated.
1 error generated.
1 error generated.
In file included from torch/csrc/autograd/python_hook.cpp:5:
In file included from /Users/ezyang/Dev/pytorch-tmp/torch/csrc/THP.h:32:
/Users/ezyang/Dev/pytorch-tmp/torch/csrc/utils.h:6:10: fatal error:
      'type_traits' file not found
#include <type_traits>
         ^
13 errors generated.
1 error generated.
1 error generated.
error: command 'gcc' failed with exit status 1

Output:
You need to setup environment variables to have Python use the correct C compiler on OS X. You should do this instead:

NO_CUDA=1 MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++  python setup.py install


Instruction: How to use the BCELoss in PyTorch?

Input: I want to write a simple autoencoder in PyTorch and use BCELoss, however, I get NaN out, since it expects the targets to be between 0 and 1. Could someone post a simple use case of BCELoss?

Output:
Update

The BCELoss function did not use to be numerically stable. See this issue https://github.com/pytorch/pytorch/issues/751. However, this issue has been resolved with Pull #1792, so that BCELoss is numerically stable now!

Old answer

If you build PyTorch from source, you can use the numerically stable function BCEWithLogitsLoss(contributed in https://github.com/pytorch/pytorch/pull/1792), which takes logits as input.

Otherwise, you can use the following function (contributed by yzgao in the above issue):

class StableBCELoss(nn.modules.Module):
       def __init__(self):
             super(StableBCELoss, self).__init__()
       def forward(self, input, target):
             neg_abs = - input.abs()
             loss = input.clamp(min=0) - input * target + (1 + neg_abs.exp()).log()
             return loss.mean()


Instruction: pytorch Network.parameters() missing 1 required positional argument: 'self'

Input:
There's a problem when I call Network.parameters() in pytorch in this line in my main function:
optimizer = optim.SGD(Network.parameters(), lr=0.001, momentum=0.9)

I get the error code:

TypeError: parameters() missing 1 required positional argument: 'self'

My network is defined in this class

class Network(nn.Module):
def __init__(self):
    super(Network, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(32, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.conv3 = nn.Conv2d(64, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(64 * 5 * 5, 512)
    self.fc2 = nn.Linear(512, 640)
    self.fc3 = nn.Linear(640, 3756)

def forward(self, x):
    x = self.pool(F.relu(self.conv(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = x.view(-1, 64 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

Pretty sure that I imported all torch modules correctly.
Any ideas of what I'm doing wrong here?

Thank you!

Output:
When doing Network.parameters() you are calling the static method parameters.

But, parameters is an instance method.

So you have to instansiate Network before calling parameters.

network = Network()
optimizer = optim.SGD(network.parameters(), lr=0.001, momentum=0.9)

Or, if you only needs Network first this particular line:

optimizer = optim.SGD(Network().parameters(), lr=0.001, momentum=0.9)


Instruction: Pytorch, TypeError: object() takes no parameters

Input:
This is probably a beginner question, but nevertheless: When running an image classifier build with pytorch, I get this error:

Traceback (most recent call last):
File "/pytorch/kanji_torch.py", line 47, in <module>
    network = Network()
  File "/pytorch/kanji_torch.py", line 113, in __init__
    self.conv1 = nn.Conv2d(1, 32, 5)
  File "/python3.5/site-packages/torch/nn/modules/conv.py", line 233, in __init__
    False, _pair(0), groups, bias)
  File "/python3.5/site-packages/torch/nn/modules/conv.py", line 32, in __init__
    out_channels, in_channels // groups, *kernel_size))
TypeError: object() takes no parameters

I define the network class like this:

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(64, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 5 * 5, 512)
        self.fc2 = nn.Linear(512, 640)
        self.fc3 = nn.Linear(640, 3756)

Pretty sure that I imported all the relevant pytorch library modules correctly.
(import torch.nn as nn and
import torch)

Any ideas of what I'm doing wrong?

Thank you!

Output:
You might have a problem with your pytorch version, when I run the code:

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(64, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 5 * 5, 512)
        self.fc2 = nn.Linear(512, 640)
        self.fc3 = nn.Linear(640, 3756)
print(network)

The output is:

Network (
  (conv1): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
  (conv2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))
  (pool2): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
  (conv3): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear (1600 -> 512)
  (fc2): Linear (512 -> 640)
  (fc3): Linear (640 -> 3756)
)

I would suggest to update/reinstall pytorch.


Instruction: How to run PyTorch on GPU by default?

Input:
I want to run PyTorch using cuda. I set model.cuda() and torch.cuda.LongTensor() for all tensors.
Do I have to create tensors using .cuda explicitly if I have used model.cuda()?
Is there a way to make all computations run on GPU by default?

Output:
I do not think you can specify that you want to use cuda tensors by default.
However you should have a look to the pytorch offical examples.

In the imagenet training/testing script, they use a wrapper over the model called DataParallel.
This wrapper has two advantages:

it handles the data parallelism over multiple GPUs
it handles the casting of cpu tensors to cuda tensors

As you can see in L164, you don't have to cast manually your inputs/targets to cuda.

Note that, if you have multiple GPUs and you want to use a single one, launch any python/pytorch scripts with the CUDA_VISIBLE_DEVICES prefix. For instance CUDA_VISIBLE_DEVICES=0 python main.py.


Instruction: Fixing a subset of weights in Neural network during training

Input:
I am considering creating a customized neural network. The basic structure is the same as usual, but I want to truncate the connections between layers. For example, if I construct a network with two hidden layers, I would like to delete some weights and keep the others, like so:

This is not conventional dropout (to avoid overfitting), since the remaining weights (connections) should be specified and fixed.
Are there any ways in python to do it? Tensorflow, pytorch, theano or any other modules?

Output:
Yes you can do this in tensorflow.

You would have some layer in your tensorflow code something like so:

m = tf.Variable( [width,height] , dtype=tf.float32  ))
b = tf.Variable( [height] , dtype=tf.float32  ))
h = tf.sigmoid( tf.matmul( x,m ) + b )

What you want is some new matrix, let's call it k for kill.  It is going to kill specific neural connections.  The neural connections are defined in m.  This would be your new configuration

k = tf.Constant( kill_matrix , dtype=tf.float32 )
m = tf.Variable( [width,height] , dtype=tf.float32  )
b = tf.Variable( [height] , dtype=tf.float32  )
h = tf.sigmoid( tf.matmul( x, tf.multiply(m,k) ) + b )

Your kill_matrix is a matrix of 1's and 0's.  Insert a 1 for every neural connection you want to keep and a 0 for every one you want to kill.


Instruction: Accuracy score in pyTorch LSTM

Input:
I have been running this LSTM tutorial on the wikigold.conll NER data set

training_data contains a list of tuples of sequences and tags, for example:

training_data = [
    ("They also have a song called \" wake up \"".split(), ["O", "O", "O", "O", "O", "O", "I-MISC", "I-MISC", "I-MISC", "I-MISC"]),
    ("Major General John C. Scheidt Jr.".split(), ["O", "O", "I-PER", "I-PER", "I-PER"])
]

And I wrote down this function

def predict(indices):
    """Gets a list of indices of training_data, and returns a list of predicted lists of tags"""
    for index in indicies:
        inputs = prepare_sequence(training_data[index][0], word_to_ix)
        tag_scores = model(inputs)
        values, target = torch.max(tag_scores, 1)
        yield target

This way I can get the predicted labels for specific indices in the training data.

However, how do I evaluate the accuracy score across all training data.

Accuracy being, the amount of words correctly classified across all sentences divided by the word count.

This is what I came up with, which is extremely slow and ugly:

y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
c=0
s=0
for i in range(len(training_data)):
    n = len(y_true[i])
    #super ugly and ineffiicient
    s+=(sum(sum(list(y_true[i].view(-1, n) == y_pred[i].view(-1, n).data))))
    c+=n

print ('Training accuracy:{a}'.format(a=float(s)/c))

How can this be done efficiently in pytorch ?

P.S:
I've been trying to use sklearn's accuracy_score unsuccessfully

Output:
I would use numpy in order to not iterate the list in pure python.

The results are the same, but it runs much faster

def accuracy_score(y_true, y_pred):
    y_pred = np.concatenate(tuple(y_pred))
    y_true = np.concatenate(tuple([[t for t in y] for y in y_true])).reshape(y_pred.shape)
    return (y_true == y_pred).sum() / float(len(y_true))

And this is how to use it:

#original code:
y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
#numpy accuracy score
print(accuracy_score(y_true, y_pred))


Instruction: Can I slice tensors with logical indexing or lists of indices?

Input:
I'm trying to slice a PyTorch tensor using a logical index on the columns. I want the columns that correspond to a 1 value in the index vector. Both slicing and logical indexing are possible, but are they possible together? If so, how? My attempt keeps throwing the unhelpful error

TypeError: indexing a tensor with an object of type ByteTensor. The
only supported types are integers, slices, numpy scalars and
torch.LongTensor or torch.ByteTensor as the only argument.

MCVE
Desired Output
import torch

C = torch.LongTensor([[1, 3], [4, 6]])
# 1 3
# 4 6

Logical indexing on the columns only:
A_log = torch.ByteTensor([1, 0, 1]) # the logical index
B = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
C = B[:, A_log] # Throws error

If the vectors are the same size, logical indexing works:
B_truncated = torch.LongTensor([1, 2, 3])
C = B_truncated[A_log]

And I can get the desired result by repeating the logical index so that it has the same size as the tensor I am indexing, but then I also have to reshape the output.
C = B[A_log.repeat(2, 1)] # [torch.LongTensor of size 4]
C = C.resize_(2, 2)

I also tried using a list of indices:
A_idx = torch.LongTensor([0, 2]) # the index vector
C = B[:, A_idx] # Throws error

If I want contiguous ranges of indices, slicing works:
C = B[:, 1:2]

Output:
I think this is implemented as the index_select function, you can try
import torch

A_idx = torch.LongTensor([0, 2]) # the index vector
B = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
C = B.index_select(1, A_idx)
# 1 3
# 4 6


Instruction: PyTorch: How to convert pretrained FC layers in a CNN to Conv layers

Input: I want to convert a pre-trained CNN (like VGG-16) to a fully convolutional network in Pytorch. How can I do so?

Output:
You can do that as follows (see comments for description):

import torch
import torch.nn as nn
from torchvision import models

# 1. LOAD PRE-TRAINED VGG16
model = models.vgg16(pretrained=True)

# 2. GET CONV LAYERS
features = model.features

# 3. GET FULLY CONNECTED LAYERS
fcLayers = nn.Sequential(
    # stop at last layer
    *list(model.classifier.children())[:-1]
)

# 4. CONVERT FULLY CONNECTED LAYERS TO CONVOLUTIONAL LAYERS

### convert first fc layer to conv layer with 512x7x7 kernel
fc = fcLayers[0].state_dict()
in_ch = 512
out_ch = fc["weight"].size(0)

firstConv = nn.Conv2d(in_ch, out_ch, 7, 7)

### get the weights from the fc layer
firstConv.load_state_dict({"weight":fc["weight"].view(out_ch, in_ch, 7, 7),
                           "bias":fc["bias"]})

# CREATE A LIST OF CONVS
convList = [firstConv]

# Similarly convert the remaining linear layers to conv layers
for layer in enumerate(fcLayers[1:]):
    if isinstance(module, nn.Linear):
        # Convert the nn.Linear to nn.Conv
        fc = module.state_dict()
        in_ch = fc["weight"].size(1)
        out_ch = fc["weight"].size(0)
        conv = nn.Conv2d(in_ch, out_ch, 1, 1)

        conv.load_state_dict({"weight":fc["weight"].view(out_ch, in_ch, 1, 1),
            "bias":fc["bias"]})

        convList += [conv]
    else:
        # Append other layers such as ReLU and Dropout
        convList += [layer]

# Set the conv layers as a nn.Sequential module
convLayers = nn.Sequential(*convList)


Instruction: how does the pytorch autograd work?

Input:
I submitted this as an issue to cycleGAN pytorch implementation, but since nobody replied me there, i will ask again here.

I'm mainly puzzled by the fact that multiple forward passes was called before one single backward pass, see the following in code cycle_gan_model

# GAN loss
# D_A(G_A(A))
self.fake_B = self.netG_A.forward(self.real_A)
pred_fake = self.netD_A.forward(self.fake_B)
self.loss_G_A = self.criterionGAN(pred_fake, True)
# D_B(G_B(B))
self.fake_A = self.netG_B.forward(self.real_B)
pred_fake = self.netD_B.forward(self.fake_A)
self.loss_G_B = self.criterionGAN(pred_fake, True)
# Forward cycle loss G_B(G_A(A))
self.rec_A = self.netG_B.forward(self.fake_B)
self.loss_cycle_A = self.criterionCycle(self.rec_A, self.real_A) * lambda_A
# Backward cycle loss G_A(G_B(B))
self.rec_B = self.netG_A.forward(self.fake_A)
self.loss_cycle_B = self.criterionCycle(self.rec_B, self.real_B) * lambda_B
# combined loss
self.loss_G = self.loss_G_A + self.loss_G_B + self.loss_cycle_A + self.loss_cycle_B + self.loss_idt_A + self.loss_idt_B
self.loss_G.backward()

The way I see it, G_A and G_B each has three forward passes, twice accepting the real data (real_A or real_B) and twice the fake data (fake_B or fake_A).

In tensorflow (I think) the backward pass is always computed w.r.t the last input data. In this case, the backpropagation of loss_G would be wrong. One should instead do backward pass thrice, each immediately following their involving forward pass.

Specifically, netG_A's gradients from loss_G_A is w.r.t real_A but its gradients from loss_cycle_B is w.r.t fake_A.

I assume this is somehow taken care of in pytorch. But how does the model know w.r.t which input data it should compute the gradients?

Output:
Pytorch uses a tape based system for automatic differentiation. that means that it will backpropagate from the last operation it did. I think that the best way to understand is make a diagram from the process. I attach one that I did by hand

Now you will see that some modules are "repeated". The way I think about them is the same way I think about RNNs; in that way, the gradients will just be added.


Instruction: Convolutional NN for text input in PyTorch

Input:
I am trying to implement a text classification model using a CNN. As far as I know, for text data, we should use 1d Convolutions. I saw an example in pytorch using Conv2d but I want to know how can I apply Conv1d for text? Or, it is actually not possible?
Here is my model scenario:
Number of in-channels: 1, Number of out-channels: 128
Kernel size : 3 (only want to consider trigrams)
Batch size : 16

So, I will provide tensors of shape, <16, 1, 28, 300> where 28 is the length of a sentence. I want to use Conv1d which will give me 128 feature maps of length 26 (as I am considering trigrams).
I am not sure, how to define nn.Conv1d() for this setting. I can use Conv2d but want to know is it possible to achieve the same using Conv1d?

Output:
This example of Conv1d and Pool1d layers into an RNN resolved my issue.

So, I need to consider the embedding dimension as the number of in-channels while using nn.Conv1d as follows.

m = nn.Conv1d(200, 10, 2) # in-channels = 200, out-channels = 10
input = Variable(torch.randn(10, 200, 5)) # 200 = embedding dim, 5 = seq length
feature_maps = m(input)
print(feature_maps.size()) # feature_maps size = 10,10,4


Instruction: OverflowError: (34, 'Numerical result out of range') in PyTorch

Input:
I am getting the following error (see the stacktrace) when I ran my code in a different GPU (Tesla K-20, cuda 7.5 installed, 6GB memory). Code works fine if I run in GeForce 1080 or Titan X GPU.

Stacktrace:

File "code/source/main.py", line 68, in <module>
    train.train_epochs(train_batches, dev_batches, args.epochs)
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 34, in train_epochs
    losses = self.train(train_batches, dev_batches, (epoch + 1))
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 76, in train
    self.optimizer.step()
  File "/gpfs/home/g/e/geniiexe/BigRed2/anaconda3/lib/python3.5/site-packages/torch/optim/adam.py", line 70, in step
    bias_correction1 = 1 - beta1 ** state['step']
OverflowError: (34, 'Numerical result out of range')

So, what can be the reason to get such error in a different GPU (Tesla K-20) while it works fine in GeForce or Titan X GPU? Moreover what the error means? Is it related to memory overflow which I don't think so.

Output:
One workaround that is suggested in discuss.pytorch.org is as follows.

Replacing the following lines in adam.py:-

bias_correction1 = 1 - beta1 ** state['step']
bias_correction2 = 1 - beta2 ** state['step']

BY

bias_correction1 = 1 - beta1 ** min(state['step'], 1022)
bias_correction2 = 1 - beta2 ** min(state['step'], 1022)


Instruction: KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'

Input:
I am getting the following error while trying to load a saved model.

KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'

This is the function I am using to load a saved model.

def load_model_states(model, tag):
    """Load a previously saved model states."""
    filename = os.path.join(args.save_path, tag)
    with open(filename, 'rb') as f:
        model.load_state_dict(torch.load(f))

The model is a sequence-to-sequence network whose init function (constructor) is given below.

def __init__(self, dictionary, embedding_index, max_sent_length, args):
    """"Constructor of the class."""
    super(Sequence2Sequence, self).__init__()
    self.dictionary = dictionary
    self.embedding_index = embedding_index
    self.config = args
    self.encoder = Encoder(len(self.dictionary), self.config)
    self.decoder = AttentionDecoder(len(self.dictionary), max_sent_length, self.config)
    self.criterion = nn.NLLLoss()  # Negative log-likelihood loss

    # Initializing the weight parameters for the embedding layer in the encoder.
    self.encoder.init_embedding_weights(self.dictionary, self.embedding_index, self.config.emsize)

When I print the model (sequence-to-sequence network), I get the following.

Sequence2Sequence (
  (encoder): Encoder (
    (drop): Dropout (p = 0.25)
    (embedding): Embedding(43723, 300)
    (rnn): LSTM(300, 300, batch_first=True, dropout=0.25)
  )
  (decoder): AttentionDecoder (
    (embedding): Embedding(43723, 300)
    (attn): Linear (600 -> 12)
    (attn_combine): Linear (600 -> 300)
    (drop): Dropout (p = 0.25)
    (out): Linear (300 -> 43723)
    (rnn): LSTM(300, 300, batch_first=True, dropout=0.25)
  )
  (criterion): NLLLoss (
  )
)

So, module.encoder.embedding is an embedding layer, and module.encoder.embedding.weight represents the associated weight matrix. So, why it says- unexpected key "module.encoder.embedding.weight" in state_dict?

Output:
I solved the problem. Actually I was saving the model using nn.DataParallel, which stores the model in module, and then I was trying to load it without DataParallel. So, either I need to add a nn.DataParallel temporarily in my network for loading purposes, or I can load the weights file, create a new ordered dict without the module prefix, and load it back.

The second workaround looks like the following.

# original saved file with DataParallel
state_dict = torch.load('myfile.pth.tar')
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
model.load_state_dict(new_state_dict)

Reference: https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686


Instruction: backward, grad function in pytorch

Input:
I'm trying to implement backward, grad function in pytorch.

But, I don't know why this value is returned.

Here is my code.

x = Variable(torch.FloatTensor([[1,2],[3,4]]), requires_grad=True)
y = x + 2
z = y * y

gradient = torch.ones(2, 2)
z.backward(gradient)
print(x.grad)

I think that result value should be [[6,8],[10,12]]

Because of dz/dx= 2*(x+2) and x=1,2,3,4

But returned value is [[7,9],[11,13]]

Why this is happened.. I want to know how gradient, grad function is doing.

Help me please..

Output:
The below piece of code on pytorch v0.12.1

import torch
from torch.autograd import Variable
x = Variable(torch.FloatTensor([[1,2],[3,4]]), requires_grad=True)
y = x + 2
z = y * y
gradient = torch.ones(2, 2)
z.backward(gradient)
print(x.grad)

returns

Variable containing:
  6   8
 10  12
[torch.FloatTensor of size 2x2]

Update your pytorch installation. This explains the working of autograd, which handles gradient computation for pytorch.


Instruction: Why should be the function backward be called only on 1 element tensor or with gradients w.r.t to Variable?

Input:
I am new to pytorch. I want to understand as to why we can't call the backward function on a variable containing a tensor of say size say [2,2].
And if we do want to call it on a variable containing tensor of say size say [2,2], we have to do that by first defining a gradient tensor and then calling the backward function on the variable containing the tensor w.r.t the defined gradients.

Output:
from the tutorial on autograd

  If you want to compute the derivatives, you can call .backward() on a
  Variable. If Variable is a scalar (i.e. it holds a one element data),
  you don’t need to specify any arguments to backward(), however if it
  has more elements, you need to specify a grad_output argument that is
  a tensor of matching shape.

Basically to start the chain rule you need a gradient AT THE OUTPUT, to get it going. In the event the output is a scalar loss function ( which it usually is - normally you are beginning the backward pass at the loss variable ) , its an implied value of 1.0

from tutorial :

  let's backprop now out.backward() is equivalent to doing
  out.backward(torch.Tensor([1.0]))

but maybe you only want to update a subgraph ( somewhere deep in the network) ... and the value of a Variable is a matrix of weights.  Then you have to tell it where the begin. From one of their chief devs ( somewhere in the links )

Yes, that's correct. We only support differentiation of scalar
 functions, so if you want to start backward form a non-scalar value
 you need to provide dout / dy

The gradients argument

https://discuss.pytorch.org/t/how-the-backward-works-for-torch-variable/907/8 ok explanation

Pytorch, what are the gradient arguments good explanation

http://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html tutorial


Instruction: PyTorch: access weights of a specific module in nn.Sequential()

Input:
When I use a pre-defined module in PyTorch, I can typically access its weights fairly easily. However, how do I access them if I wrapped the module in nn.Sequential() first? r.g:
class My_Model_1(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_1, self).__init__()
        self.layer = nn.Linear(D_in,D_out)
    def forward(self,x):
        out = self.layer(x)
        return out

class My_Model_2(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_2, self).__init__()
        self.layer = nn.Sequential(nn.Linear(D_in,D_out))
    def forward(self,x):
        out = self.layer(x)
        return out

model_1 = My_Model_1(10,10)
print(model_1.layer.weight)
model_2 = My_Model_2(10,10)

How do I print the weights now?
model_2.layer.0.weight doesn't work.

Output:
From the PyTorch forum, this is the recommended way:

model_2.layer[0].weight


Instruction: Deep learning: save and load a universal machine model through different libraries

Input:
My questions can be divided into two parts.

Is there a format of machine learning model file that can be used through different libraries? For example, I saved a model by pytorch, then load it using tensorflow?
If not, is there a library that can help transfer the formats so that a pytorch machine learning model can be used directly in keras?

The reason why I ask this question is that recently I need to adjust some of my previous trained models in tensorflow to pytorch.

An update for this question:
Facebook and Microsoft are going to launch a model standard called ONNX, which is used for transferring models between different framworks, for example between Pytorch to Caffe2. Link in the following:

https://research.fb.com/facebook-and-microsoft-introduce-new-open-ecosystem-for-interchangeable-ai-frameworks/

An further update for this question:
Tensorflow itself use Protocol Buffer format to store model file, which can be used for transfer between different models. Link in the following:

https://www.tensorflow.org/extend/tool_developers/

Output:
Very interesting question. A neural network is a mathematical abstraction consisting of a network of layers (convolution, recurrent, ...), operations (dot product, non-linearity, ...) and their respective parameters (weights, biases).

AFAIK, there's not an universal model file. Nonetheless, different libraries allow users to save their models in a binary format.
There's no library for conversion but there's effort on github repo that addresses this question.


Instruction: How to get a uniform distribution in a range [r1,r2] in PyTorch?

Input: I want to get a 2-D torch.Tensor with size [a,b] filled with values from a uniform distribution (in range [r1,r2]) in PyTorch.

Output:
If U is a random variable uniformly distributed on [0, 1], then (r1 - r2) * U + r2 is uniformly distributed on [r1, r2].

Thus, you just need:

(r1 - r2) * torch.rand(a, b) + r2

Alternatively, you can simply use:

torch.FloatTensor(a, b).uniform_(r1, r2)

To fully explain this formulation, let's look at some concrete numbers:

r1 = 2 # Create uniform random numbers in half-open interval [2.0, 5.0)
r2 = 5

a = 1  # Create tensor shape 1 x 7
b = 7

We can break down the expression (r1 - r2) * torch.rand(a, b) + r2 as follows:

torch.rand(a, b) produces an a x b (1x7) tensor with numbers uniformly distributed in the range [0.0, 1.0).

x = torch.rand(a, b)
print(x)
# tensor([[0.5671, 0.9814, 0.8324, 0.0241, 0.2072, 0.6192, 0.4704]])

(r1 - r2) * torch.rand(a, b) produces numbers distributed in the uniform range [0.0, -3.0)

print((r1 - r2) * x)
tensor([[-1.7014, -2.9441, -2.4972, -0.0722, -0.6216, -1.8577, -1.4112]])

(r1 - r2) * torch.rand(a, b) + r2 produces numbers in the uniform range [5.0, 2.0)

print((r1 - r2) * x + r2)
tensor([[3.2986, 2.0559, 2.5028, 4.9278, 4.3784, 3.1423, 3.5888]])

Now, let's break down the answer suggested by @Jonasson: (r2 - r1) * torch.rand(a, b) + r1

Again, torch.rand(a, b) produces (1x7) numbers uniformly distributed in the range [0.0, 1.0).

x = torch.rand(a, b)
print(x)
# tensor([[0.5671, 0.9814, 0.8324, 0.0241, 0.2072, 0.6192, 0.4704]])

(r2 - r1) * torch.rand(a, b) produces numbers uniformly distributed in the range [0.0, 3.0).

print((r2 - r1) * x)
# tensor([[1.7014, 2.9441, 2.4972, 0.0722, 0.6216, 1.8577, 1.4112]])

(r2 - r1) * torch.rand(a, b) + r1 produces numbers uniformly distributed in the range [2.0, 5.0)

print((r2 - r1) * x + r1)
tensor([[3.7014, 4.9441, 4.4972, 2.0722, 2.6216, 3.8577, 3.4112]])

In summary, (r1 - r2) * torch.rand(a, b) + r2 produces numbers in the range [r2, r1), while (r2 - r1) * torch.rand(a, b) + r1 produces numbers in the range [r1, r2).


Instruction: How can I install python modules in a docker image?

Input: I have an image called: Image and a running container called: container. I want to install pytorch and anacoda. what's the easiest way to do this? Do I have to change the dockerfile and build a new image? Thanks a lot.

Output:
Yes, the best thing is to build your image in such a way it has the python modules are in there.

Here is an example. I build an image with the build dependencies:

$ docker build -t oz123/alpine-test-mycoolapp:0.5 - < Image
Sending build context to Docker daemon  2.56 kB
Step 1 : FROM alpine:3.5
 ---> 88e169ea8f46
Step 2 : ENV MB_VERSION 3.1.4
 ---> Running in 4587d36fa4ae
 ---> b7c55df49803
Removing intermediate container 4587d36fa4ae
Step 3 : ENV CFLAGS -O2
 ---> Running in 19fe06dcc314
 ---> 31f6a4f27d4b
Removing intermediate container 19fe06dcc314
Step 4 : RUN apk add --no-cache python3 py3-pip gcc python3-dev py3-cffi    file git curl autoconf automake py3-cryptography linux-headers musl-dev libffi-dev openssl-dev build-base
 ---> Running in f01b60b1b5b9
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
(1/57) Upgrading musl (1.1.15-r5 -> 1.1.15-r6)
(2/57) Upgrading zlib (1.2.8-r2 -> 1.2.11-r0)
(3/57) Installing m4 (1.4.17-r1)
(4/57) Installing perl (5.24.0-r0)
(5/57) Installing autoconf (2.69-r0)
(6/57) Installing automake (1.15-r0)
(7/57) Installing binutils-libs (2.27-r1)
...

Note, I am installing Python's pip inside the image, so later I can download packages from pypi. Packages like numpy might require a C compiler and tool chain, so I am installing these too.

After building the packages which require the build tools chain I remove the tool chain packages:

RUN apk del file pkgconf autoconf m4 automake perl g++ libstdc++

After you have your base image, you can run your application code in
an image building on top of it:

$ cat Dockerfile

    FROM oz123/alpine-test-mycoolapp
    ADD . /code
    WORKDIR /code
    RUN pip3 install -r requirements.txt -r requirements_dev.txt
    RUN pip3 install -e .
    RUN make clean
    CMD ["pytest", "-vv", "-s"]

I simply run this with docker.


Instruction: How to convert Pytorch autograd.Variable to Numpy?

Input:
The title says it all. I want to convert a PyTorch autograd.Variable to its equivalent numpy array. In their official documentation they advocated using a.numpy() to get the equivalent numpy array (for PyTorch tensor). But this gives me the following error:

  Traceback (most recent call last): File "stdin", line 1, in module
  File
  "/home/bishwajit/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py",
  line 63, in getattr    raise AttributeError(name) AttributeError:
  numpy

Is there any way I can circumvent this?

Output:
Two possible case

Using GPU: If you  try to convert a cuda float-tensor directly to numpy like shown below,it will throw an error.

  x.data.numpy()

  RuntimeError: numpy conversion for FloatTensor is not supported

So, you cant covert a cuda float-tensor directly to numpy, instead you have to convert it into a cpu float-tensor first, and try converting into numpy, like shown below.

  x.data.cpu().numpy()

Using CPU: Converting a CPU tensor is straight forward.

  x.data.numpy()


Instruction: PyTorch Linear layer input dimension mismatch

Input:
Im getting this error when passing the input data to the Linear (Fully Connected Layer) in PyTorch:

matrices expected, got 4D, 2D tensors

I fully understand the problem since the input data has a shape (N,C,H,W) (from a Convolutional+MaxPool layer) where:

N: Data Samples
C: Channels of the data
H,W: Height and Width

Nevertheless I was expecting PyTorch to do the "reshaping" of the data form:

[ N , D1,...Dn] --> [ N, D] where D = D1*D2*....Dn

I try to reshape the Variable.data, but I've read that this approach is not recommended since the gradients will conserve the previous shape, and that in general you should not mutate a Variable.data shape.

I am pretty sure there is a simple solution that goes along with the framework, but i haven't find it.

Is there a good solution for this?

PD: The Fully connected layer has as input size the value C * H * W

Output:
After reading some Examples I found the solution. here is how you do it without messing up the forward/backward pass flow:
(_, C, H, W) = x.data.size()
x = x.view( -1 , C * H * W)


Instruction: What is the relationship between PyTorch and Torch?

Input:
There are two PyTorch repositories :

https://github.com/hughperkins/pytorch
https://github.com/pytorch/pytorch

The first clearly requires Torch and lua and is a wrapper, but the second doesn't make any reference to the Torch project except with its name.
How is it related to the Lua Torch?

Output:
Here a short comparison on pytorch and torch.
Torch:

A Tensor library like numpy, unlike numpy it has strong GPU support.
Lua is a wrapper for Torch (Yes! you need to have a good understanding of Lua), and for that you will need LuaRocks package manager.

PyTorch:

No need for the LuaRocks package manager, no need to write code in Lua. And because we are using Python, we can develop Deep Learning models with utmost flexibility. We can also exploit major Python packages likes scipy, numpy, matplotlib and Cython with PyTorch's own autograd.

There is a detailed discussion on this on pytorch forum. Adding to that both PyTorch and Torch use THNN. Torch provides lua wrappers to the THNN library while Pytorch provides Python wrappers for the same.
PyTorch's recurrent nets, weight sharing and memory usage with the flexibility of interfacing with C, and the current speed of Torch.
For more insights, have a look at this discussion session here.


Instruction: Handling C++ arrays in Cython (with numpy and pytorch)

Input:
I am trying to use cython to wrap a C++ library (fastText, if its relevant). The C++ library classes load a very large array from disk. My wrapper instantiates a class from the C++ library to load the array, then uses cython memory views and numpy.asarray to turn the array into a numpy array, then calls torch.from_numpy to create a tensor.

The problem arising is how to handle deallocating the memory for the array.

Right now, I get pointer being freed was not allocated when the program exits. This is, I expect, because both the C++ code and numpy/pytorch are trying to manage the same chunk of RAM.

I could simply comment out the destructor in the C++ library, but that feels like its going to cause me a different problem down the road.

How should I approach the issue?  Is there any kind of best practices documentation somewhere on how to handle memory sharing with C++ and cython?

If I modify the C++ library to wrap the array in a shared_ptr, will cython (and numpy, pytorch, etc.) share the shared_ptr properly?

I apologize if the question is naive; Python garbage collection is very mysterious to me.

Any advice is appreciated.

Output:
I can think of three sensible ways of doing it. I'll outline them below (i.e. none of the code will be complete but hopefully it will be clear how to finish it).

1. C++ owns the memory; Cython/Python holds a shared pointer to the C++ class

(This is looks to be the lines you're already thinking along).

Start by creating a Cython class that holds a shared pointer

from libcpp.memory cimport shared_ptr

cdef class Holder:
    cdef shared_ptr[cpp_class] ptr

    @staticmethod
    cdef make_holder(shared_ptr[cpp_class] ptr):
       cdef holder = Holder() # empty class
       holder.ptr = ptr
       return holder

You then need to define the buffer protocol for Holder. This allows direct access to the memory allocated by cpp_class in a way that both numpy arrays and Cython memoryviews can understand. Thus they hold a reference to a Holder instance, which in turn keeps a cpp_class alive. (Use np.asarray(holder_instance) to create a numpy array that uses the instance's memory)

The buffer protocol is a little involved but Cython has fairly extensive documentation and you should largely be able to copy and paste their examples. The two methods you need to add to Holder are __getbuffer__ and __releasebuffer__.

2. Python owns the memory; Your C++ class holds a pointer to the Python object

In this version you allocate the memory as a numpy array (using the Python C API interface). When your C++ class is destructed in decrements the reference count of the array, however if Python holds references to that array then the array can outlive the C++ class.

#include <numpy/arrayobject.h>
#include <Python.h>

class cpp_class {
   private:
     PyObject* arr;
     double* data;
   public:
     cpp_class() {
       arr = PyArray_SimpleNew(...); // details left to be filled in
       data = PyArray_DATA(reinterpret_cast<PyArrayObject*>(arr));
       # fill in the data
     }

     ~cpp_class() {
         Py_DECREF(arr); // release our reference to it
     }

     PyObject* get_np_array() {
         Py_INCREF(arr); // Cython expects this to be done before it receives a PyObject
         return arr;
     }
};

See the numpy documentation for details of the how to allocate numpy arrays from C/C++. Be careful of reference counting if you define copy/move constructors.

The Cython wrapper then looks like:

cdef extern from "some_header.hpp":
    cdef cppclass cpp_class:
       # whatever constructors you want to allow
       object get_np_array()

3. C++ transfers ownership of the data to Python/Cython

In this scheme C++ allocates the array, but Cython/Python is responsible for deallocating it. Once ownership is transferred C++ no longer has access to the data.

class cpp_class {
   public:
     double* data; // for simplicity this is public - you may want to use accessors
     cpp_class() :
     data(new double[50])
     {/* fill the array as needed */}

     ~cpp_class() {
       delete [] data;
     }
};

// helper function for Cython
inline void del_cpp_array(double* a) {
   delete [] a;
}

You then use the cython.view.array class to capture the allocated memory. This has a callback function which is used on destruction:

from cython cimport view

cdef extern from "some_header.hpp":
   cdef cppclass cpp_class:
      double* data
      # whatever constructors and other functions
   void del_cpp_array(double*)

# later
cdef cpp_class cpp_instance # create this however you like
# ...
# modify line below to match your data
arr = view.array(shape=(10, 2), itemsize=sizeof(double), format="d",
                 mode="C", allocate_buffer=False)
arr.data = <char*>cpp_instance.data
cpp_instance.data = None # reset to NULL pointer
arr.callback_free_data = del_cpp_array

arr can then be used with a memoryview or a numpy array.

You may have to mess about a bit with casting from void* or char* with del_cpp_array - I'm not sure exactly what types the Cython interface requires.

The first option is probably most work to implement but requires few changes to the C++ code. The second option may require changes to your C++ code that you don't want to make. The third option is simple but means that C++ no longer has access to the data, which might be a disadvantage.


Instruction: Implementing Adagrad in Python

Input:
I'm trying to implement Adagrad in Python. For learning purposes, I am using matrix factorisation as an example. I'd be using Autograd for computing the gradients.

My main question is if the implementation is fine.

Problem description

Given a matrix A (M x N) having some missing entries, decompose into W and H having sizes (M x k) and (k X N) respectively. Goal would to learn W and H using Adagrad. I'd be following this guide for the Autograd implementation.

NB: I very well know that ALS based implementation are well-suited. I'm using Adagrad only for learning purposes

Customary imports

import autograd.numpy as np
import pandas as pd

Creating the matrix to be decomposed

A = np.array([[3, 4, 5, 2],
                   [4, 4, 3, 3],
                   [5, 5, 4, 3]], dtype=np.float32).T

Masking one entry

A[0, 0] = np.NAN

Defining the cost function

def cost(W, H):
    pred = np.dot(W, H)
    mask = ~np.isnan(A)
    return np.sqrt(((pred - A)[mask].flatten() ** 2).mean(axis=None))

Decomposition params

rank = 2
learning_rate=0.01
n_steps = 10000

Gradient of cost wrt params W and H

from autograd import grad, multigrad
grad_cost= multigrad(cost, argnums=[0,1])

Main Adagrad routine (this needs to be checked)

shape = A.shape

# Initialising W and H
H =  np.abs(np.random.randn(rank, shape[1]))
W =  np.abs(np.random.randn(shape[0], rank))

# gt_w and gt_h contain accumulation of sum of gradients
gt_w = np.zeros_like(W)
gt_h = np.zeros_like(H)

# stability factor
eps = 1e-8
print "Iteration, Cost"
for i in range(n_steps):

    if i%1000==0:
        print "*"*20
        print i,",", cost(W, H)

    # computing grad. wrt W and H
    del_W, del_H = grad_cost(W, H)

    # Adding square of gradient
    gt_w+= np.square(del_W)
    gt_h+= np.square(del_H)

    # modified learning rate
    mod_learning_rate_W = np.divide(learning_rate, np.sqrt(gt_w+eps))
    mod_learning_rate_H = np.divide(learning_rate, np.sqrt(gt_h+eps))
    W =  W-del_W*mod_learning_rate_W
    H =  H-del_H*mod_learning_rate_H

While the problem converges and I get a reasonable solution, I was wondering if the implementation is correct. Specifically, if the understanding of sum of gradients and then computing the adaptive learning rate is correct or not?

Output:
At a cursory glance, your code closely matches that at https://github.com/benbo/adagrad/blob/master/adagrad.py

del_W, del_H = grad_cost(W, H)

matches

grad=f_grad(w,sd,*args)

gt_w+= np.square(del_W)
gt_h+= np.square(del_H)

matches

gti+=grad**2

mod_learning_rate_W = np.divide(learning_rate, np.sqrt(gt_w+eps))
mod_learning_rate_H = np.divide(learning_rate, np.sqrt(gt_h+eps))

matches

adjusted_grad = grad / (fudge_factor + np.sqrt(gti))

W =  W-del_W*mod_learning_rate_W
H =  H-del_H*mod_learning_rate_H

matches

w = w - stepsize*adjusted_grad

So, assuming that adagrad.py is correct and the translation is correct, would make your code correct. (consensus does not prove your code to be right, but it could be a hint)


Instruction: pytorch custom layer "is not a Module subclass"

Input:
I am new to PyTorch, trying it out after using a different toolkit for a while.

I would like understand how to program custom layers and functions. And as a simple test, I wrote this:

class Testme(nn.Module):         ## it _is_ a sublcass of module ##
    def __init__(self):
        super(Testme, self).__init__()

    def forward(self, x):
        return x / t_.max(x)

which is intended to cause the data passing through it to sum to 1. Not actually useful, just at test.

Then I plug it to the example code from the PyTorch Playground:

def make_layers(cfg, batch_norm=False):
    layers = []
    in_channels = 3
    for i, v in enumerate(cfg):
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            padding = v[1] if isinstance(v, tuple) else 1
            out_channels = v[0] if isinstance(v, tuple) else v
            conv2d = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=padding)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(out_channels, affine=False), nn.ReLU()]
            else:
                layers += [conv2d, nn.ReLU()]
            layers += [Testme]                           # here <------------------
            in_channels = out_channels
    return nn.Sequential(*layers)

The result is an error!

TypeError: model.Testme is not a Module subclass

Maybe this needs to be a Function rather than a Module? Also not clear what the difference is between Function, Module.

For example, why does a Function need a backward(), even if it is constructed entirely from standard pytorch primitive, whereas a Module does not need this?

Output:
That's a simple one. You almost got it, but you forgot to actually create an instance of your new class Testme. You need to do this, even if the creation of an instance of a particular class doesn't take any parameters (as for Testme). But it's easier to forget than for a convolutional layer, to which you typically pass a lot of arguments.

Change the line you have indicated to the following and your problem is resolved.

layers += [Testme()]


Instruction: when is a pytorch custom function needed (rather than only a module)?

Input:
Pytorch beginner here! Consider the following custom Module:

class Testme(nn.Module):
    def __init__(self):
        super(Testme, self).__init__()

def forward(self, x):
    return x / t_.max(x).expand_as(x)

As far as I understand the documentation:
I believe this could also be implemented as a custom Function.  A subclass of Function requires a backward() method, but the
Module does not.  As well, in the doc example of a Linear Module, it depends on a Linear Function:

class Linear(nn.Module):
    def __init__(self, input_features, output_features, bias=True):
        ...
    def forward(self, input):
        return Linear()(input, self.weight, self.bias)

The question: I do not understand the relation between Module and Function. In the first listing above (module Testme), should it have an associated function? If not, then it is possible to implement this without a backward method by subclassing Module, so why does a Function always require a backward method?

Perhaps Functions are intended only for functions that are not composed out of existing torch functions? Say differently: maybe modules do not need the associated Function if their forward method is composed entirely from previously defined torch functions?

Output:
This information is gathered and summarised from the official PyTorch Documentaion.

torch.autograd.Functionreally lies at the heart of the autograd package in PyTorch. Any graph you build in PyTorch and any operation you conduct on Variables in PyTorch is based on a Function. Any function requires an  __init__(), forward() and backward() method (see more here: http://pytorch.org/docs/notes/extending.html) . This enables PyTorch to compute results and compute gradients for Variables.

nn.Module()in contrast is really just a convenience for organising your model, your different layers, etc. For example, it organises all the trainable parameters in your model in .parameters()and allows you to add another layer to a model easily, etc. etc. It is not the place where you define a backward method, because in the forward() method, you're supposed to use subclasses of Function(), for which you have already defined backward(). Hence, if you have specified the order of operations in forward(), PyTorch already knows how to back-propagate gradients.

Now, when should you use what?

If you have an operation that is just a composition of existing implemented functions in PyTorch (like your thing above), there's really no point adding any subclass to Function() yourself. Because you can just stack operations up and build a dynamic graph. It's however a sensible idea to bunch these operations together. If any operation involves trainable parameters (for example a linear layer of a neural network), you should subclass nn.Module() and bunch your operations together in the forward method. This allows you to easily access parameters (as outlined above) for use of torch.optim, etc. If you don't have any trainable parameters, I would probably still bunch them together, but a standard Python function, where you take care of the instantination of each operation you use would be sufficient.

If you have a new custom operation (e.g. a new stochastic layer with some complicated sampling procedure), you should subclass Function() and define __init__(), forward() and backward() to tell PyTorch how to compute results and how to compute gradients, when you use this operation. Afterwards, you should either create a functional version to take care of instantinating the function and use your operation or create a module, if your operation has trainable parameters. Again, you can read more about this in the link above.


Instruction: How to load a list of numpy arrays to pytorch dataset loader?

Input:
I have a huge list of numpy arrays, where each array represents an image and I want to load it using torch.utils.data.Dataloader object. But the documentation of torch.utils.data.Dataloader mentions that it loads data directly from a folder. How do I modify it for my cause? I am new to pytorch and any help would be greatly appreciated.
my numpy array for a single image looks something like this. The image is RBG image.
[[[ 70  82  94]
  [ 67  81  93]
  [ 66  82  94]
  ...,
  [182 182 188]
  [183 183 189]
  [188 186 192]]

 [[ 66  80  92]
  [ 62  78  91]
  [ 64  79  95]
  ...,
  [176 176 182]
  [178 178 184]
  [180 180 186]]

 [[ 62  82  93]
  [ 62  81  96]
  [ 65  80  99]
  ...,
  [169 172 177]
  [173 173 179]
  [172 172 178]]

 ...,

Output:
I think what DataLoader actually requires is an input that subclasses Dataset. You can either write your own dataset class that subclasses Datasetor use TensorDataset as I have done below:

import torch
import numpy as np
from torch.utils.data import TensorDataset, DataLoader

my_x = [np.array([[1.0,2],[3,4]]),np.array([[5.,6],[7,8]])] # a list of numpy arrays
my_y = [np.array([4.]), np.array([2.])] # another list of numpy arrays (targets)

tensor_x = torch.Tensor(my_x) # transform to torch tensor
tensor_y = torch.Tensor(my_y)

my_dataset = TensorDataset(tensor_x,tensor_y) # create your datset
my_dataloader = DataLoader(my_dataset) # create your dataloader

Works for me. Hope it helps you.


Instruction: Creating one hot vector from indices given as a tensor

Input:
I have a tensor of size 4 x 6 where 4 is batch size and 6 is sequence length. Every element of the sequence vectors are some index (0 to n). I want to create a 4 x 6 x n tensor where the vectors in 3rd dimension will be one hot encoding of the index which means I want to put 1 in the specified index and rest of the values will be zero.

For example, I have the following tensor:

[[5, 3, 2, 11, 15, 15],
[1, 4, 6, 7, 3, 3],
[2, 4, 7, 8, 9, 10],
[11, 12, 15, 2, 5, 7]]

Here, all the values are in between (0 to n) where n = 15. So, I want to convert the tensor to a 4 X 6 X 16 tensor where the third dimension will represent one hot encoding vector.

How can I do that using PyTorch functionalities? Right now, I am doing this with loop but I want to avoid looping!

Output:
NEW ANSWER
As of PyTorch 1.1, there is a one_hot function in torch.nn.functional. Given any tensor of indices indices and a maximal index n, you can create a one_hot version as follows:

n = 5
indices = torch.randint(0,n, size=(4,7))
one_hot = torch.nn.functional.one_hot(indices, n) # size=(4,7,n)

Very old Answer

At the moment, slicing and indexing can be a bit of a pain in PyTorch from my experience. I assume you don't want to convert your tensors to numpy arrays. The most elegant way I can think of at the moment is to use sparse tensors and then convert to a dense tensor. That would work as follows:

from torch.sparse import FloatTensor as STensor

batch_size = 4
seq_length = 6
feat_dim = 16

batch_idx = torch.LongTensor([i for i in range(batch_size) for s in range(seq_length)])
seq_idx = torch.LongTensor(list(range(seq_length))*batch_size)
feat_idx = torch.LongTensor([[5, 3, 2, 11, 15, 15], [1, 4, 6, 7, 3, 3],
                             [2, 4, 7, 8, 9, 10], [11, 12, 15, 2, 5, 7]]).view(24,)

my_stack = torch.stack([batch_idx, seq_idx, feat_idx]) # indices must be nDim * nEntries
my_final_array = STensor(my_stack, torch.ones(batch_size * seq_length),
                         torch.Size([batch_size, seq_length, feat_dim])).to_dense()

print(my_final_array)

Note: PyTorch is undergoing some work currently, that will add numpy style broadcasting and other functionalities within the next two or three weeks and other functionalities. So it's possible, there'll be better solutions available in the near future.

Hope this helps you a bit.


Instruction: tensor division in pytorch. Assertion error

Input:
This is kind of a pytorch beginner question. In pytorch I'm trying to do element wise division with two tensors of size [5,5,3]. In numpy it works fine using np.divide(), but somehow I get an error here. I'm using PyTorch version 0.1.12 for Python 3.5.

c = [torch.DoubleTensor of size 5x5x3]

input_patch = [torch.FloatTensor of size 5x5x3]

input_patch is a slice of a torch.autograd Variable, and c is made by doing c = torch.from_numpy(self.patch_filt[:, :, :, 0]).float()

When doing:

torch.div(input_patch, c)

I get this error that I don't understand.

line 317, in div
assert not torch.is_tensor(other)
AssertionError

Does it mean that variable c should not be a torch_tensor? After casting c to also be a FloatTensor gives the same error still.

Thank you!

Output:
Input_patch is a slice of a torch.autograd Variable, and c is made by doing

c = torch.from_numpy(self.patch_filt[:, :, :, 0]).float()

Anyway, mexmex, thanks to your comment I've solved it by defining c as

Variable(torch.from_numpy(self.patch_filt[:, :, :, 0])).float()


Instruction: How do I multiply matrices in PyTorch?

Input:
With numpy, I can do a simple matrix multiplication like this:
a = numpy.ones((3, 2))
b = numpy.ones((2, 1))
result = a.dot(b)

However, this does not work with PyTorch:
a = torch.ones((3, 2))
b = torch.ones((2, 1))
result = torch.dot(a, b)

This code throws the following error:

RuntimeError: 1D tensors expected, but got 2D and 2D tensors

How do I perform matrix multiplication in PyTorch?

Output:
Use torch.mm:
torch.mm(a, b)

torch.dot() behaves differently to np.dot(). There's been some discussion about what would be desirable here. Specifically, torch.dot() treats both a and b as 1D vectors (irrespective of their original shape) and computes their inner product. The error is thrown because this behaviour makes your a a vector of length 6 and your b a vector of length 2; hence their inner product can't be computed. For matrix multiplication in PyTorch, use torch.mm(). Numpy's np.dot() in contrast is more flexible; it computes the inner product for 1D arrays and performs matrix multiplication for 2D arrays.
torch.matmul performs matrix multiplications if both arguments are 2D and computes their dot product if both arguments are 1D. For inputs of such dimensions, its behaviour is the same as np.dot. It also lets you do broadcasting or matrix x matrix, matrix x vector and vector x vector operations in batches.
# 1D inputs, same as torch.dot
a = torch.rand(n)
b = torch.rand(n)
torch.matmul(a, b) # torch.Size([])

# 2D inputs, same as torch.mm
a = torch.rand(m, k)
b = torch.rand(k, j)
torch.matmul(a, b) # torch.Size([m, j])


Instruction: python child process exit unexpectedly with exit code -9

Input:
I have a PyTorch script with 16 processes. Following is a code snippet from the main process:

procs = [mp.Process(target=self.worker_wrapper, args=(i, )) for i in range(self.n_workers)]
for p in procs: p.start()
while True:
    time.sleep(60)
    for i, p in enumerate(procs):
        self.logger.info('Check: id %d, exitcode %s, alive %s' % (
            i, str(p.exitcode), str(p.is_alive())))

the worker_wrapper is this:

def worker_wrapper(self, id):
    try:
        self.worker(id)
    except Exception as e:
        self.logger.info(e)
        self.logger.error(traceback.format_exc())

The worker will do some stuff and should never stop. It works well at beginning, but usually after 1 days running, some child processes (not all) exit unexpectedly, I get the output like this:

[2017-06-15 08:45:21,540] Check: id 13, exitcode -9, alive False

And it didn't catch any exception. I tried to search exitcode -9 but didn't get anything useful. Does anyone meet this before? Or do you have any idea about how to debug this? I'm sorry I can't provide a minimal reproducible code snippet, because it usually takes one day to reproduce. And I'm using python2.7

Output:
Exit code -9 means the process was killed via SIGKILL.

It's probably due to your machine running out of memory and the OS kernel was triggering the OOM killer.

To verify that, check the kernel logs via dmesg you should notice the OOM killer targeting your worker processes.

You need to reduce the memory footprint of your application (or increase the memory of your server/workstation). Also verify your application is not leaking memory.


Instruction: CUDA vs. DataParallel: Why the difference?

Input:
I have a simple neural network model and I apply either cuda() or DataParallel() on the model like following.

model = torch.nn.DataParallel(model).cuda()

OR,

model = model.cuda()

When I don't use DataParallel, rather simply transform my model to cuda(), I need to explicitly convert the batch inputs to cuda() and then give it to the model, otherwise it returns the following error.

  torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, torch.LongTensor)

But with DataParallel, the code works fine. Rest of the other things are same. Why this happens? Why when I use DataParallel, I don't need to transform the batch inputs explicitly to cuda()?

Output:
Because, DataParallel allows CPU inputs, as it's first step is to transfer inputs to appropriate GPUs.

Info source: https://discuss.pytorch.org/t/cuda-vs-dataparallel-why-the-difference/4062/3


Instruction: Custom loss function in PyTorch

Input:
I have three simple questions.

What will happen if my custom loss function is not differentiable? Will pytorch through error or do something else?
If I declare a loss variable in my custom function which will represent the final loss of the model, should I put requires_grad = True for that variable? or it doesn't matter? If it doesn't matter, then why?
I have seen people sometimes write a separate layer and compute the loss in the forward function. Which approach is preferable, writing a function or a layer? Why?

I need a clear and nice explanation to these questions to resolve my confusions. Please help.

Output:
Let me have a go.

This depends on what you mean by "non-differentiable". The first definition that makes sense here is that PyTorch doesn't know how to compute gradients. If you try to compute gradients nevertheless, this will raise an error. The two possible scenarios are:

a) You're using a custom PyTorch operation for which gradients have not been implemented, e.g. torch.svd(). In that case you will get a TypeError:

import torch
from torch.autograd import Function
from torch.autograd import Variable

A = Variable(torch.randn(10,10), requires_grad=True)
u, s, v = torch.svd(A) # raises TypeError

b) You have implemented your own operation, but did not define backward(). In this case, you will get a NotImplementedError:

class my_function(Function): # forgot to define backward()

    def forward(self, x):
        return 2 * x

A = Variable(torch.randn(10,10))
B = my_function()(A)
C = torch.sum(B)
C.backward() # will raise NotImplementedError

The second definition that makes sense is "mathematically non-differentiable". Clearly, an operation which is mathematically not differentiable should either not have a backward() method implemented or a sensible sub-gradient. Consider for example torch.abs() whose backward() method returns the subgradient 0 at 0:

A = Variable(torch.Tensor([-1,0,1]),requires_grad=True)
B = torch.abs(A)
B.backward(torch.Tensor([1,1,1]))
A.grad.data

For these cases, you should refer to the PyTorch documentation directly and dig out the backward() method of the respective operation directly.
It doesn't matter. The use of requires_gradis to avoid unnecessary computations of gradients for subgraphs. If there’s a single input to an operation that requires gradient, its output will also require gradient. Conversely, only if all inputs don’t require gradient, the output also won’t require it. Backward computation is never performed in the subgraphs, where all Variables didn’t require gradients.

Since, there are most likely some Variables (for example parameters of a subclass of nn.Module()), your loss Variable will also require gradients automatically. However, you should notice that exactly for how requires_grad works (see above again), you can only change requires_grad for leaf variables of your graph anyway.
All the custom PyTorch loss functions, are subclasses of _Loss which is a  subclass of nn.Module. See here. If you'd like to stick to this convention, you should subclass _Loss when defining your custom loss function. Apart from consistency, one advantage is that your subclass will raise an AssertionError, if you haven't marked your target variables as volatile or requires_grad = False. Another advantage is that you can nest your loss function in nn.Sequential(), because its a nn.Module I would recommend this approach for these reasons.


Instruction: Is Torch7 defined-by-run like Pytorch?

Input:
Pytorch have Dynamic Neural Networks (defined-by-run) as opposed to Tensorflow which have to compile the computation graph before run.

I see that both Torch7 and PyTorch depend on TH, THC, THNN, THCUNN (C library). Does Torch7 have Dynamic Neural Networks (defined-by-run) feature ?

Output: No, Torch7 use static computational graphs, as in Tensorflow. It is one of the major differences between PyTorch and Torch7.


Instruction: subtraction of scalar from tensor yields 'inconsistent tensor size' in pytorch

Input:
I'm using pytorch and my variables are

x = [torch.FloatTensor of size 1x3x32x32]
mean = Variable containing:
1.00000e-02 *
  2.0518
[torch.FloatTensor of size 1]

what I want to do is subtract the scalar mean from x by doing

x = x - mean

However, I'm getting this error:

RuntimeError: inconsistent tensor size at /py/conda-
bld/pytorch_1493670682084/work/torch/lib/TH/generic/THTensorMath.c:831

What am I doing wrong?
Thanks a lot

Output:
what you are trying only works if mean is truly a scalar, i.e. a float() (in this case) and not a torch.FloatTensor of size 1. You can either extract a true scalar from mean or expand mean to the size of x in order to perform the subtraction.

To extract the float from mean, do:

x = x - mean[0]

To expand mean to the size of x, do:

x = x - mean.expand_as(x)

Note that both of these methods subtract the mean from each element in your tensor.