Instruction: How to correctly give inputs to Embedding, LSTM and Linear layers in PyTorch?

Input:
I need some clarity on how to correctly prepare inputs for batch-training using different components of the torch.nn module. Specifically, I'm looking to create an encoder-decoder network for a seq2seq model.

Suppose I have a module with these three layers, in order:

nn.Embedding
nn.LSTM
nn.Linear

nn.Embedding

Input: batch_size * seq_length
Output: batch_size * seq_length * embedding_dimension

I don't have any problems here, I just want to be explicit about the expected shape of the input and output.

nn.LSTM

Input: seq_length * batch_size * input_size (embedding_dimension in this case)
Output: seq_length * batch_size * hidden_size
last_hidden_state: batch_size * hidden_size
last_cell_state: batch_size * hidden_size

To use the output of the Embedding layer as input for the LSTM layer, I need to transpose axis 1 and 2.

Many examples I've found online do something like x = embeds.view(len(sentence), self.batch_size , -1), but that confuses me. How does this view ensure that elements of the same batch remain in the same batch? What happens when len(sentence) and self.batch size are of same size?

nn.Linear

Input: batch_size x input_size (hidden_size of LSTM in this case or ??)
Output: batch_size x output_size

If I only need the last_hidden_state of LSTM, then I can give it as input to nn.Linear.

But if I want to make use of Output (which contains all intermediate hidden states as well) then I need to change nn.Linear's input size to seq_length * hidden_size and to use Output as input to Linear module I need to transpose axis 1 and 2 of output and then I can view with Output_transposed(batch_size, -1).

Is my understanding here correct? How do I carry out these transpose operations in tensors (tensor.transpose(0, 1))?

Output:
Your understanding of most of the concepts is accurate, but, there are some missing points here and there.

Interfacing embedding to LSTM (Or any other recurrent unit)

You have embedding output in the shape of (batch_size, seq_len, embedding_size). Now, there are various ways through which you can pass this to the LSTM.
* You can pass this directly to the LSTM, if LSTM accepts input as batch_first. So, while creating your LSTM pass argument batch_first=True.
* Or, you can pass input in the shape of (seq_len, batch_size, embedding_size). So, to convert your embedding output to this shape, you’ll need to transpose the first and second dimensions using torch.transpose(tensor_name, 0, 1), like you mentioned.

  Q. I see many examples online which do something like x = embeds.view(len(sentence), self.batch_size , -1) which confuses me.
  A. This is wrong. It will mix up batches and you will be trying to learn a hopeless learning task. Wherever you see this, you can tell the author to change this statement and use transpose instead.

There is an argument in favor of not using batch_first, which states that the underlying API provided by Nvidia CUDA runs considerably faster using batch as secondary.

Using context size

You are directly feeding the embedding output to LSTM, this will fix the input size of LSTM to context size of 1. This means that if your input is words to LSTM, you will be giving it one word at a time always. But, this is not what we want all the time. So, you need to expand the context size. This can be done as follows -

# Assuming that embeds is the embedding output and context_size is a defined variable
embeds = embeds.unfold(1, context_size, 1)  # Keeping the step size to be 1
embeds = embeds.view(embeds.size(0), embeds.size(1), -1)

Unfold documentation
Now, you can proceed as mentioned above to feed this to the LSTM, just remembed that seq_len is now changed to seq_len - context_size + 1 and embedding_size (which is the input size of the LSTM) is now changed to context_size * embedding_size

Using variable sequence lengths

Input size of different instances in a batch will not be the same always. For example, some of your sentence might be 10 words long and some might be 15 and some might be 1000. So, you definitely want variable length sequence input to your recurrent unit. To do this, there are some additional steps that needs to be performed before you can feed your input to the network. You can follow these steps -
1. Sort your batch from largest sequence to the smallest.
2. Create a seq_lengths array that defines the length of each sequence in the batch. (This can be a simple python list)
3. Pad all the sequences to be of equal length to the largest sequence.
4. Create LongTensor Variable of this batch.
5. Now, after passing the above variable through embedding and creating the proper context size input, you’ll need to pack your sequence as follows -

# Assuming embeds to be the proper input to the LSTM
lstm_input = nn.utils.rnn.pack_padded_sequence(embeds, [x - context_size + 1 for x in seq_lengths], batch_first=False)

Understanding output of LSTM

Now, once you have prepared your lstm_input acc. To your needs, you can call lstm as

lstm_outs, (h_t, h_c) = lstm(lstm_input, (h_t, h_c))

Here, (h_t, h_c) needs to be provided as the initial hidden state and it will output the final hidden state. You can see, why packing variable length sequence is required, otherwise LSTM will run the over the non-required padded words as well.
Now, lstm_outs will be a packed sequence which is the output of lstm at every step and (h_t, h_c) are the final outputs and the final cell state respectively. h_t and h_c will be of shape (batch_size, lstm_size). You can use these directly for further input, but if you want to use the intermediate outputs as well you’ll need to unpack the lstm_outs first as below

lstm_outs, _ = nn.utils.rnn.pad_packed_sequence(lstm_outs)

Now, your lstm_outs will be of shape (max_seq_len - context_size + 1, batch_size, lstm_size). Now, you can extract the intermediate outputs of lstm according to your need.

  Remember that the unpacked output will have 0s after the size of each batch, which is just padding to match the length of the largest sequence (which is always the first one, as we sorted the input from largest to the smallest).

  Also note that, h_t will always be equal to the last element for each batch output.

Interfacing lstm to linear

Now, if you want to use just the output of the lstm, you can directly feed h_t to your linear layer and it will work. But, if you want to use intermediate outputs as well, then, you’ll need to figure out, how are you going to input this to the linear layer (through some attention network or some pooling). You do not want to input the complete sequence to the linear layer, as different sequences will be of different lengths and you can’t fix the input size of the linear layer. And yes, you’ll need to transpose the output of lstm to be further used (Again you cannot use view here).

  Ending Note: I have purposefully left some points, such as using bidirectional recurrent cells, using step size in unfold, and interfacing attention, as they can get quite cumbersome and will be out of the scope of this answer.


Instruction: How do you use next_functions[0][0] on grad_fn correctly in pytorch?

Input:
I was given this nn structure in the offical pytorch tutorial:

  input
  -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d
       -> view -> linear -> relu -> linear -> relu -> linear
       -> MSELoss
       -> loss

then an example of how to follow the grad backwards using built-in .grad_fn from Variable.

# Eg:
print(loss.grad_fn)  # MSELoss
print(loss.grad_fn.next_functions[0][0])  # Linear
print(loss.grad_fn.next_functions[0][0].next_functions[0][0])  # ReLU

So I thought I can reach the grad object for Conv2d by pasting next_function[0][0] 9 times because of the given examples but I got the error tuple out of index. So how can I index these backprop objects correctly?

Output:
In the PyTorch CNN tutorial after running the following from the tutorial:

output = net(input)
target = torch.randn(10)  # a dummy target, for example
target = target.view(1, -1)  # make it the same shape as output
criterion = nn.MSELoss()

loss = criterion(output, target)
print(loss)

The following code snippet will print the full graph:

def print_graph(g, level=0):
    if g == None: return
    print('*'*level*4, g)
    for subg in g.next_functions:
        print_graph(subg[0], level+1)

print_graph(loss.grad_fn, 0)


Instruction: Pytorch save embeddings as part of encoder class or not

Input:
So I'm using pytorch for the first time. I'm trying to save weights to a file. I'm using a Encoder class that has a GRU and a embedding component. I want to make sure when I save the Encoder values that I will get the embedding values. Initially my code uses state_dict() to copy values to a dictionary of my own which I pass to torch.save(). Should I be looking for some way to save this embedding component or is it part of the larger encoder? The Encoder is a subclass of nn.Module . here's a link:

http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py

def make_state(self, converted=False):
    if not converted:
        z = [
            {
                'epoch':0,
                'arch': None,
                'state_dict': self.model_1.state_dict(),
                'best_prec1': None,
                'optimizer': self.opt_1.state_dict(),
                'best_loss': self.best_loss
            },
            {
                'epoch':0,
                'arch':None,
                'state_dict':self.model_2.state_dict(),
                'best_prec1':None,
                'optimizer': self.opt_2.state_dict(),
                'best_loss': self.best_loss
            }
        ]
    else:
        z = [
            {
                'epoch': 0,
                'arch': None,
                'state_dict': self.model_1.state_dict(),
                'best_prec1': None,
                'optimizer': None , # self.opt_1.state_dict(),
                'best_loss': self.best_loss
            },
            {
                'epoch': 0,
                'arch': None,
                'state_dict': self.model_2.state_dict(),
                'best_prec1': None,
                'optimizer': None, # self.opt_2.state_dict(),
                'best_loss': self.best_loss
            }
        ]
    #print(z)
    return z
    pass

def save_checkpoint(self, state=None, is_best=True, num=0, converted=False):
    if state is None:
        state = self.make_state(converted=converted)
        if converted: print(converted, 'is converted.')
    basename = hparams['save_dir'] + hparams['base_filename']
    torch.save(state, basename + '.' + str(num)+ '.pth.tar')
    if is_best:
        os.system('cp '+ basename + '.' + str(num) + '.pth.tar' + ' '  +
                  basename + '.best.pth.tar')

https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610/3

Here is another link

Output:
No, you do not need to save the embedding values explicitly. Saving a model’s state_dict will save all the variables pertaining to that model, including the embedding weights.
You can look for what a state dict contains by looping over it as -

for var_name in model.state_dict():
    print(var_name)


Instruction: Masking diagonal to a specific value with PyTorch tensors

Input:
How do I fill the diagonal with a value in torch? In numpy you can do:
a = np.zeros((3, 3), int)
np.fill_diagonal(a, 5)

array([[5, 0, 0],
       [0, 5, 0],
       [0, 0, 5]])

I know that torch.diag() returns the diagonal, but how to use this as a mask to assign new values is beyond me. I haven't been able to find the answer here or in the PyTorch documentation.

Output:
You can do this in PyTorch using fill_diagonal_:
>>> a = torch.zeros(3, 3)
>>> a.fill_diagonal_(5)
tensor([[5, 0, 0],
        [0, 5, 0],
        [0, 0, 5]])


Instruction: Exploding loss in pyTorch

Input:
I am trying to train a latent space model in pytorch. The model is relatively simple and just requires me to minimize my loss function but I am getting an odd error. After running for a short while the loss suddenly explodes upwards.

import numpy as np
import scipy.sparse.csgraph as csg
import torch
from torch.autograd import Variable
import torch.autograd as autograd
import matplotlib.pyplot as plt
%matplotlib inline

def cmdscale(D):
    # Number of points
    n = len(D)

    # Centering matrix
    H = np.eye(n) - np.ones((n, n))/n

    # YY^T
    B = -H.dot(D**2).dot(H)/2

    # Diagonalize
    evals, evecs = np.linalg.eigh(B)

    # Sort by eigenvalue in descending order
    idx   = np.argsort(evals)[::-1]
    evals = evals[idx]
    evecs = evecs[:,idx]

    # Compute the coordinates using positive-eigenvalued components only
    w, = np.where(evals > 0)
    L  = np.diag(np.sqrt(evals[w]))
    V  = evecs[:,w]
    Y  = V.dot(L)

    return Y, evals

Y = np.array([[0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
              [0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
              [1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0.]])

temp = Y[~np.all(Y == 0, axis=1)]
temp = temp[:,~np.all(Y == 0, axis=1)]
Y = temp

n = np.shape(Y)[0]
k = 2

D = csg.shortest_path(Y, directed=True)
Z = cmdscale(D)[0][:,0:k]
Z = Z - Z.mean(axis=0, keepdims=True)

tZ = autograd.Variable(torch.Tensor(Z), requires_grad=True)
B = autograd.Variable(torch.Tensor([0]), requires_grad=True)
tY = torch.autograd.Variable(torch.Tensor(Y), requires_grad=False)

#calculating pairwise euclidean distance
def distMatrix(m):
    n = m.size(0)
    d = m.size(1)
    x = m.unsqueeze(1).expand(n, n, d)
    y = m.unsqueeze(0).expand(n, n, d)
    return torch.sqrt(torch.pow(x - y, 2).sum(2) + 1e-4)

def loss(tY):
    d = -distMatrix(tZ)+B
    sigmoidD = torch.sigmoid(d)
    #removing diagonal
    reduce = tY*torch.log(sigmoidD)+(1-tY)*torch.log(1-sigmoidD)
    reduce[torch.eye(n).byte()] = 0
    return -reduce.sum()

losses = []
learning_rate = 1e-4
l = loss(tY)
stepSize = 1000

for i in range(stepSize):
    l.backward(retain_graph=True)
    losses.append(float(loss(tY)))
    tZ.data = tZ.data - learning_rate * tZ.grad.data
    B.data = B.data - learning_rate * B.grad.data

    tZ.grad.data.zero_()
    B.grad.data.zero_()

plt.subplot(122)
plt.plot(losses)
plt.title('Loss')
plt.xlabel('Iteration')
plt.ylabel('loss')

plt.show()

shouldnt the loss keep going down? or atleast converge to some point? I must've done something wrong, I am new to pytorch, any hints or nudges in the right direction would be highly appreciated!

Output:
The issue was that I defined my loss

l = loss(tY)

outside of the loop that ran and updated my gradients, I am not entirely sure why it had the effect that it did, but moving the loss function definition inside of the loop solved the problem, resulting in this loss:


Instruction: Force GPU memory limit in PyTorch

Input: Is there a way to force a maximum value for the amount of GPU memory that I want to be available for a particular Pytorch instance? For example, my GPU may have 12Gb available, but I'd like to assign 4Gb max to a particular process.

Output:
Update (04-MAR-2021): it is now available in the stable 1.8.0 version of PyTorch. Also, in the docs
Original answer follows.

This feature request has been merged into PyTorch master branch. Yet, not introduced in the stable release.
Introduced as set_per_process_memory_fraction

Set memory fraction for a process.
The fraction is used to limit an caching allocator to allocated memory on a CUDA device.
The allowed value equals the total visible memory multiplied fraction.
If trying to allocate more than the allowed value in a process, will raise an out of
memory error in allocator.

You can check the tests as usage examples.


Instruction: Pytorch errors when given numpy integer types only in python 3 (not in python 2)

Input:
For example, the torch.randn function, among others, gets mad when given a numpy.int64 type:

Python 3.5.5 |Anaconda custom (64-bit)| (default, Mar 12 2018, 23:12:44)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import numpy
>>> torch.randn(numpy.int64(4))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.randn received an invalid combination of arguments - got (numpy.int64), but expected one of:
 * (int ... size)
      didn't match because some of the arguments have invalid types: (numpy.int64)
 * (torch.Size size)
      didn't match because some of the arguments have invalid types: (numpy.int64)
 * (torch.Generator generator, int ... size)
 * (torch.Generator generator, torch.Size size)

But in python 2, this works just fine:

Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import numpy
>>> torch.randn(numpy.int64(3))

-2.0513
 0.5409
-0.0814
[torch.FloatTensor of size 3]

I couldn't find anyone else running into this issue. Is this known? Is this something about my setup? Is there any way to work around this without completely dropping numpy?

I'm using version 0.3.1 of pytorch and version 1.14.2 of numpy.

Output:
On Python 2, on an OS where a C long is 64-bit, numpy.int64 is a subclass of int, so most things that want ints will accept numpy.int64 even if they're not written to handle int-like types.

On Python 3, that doesn't happen any more. If you need to use a library that wants real ints, call int:

torch.randn(int(some_numpy_integer))


Instruction: PyTorch: Variable data has to be a tensor -- data is already as tenors

Input:
I am trying to load data using pytorch's Dataset and DataLoader classes. I use torch.from_numpyto convert each array to a tensor in the torch Dataset and from looking into the data, each X and y is indeed a tensor

# At this point dataset is {'X': numpy array of arrays, 'y': numpy array of arrays  }

class TorchDataset(torch.utils.data.Dataset):
    def __init__(self, dataset):
        self.X_train = torch.from_numpy(dataset['X'])
        self.y_train = torch.from_numpy(dataset['y'])

    def __len__(self):
        return len(self.X_train)

    def __getitem__(self, index):
        return {'X': self.X_train[index], 'y': self.y_train[index]}

torch_dataset = TorchDataset(dataset)
dataloader = DataLoader(torch_dataset, batch_size=4, shuffle=True, num_workers=4)

for epoch in range(num_epochs):
    for X, y in enumerate(dataloader):
        features = Variable(X)
        labels = Variable(y)
        ....

However on features = Variable(X) i get:

RuntimeError: Variable data has to be a tensor, but got int

An example of an X and y in the dataset are:

In [1]: torch_dataset[1]
Out[1]:
{'X':
 -2.5908 -3.1123 -2.9460  ...  -3.9898 -4.0000 -3.9975
 -3.0867 -2.9992 -2.5254  ...  -4.0000 -4.0000 -4.0000
 -2.7665 -2.5318 -2.7035  ...  -4.0000 -4.0000 -4.0000
       ...             ⋱             ...
 -2.4784 -2.6061 -1.6280  ...  -4.0000 -4.0000 -4.0000
 -2.2046 -2.1778 -1.5626  ...  -3.9597 -3.9366 -3.9497
 -1.9623 -1.9468 -1.5352  ...  -3.8485 -3.8474 -3.8474
 [torch.DoubleTensor of size 1024x1024], 'y':
  107
 [torch.LongTensor of size 1]}

which is why it is very confusing for me that torch thinks X is an int. Any help would be much appreciated - thanks!

Output:
There is error in your use of enumerate which caused the error because the first return value of enumerate is the batch index, not the actual data. There are two ways you can make your script work.

First way

Since your X and y is do not need special process. You can just return a sample of X and y. Change your __getitem__ method to

 def __getitem__(self, index):
        return self.X_train[index], self.y_train[index]

Also, change your training loop a little bit:

for epoch in range(num_epochs):
    for batch_id, (x, y) in enumerate(dataloader):
           x = Variable(x)
           y = Variable(y)
           # then do whatever you want to do

Second way

You can return a dict in the __getitem__ method and extract the actual data in the training loop. In this case, you do not need to change the __getitem__ method. Just change your training loop:

for epoch in range(num_epochs):
    for batch_id, data in enumerate(dataloader):
        # data will be dict
        x = Variable(data['X'])
        y = Variable(data['y'])
        # then do whatever you want to do


Instruction: How does PyTorch module do the back prop

Input: While following the instructions on extending PyTorch - adding a module, I noticed while extending Module, we don't really have to implement the backward function. The only thing we need is to apply the Function instance in the forward function and PyTorch can automatically call the backward one in the Function instance when doing the back prop. This seems like magic to me as we didn't even register the Function instance we used. I looked into the source code but didn't find anything related. Could anyone kindly point me out a place that all those actually happened?

Output:
Not having to implement backward() is the reason PyTorch or any other DL framework is so valuable. In fact, implementing backward() should only be done in very specific cases where you need to mess with the network's gradient (or when you create a custom Function that can't be expressed using PyTorch's built-in functions).

PyTorch computes backward gradients using a computational graph which keeps track of what operations have been done during your forward pass. Any operation done on a Variable implicitly get registered here. Then it's a matter of traversing the graph backward from the variable where it was called, and applying derivative chain rule to compute the gradients.

PyTorch's About page has a nice visualization of the graph and how it generally works. I'd also recommend looking up compute graphs and autograd mechanism on Google if you want more details.

EDIT: The source code where all this happens would be in the C part of PyTorch's codebase, where the actual graph is implemented. After some digging around, I found this:

/// Evaluates the function on the given inputs and returns the result of the
/// function call.
variable_list operator()(const variable_list& inputs) {
    profiler::RecordFunction rec(this);
    if (jit::tracer::isTracingVar(inputs)) {
        return traced_apply(inputs);
    }
    return apply(inputs);
}

So in each Function, PyTorch first checks if its inputs needs tracing, and performs trace_apply() as implemented here. You can see the node being created and appended to the graph:

// Insert a CppOp in the trace.
auto& graph = state->graph;
std::vector<VariableFlags> var_flags;
for(auto & input: inputs) {
    var_flags.push_back(VariableFlags::of(input));
}
auto* this_node = graph->createCppOp(get_shared_ptr(), std::move(var_flags));
// ...
for (auto& input: inputs) {
    this_node->addInput(tracer::getValueTrace(state, input));
}
graph->appendNode(this_node);

My best guess here is that every Function object registers itself and its inputs (if needed) upon execution. Every non-functional calls (eg. variable.dot()) simply defers to the corresponding function, so this still applies.

NOTE: I don't take part in PyTorch's development and is in no way an expert on its architecture. Any corrections or addition would be welcomed.


Instruction: Find a GPU with enough memory

Input:
I want to programmatically find out the available GPUs and their current memory usage and use one of the GPUs based on their memory availability. I want to do this in PyTorch.

I have seen the following solution in this post:

import torch.cuda as cutorch

for i in range(cutorch.device_count()):
    if cutorch.getMemoryUsage(i) > MEM:
        opts.gpuID = i
        break

but it is not working in PyTorch 0.3.1 (there is no function called, getMemoryUsage). I am interested in a PyTorch based (using the library functions) solution. Any help would be appreciated.

Output:
In the webpage you give, there exist an answer:

#!/usr/bin/env python
# encoding: utf-8

import subprocess

def get_gpu_memory_map():
    """Get the current gpu usage.

    Returns
    -------
    usage: dict
        Keys are device ids as integers.
        Values are memory usage as integers in MB.
    """
    result = subprocess.check_output(
        [
            'nvidia-smi', '--query-gpu=memory.used',
            '--format=csv,nounits,noheader'
        ])
    # Convert lines into a dictionary
    gpu_memory = [int(x) for x in result.strip().split('\n')]
    gpu_memory_map = dict(zip(range(len(gpu_memory)), gpu_memory))
    return gpu_memory_map

print get_gpu_memory_map()


Instruction: How to resolve runtime error due to size mismatch in PyTorch?

Input:
I am trying to implement a simple autoencoder using PyTorch. My dataset consists of 256 x 256 x 3 images. I have built a torch.utils.data.dataloader.DataLoader object which has the image stored as tensor. When I run the autoencoder, I get a runtime error:

  size mismatch, m1: [76800 x 256], m2: [784 x 128] at
  /Users/soumith/minicondabuild3/conda-bld/pytorch_1518371252923/work/torch/lib/TH/generic/THTensorMath.c:1434

These are my hyperparameters:

batch_size=100,
learning_rate = 1e-3,
num_epochs = 100

Following is the architecture of my auto-encoder:

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(3*256*256, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(True),
            nn.Linear(64, 12),
            nn.ReLU(True),
            nn.Linear(12, 3))

        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.Linear(128, 3*256*256),
            nn.ReLU())

def forward(self, x):
    x = self.encoder(x)
    #x = self.decoder(x)
    return x

This is the code I used to run the model:

for epoch in range(num_epochs):
for data in dataloader:
    img = data['image']
    img = Variable(img)
    # ===================forward=====================
    output = model(img)
    loss = criterion(output, img)
    # ===================backward====================
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
# ===================log========================
print('epoch [{}/{}], loss:{:.4f}'
      .format(epoch+1, num_epochs, loss.data[0]))
if epoch % 10 == 0:
    pic = show_img(output.cpu().data)
    save_image(pic, './dc_img/image_{}.jpg'.format(epoch))

Output:
If your input is 3 x 256 x 256, then you need to convert it to B x N to pass it through the linear layer: nn.Linear(3*256*256, 128) where B is the batch_size and N is the linear layer input size.
If you are giving one image at a time, you can convert your input tensor of shape 3 x 256 x 256 to 1 x (3*256*256) as follows.

img = img.view(1, -1) # converts [3 x 256 x 256] to 1 x 196608
output = model(img)


Instruction: Pytorch, pipenv and numpy support?

Input:
I am using pytorch and installed it follow these instructions:

pipenv install git+https://github.com/pytorch/pytorch#egg=pytorch
pipenv install git+https://github.com/pytorch/vision#egg=torchvision

in order to install pytorch with pipenv. This runs some of the examples, but for instance, this MNIST code:

from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

# Training settings
parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                    help='input batch size for training (default: 64)')
parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                    help='input batch size for testing (default: 1000)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
                    help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                    help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                    help='SGD momentum (default: 0.5)')
parser.add_argument('--no-cuda', action='store_true', default=False,
                    help='disables CUDA training')
parser.add_argument('--seed', type=int, default=1, metavar='S',
                    help='random seed (default: 1)')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                    help='how many batches to wait before logging training status')
args = parser.parse_args()
args.cuda = not args.no_cuda and torch.cuda.is_available()

torch.manual_seed(args.seed)
if args.cuda:
    torch.cuda.manual_seed(args.seed)

kwargs = {'num_workers': 1, 'pin_memory': True} if args.cuda else {}
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.test_batch_size, shuffle=True, **kwargs)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

model = Net()
if args.cuda:
    model.cuda()

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data[0] # sum up batch loss
        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

for epoch in range(1, args.epochs + 1):
    train(epoch)
    test()

Does not run giving the error:

RuntimeError: PyTorch was compiled without NumPy support

Running:

pipenv install numpy

does not resolve the error. Is there a way to use pipenv and install pytorch with numpy support?

Output:
I managed to get it. I slightly changed:

pipenv install git+https://github.com/pytorch/pytorch#egg=pytorch
pipenv install git+https://github.com/pytorch/vision#egg=torchvision

To:

pipenv install http://download.pytorch.org/whl/cu91/torch-0.3.1-cp36-cp36m-linux_x86_64.whl
pipenv install git+https://github.com/pytorch/vision#egg=torchvision

The first command is what pytorch.org recomends when I select Linux, pip, python 3.6, and CUDA 9.1. For some reason, pipenv install torchvision doesn't work.

For reference here is the resulting Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]

"633048b" = {file = "http://download.pytorch.org/whl/cu91/torch-0.3.1-cp36-cp36m
-linux_x86_64.whl"}
torchvision = {git = "https://github.com/pytorch/vision"}

[dev-packages]


Instruction: What's the difference between reshape and view in pytorch?

Input:
In numpy, we use ndarray.reshape() for reshaping an array.

I noticed that in pytorch, people use torch.view(...) for the same purpose, but at the same time, there is also a torch.reshape(...) existing.

So I am wondering what the differences are between them and when I should use either of them?

Output:
torch.view has existed for a long time. It will return a tensor with the new shape. The returned tensor will share the underling data with the original tensor.
See the documentation here.

On the other hand, it seems that torch.reshape has been introduced recently in version 0.4. According to the document, this method will

  Returns a tensor with the same data and number of elements as input, but with the specified shape. When possible, the returned tensor will be a view of input. Otherwise, it will be a copy. Contiguous inputs and inputs with compatible strides can be reshaped without copying, but you should not depend on the copying vs. viewing behavior.

It means that torch.reshape may return a copy or a view of the original tensor. You can not count on that to return a view or a copy. According to the developer:

  if you need a copy use clone() if you need the same storage use view(). The semantics of reshape() are that it may or may not share the storage and you don't know beforehand.

Another difference is that reshape() can operate on both contiguous and non-contiguous tensor while view() can only operate on contiguous tensor. Also see here about the meaning of contiguous.


Instruction: Downloading pretrained TimeNet RNN?

Input:
I've been looking for the pretrained net behind TimeNet - a resnet equivalent for time series analysis via RNNs.

There's some articles and several publications of the same paper, but the weights and architecture themselves are not easy to find.

Anyone knows how to find it? Is it available to the public?

Output:
It looks like someone answered this on the top response to this Quora question.

Why are there no pre-trained RNN models

It's not available to the public just yet.


Instruction: pytorch - net.cuda() seems don't work

Input:
I wrote a cnn module to do digit recognition using pytorch, then try to train the network with gpu but got following error.

Traceback (most recent call last):
  File "main.py", line 51, in <module>
    outputs = cnn(inputs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/Code/kaggle-competitions/digit-recognizer/Net.py", line 40, in forward
    x = self.pool(F.relu(self.conv[i](x)))
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/conv.py", line 282, in forward
    self.padding, self.dilation, self.groups)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 90, in conv2d
    return f(input, weight, bias)
RuntimeError: Input type (CUDAFloatTensor) and weight type (CPUFloatTensor) should be the same

here is my source code

It seems that cnn.cuda() didn't work properly because I got the same error after removing it. But I have no idea how to fix it.

Output:
Daniel's answer to his own question seems to be correct. The problem is indeed that modules are not recognized if they are appended to a list. However, Pytorch also provides built-in solutions to this problem:
nn.ModuleList and nn.ModuleDict are two container types that keep track of the added content and their parameters.
Both have the same functionality as their Python equivalents, but the dictionary uses named arguments and can be used to keep track of for example task-specific layers.

A working example would be:

    self.conv = torch.nn.ModuleList()
    self.conv.append(nn.Conv2d(1, channels[0], kernel_sizes[0]))
    self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[0]-1))/2)

    for i in range(1, self.conv_layer_size):
        self.conv.append(nn.Conv2d(channels[i-1], channels[i], kernel_sizes[i]))
        self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[i]-1))/2)
        # Modules are automatically added to the model parameters


Instruction: Resolved package not found

Input:
When I try to execute conda env create -f virtual_platform_windows.yml
It shows

ResolvePackageNotFound:
  - pytorch==0.1.12=py35_0.1.12cu80

I tried installing pytorch for windows and error still comes.How to solve this??

Output:
Open: virtual_platform_windows.yml in Notepad
Delete: - pytorch=0.1.12=py35_0.1.12cu80
Delete: - torch==0.1.12
Save

Using Anaconda prompt:

Execute the code: conda env create -f virtual_platform_windows.yml

Activate virtual environment: source activate virtual_platform

Install Pytorch seperately with conda install -c peterjc123 pytorch cuda80

Copy haas files into script directory


Instruction: PyTorch - WSD using LSTM

Input:
I'm trying to replicate Google's research paper on WSD with neural models using PyTorch.

I'm having some issues traying to overfit the model before training on large datasets.

Using this training set:

  The film was also intended to be the first in a trilogy.

this model definition:

class WordGuesser(nn.Module):
    def __init__(self, hidden_dim, context_dim, embedding_dim, vocabulary_dim, batch_dim, window_dim):
        super(WordGuesser, self).__init__()
        self.hidden_dim = hidden_dim
        self.batch_dim = batch_dim
        self.window_dim = window_dim
        self.word_embeddings = nn.Embedding(vocabulary_dim, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        #self.extract_context = nn.Linear((2 * window_dim + 1) * hidden_dim, context_dim)
        self.extract_context = nn.Linear(hidden_dim, context_dim)
        self.predict = nn.Linear(context_dim, vocabulary_dim)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (autograd.Variable(torch.zeros(1, self.batch_dim, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1, self.batch_dim, self.hidden_dim).cuda()))

    def forward(self, sentence, hidden):
        embeddings = self.word_embeddings(sentence)
        out, self.hidden = self.lstm(embeddings.permute(1, 0, 2), hidden)
        lstm_out = out[-1]
        context = self.extract_context(lstm_out)
        prediction = self.predict(context)
        return prediction, context

and this training routine:

num_epoch = 100
hidden_units = 512
embedding_dim = 256
context_dim = 256

def mytrain():
    lines = open('training/overfit.txt').readlines()
    sentences = data.split_to_sentences(lines) #uses spaCy to detect sentences from each line
    word2idx=dict() #dictionary is built from the training set
    idx2word =dict()
    i = 0
    for s in sentences:
        for t in s.split(' '):
            if t in word2idx:
                continue
            word2idx[t] = i
            idx2word[i] = t
            i += 1
    word2idx['$'] = i #the token to guess the missing word in a sentence
    idx2word[i] = '$'
    X = list()
    Y = list()
    for sentence in sentences:
        sentence = sentence.split(' ')
        for i in range(len(sentence)):
            newsentence = list(sentence)
            newsentence[i] = '$'
            if not sentence[i] in word2idx:
                continue
            indices = [word2idx[w] for w in newsentence]
            label = word2idx[sentence[i]]
            X.append(indices)
            Y.append(label)
    model = WordGuesser(hidden_units, context_dim, embedding_dim, len(word2idx), len(X), len(X[0]))
    model.train()
    model.cuda()
    input = torch.LongTensor(X).cuda()
    output = torch.LongTensor(Y).cuda()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    model.hidden = model.init_hidden()
    for epoch in range(num_epoch):
        model.hidden = model.init_hidden()
        model.zero_grad()
        input_tensor = autograd.Variable(input)
        target_tensor = autograd.Variable(output)

        predictions, context = model(input_tensor, model.hidden)
        for i, prediction in enumerate(predictions):
            sorted_val = sorted(enumerate(np.array(prediction.data)), key=lambda x : x[1], reverse=True)
            print([(idx2word[x[0]], x[1]) for x in sorted_val[:5]], idx2word[Y[i]])
        loss = criterion(predictions, target_tensor)
        loss.backward()
        optimizer.step()

        print(epoch, loss.data[0])

    torch.save(model, "train2.pt")

during the training it seems that the model is able to overfit just after the 21st epoch, as you can see from the following scores (top 5 words from the predictions and the last word in a line is the label for that sentence):

  [('The', 11.362326), ('film', 11.356865), ('also', 7.5573149), ('to',
  5.3518314), ('intended', 4.3520432)] The

  [('film', 11.073805), ('The', 10.451499), ('also', 7.5498624), ('was',
  4.9684553), ('be', 4.0730805)] film

  [('was', 11.232123), ('also', 9.9741745), ('the', 6.0156212), ('be',
  4.9949703), ('The', 4.5516477)] was

  [('also', 9.6998224), ('was', 9.6202812), ('The', 6.345758), ('film',
  4.9122157), ('be', 2.6727715)] also

  [('intended', 18.344809), ('to', 16.410078), ('film', 10.147289),
  ('The', 9.8423424), ('$', 9.6181822)] intended

  [('to', 12.442947), ('intended', 10.900065), ('film', 8.2598763),
  ('The', 8.0493736), ('$', 4.4901967)] to

  [('be', 12.189278), ('also', 7.7172523), ('was', 7.5415096), ('the',
  5.2521734), ('The', 4.1723843)] be

  [('the', 15.59604), ('be', 9.3750105), ('first', 8.9820032), ('was',
  8.6859236), ('also', 5.0665498)] the

  [('first', 10.191225), ('the', 5.1829329), ('in', 3.6020348), ('be',
  3.4108081), ('a', 1.5569853)] first

  [('in', 14.731103), ('first', 9.3131113), ('a', 5.982264), ('trilogy',
  4.2928643), ('be', 0.49548936)] in

  [('a', 14.357709), ('in', 8.3088198), ('trilogy', 6.3918238),
  ('first', 6.2178354), ('intended', 0.95656234)] a

  [('trilogy', 14.351434), ('a', 4.5073452), ('in', 4.2348137), ('$',
  3.7552347), ('intended', 3.5101018)] trilogy

  [('.', 18.152126), ('$', 12.028764), ('to', 9.6003456), ('intended',
  8.1202478), ('The', 4.9225812)] .

When running another Python script which loads the model and queries it for the following words (using the same code to print out the scores during the training):

  The film was also intended to $ the first in a trilogy.   be

  The film $ also intended to be the first in a trilogy.    was

  $ film was also intended to be the first in a trilogy.    The

I'm getting these scores:

  [('film', 24.066889), ('$', 20.107487), ('was', 16.855488), ('a',
  12.969441), ('in', 8.1248817)] be

  [('film', 24.089062), ('$', 20.116539), ('was', 16.891994), ('a',
  12.982826), ('in', 8.1167336)] was

  [('film', 23.993624), ('$', 20.108011), ('was', 16.891005), ('a',
  12.960193), ('in', 8.1577587)] The

I've also tried setting to False the model.train() mode, using model.eval() as well as calling topk on the LSTM scores, but the results aren't satifying,

Output:
Solved it by saving only the model's state_dict() via torch.save() and then loading it back in the evaluation phase using model.load_state_dict().

Furthermore, I wrapped the sentence querying loop in another loop, acting as a warm-up (got it from here) and once it was at its last time looping, I set model.eval() and printed the scores, which turned out to be correct.


Instruction: PyTorch / Gensim - How do I load pre-trained word embeddings?

Input:
I want to load a pre-trained word2vec embedding with gensim into a PyTorch embedding layer.
How do I get the embedding weights loaded by gensim into the PyTorch embedding layer?

Output:
I just wanted to report my findings about loading a gensim embedding with PyTorch.

Solution for PyTorch 0.4.0 and newer:

From v0.4.0 there is a new function from_pretrained() which makes loading an embedding very comfortable.
Here is an example from the documentation.

import torch
import torch.nn as nn

# FloatTensor containing pretrained weights
weight = torch.FloatTensor([[1, 2.3, 3], [4, 5.1, 6.3]])
embedding = nn.Embedding.from_pretrained(weight)
# Get embeddings for index 1
input = torch.LongTensor([1])
embedding(input)

The weights from gensim can easily be obtained by:

import gensim
model = gensim.models.KeyedVectors.load_word2vec_format('path/to/file')
weights = torch.FloatTensor(model.vectors) # formerly syn0, which is soon deprecated

As noted by @Guglie: in newer gensim versions the weights can be obtained by model.wv:

weights = model.wv

Solution for PyTorch version 0.3.1 and older:

I'm using version 0.3.1 and from_pretrained() isn't available in this version.

Therefore I created my own from_pretrained so I can also use it with 0.3.1.

Code for from_pretrained for PyTorch versions 0.3.1 or lower:

def from_pretrained(embeddings, freeze=True):
    assert embeddings.dim() == 2, \
         'Embeddings parameter is expected to be 2-dimensional'
    rows, cols = embeddings.shape
    embedding = torch.nn.Embedding(num_embeddings=rows, embedding_dim=cols)
    embedding.weight = torch.nn.Parameter(embeddings)
    embedding.weight.requires_grad = not freeze
    return embedding

The embedding can be loaded then just like this:

embedding = from_pretrained(weights)

I hope this is helpful for someone.


Instruction: Error Utilizing Pytorch Transforms and Custom Dataset

Input:
This question mainly concerns the return value of __getitem__ in a pytorch Dataset which I've seen as both a tuple and a dict in the source code.

I have been following this tutorial for creating a dataset class within my code, which is following this tutorial on transfer learning. It has the following definition of a dataset.

class FaceLandmarksDataset(Dataset):
"""Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:].as_matrix()
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample

As you can see, __getitem__ returns a dictionary with two entries.
In the transfer learning tutorial, the following calls are made to transform a dataset:

    data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

use_gpu = torch.cuda.is_available()

inputs, classes = next(iter(dataloaders['train']))

That last line of code causes an error in my code by attempting to run transform on a sample in my custom dataset.

  'dict' object has no attribute 'size'

But if the tutorial dataset is implemented correctly, shouldn't it function correctly with a transform? My own hybrid implementation is below:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy
from torch.utils.data import *
from skimage import io, transform
plt.ion()

class NumsDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.docs = []
        for file in os.listdir(root_dir):
            #print(file)
            if file.endswith(".txt"):
                path = os.path.join(root_dir, file)
                with open(path, 'r') as f:
                    self.docs.append( (  file , list(f.read()) ) ) #tup containing file, image values pairs
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self): #returns number of images
        i = 0
        for j in self.docs:
            i += len(j[1])
        return i

    def len2(self): #returns number of batches
        return len(self.docs)

    def __getitem__(self, idx):
        idx1 = idx // self.len2()
        idx2 = idx % self.len2()
        imglabel = self.docs[idx1][0] #label with filename for batch error calculation later
        imgdir = os.path.join(self.root_dir, self.docs[idx1][0].strip(".txt"))
        img = None
        l = idx2

        for file in os.listdir(imgdir):
            file = os.path.join(imgdir, file)
            if(l == 0):
                img = io.imread(file)
            l -= 1
        sample = (img , imglabel)
        sample ={'image': img, 'label': imglabel}
        if self.transform:
            sample = self.transform(sample)

        return sample

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}
data_dir = "images"
image_datasets = {x: NumsDataset(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=5)
              for x in ['train', 'val']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = ["one", "two", "four"]

use_gpu = torch.cuda.is_available()
# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))

directory structure:

images
     /train
        /file1
            *.jpg
        /file2...
            *.jpg
        file1.txt
        file2.txt...
     /val
        /file1
            *.jpg
        /file2...
            *.jpg
        file1.txt
        file2.txt...

Is the sample I'm returning formatted incorrectly?

Output: The particular way the tutorial on dataloading uses the custom dataset is with self defined transforms. The transforms must be designed to fit the dataset. As such, the dataset must output a sample compatible with the library transform functions, or transforms must be defined for the particular sample case. Choosing the latter, among other things has resulted in completely functional code.


Instruction: pytorch doesn't give expected output

Input:
Firstly, a bunch of data is classified by the CNN model. Then, I'm trying to make prediction on correctly classified data from first step, which is expected to give an accuracy of 100%. However, I found the result is unstable, sometimes 99+%, but not 100%. Is there anybody know what is the problem with my code? Thank you very much in advance, it has troubled me several days ~ ~

torch.version

'0.3.1.post2'

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable

n = 2000
data = np.random.randn(n, 1, 10, 10)
label = np.random.randint(2, size=(n, ))

def test_pred(model, data_test, label_test):

    data_batch = data_test
    labels_batch = label_test

    images = torch.autograd.Variable(torch.FloatTensor(data_batch))
    labels = torch.autograd.Variable(torch.FloatTensor(labels_batch))

    outputs = model(images)

    _, predicted = torch.max(outputs.data, 1)

    correct = (np.array(predicted) == labels_batch).sum()

    label_pred = np.array(predicted)

    acc = correct/len(label_test)
    print(" acc:", acc)

    return acc, label_pred

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.fc = nn.Linear(128, 2)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

cnn = CNN()

[_, label_pred] = test_pred(cnn, data, label)

print("Acc:", np.mean(label_pred==label))
# Given the correctly classified data in previous step, expect to get 100% accuracy
# Why it sometimes doesn't give a 100% accuracy ?
print("Using selected data size {}:".format(data[label_pred==label].shape))
_, _ = test_pred(cnn, data[label_pred==label], label[label_pred==label])

output:

acc: 0.482

Acc: 0.482

Using selected data size (964, 1, 10, 10):

acc: 0.9979253112033195

Output:
Seems like you did not set the network to evaluation mode which might be causing some problems, specially with the BatchNorm layers. Do

cnn = CNN()
cnn.eval()

and it should work.


Instruction: How are PyTorch's tensors implemented?

Input:
I am building my own Tensor class in Rust, and I am trying to make it like PyTorch's implementation.

What is the most efficient way to store tensors programmatically, but, specifically, in a strongly typed language like Rust? Are there any resources that provide good insights into how this is done?

I am currently building a contiguous array, so that, given dimensions of 3 x 3 x 3, my array would just have 3^3 elements in it, which would represent the tensor. However, this does make some of the mathematical operations and manipulations of the array harder.

The dimension of the tensor should be dynamic, so that I could have a tensor with n dimensions.

Output:
Contiguous array

The commonly used way to store such data is in a single array that is laid out as a single, contiguous block within memory. More concretely, a 3x3x3 tensor would be stored simply as a single array of 27 values, one after the other.

The only place where the dimensions are used is to calculate the mapping between the (many) coordinates and the offset within that array. For example, to fetch the item [3, 1, 1] you would need to know if it is a 3x3x3 matrix, a 9x3x1 matrix, or a 27x1x1 matrix - in all cases the "storage" would be 27 items long, but the interpretation of "coordinates" would be different. If you use zero-based indexing, the calculation is trivial, but you need to know the length of each dimension.

This does mean that resizing and similar operations may require copying the whole array, but that's ok, you trade off the performance of those (rare) operations to gain performance for the much more common operations, e.g. sequential reads.


Instruction: Get all 2D diagonals of a 3D tensor in numpy

Input:
I have a 3D tensor A x B x C. For each matrix B x C, I want to extract the leading diagonal.

Is there a vectorized way of doing this in numpy or pytorch instead of looping over A?

Output:
You can use numpy.diagonal()

np.diagonal(a, axis1=1, axis2=2)

Example:

In [10]: a = np.arange(3*4*5).reshape(3,4,5)

In [11]: a
Out[11]:
array([[[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14],
        [15, 16, 17, 18, 19]],

       [[20, 21, 22, 23, 24],
        [25, 26, 27, 28, 29],
        [30, 31, 32, 33, 34],
        [35, 36, 37, 38, 39]],

       [[40, 41, 42, 43, 44],
        [45, 46, 47, 48, 49],
        [50, 51, 52, 53, 54],
        [55, 56, 57, 58, 59]]])

In [12]: np.diagonal(a, axis1=1, axis2=2)
Out[12]:
array([[ 0,  6, 12, 18],
       [20, 26, 32, 38],
       [40, 46, 52, 58]])


Instruction: How to form a sequence of consecutive numbers in Pytorch?

Input:
How to convert the Matlab code
v = [1: n]

to pytorch?
Writing a whole loop for that seems inefficient.

Output:
You can directly use the arange method from Pytorch.

torch_v = torch.arange(1,n)

Reference: https://pytorch.org/docs/master/torch.html?highlight=arange#torch.arange


Instruction: What is the benefit of random image crop on Convolutional Network?

Input:
I`m studying about transfer learning with the pytorch tutorial. I found pytorch tutorial author uses the different approach to train set and validation set.

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

As above, training session uses the Randomly cropped image on training. But I found that after that transformation, some of the images are totally cut off the target object and remains no useful feature for detection. I am curious why the author has chosen that method, even if it is more difficult to define different pipeline to train&validation dataset. Is there any benefit of the random cropping?

Output:
Couple of ideas behind random cropping:

In short:

Extending the amount of data for training
Making NN more robust

More detail:

The semantics of the image are preserved but the activation values of the conv net are different. The conv net learns to associate a broader range of spatial activations with a certain class label and improves the robustness of the feature detectors in conv nets.

Look at this excerpt https://link.springer.com/chapter/10.1007/978-3-319-31293-4_25


Instruction: Pytorch tensor to numpy array

Input:
I have a pytorch Tensor of shape [4, 3, 966, 1296]. I want to convert it to numpy array using the following code:
imgs = imgs.numpy()[:, ::-1, :, :]

How does that code work?

Output:
There are 4 dimensions of the tensor you want to convert.

[:, ::-1, :, :]

: means that the first dimension should be copied as it is and converted, same goes for the third and fourth dimension.

::-1 means that for the second axes it reverses the the axes


Instruction: Pytorch hidden state LSTM

Input:
Why do we need to initialize the hidden state h0 in LSTM in pytorch. As h0 will anyways be calculated and get overwritten ? Isn't it like

int a
a = 0

a = 4

Even if we do not do a=0, it should be fine..

Output:
The point is that you are able to supply the initial state, it is a feature. They could have implemented it as a default but by letting you control the allocation of the tensor you can save some memory (allocating once, zeroing on every invocation).

Why would you need to set h? Sequence-to-sequence models require this (compress input to one vector, use this vector as hidden state for the decoder) or you might want to make the initial state learnable.


Instruction: How is the print and view functions works in pytorch?

Input:
This is a convolutional neural network which I found in the web

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(500, 50)
        self.fc2 = nn.Linear(50, 64)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 500)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

and its summary

print(net)
Net(
  (conv1): Conv2d(3, 10, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
  (conv2_drop): Dropout2d(p=0.5)
  (fc1): Linear(in_features=500, out_features=50, bias=True)
  (fc2): Linear(in_features=50, out_features=64, bias=True)
)

What is x.view does? Is it similar to the Flatten function in keras. The other query is reagarding how pytorch prints summary of a model. Eventhough the model uses two dropouts nn.Dropout2d() and F.dropout. When printing the model we can see only one (conv2_drop): Dropout2d(p=0.5), why?. The last question is why pytorch  dosen't print F.max_pool2d layer?

Output:
1) x.view can do more than just flatten: It will keep the same data while reshaping the dimension. So using x.view(batch_size, -1)will be equivalent to Flatten

2) In the __repr__function of nn.Module, the elements that are printed are the modules in self._modules.items() which are its children.

F.dropoutand F.max_pool2d are functions and not children of nn.Module, thus they are not layers and will not be printed. For pooling and dropout however, there is a module in torch.nn which you already used for the first dropout.


Instruction: How does one manually compute the error of the whole data set in pytorch?

Input:
I was trying to track the error of the whole data set and compute the error of the whole data set in pytorch. I wrote the following (reproducible example and fully contained) in cifar10 pytorch 0.3.1:

import torch
from torch.autograd import Variable
import torch.optim as optim

import torchvision
import torchvision.transforms as transforms

from math import inf

from pdb import set_trace as st

def error_criterion(outputs,labels):
    max_vals, max_indices = torch.max(outputs,1)
    train_error = (max_indices != labels).sum().data[0]/max_indices.size()[0]
    return train_error

def evalaute_mdl_data_set(loss,error,net,dataloader,enable_cuda,iterations=inf):
    '''
    Evaluate the error of the model under some loss and error with a specific data set.
    '''
    running_loss,running_error = 0,0
    for i,data in enumerate(dataloader):
        if i >= iterations:
            break
        inputs, labels = extract_data(enable_cuda,data,wrap_in_variable=True)
        outputs = net(inputs)
        running_loss += loss(outputs,labels).data[0]
        running_error += error(outputs,labels)
    return running_loss/(i+1),running_error/(i+1)

def extract_data(enable_cuda,data,wrap_in_variable=False):
    inputs, labels = data
    if enable_cuda:
        inputs, labels = inputs.cuda(), labels.cuda() #TODO potential speed up?
    if wrap_in_variable:
        inputs, labels = Variable(inputs), Variable(labels)
    return inputs, labels

def train_and_track_stats(enable_cuda, nb_epochs, trainloader,testloader, net,optimizer,criterion,error_criterion, iterations=inf):
    ''' Add stats before training '''
    train_loss_epoch, train_error_epoch = evalaute_mdl_data_set(criterion, error_criterion, net, trainloader, enable_cuda, iterations)
    test_loss_epoch, test_error_epoch = evalaute_mdl_data_set(criterion, error_criterion, net, testloader, enable_cuda, iterations)
    print(f'[-1, -1], (train_loss: {train_loss_epoch}, train error: {train_error_epoch}) , (test loss: {test_loss_epoch}, test error: {test_error_epoch})')
    ##
    ''' Start training '''
    print('about to start training')
    for epoch in range(nb_epochs):  # loop over the dataset multiple times
        running_train_loss,running_train_error = 0.0,0.0
        for i,data_train in enumerate(trainloader):
            ''' zero the parameter gradients '''
            optimizer.zero_grad()
            ''' train step = forward + backward + optimize '''
            inputs, labels = extract_data(enable_cuda,data_train,wrap_in_variable=True)
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_train_loss += loss.data[0]
            running_train_error += error_criterion(outputs,labels)
        ''' End of Epoch: collect stats'''
        train_loss_epoch, train_error_epoch = running_train_loss/(i+1), running_train_error/(i+1)
        #train_loss_epoch, train_error_epoch = evalaute_mdl_data_set(criterion,error_criterion,net,trainloader,enable_cuda,iterations)
        test_loss_epoch, test_error_epoch = evalaute_mdl_data_set(criterion,error_criterion,net,testloader,enable_cuda,iterations)
        print(f'[{epoch}, {i+1}], (train_loss: {train_loss_epoch}, train error: {train_error_epoch}) , (test loss: {test_loss_epoch}, test error: {test_error_epoch})')
    return train_loss_epoch, train_error_epoch, test_loss_epoch, test_error_epoch

class Flatten(torch.nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

def main():
    enable_cuda = True
    print('running main')
    num_workers = 0
    ''' Get Data set '''
    batch_size_test = 10000
    batch_size_train = 10000
    data_path = './data'
    transform = [transforms.ToTensor(),transforms.Normalize( (0.5, 0.5, 0.5), (0.5, 0.5, 0.5) )]
    transform = transforms.Compose(transform)
    trainset = torchvision.datasets.CIFAR10(root=data_path, train=True,download=False, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size_train,shuffle=True, num_workers=num_workers)
    testset = torchvision.datasets.CIFAR10(root=data_path, train=False,download=False, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size_test,shuffle=False, num_workers=num_workers)
    ''' Get model '''
    net = torch.nn.Sequential(
        torch.nn.Conv2d(3,13,5), #(in_channels, out_channels, kernel_size),
        Flatten(),
        torch.nn.Linear(28*28*13, 13),
        torch.nn.Linear(13, 10)
    )
    net.cuda()
    ''' Train '''
    nb_epochs = 10
    lr = 0.1
    err_criterion = error_criterion
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=lr, momentum=0.0)
    train_and_track_stats(enable_cuda, nb_epochs, trainloader,testloader, net,optimizer,criterion,err_criterion, iterations=inf)
    ''' Done '''
    print('Done')

if __name__ == '__main__':
    main()

When I run it I get the following error:

python my_cifar10.py
running main
[-1, -1], (train_loss: 2.3172860145568848, train error: 0.0054) , (test loss: 2.317185878753662, test error: 0.0038)
about to start training
[0, 5], (train_loss: 2.22599835395813, train error: 0.015160000000000002) , (test loss: 2.0623881816864014, test error: 0.0066)
[1, 5], (train_loss: 2.014406657218933, train error: 0.00896) , (test loss: 1.9619578123092651, test error: 0.0195)
[2, 5], (train_loss: 1.9428715705871582, train error: 0.01402) , (test loss: 1.918603539466858, test error: 0.0047)
[3, 5], (train_loss: 1.9434458494186402, train error: 0.01192) , (test loss: 1.9194672107696533, test error: 0.0125)
[4, 5], (train_loss: 1.8804980754852294, train error: 0.00794) , (test loss: 1.8549214601516724, test error: 0.004)
[5, 5], (train_loss: 1.8573726177215577, train error: 0.010159999999999999) , (test loss: 1.8625996112823486, test error: 0.0158)
[6, 5], (train_loss: 1.8454653739929199, train error: 0.01524) , (test loss: 1.8155865669250488, test error: 0.0122)
[7, 5], (train_loss: 1.8140610456466675, train error: 0.01066) , (test loss: 1.808283805847168, test error: 0.0101)
[8, 5], (train_loss: 1.8036894083023072, train error: 0.00832) , (test loss: 1.799634575843811, test error: 0.007)
[9, 5], (train_loss: 1.8023016452789307, train error: 0.0077399999999999995) , (test loss: 1.8030155897140503, test error: 0.0114)
Done

Clearly it has to be wrong cuz the test error is nearly zero with a model that is ridiculously small and simple (1 conv 2 fcs).

the code seems so simple that I can't figure out what is going wrong. I've been doing stuff and changing things for a few days now. Any new suggestions what to try?

Output:
If your batch size is too large, with your code the values of

(max_indices == labels).sum()
(max_indices != labels).sum()

do not add up to the batch size. This is due to the fact, that you use a torch.ByteTensor which will overflow for values > 255 when summing.

using

(max_indices != labels).int().sum()

will resolve the issue by casting the Tensor to int before summing.


Instruction: what is the default weight initializer for conv in pytorch?

Input: The question How to initialize weights in PyTorch? shows how to initialize the weights in Pytorch.  However, what is the default weight initializer for Convand Dense in Pytorch? What distribution does Pytorch use?

Output:
Each pytorch layer implements the method reset_parameters which is called at the end of the layer initialization to initialize the weights.
You can find the implementation of the layers here.

For the dense layer which in pytorch is called linear for example, weights are initialized uniformly

stdv = 1. / math.sqrt(self.weight.size(1))
self.weight.data.uniform_(-stdv, stdv)

where self.weight.size(1) is the number of inputs. This is done to keep the variance of the distributions of each layer relatively similar at the beginning of training by normalizing it to one. You can read a more detailed explanation here.

For the convolutional layer the initialization is basically the same. You just compute the number of inputs by multiplying the number of channels with the kernel size.


Instruction: PyTorch Getting Custom Loss Function Running

Input:
I'm trying to use a custom loss function by extending nn.Module, but I can't get past the error

  element 0 of variables does not require grad and does not have a grad_fn

Note: my labels are lists of size: num_samples, but each batch will have the same labels throughout the batch, so we shrink labels for the whole batch to be a single label by calling .diag()

My code is as follows and is based on the transfer learning tutorial:

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train(True)  # Set model to training mode
            else:
                model.train(False)  # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for data in dataloaders[phase]:
                # get the inputs
                inputs, labels = data
                inputs = inputs.float()

                # wrap them in Variable
                if use_gpu:
                    inputs = Variable(inputs.cuda())
                    labels = Variable(labels.cuda())
                else:
                    inputs = Variable(inputs)
                    labels = Variable(labels)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                outputs = model(inputs)
                #outputs = nn.functional.sigmoid(outputs).round()
                _, preds = torch.max(outputs, 1)
                label = labels.diag().float()
                preds = preds.float()
                loss = criterion(preds, label)
                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

                # statistics
                running_loss += loss.data[0] * inputs.size(0)
                running_corrects += torch.sum(pred == label.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

and my loss function is defined below:

class CustLoss(nn.Module):
    def __init__(self):
        super(CustLoss, self).__init__()
    def forward(self, outputs, labels):
        return cust_loss(outputs, labels)

def cust_loss(pred, targets):
    '''preds are arrays of size classes with floats in them'''
    '''targets are arrays of all the classes from the batch'''
    '''we sum the classes from the batch and find the num correct'''
    r = torch.sum(pred == targets)
    return r

Then I run the following to run the model:

model_ft = models.resnet18(pretrained=True)
for param in model_ft.parameters():
    param.requires_grad = False

num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 3)

if use_gpu:
    model_ft = model_ft.cuda()

criterion = CustLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

I tried getting it to work with other loss functions to no avail. I always get the same error when loss.backward() is called.

It was my understanding that I wouldn't need a custom implementation of loss.backward if I extend nn.Module.

Output:
You are subclassing nn.Module to define a function, in your case Loss function. So, when you compute loss.backward(), it tries to store the gradients in the loss itself, instead of the model and there is no variable in the loss for which to store the gradients. Your loss needs to be a function and not a module. See Extending autograd.

You have two options here -

The easiest one is to directly pass cust_loss function as criterion parameter to train_model.
You can extend torch.autograd.Function to define the custom loss (and if you wish, the backward function as well).

P.S. - It is mentioned that you need to implement the backward of the custom loss functions. This is not always the case. It is required only when your loss function is non-differentiable at some point. But, I do not think so that you’ll need to do that.


Instruction: Variable size input for LSTM in Pytorch

Input:
I am using features of variable length videos to train one layer LSTM. Video sizes are changing from 10 to 35 frames. I am using batch size of 1.
I have the following code:

lstm_model = LSTMModel(4096, 4096, 1, 64)
for step, (video_features, label) in enumerate(data_loader):
    bx = Variable(score.view(-1, len(video_features), len(video_features[0]))) #examples = 1x12x4096, 1x5x4096
    output = lstm_model(bx)

Lstm model is;

class LSTMModel(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(LSTMModel, self).__init__()
    self.l1 = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
    self.out = nn.Linear(hidden_size, num_classes)
def forward(self, x):
    r_out, (h_n, h_c) = self.l1(x, None) #None represents zero initial hidden state
    out = self.out(r_out[:, -1, :])
    return out

I just want to ask; am I doing the right for training LSTM with variable size input. The code works okay and loss decreases but I am not sure if I am doing the right thing. Because I haven't used LSTMs in Pytorch before.

Output:
Yes, you code is correct and will work always for a batch size of 1. But, if you want to use a batch size other than 1, you’ll need to pack your variable size input into a sequence, and then unpack after LSTM. You can find more details in my answer to a similar question.

P.S. - You should post such questions to codereview


Instruction: What is volatile variable in Pytorch

Input:
What is volatile attribute of a Variable in Pytorch? Here's a sample code for defining a variable in PyTorch.

datatensor = Variable(data, volatile=True)

Output:
Basically, set the input to a network to volatile if you are doing inference only and won't be running backpropagation in order to conserve memory.

From the docs:

  Volatile is recommended for purely inference mode, when you’re sure
  you won’t be even calling .backward(). It’s more efficient than any
  other autograd setting - it will use the absolute minimal amount of
  memory to evaluate the model. volatile also determines that
  requires_grad is False.

Edit: The volatile keyword has been deprecated as of pytorch version 0.4.0


Instruction: How to convert from a Pytroch Tensor from mathematically a column vector to a column matrix?

Input:
I am working with tensors in pytorch.
How can I convert a tensor corresponding to a column vector to a tensor corresponding to its transpose?

import numpy as np
coef = torch.from_numpy(np.arange(1.0, 5.0)).float()
print(coef)
print(coef.size())

Currently the size of coef is [4] but I want it to be [4, 1] with the same content.

Output:
It is easy to achieve in PyTorch. You can use the view() method.

coef = coef.view(4, 1)
print(coef.size()) # now the shape will be [4, 1]


Instruction: Tensorflow : What is actually tf.nn.dropout output_keep_prob?

Input:
I am trying to understand concept of output_keep_prob:

So if my example is simple RNN :

    with tf.variable_scope('encoder') as scope:
        cells = rnn.LSTMCell(num_units=500)
        cell = rnn.DropoutWrapper(cell=cells, output_keep_prob=0.5)

        model = tf.nn.bidirectional_dynamic_rnn(cell, cell, inputs=embedding_lookup, sequence_length=sequence_le,
                                                dtype=tf.float32)

My confusion is if I am giving output_keep_prob=0.5 what actually it means? I know that it makes less prone to overfitting (called regularizing) by adding a dropout. It  randomly turns off activations of neurons during training, ok I got this point but I am confused when I give

output_keep_prob=0.5 and my no_of_nodes = 500 then 0.5 means it will randomly turn of 50% nodes at each iteration or it means it will keep only those connections which have more or equal probability of 0.5

keep_layers whose probability =>0.5

or

turn off 50% randomly nodes unit at each iteration ??

I tried to understand the concept by this stackoverflow answer but there is also same confusion that what actually 0.5 means? it should drop 50% nodes at each iteration or keep only those nodes which have probability more or equal to 0.5

if the answer is second keep only those nodes which have probability more or equal to 0.5 :

then it means suppose I have given 500 nodes units and only 30 nodes have 0.5 probability so it will turn of rest 470 nodes and will use only 30 nodes for incoming and outgoing connections?

Because  this answer says :

  Suppose you have 10 units in the layer and set the keep_prob to 0.1,
  Then the activation of 9 randomly chosen units out of 10 will be set
  to 0, and the remaining one will be scaled by a factor of 10. I think
  a more precise description is that you only keep the activation of 10
  percent of the nodes.

While other side this answer by @mrry says :

  it means that each connection between layers (in this case between the
  last densely connected layer and the readout layer) will only be used
  with probability 0.5 when training.

can anyone give a clear explanation which one is correct and what actually this value represent in keep_prob?

Output:
Keep_prop means the probability of any given neuron's output to be preserved (as opposed to dropped, that is zeroed out.) In other words, keep_prob = 1 - drop_prob.

The tf.nn.dropout() description states that

  By default, each element is kept or dropped independently.

So if you think about it, if you have a large amount of neurons, like 10,000 in a layer, and the keep_prob is let's say, 0.3, then 3,000 is the expected value of the number of neurons kept. So it's more or less the same thing to say that a keep_prob of 0.3 means to keep the value of 3,000 randomly chosen ones of the 10,000 neurons. But not exactly, because the actual number might vary a bit from 3,000.

Scaling comes into the picture because if you drop a certain number of neurons, then the expected sum of the layer will be reduced. So the remaining ones are multiplied to feed forward the same magnitude of values as they would otherwise. This is especially important if you load a pretrained network and want to continue training but with a different keep_prob value now.

(Please note, you can decide to introduce non-independence into the drop probabilities with the noise_shape argument, please see the tf.nn.drouput() description, but that is outside of the scope of this question.)

The random decision to drop a neuron or not is recalculated for each invocation of the network, so you will have a different set of neurons dropped on every iteration. The idea behind dropout is that subsequent layers cannot overfit and learn to watch for arbitrary constellations of certain activations. You ruin the "secret plan of lazy neurons to overfit" by always changing which previous activations are available.


Instruction: Running through a dataloader in Pytorch using Google Colab

Input:
I am trying to use Pytorch to run classification on a dataset of images of cats and dogs. In my code I am so far downloading the data and going into the folder train which has two folders in it called "cats" and "dogs." I am then trying to load this data into a dataloader and iterate through batches, but it is giving me some error I don't understand in the iteration step.

Since it is Google Colabs I have code in there for downloading data and installing libraries. Any other advice on my code so far would be appreciated as well.

!pip install torch
!pip install torchvision

from __future__ import print_function, division
import os
import torch
import pandas as pd
import numpy as np
# For showing and formatting images
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# For importing datasets into pytorch
import torchvision.datasets as dataset

# Used for dataloaders
import torch.utils.data as data

# For pretrained resnet34 model
import torchvision.models as models

# For optimisation function
import torch.nn as nn
import torch.optim as optim

!wget http://files.fast.ai/data/dogscats.zip
!unzip dogscats.zip

batch_size = 256

train_raw = dataset.ImageFolder(PATH+"train", transform=transforms.ToTensor())
train_loader = data.DataLoader(train_raw, batch_size=batch_size, shuffle=True)

for batch_idx, (data, target) in enumerate(train_loader):
  print("Data: ", batch_idx)

The error comes up on the last lines and is below:

RuntimeErrorTraceback (most recent call last)
<ipython-input-66-c32dd0c1b880> in <module>()
----> 1 for batch_idx, (data, target) in enumerate(train_loader):
      2   print("Data: ", batch_idx)
      3

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in __next__(self)
    257         if self.num_workers == 0:  # same-process loading
    258             indices = next(self.sample_iter)  # may raise StopIteration
--> 259             batch = self.collate_fn([self.dataset[i] for i in indices])
    260             if self.pin_memory:
    261                 batch = pin_memory_batch(batch)

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in default_collate(batch)
    133     elif isinstance(batch[0], collections.Sequence):
    134         transposed = zip(*batch)
--> 135         return [default_collate(samples) for samples in transposed]
    136
    137     raise TypeError((error_msg.format(type(batch[0]))))

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in default_collate(batch)
    110             storage = batch[0].storage()._new_shared(numel)
    111             out = batch[0].new(storage)
--> 112         return torch.stack(batch, 0, out=out)
    113     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
    114             and elem_type.__name__ != 'string_':

/usr/local/lib/python2.7/dist-packages/torch/functional.pyc in stack(sequence, dim, out)
     62     inputs = [t.unsqueeze(dim) for t in sequence]
     63     if out is None:
---> 64         return torch.cat(inputs, dim)
     65     else:
     66         return torch.cat(inputs, dim, out=out)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 400 and 487 in dimension 2 at /pytorch/torch/lib/TH/generic/THTensorMath.c:2897

Thanks

Output:
I think the main problem was images being of different size . I may have understood ImageFolder in other way but, i think you don't need labels for images if the directory structure is as specified in pytorch and  pytorch will figure out the labels for you.
I would also add more things to your transform that automatically resizes every images from the folder such as:

   normalize = transforms.Normalize(
                        mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225]
                        )
   transform = transforms.Compose(
        [transforms.ToTensor(),transforms.Resize((224,224)),
         normalize])

Also you can use other tricks to make your DataLoader much faster such as adding batch_size and number of cpu workers such as:

    testloader = DataLoader(testset, batch_size=16,
                         shuffle=False, num_workers=4)

I think this will make you pipeline much faster.


Instruction: Pytorch function name demystification: gels for least squares estimation

Input:
What does "gels" stand for in Pytorch?
It solves least squares, but what does the name stand for?

It is hard to get comfortable with a function without getting its name and it is surprising that these are not explained in the documentation.

Output:
gels is actually a function from LAPACK (Linear Algebra Package) and stands for GEneralalized Least Squares meaning that it works on general matrices:

  General matrix

  A general real or complex m by n matrix is represented by a real or complex matrix of size (m, n).


Instruction: KL Divergence for two probability distributions in PyTorch

Input: I have two probability distributions. How should I find the KL-divergence between them in PyTorch? The regular cross entropy only accepts integer labels.

Output:
Yes, PyTorch has a method named kl_div under torch.nn.functional to directly compute KL-devergence between tensors. Suppose you have tensor a and b of same shape. You can use the following code:

import torch.nn.functional as F
out = F.kl_div(a, b)

For more details, see the above method documentation.


Instruction: PyTorch replace torch.nn.Conv2d with torch.nn.functional.conv2d

Input:
So I have this MNIST example for PyTorch.
I wanted to replace conv2d with functional method. But got unexpected error.

I replace self.conv1 = nn.Conv2d(1, 32, 5, padding=2) with self.w_conv1 = Variable(torch.randn(1, 32, 5))

In the forward method I replace x = F.max_pool2d(F.relu(self.conv1(x)), 2) with x = F.max_pool2d(F.relu(F.conv2d(x, self.w_conv1, padding=2),2))

And then it will give me an error:

Expected 4-dimensional input for 4-dimensional weight [1, 32, 5], but got input of size [50, 1, 28, 28] instead

The code worked before, and I thought I'd replace the class with it's functional equivalent.

Output:
albanD answerd the question in https://discuss.pytorch.org/t/pytorch-replace-torch-nn-conv2d-with-torch-nn-functional-conv2d/16596

  Hi,

  The error message is not very clear I’m afraid because it comes from
  deep within the C backend. The problem here is that when you do a
  convolution on a 2D image with size (batch, in_chan, width, height),
  and you want an output of size (batch, out_chan, width’, height’),
  your weights for the convolution should be (out_chan, in_chan,
  width_kern_size, height_kern_size), basically when you use a kernel
  size of 5 for the Conv2d function, it is the same as having a kernel
  of width 5 and height 5. Thus you should have self.w_conv1 =
  Variable(torch.randn(32, 1, 5, 5)). See the doc for more details.


Instruction: Is there analog of theano.tensor.switch in pytorch?

Input:
I'd like to force to zero all elements of a vector which are below a certain threshold. And I'd like to do it so that I can still propagate gradient through non-zero ones.

For example, in theano I could write:

B = theano.tensor.switch(A < .1, 0, A)

Is there a solution for that in pytorch?

Output:
As of pytorch 0.4+, you can do it easily with torch.where(see doc,Merged PR)

It is as easy as in Theano. See yourself with an example:

import torch
from torch.autograd import Variable

x = Variable(torch.arange(0,4), requires_grad=True) # x     = [0 1 2 3]
zeros = Variable(torch.zeros(*x.shape))             # zeros = [0 0 0 0]

y = x**2                         # y = [0 1 4 9]
z = torch.where(y < 5, zeros, y) # z = [0 0 0 9]

# dz/dx = (dz/dy)(dy/dx) = (y < 5)(0) + (y ≥ 5)(2x) = 2x(x**2 ≥ 5)
z.backward(torch.Tensor([1.0]))
x.grad # (dz/dx) = [0 0 0 6]


Instruction: How to calculate the distance between a mini batch and a set of filters in PyTorch

Input:
I have a mini-batch of size NxDxWxH, where N is the size of the mini-batch, D is the dimension, and W and H are the width and height respectively. Assume that I have a set of filters F, each with dimension Dx1x1. I need to calculate the pairwise distance between the mini-batch and the filters. The size of the output should be NxFxWxH.

      input:   NxDxWxH
      filters: FxDx1x1
      output:  NxFxWxH
      Lets assume a is a vector of size D extracted at the location (x,y)
      of the input and f is filter of size Dx1x1. Each value in the output
      should be \sum_{d=1}^D (x_d - f_c)^2

In other words, instead of convolution I am trying to find the pair-wise L2 distances.

How can I do this in pytorch?

Output:
You can do this by expanding the input and filters for proper automatic shape casting.

# Assuming that input.size() is (N, D, W, H) and filters.size() is (F, D, 1, 1)
input.unsqueeze_(1)
filters.unsqueeze_(0)
output = torch.sum((input - filters)**2, dim=2)


Instruction: pytorch multiply 4*1 matrix and 1 size variable occur error

Input:
import torch
from torch.autograd import Variable
import numpy as np

x = np.transpose(np.array([[1, 2, 3, 4]]))
a = Variable(torch.rand(1), requires_grad=True)

print(a * x) # error!

I want result like x = [[2][4][6][8]] if a = 2

is there any solution?

Output:
What you are looking for is the dot scalar product in matrix multiplication.

try:

x = np.transpose(np.array([[1, 2, 3, 4]]))
a = 2
x.dot(a)

This outputs a matrix [[2][4][6][8]]


Instruction: AttributeError: 'collections.OrderedDict' object has no attribute 'eval'

Input:
I have a model file which looks like this
OrderedDict([('inp.conv1.conv.weight',
          (0 ,0 ,0 ,.,.) =
           -1.5073e-01  6.4760e-02  1.9156e-01
            1.2175e-01  3.5886e-02  1.3992e-01
           -1.5903e-01  8.2055e-02  1.7820e-01

          (0 ,0 ,1 ,.,.) =
            1.0604e-01 -1.3653e-01  1.4803e-01
            6.0276e-02 -1.4674e-02  2.3059e-06
           -6.2192e-02 -5.1061e-03 -7.4145e-03

          (0 ,0 ,2 ,.,.) =
           -5.5632e-02  3.5326e-02  6.5108e-02
            1.1411e-01 -4.4160e-02  8.2610e-02
            8.9979e-02 -3.5454e-02  4.2549e-02

          (1 ,0 ,0 ,.,.) =
            4.8523e-02 -4.3961e-02  5.3614e-02
           -1.2644e-01  1.2777e-01  8.9547e-02
            3.8392e-02  2.7016e-02 -1.4552e-01

          (1 ,0 ,1 ,.,.) =
            9.5537e-02  2.8748e-02  3.9772e-02
           -6.2410e-02  1.1264e-01  7.8663e-02
           -2.6374e-02  1.4401e-01 -1.7109e-01

          (1 ,0 ,2 ,.,.) =
            5.1791e-02 -1.6388e-01 -1.7605e-01
            3.5028e-02  7.7164e-02 -1.4499e-01
           -2.9189e-02  2.7064e-03 -2.3228e-02

          (2 ,0 ,0 ,.,.) =
           -7.4446e-03 -9.7202e-02 -1.4704e-01
           -1.0019e-02  8.1780e-02 -5.3530e-02
           -1.8412e-01  1.5988e-01 -1.3450e-01

          (2 ,0 ,1 ,.,.) =
           -1.1075e-01 -5.2478e-02  6.0658e-02
            1.6739e-01 -2.9360e-02  1.2621e-01
            2.0686e-02  1.1468e-01  1.2282e-01

I want to do inference on this model, but when i do model.eval() i get,
AttributeError: 'collections.OrderedDict' object has no attribute 'eval

Output:
It is not a model file, instead, this is a state file. In a model file, the complete model is stored, whereas in a state file only the parameters are stored.
So, your OrderedDict are just values for your model. You will need to create the model and then need to load these values into your model. So, the process will be something in form of

import torch
import torch.nn as nn

class TempModel(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3, 5, (3, 3))
    def forward(self, inp):
        return self.conv1(inp)

model = TempModel()
model.load_state_dict(torch.load(file_path))
model.eval()

You'll need to define your model properly. The one given in the example above is just a dummy. If you construct your model yourself, you might need to update the keys of the saved dict file as mentioned here. The best course of action is to define your model in exactly the same way from when the state_dict was saved and then directly executing model.load_state_dict will work.


Instruction: Installing PyTorch via Conda

Input:
Objective: Create a conda environment with pytorch and torchvision. Anaconda Navigator 1.8.3, python 3.6, MacOS 10.13.4.

What I've tried:

In Navigator, created a new environment. Tried to install pytorch and torchvision but could not because the UI search for packages does not find any packages available matching pytorch, torch, torchvision, or similar strings.
conda install pytorch torchvision -c pytorch
conda update --all

pytorch 0.3.1, torch 0.3.1, and torchvision 0.2.0 now appear as installed in the root environment. However, the root environment is no longer cloneable; the clone button is gray/disabled (it used be enabled/cloneable). I could use the root environment as a fallback but the main point of conda is to be able to create separate and disposable environments. What am I missing?

UPDATE -----------------

Running conda install -c pytorch pytorch yields:
# All requested packages already installed. But if I activate the pytorch environment and list the packages therein, there is no package containing the word "torch". If I then do conda search pytorch I get PackagesNotFoundError: The following packages are not available from current channels: - pytorch. If I activate the base environment and then do conda list then pytorch is in the package list for base. So how does one create a separate environment containing pytorch?

Output:
You seem to have installed PyTorch in your base environment, you therefore cannot use it from your other "pytorch" env.

Either:

directly create a new environment (let's call it pytorch_env) with PyTorch: conda create -n pytorch_env -c pytorch pytorch torchvision
switch to the pytorch environment you have already created with: source activate pytorch_env and then install PyTorch in it: conda install -c pytorch pytorch torchvision