Instruction: deep reinforcement learning parameters and training time for a simple game

Input:
I want to learn how deep reinforcement algorithm works and what time it takes to train itself for any given environment.
I came up with a very simple example of environment:

There is a counter which holds an integer between 0 to 100.
counting to 100 is its goal.

there is one parameter direction whose value can be +1 or -1.
it simply show the direction to move.

out neural network takes this direction as input and 2 possible action as output.

Change the direction
Do not change the direction

1st action will simply flip the direction (+1 => -1 or -1 =>+1). 2nd action will keep the direction as it is.

I am using python for backend and javascript for frontend.
It seems to take too much time, and still it is pretty random. i have used 4 layer perceptron. training rate of 0.001 . memory learning with batch of 100. Code is of Udemy tutorial of Artificial Intelligence and is working properly.

My question is, What should be the reward for completion and for each state.? and  how much time it is required to train simple example as that.?

Output:
In Reinforcement Learning the underlining reward function is what defines the game. Different reward functions lead to different games with different optimal strategies.

In your case there are a few different possibilities:

Give +1 for reaching 100 and only then.
Give +1 for reaching 100 and -0.001 for every time step it is not at 100.
Give +1 for going up -1 for going down.

The third case is way too easy there is no long term planing involved. In the first too cases the agent will only start learning once it accidentally reaches 100 and sees that it is good. But in the first case once it learns to go up it doesn't matter how long it takes to get there. The second is the most interesting where it needs to get there as fast as possible.

There is no right answer for what reward to use, but ultimately the reward you choose defines the game you are playing.

Note: 4 layer perceptron for this problem is Big Time Overkill. One layer should be enough (this problem is very simple). Have you tried the reinforcement learning environments at OpenAI's gym? Highly recommend it, they have all the "classical" reinforcement learning problems.


Instruction: pip to install Pytorch for usr/bin/python

Input:
I have set up Malmo on my mac and got into a trouble. I am using Malmo-0.17.0-Mac-64bit, python 2.7, macOS Sierra.

As it is explained on the Malmo installation instruction, mac users require to run the python scripts with the default mac python located on /usr/bin/python. I am aiming to implement a neural nets with Pytorch and doing some experiments. However, when I run the command /usr/bin/python -m pip install http://download.pytorch.org/whl/torch-0.2.0.post3-cp27-none-macosx_10_7_x86_64.whl to install Pytorch I am getting this error:

Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/c7/24z_lgxx4b7107ynf4cy34qh0000gn/T/pip-build-kLOmTh/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/c7/24z_lgxx4b7107ynf4cy34qh0000gn/T/pip-YxAJnq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/c7/24z_lgxx4b7107ynf4cy34qh0000gn/T/pip-build-kLOmTh/pyyaml/.

I was wondering if anyone has faced the same problem.

Many Thanks,

Output: I figured it out what the problem was. It is so stupid. I just needed to add sudo at the beginning of the command.


Instruction: Neural network in pytorch predict two binary variables

Input:
Suppose I want to have the general neural network architecture:

                     ---> BinaryOutput_A
                    /
Input --> nnLayer -
                    \
                     ---> BinaryOutput_B

An input is put through a neural network layer which then goes to predict two binary variables (i.e., A is [0 or 1] and B is [0 or 1].

In pytorch, you can make such a network with:

class NN(nn.Module):

    def __init__(self, inputs):
        super(NN, self).__init__()

        # -- first layer
        self.lin = nn.Linear(inputs,10)

        # -- firstLayer --> binaryOutputA
        self.l2a = nn.Linear(10,2)

        # -- firstLayer --> binaryOutputB
        self.l2b = nn.Linear(10,2)

    def forward(self, inputs):
        o = self.lin(inputs)
        o1 = F.log_softmax(self.l2a(o))
        o2 = F.log_softmax(self.l2b(o))
        return o1, o2

In my train function, I calculate the loss with loss = loss_function(output, target).  If that's the case, to properly backpropagate the loss to both the l2a and l2b layers using loss.backward(), could I simply concat the target with the proper label for l2a and l2b?  In that sense, the output would be [outputPredictionA, outputPredictionB] and I could make the target be [labelA, labelB], Would pytorch know to properly assign the loss to each layer?

Output:
It turns out that torch is actually really smart, and you can just calculate the total loss as:

loss = 0
loss += loss_function(pred_A, label_A)
loss += loss_function(pred_B, label_B)

loss.backward()

and the error will be properly backpropagated through the network.  No torch.cat() needed or anything.


Instruction: Cross entropy loss in pytorch nn.CrossEntropyLoss()

Input:
maybe someone is able to help me here. I am trying to compute the cross entropy loss of a given output of my network

print output
Variable containing:
1.00000e-02 *
-2.2739  2.9964 -7.8353  7.4667  4.6921  0.1391  0.6118  5.2227  6.2540
-7.3584
[torch.FloatTensor of size 1x10]

and the desired label, which is of the form

print lab
Variable containing:
x
[torch.FloatTensor of size 1]

where x is an integer between 0 and 9.
According to the pytorch documentation (http://pytorch.org/docs/master/nn.html)

criterion = nn.CrossEntropyLoss()
loss = criterion(output, lab)

this should work, but unfortunately I get a weird error

TypeError: FloatClassNLLCriterion_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, !torch.FloatTensor!, torch.FloatTensor, bool, NoneType, torch.FloatTensor, int), but expected (int state, torch.FloatTensor input, torch.LongTensor target, torch.FloatTensor output, bool sizeAverage, [torch.FloatTensor weights or None], torch.FloatTensor total_weight, int ignore_index)

Can anyone help me? I am really confused and tried almost everything I could imagined to be helpful.

Best

Output:
Please check this code

import torch
import torch.nn as nn
from torch.autograd import Variable

output = Variable(torch.rand(1,10))
target = Variable(torch.LongTensor([1]))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

This will print out the loss nicely:

Variable containing:
 2.4498
[torch.FloatTensor of size 1]


Instruction: Elegant way to compare to torch.FloatTensor on GPU

Input:
I try to compare two torch.FloatTensor (with only one entry) lying on GPU like this:

  if (FloatTensor_A > FloatTensor_B): do something

The problem is, that (FloatTensor_A > FloatTensor_B) gives ByteTensor back. Is there a way to do boolean comparison between these two scalar FloatTensors, without loading the tensors on CPU and converting them back to numpy or conventional floats?

Output:
The comparison operations in PyTorch return ByteTensors (see docs). In order to convert your result back to a float datatype, you can call .float() on your result. For example:

(t1 > t2).float()

(t1 > t2) will return a ByteTensor.

The inputs to the operation must be on the same memory (either CPU or GPU). The return result will be on the same memory. Of course, any Tensor can be moved to the respective memory by callin .cpu() or .cuda() on it.


Instruction: How do I use a ByteTensor in a contrastive cosine loss function?

Input:
I'm trying to implement the loss function in http://anthology.aclweb.org/W16-1617 in PyTorch. It is shown as follows:

I've implemented the loss as follows:

class CosineContrastiveLoss(nn.Module):
    """
    Cosine contrastive loss function.
    Based on: http://anthology.aclweb.org/W16-1617
    Maintain 0 for match, 1 for not match.
    If they match, loss is 1/4(1-cos_sim)^2.
    If they don't, it's cos_sim^2 if cos_sim < margin or 0 otherwise.
    Margin in the paper is ~0.4.
    """

    def __init__(self, margin=0.4):
        super(CosineContrastiveLoss, self).__init__()
        self.margin = margin

    def forward(self, output1, output2, label):
        cos_sim = F.cosine_similarity(output1, output2)
        loss_cos_con = torch.mean((1-label) * torch.div(torch.pow((1.0-cos_sim), 2), 4) +
                                    (label) * torch.pow(cos_sim * torch.lt(cos_sim, self.margin), 2))
        return loss_cos_con

However, I'm getting an error saying:

TypeError: mul received an invalid combination of arguments - got (torch.cuda.ByteTensor), but expected one of:
 * (float value)
      didn't match because some of the arguments have invalid types: (torch.cuda.ByteTensor)
 * (torch.cuda.FloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.cuda.ByteTensor)

I know that torch.lt() returns a ByteTensor, but if I try to coerce it to a FloatTensor with torch.Tensor.float() I get AttributeError: module 'torch.autograd.variable' has no attribute 'FloatTensor'.

I'm really not sure where to go from here. It seems logical to me to do an element-wise multiplication between the cosine similarity tensor and a tensor with 0 or 1 based on a less-than rule.

Output:
Maybe you can try float() method on the variable directly?
Variable(torch.zeros(5)).float() - works for me, for instance


Instruction: Upsampling in Semantic Segmentation

Input:
I am trying to implement a paper on Semantic Segmentation and I am confused about how to Upsample the prediction map produced by my segmentation network to match the input image size.

For example, I am using a variant of Resnet101 as the segmentation network (as used by the paper). With this network structure, an input of size 321x321 (again used in the paper) produces a final prediction map of size 41x41xC (C is the number of classes). Because I have to make pixel-level predictions, I need to upsample it to 321x321xC. Pytorch provides function to Upsample to an output size which is a multiple of the prediction map size. So, I can not directly use that method here.

Because this step is involved in every semantic segmentation network, I am sure there should be a standard way to implement this.

I would appreciate any pointers. Thanks in advance.

Output:
Maybe the simpliest thing you can try is:

upsample 8 times. Then you 41x41 input turns into 328x328
perform center cropping to get your desired shape 321x321 (for instance, something like this input[3:,3:,:-4,:-4])


Instruction: how to calculate loss over a number of images and then back propagate the average loss and update network weight

Input:
I am doing a task where the batch size is 1, i.e, each batch contains only 1 image. So I have to do manual batching: when the the number of accumulated losses reach a number, average the loss and then do back propagation.
My original code is:

real_batchsize = 200

for epoch in range(1, 5):
    net.train()

    total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)

    iter_count = 0
    for batch_idx, (input, target) in enumerate(train_loader):

        input, target = Variable(input.cuda()), Variable(target.cuda())
        output = net(input)

        loss = F.nll_loss(output, target)

        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:
            iter_count += 1

            ave_loss = total_loss/real_batchsize
            ave_loss.backward()
            optimizer.step()

            if iter_count % 10 == 0:
                print("Epoch:{}, iteration:{}, loss:{}".format(epoch,
                                                           iter_count,
                                                           ave_loss.data[0]))
            total_loss.data.zero_()
            optimizer.zero_grad()

This code will give the error message

  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I have tried the following way,

First way (failed)

I read some post about this error message, but can not understand it fully. Change ave_loss.backward() to ave_loss.backward(retain_graph=True) prevent the error message, but the loss doesn't improve the soon becomes nan.

Second way (failed)

I also tried to change total_loss = total_loss + loss.data[0], this will also prevent the error message. But the loss are always the same. So there must be something wrong.

Third way (success)

Following the instruction in this post, for each image's loss,I divide the loss by real_batchsize and backprop it. When the number of input image reach the real_batchsize, I do one update of parameter using optimizer.step().  The loss is slowly decreasing as the training process goes. But the training speed is really slow, because we backprop for each image.

My question

What does the error message mean in my case? Also, why doesn't first way and second way work?  How to write the code correctly so that we can backprop gradient every real_batchsize images and update gradient once so that the training speed a faster?  I know my code is nearly correct, but I just do not know how to change it.

Output:
The problem you encounter here is related to how PyTorch accumulates gradients over different passes. (see here for another post on a similar question)
So let's have a look at what happens when you have code of the following form:

loss_total = Variable(torch.zeros(1).cuda(), requires_grad=True)
for l in (loss_func(x1,y1), loss_func(x2, y2), loss_func(x3, y3), loss_func(x4, y4)):
    loss_total = loss_total + l
    loss_total.backward()

Here, we do a backward pass when loss_total has the following values over the different iterations:

total_loss = loss(x1, y1)
total_loss = loss(x1, y1) + loss(x2, y2)
total_loss = loss(x1, y1) + loss(x2, y2) + loss(x3, y3)
total_loss = loss(x1, y1) + loss(x2, y2) + loss(x3, y3) + loss(x4, y4)

so when you call .backward() on total_loss each time, you actually call .backward() on loss(x1, y1) four times! (and on loss(x2, y2) three times, etc).

Combine that with what is discussed in the other post, namely that to optimize memory usage, PyTorch will free the graph attached to a Variable when calling .backward() (and thereby destroying the gradients connecting x1 to y1, x2 to y2, etc), you can see what the error message means - you try to do backward passes over a loss for several times, but the underlying graph was freed up after the first pass. (unless to specify retain_graph=True, of course)

As for the specific variations you have tried:
First way: here, you will accumulate (i.e. sum up - again, see the other post) gradients forever, with them (potentially) adding up to inf.
Second way: here, you convert loss to a tensor by doing loss.data, removing the Variable wrapper, and thereby deleting the gradient information (since only Variables hold gradients).
Third way: here, you only do one pass through each xk, yk tuple, since you immediately do a backprop step, avoiding the above problem alltogether.

SOLUTION: I have not tested it, but from what I gather, the solution should be pretty straightforward: create a new total_loss object at the beginning of each batch, then sum all of the losses into that object, and then do one final backprop step at the end.


Instruction: Calculating input and output size for Conv2d in PyTorch for image classification

Input:
I'm trying to run the PyTorch tutorial on CIFAR10 image classification here - http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

I've made a small change and I'm using a different dataset. I have images from the Wikiart dataset that I want to classify by artist (label = artist name).

Here is the code for the Net -

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Then there is this section of the code where I start training the Net.

for epoch in range(2):
     running_loss = 0.0

     for i, data in enumerate(wiki_train_dataloader, 0):
        inputs, labels = data['image'], data['class']
        print(inputs.shape)
        inputs, labels = Variable(inputs), Variable(labels)

        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.data[0]
        if i % 2000 == 1999:  # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

This line print(inputs.shape) gives me torch.Size([4, 32, 32, 3]) with my Wikiart dataset whereas in the original example with CIFAR10, it prints torch.Size([4, 3, 32, 32]).

Now, I'm not sure how to change the Conv2d in my Net to be compatible with torch.Size([4, 32, 32, 3]).

I get this error:

RuntimeError: Given input size: (3 x 32 x 3). Calculated output size: (6 x 28 x -1). Output size is too small at /opt/conda/conda-bld/pytorch_1503965122592/work/torch/lib/THNN/generic/SpatialConvolutionMM.c:45

While reading the images for the Wikiart dataset, I resize them to (32, 32) and these are 3-channel images.

Things I tried:

1) The CIFAR10 tutorial uses a transform which I am not using. I could not incorporate the same into my code.

2) Changing self.conv2 = nn.Conv2d(6, 16, 5) to self.conv2 = nn.Conv2d(3, 6, 5). This gave me the same error as above. I was only changing this to see if the error message changes.

Any resources on how to calculate input & output sizes in PyTorch or automatically reshape Tensors would be really appreciated. I just started learning Torch & I find the size calculations complicated.

Output:
You have to shape your input to this format (Batch, Number Channels, height, width).
Currently you have format (B,H,W,C)  (4, 32, 32, 3), so you need to swap 4th and 2nd axis to shape your data with (B,C,H,W).
You can do it this way:

inputs, labels = Variable(inputs), Variable(labels)
inputs = inputs.transpose(1,3)
... the rest


Instruction: Error reading images with pandas (+pyTorch,scikit)

Input:
I'm trying to read images to work with a CNN, but I'm getting a pandas error while trying to load the images.
This is some of the code (omitted imports and irrelevant nn class for clarity):

file_name = "annotation.csv"
image_files = pd.read_csv(file_name)

class SimpsonsDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        self.image_file = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.image_file)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir, self.image_file.iloc[idx,0][1:])
        image = io.imread(img_name)
        sample = {'image': image}
        if self.transform:
            sample = self.transform(sample)
        return sample

simpsons = SimpsonsDataset(csv_file=image_files,root_dir="folder/")

I use the iloc[idx,0][1:] to format the filepath, and the filepath is joined, with the folders and filenames matching.

However, when I try to run the file, I get the following error:

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/.../image_extractor.py", line 41, in <module>
    simpsons = SimpsonsDataset(csv_file=image_files,root_dir="folder/")

  File "C:/.../image_extractor.py", line 26, in __init__
    self.image_file = pd.read_csv(csv_file)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
    return _read(filepath_or_buffer, kwds)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 392, in _read
    filepath_or_buffer, encoding, compression)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\common.py", line 210, in get_filepath_or_buffer
    raise ValueError(msg.format(_type=type(filepath_or_buffer)))

ValueError: Invalid file path or buffer object type: <class 'pandas.core.frame.DataFrame'>

Would love some insights on why this is happening. Thanks!

Output:
Your variable image_files is a pandas DataFrame, since it holds the return value of pd.read_csv(), which returns a DataFrame. Try deleting the line

image_files = pd.read_csv(file_name)

and changing the last line to this:

simpsons = SimpsonsDataset(csv_file=file_name, root_dir="folder/")


Instruction: pytorch: Convert a tuple of FloatTensors into a numpy array

Input:
I just got a torchvision.datasets object with MNIST data

train_dataset= dsets.MNIST(root='./data',train=True,transform=transforms.ToTensor(),download=True)

I want to convert this tuple into a set of numpy arrays of shape 60000x28x28 and labels of 60000

I know that the form that the data is provided, can be directly applied into a pytorch neuron in order to be used as training data, but I would like to convert this data into numpy arrays.

the first thing I did was to divide the tuples of (data,labels) with zip(*train_dataset)

data,labels = zip(*train_dataset)

labels is easy to convert into a numpy array, however I have not been able to convert "data" into a numpy array the way I would like. When I try to convert all of the data into numpy.array like

data[:].numpy()

I receive an error telling me 'tuple' object has no attribute 'numpy'. and if I convert this data[:] into numpy, just the first dimension (the 60000 data) is converted into numpy.array but the rest remains as Tensors.

I can convert all of the data using a loop (actually I did it) and it works, however it is really slow. Has anyone done any conversion like this before?

Thanks.

Output:
Maybe something like this will work for you:

train_dataset.train_data.numpy()  #contains (60000, 28, 28) numpy array
train_dataset.train_labels.numpy() # contains labels


Instruction: Pytorch-tutorial: Strange input argument in class definition

Input:
I'm reading through some pytorch tutorials. Below is the definition of a residual block. However in the forward method each function handle only takes one argument out while in the __init__ function these functions have different number of input arguments:

# Residual Block
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

Does anyone know how this works?
Is it a standard python class inheritance feature or is this specific to pytorch?

Output: you define the layer in the init function, which means the parameters. In the forward function you only input the data that needs to be processed with the predefined settings from init. The nn.whatever builds a function with the settings you pass to it. Then this function can be used in forward and this function only takes one argument.


Instruction: Pytorch Tensor help in LongTensor

Input:
a=[1,2,3];
context_var = autograd.Variable(torch.LongTensor(a))

This is giving an error

RuntimeError: tried to construct a tensor from a int sequence, but found an item of type numpy.int32 at index

I am not able to figure out how to get over this.

Output:
Your code works perfectly fine in the recent version of pytorch. But for older versions, you can convert the numpy array to list using .tolist() method as follows to get rid of the error.

a=[1,2,3];
context_var = autograd.Variable(torch.LongTensor(a.tolist()))


Instruction: How to do fully connected batch norm in PyTorch?

Input: torch.nn has classes BatchNorm1d, BatchNorm2d, BatchNorm3d, but it doesn't have a fully connected BatchNorm class? What is the standard way of doing normal Batch Norm in PyTorch?

Output:
Ok. I figured it out. BatchNorm1d can also handle Rank-2 tensors, thus it is possible to use BatchNorm1d for the normal fully-connected case.

So for example:

import torch.nn as nn

class Policy(nn.Module):
def __init__(self, num_inputs, action_space, hidden_size1=256, hidden_size2=128):
    super(Policy, self).__init__()
    self.action_space = action_space
    num_outputs = action_space

    self.linear1 = nn.Linear(num_inputs, hidden_size1)
    self.linear2 = nn.Linear(hidden_size1, hidden_size2)
    self.linear3 = nn.Linear(hidden_size2, num_outputs)
    self.bn1 = nn.BatchNorm1d(hidden_size1)
    self.bn2 = nn.BatchNorm1d(hidden_size2)

def forward(self, inputs):
    x = inputs
    x = self.bn1(F.relu(self.linear1(x)))
    x = self.bn2(F.relu(self.linear2(x)))
    out = self.linear3(x)

    return out


Instruction: Embedding 3D data in Pytorch

Input:
I want to implement character-level embedding.

This is usual word embedding.

Word Embedding

Input: [ [‘who’, ‘is’, ‘this’] ]
-> [ [3, 8, 2] ]     # (batch_size, sentence_len)
-> // Embedding(Input)
 # (batch_size, seq_len, embedding_dim)

This is what i want to do.

Character Embedding

Input: [ [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ] ]
-> [ [ [2, 3, 9, 0], [ 11, 4, 0, 0], [21, 10, 8, 9] ] ]      # (batch_size, sentence_len, word_len)
-> // Embedding(Input) # (batch_size, sentence_len, word_len, embedding_dim)
-> // sum each character embeddings  # (batch_size, sentence_len, embedding_dim)
The final output shape is same as Word embedding. Because I want to concat them later.

Although I tried it, I am not sure how to implement 3-D embedding. Do you know how to implement such a data?

def forward(self, x):
    print('x', x.size()) # (N, seq_len, word_len)
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    embd_list = []
    for i, elm in enumerate(x):
        tmp = torch.zeros(1, word_len, self.embd_size)
        for chars in elm:
            tmp = torch.add(tmp, 1.0, self.embedding(chars.unsqueeze(0)))

Above code got an error because output of self.embedding is Variable.

TypeError: torch.add received an invalid combination of arguments - got (torch.FloatTensor, float, Variable), but expected one of:
 * (torch.FloatTensor source, float value)
 * (torch.FloatTensor source, torch.FloatTensor other)
 * (torch.FloatTensor source, torch.SparseFloatTensor other)
 * (torch.FloatTensor source, float value, torch.FloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)
 * (torch.FloatTensor source, float value, torch.SparseFloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)

Update

I could do this. But for is not effective for batch. Do you guys know more efficient way?

def forward(self, x):
    print('x', x.size()) # (N, seq_len, word_len)
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    embd = Variable(torch.zeros(bs, seq_len, self.embd_size))
    for i, elm in enumerate(x): # every sample
        for j, chars in enumerate(elm): # every sentence. [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ]
            chars_embd = self.embedding(chars.unsqueeze(0)) # (N, word_len, embd_size) [‘w’,‘h’,‘o’,0]
            chars_embd = torch.sum(chars_embd, 1) # (N, embd_size). sum each char's embedding
            embd[i,j] = chars_embd[0] # set char_embd as word-like embedding

    x = embd # (N, seq_len, embd_dim)

Update2

This is my final code. Thank you, Wasi Ahmad!

def forward(self, x):
    # x: (N, seq_len, word_len)
    input_shape = x.size()
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    x = x.view(-1, word_len) # (N*seq_len, word_len)
    x = self.embedding(x) # (N*seq_len, word_len, embd_size)
    x = x.view(*input_shape, -1) # (N, seq_len, word_len, embd_size)
    x = x.sum(2) # (N, seq_len, embd_size)

    return x

Output:
I am assuming you have a 3d tensor of shape BxSxW where:

B = Batch size
S = Sentence length
W = Word length

And you have declared embedding layer as follows.

self.embedding = nn.Embedding(dict_size, emsize)

Where:

dict_size = No. of unique characters in the training corpus
emsize = Expected size of embeddings

So, now you need to convert the 3d tensor of shape BxSxW to a 2d tensor of shape BSxW and give it to the embedding layer.

emb = self.embedding(input_rep.view(-1, input_rep.size(2)))

The shape of emb will be BSxWxE where E is the embedding size. You can convert the resulting 3d tensor to a 4d tensor as follows.

emb = emb.view(*input_rep.size(), -1)

The final shape of emb will be BxSxWxE which is what you are expecting.


Instruction: How to get the value of a feature in a layer that match a the state dict in PyTorch?

Input:
I have some cnn, and I want to fetch the value of some intermediate layer corresponding to a some key from the state dict.
How could this be done?
Thanks.

Output:
I think you need to create a new class that redefines the forward pass through a given model. However, most probably you will need to create the code regarding the architecture of your model. You can find here an example:

class extract_layers():

    def __init__(self, model, target_layer):
        self.model = model
        self.target_layer = target_layer

    def __call__(self, x):
        return self.forward(x)

    def forward(self, x):
        module = self.model._modules[self.target_layer]

        # get output of the desired layer
        features = module(x)

        # get output of the whole model
        x = self.model(x)

        return x, features

model = models.vgg19(pretrained=True)
target_layer = 'features'
extractor = extract_layers(model, target_layer)

image = Variable(torch.randn(1, 3, 244, 244))
x, features = extractor(image)

In this case, I am using the pre-defined vgg19 network given in the pytorch models zoo. The network has the layers structured in two modules the features for the convolutional part and the classifier for the fully-connected part. In this case, since features wraps all the convolutional layers of the network it is straightforward. If your architecture has several layers with different names, you will need to store their output using something similar to this:

 for name, module in self.model._modules.items():
    x = module(x)  # forward the module individually
    if name in self.target_layer:
        features = x  # store the output of the desired layer

Also, you should keep in mind that you need to reshape the output of the layer that connects the convolutional part to the fully-connected one. It should be easy to do if you know the name of that layer.


Instruction: Pytorch FloatClassNLLCriterion_updateOutput error

Input:
I am getting the following error message while computing the loss of my neural net: TypeError: FloatClassNLLCriterion_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, !torch.FloatTensor!, torch.FloatTensor, bool, NoneType, torch.FloatTensor, int), but expected (int state, torch.FloatTensor input, torch.LongTensor target, torch.FloatTensor output, bool sizeAverage, [torch.FloatTensor weights or None], torch.FloatTensor total_weight, int ignore_index) on this line: loss = criterion(outputs,one_hot_target).

I have tried several things and searched the web, but I can't seem to find my mistake. Does someone have an idea?

The code used:

class Net(nn.Module):
def _init_(self,input_size,hidden_size, num_classes):
    super(Net, self)._init_()
    self.l1 = nn.Linear(input_size,hidden_size)
    self.l2 = nn.Linear(hidden_size,num_classes)

def forward(self,x):
    x = self.l1(x)
    x = F.tanh(x)
    x = self.l2(x)
    x = F.softmax(x)
    return x

mlp = Net(input_size,hidden_size,num_classes)

criterion = nn.NLLLoss()
optimizer = torch.optim.SGD(mlp.parameters(), lr=learning_rate)

for i in range(N-1,len(word_list)):

  # Define input vector x
  x = None

  for j in range(0,N-1):
      try:
          x = np.r_[x,w2v[word_list[j]]]
      except:
          x = w2v[word_list[j-N+1]]

  # Done with defining  x

  np.reshape(x,(len(x),1))
  x = autograd.Variable(torch.FloatTensor(x))
  optimizer.zero_grad()

  outputs = mlp(x)
  outputs = outputs.unsqueeze(0)
  outputs = outputs.transpose(0,1)

  index = w2i[word_list[i]]
  one_hot_target = np.zeros([num_classes,1],dtype=float)

  one_hot_target[index] = float(1)

  one_hot_target = autograd.Variable(torch.Tensor(one_hot_target))
  print (one_hot_target)
  loss = criterion(outputs,one_hot_target)
  #loss.backward()
  #optimizer.step()

Output:
The target tensor you are passing to your loss function is of type Float Tensor. That's happening here:
one_hot_target = autograd.Variable(torch.Tensor(one_hot_target))

That is, because the default Tensor type in PyTorch is FloatTensor.
However, the NLLLoss() function is expecting a LongTensor as target. You can double check in the example in the documentation, which you can find here: NLLLoss Pytorch docs.

You can simply convert your target Tensor to a LongTensor like this:

one_hot_target = autograd.Variable(torch.Tensor(one_hot_target)).long()


Instruction: How to generate a new Tensor with different vectors in PyTorch?

Input:
I want to generate new a○b vector with a and b (○ means element wise multiply). My code is below, but the performance looks bad because of for. Are there any efficient way?

a = torch.rand(batch_size, a_len, hid_dim)
b = torch.rand(batch_size, b_len, hid_dim)
# a_elmwise_mul_b = torch.zeros(batch_size, a_len, b_len, hid_dim)
for sample in range(batch_size):
    for ai in range(a_len):
        for bi in range(b_len):
            a_elmwise_mul_b[sample, ai, bi] = torch.mul(a[sample, ai], b[sample, bi])

Update

I updated my code refer to Ahmad! Thank you.

N = 16
hid_dim = 50
a_seq_len = 10
b_seq_len = 20
a = torch.randn(N, a_seq_len, hid_dim)
b = torch.randn(N, b_seq_len, hid_dim)
shape = (N, a_seq_len, b_seq_len, hid_dim)

a_dash = a.unsqueeze(2) # (N, a_len, 1,     hid_dim)
b_dash = b.unsqueeze(1) # (N, 1,     b_len, hid_dim)
a_dash = a_dash.expand(shape)
b_dash = b_dash.expand(shape)
print(a_dash.size(), b_dash.size())
mul = a_dash * b_dash
print(mul.size())
----------
torch.Size([16, 10, 20, 50]) torch.Size([16, 10, 20, 50])
torch.Size([16, 10, 20, 50])

Output:
From your problem definition, it looks like you want to multiply two tensors, say A and B of shape AxE and BxE and want to get a tensor of shape AxBxE. It means you want to multiply, each row of tensor A with the whole tensor B. If it is correct, then we don't call it element-wise multiplication.

You can accomplish your goal as follows.

import torch

# batch_size = 16, a_len = 10, b_len = 20, hid_dim = 50
a = torch.rand(16, 10, 50)
b = torch.rand(16, 20, 50)

c = a.unsqueeze(2).expand(*a.size()[:-1], b.size(1), a.size()[-1])
d = b.unsqueeze(1).expand(b.size()[0], a.size(1), *b.size()[1:])
print(c.size(), d.size())
print(c.size(), d.size())

mul = c * d       # shape of c, d: 16 x 10 x 20 x 50
print(mul.size()) # 16 x 10 x 20 x 50

Here, mul tensor is your desired result. Just to clarify, the above two lines realted to c and d computation, are equivalent to:

c = a.unsqueeze(2).expand(a.size(0), a.size(1), b.size(1), a.size(2))
d = b.unsqueeze(1).expand(b.size(0), a.size(1), b.size(1), b.size(2))


Instruction: Assign torch.cuda.FloatTensor

Input:
I'd like to know how can I do the following code, but now using pytorch,
where dtype = torch.cuda.FloatTensor. There's the code straight python (using numpy):

import numpy as np
import random as rand
xmax, xmin = 5, -5
pop = 30
x = (xmax-xmin)*rand.random(pop,1)
y = x**2
[minz, indexmin] = np.amin(y), np.argmin(y)
best = x[indexmin]

This is my attempt to do it:

import torch
dtype = torch.cuda.FloatTensor
def fit (position):
    return  position**2
def main():
    pop = 30
    xmax, xmin = 5, -5
    x= (xmax-xmin)*torch.rand(pop, 1).type(dtype)+xmin
    y = fit(x)
    [miny, indexmin] = torch.min(y,0)
    best = x[indexmin]
    print(best)

The last part where I define the variable best as the value of x with index equal to indexmin it doesn't work. What am I doing wrong here.

The following messenge appears:

RuntimeError: expecting vector of indices at /opt/conda/conda-bld/pytorch_1501971235237/work/pytorch-0.1.12/torch/lib/THC/generic/THCTensorIndex.cu:405

Output:
The above code works fine in pytorch 0.2. Let me analyze your code so that you can identify the problem.

x= (xmax-xmin)*torch.rand(pop, 1).type(dtype)+xmin
y = fit(x)

Here, x and y is a 2d tensor of shape 30x1. In the next line:

[miny, indexmin] = torch.min(y,0)

The returned tensor miny is a 2d tensor of shape 30x1 and indexmin is a 1d tensor of size 1. So, when you execute:

best = x[indexmin]

It (probably) gives error (in old pytorch version) because x is a 2d tensor of shape 30x1 and indexmin is a 1d tensor of size 1. To resolve this error, you can simply do:

best = x.squeeze()[indexmin] # x.squeeze() returns a 1d tensor of size `30`

Please note, a 2d tensor of shape 30x1 is same as a 1d tensor of size 30. So, you can modify your program as follows.

import torch
dtype = torch.cuda.FloatTensor
def main():
    pop, xmax, xmin = 30, 5, -5
    x= (xmax-xmin)*torch.rand(pop).type(dtype)+xmin
    y = torch.pow(x, 2)
    minz, indexmin = y.min(0)
    best = x[indexmin]
    print(best)

main()


Instruction: Assign variable in pytorch

Input:
I'd like to know if it is possble to the following code, but now using pytorch, where dtype = torch.cuda.FloatTensor. There's the code straight python (using numpy): Basically I want to get the value of x that produces the min value of fitness.

import numpy as np
import random as rand
xmax, xmin       = 5, -5
pop              = 30
x                = (xmax-xmin)*rand.random(pop,1)
y                = x**2
[minz, indexmin] = np.amin(y), np.argmin(y)
best             = x[indexmin]

This is my attempt to do it:

import torch
dtype = torch.cuda.FloatTensor
def fit (x):
    return  x**2
def main():
    pop              = 30
    xmax, xmin       = 5, -5
    x                = (xmax-xmin)*torch.rand(pop, 1).type(dtype)+xmin
    y                = fit(x)
    [miny, indexmin] = torch.min(y,0)
    best             = x[indexmin]
main()

The last part where I define the variable best as the value of x with index equal to indexmin it doesn't work. What am I doing wrong here.

The following messenge appears: RuntimeError:

expecting vector of indices at /opt/conda/conda-bld/pytorch_1501971235237/work/pytorch-0.1.12/torch/lib/THC/generic/THCTensorIndex.cu:405

Output:
You can simply do as follows.

import torch
dtype = torch.cuda.FloatTensor
def main():
    pop, xmax, xmin  = 30, 5, -5
    x                = (xmax-xmin)*torch.rand(pop, 1).type(dtype)+xmin
    y                = torch.pow(x, 2)
    [miny, indexmin] = y.min(0)
    best             = x.squeeze()[indexmin] # squeeze x to make it 1d

main()


Instruction: Add a constant variable to a cuda.FloatTensor

Input:
I have two question:

  1) I'd like to know how can I add/subtract a constante torch.FloatTensor of size 1 to all of the elemets of a torch.FloatTensor of size 30.

  2) How can I multiply each element of a torch.FloatTensor of size 30 by a random value (different or not for each).

My code:

import torch
dtype = torch.cuda.FloatTensor
def main():
     pop, xmax, xmin   = 30, 5, -5
     x                 = (xmax-xmin)*torch.rand(pop).type(dtype)+xmin
     y                 = torch.pow(x, 2)
     [miny, indexmin]  = y.min(0)
     gxbest            = x[indexmin]
     pxbest            = x
     pybest            = y
     v = torch.rand(pop)
     vnext = torch.rand()*v + torch.rand()*(pxbest - x) + torch.rand()*(gxbest - x)

main()

What is the best way to do it? I think I should so how convert the gxbest into a torch.FloatTensor of size 30 but how can I do that?
I've try to create a vector:

  Variable(torch.from_numpy(np.ones(pop)))*gxbest

But it did not work. The multiplication is not working also.

RuntimeError: inconsistent tensor size

Thank you all for your help!

Output:
  1) How can I add/subtract a constant torch.FloatTensor of size 1 to all of the elements of a torch.FloatTensor of size 30?

You can do it directly in pytorch 0.2.

import torch

a = torch.randn(30)
b = torch.randn(1)
print(a-b)

In case if you get any error due to size mismatch, you can make a small change as follows.

print(a-b.expand(a.size(0))) # to make both a and b tensor of same shape

  2) How can I multiply each element of a torch.FloatTensor of size 30 by a random value (different or not for each)?

In pytorch 0.2, you can do it directly as well.

import torch

a = torch.randn(30)
b = torch.randn(1)
print(a*b)

In case, if you get an error due to size mismatch, do as follows.

print(a*b.expand(a.size(0)))

So, in your case you can simply change the size of gxbest tensor from 1 to 30 as follows.

gxbest = gxbest.expand(30)


Instruction: How to `dot` weights to batch data in PyTorch?

Input:
I have batch data and want to dot() to the data. W is trainable parameters.
How to dot between batch data and weights?

hid_dim = 32
data = torch.randn(10, 2, 3, hid_dim)
data = data.view(10, 2*3, hid_dim)
W = torch.randn(hid_dim) # assume trainable parameters via nn.Parameter
result = torch.bmm(data, W).squeeze() # error, want (N, 6)
result = result.view(10, 2, 3)

Update

How about this one?

hid_dim = 32
data = torch.randn(10, 2, 3, hid_dim)
data = tdata.view(10, 2*3, hid_dim)
W = torch.randn(hid_dim, 1) # assume trainable parameters via nn.Parameter
W = W.unsqueeze(0).expand(10, hid_dim, 1)
result = torch.bmm(data, W).squeeze() # error, want (N, 6)
result = result.view(10, 2, 3)

Output:
Expand W tensor to match the shape of data tensor. The following should work.

hid_dim = 32
data = torch.randn(10, 2, 3, hid_dim)
data = data.view(10, 2*3, hid_dim)
W = torch.randn(hid_dim)
W = W.unsqueeze(0).unsqueeze(0).expand(*data.size())
result = torch.sum(data * W, 2)
result = result.view(10, 2, 3)

Edit: Your updated code is correct. Since you are converting W to a Bxhid_dimx1 and your data is of shape Bxdxhid_dim, so doing batch matrix multiplication will result in Bxdx1 which is essentially the dot product between W parameter and all the row vectors in data (dxhid_dim).


Instruction: Generating new images with PyTorch

Input:
I am studying GANs I've completed the one course which gave me an example of a program that generates images based on examples inputed.

The example can be found here:

https://github.com/davidsonmizael/gan

So I decided to use that to generate new images based on a dataset of frontal photos of faces, but I am not having any success. Differently from the example above, the code only generates noise, while the input has actual images.

Actually I don't have any clue about what should I change to make the code point to the right direction and learn from images. I haven't change a single value on the code provided in the example, yet it does not work.

If anyone can help me understand this and point me to the right direction would be very helpful. Thanks in advance.

My Discriminator:

class D(nn.Module):

    def __init__(self):
        super(D, self).__init__()
        self.main = nn.Sequential(
                nn.Conv2d(3, 64, 4, 2, 1, bias = False),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(64, 128, 4, 2, 1, bias = False),
                nn.BatchNorm2d(128),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(128, 256, 4, 2, 1, bias = False),
                nn.BatchNorm2d(256),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(256, 512, 4, 2, 1, bias = False),
                nn.BatchNorm2d(512),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(512, 1, 4, 1, 0, bias = False),
                nn.Sigmoid()
                )

    def forward(self, input):
        return self.main(input).view(-1)

My Generator:

class G(nn.Module):

    def __init__(self):
        super(G, self).__init__()
        self.main = nn.Sequential(
                nn.ConvTranspose2d(100, 512, 4, 1, 0, bias = False),
                nn.BatchNorm2d(512),
                nn.ReLU(True),
                nn.ConvTranspose2d(512, 256, 4, 2, 1, bias = False),
                nn.BatchNorm2d(256),
                nn.ReLU(True),
                nn.ConvTranspose2d(256, 128, 4, 2, 1, bias = False),
                nn.BatchNorm2d(128),
                nn.ReLU(True),
                nn.ConvTranspose2d(128, 64, 4, 2, 1, bias = False),
                nn.BatchNorm2d(64),
                nn.ReLU(True),
                nn.ConvTranspose2d(64, 3, 4, 2, 1, bias = False),
                nn.Tanh()
                )

    def forward(self, input):
        return self.main(input)

My function to start the weights:

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

Full code can be seen here:

https://github.com/davidsonmizael/criminal-gan

Noise generated on epoch number 25:

Input with real images:

Output:
The code from your example (https://github.com/davidsonmizael/gan) gave me the same noise as you show. The loss of the generator decreased way too quickly.

There were a few things buggy, I'm not even sure anymore what - but I guess it's easy to figure out the differences yourself. For a comparison, also have a look at this tutorial:
GANs in 50 lines of PyTorch

.... same as your code
print("# Starting generator and descriminator...")
netG = G()
netG.apply(weights_init)

netD = D()
netD.apply(weights_init)

if torch.cuda.is_available():
    netG.cuda()
    netD.cuda()

#training the DCGANs
criterion = nn.BCELoss()
optimizerD = optim.Adam(netD.parameters(), lr = 0.0002, betas = (0.5, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr = 0.0002, betas = (0.5, 0.999))

epochs = 25

timeElapsed = []
for epoch in range(epochs):
    print("# Starting epoch [%d/%d]..." % (epoch, epochs))
    for i, data in enumerate(dataloader, 0):
        start = time.time()
        time.clock()

        #updates the weights of the discriminator nn
        netD.zero_grad()

        #trains the discriminator with a real image
        real, _ = data

        if torch.cuda.is_available():
            inputs = Variable(real.cuda()).cuda()
            target = Variable(torch.ones(inputs.size()[0]).cuda()).cuda()
        else:
            inputs = Variable(real)
            target = Variable(torch.ones(inputs.size()[0]))

        output = netD(inputs)
        errD_real = criterion(output, target)
        errD_real.backward() #retain_graph=True

        #trains the discriminator with a fake image
        if torch.cuda.is_available():
            D_noise = Variable(torch.randn(inputs.size()[0], 100, 1, 1).cuda()).cuda()
            target = Variable(torch.zeros(inputs.size()[0]).cuda()).cuda()
        else:
            D_noise = Variable(torch.randn(inputs.size()[0], 100, 1, 1))
            target = Variable(torch.zeros(inputs.size()[0]))
        D_fake = netG(D_noise).detach()
        D_fake_ouput = netD(D_fake)
        errD_fake = criterion(D_fake_ouput, target)
        errD_fake.backward()

        # NOT:backpropagating the total error
        # errD = errD_real + errD_fake

        optimizerD.step()

    #for i, data in enumerate(dataloader, 0):

        #updates the weights of the generator nn
        netG.zero_grad()

        if torch.cuda.is_available():
            G_noise = Variable(torch.randn(inputs.size()[0], 100, 1, 1).cuda()).cuda()
            target = Variable(torch.ones(inputs.size()[0]).cuda()).cuda()
        else:
            G_noise = Variable(torch.randn(inputs.size()[0], 100, 1, 1))
            target = Variable(torch.ones(inputs.size()[0]))

        fake = netG(G_noise)
        G_output = netD(fake)
        errG  = criterion(G_output, target)

        #backpropagating the error
        errG.backward()
        optimizerG.step()

        if i % 50 == 0:
            #prints the losses and save the real images and the generated images
            print("# Progress: ")
            print("[%d/%d][%d/%d] Loss_D: %.4f Loss_G: %.4f" % (epoch, epochs, i, len(dataloader), errD_real.data[0], errG.data[0]))

            #calculates the remaining time by taking the avg seconds that every loop
            #and multiplying by the loops that still need to run
            timeElapsed.append(time.time() - start)
            avg_time = (sum(timeElapsed) / float(len(timeElapsed)))
            all_dtl = (epoch * len(dataloader)) + i
            rem_dtl = (len(dataloader) - i) + ((epochs - epoch) * len(dataloader))
            remaining =  (all_dtl - rem_dtl) * avg_time
            print("# Estimated remaining time: %s" % (time.strftime("%H:%M:%S", time.gmtime(remaining))))

        if i % 100 == 0:
            vutils.save_image(real, "%s/real_samples.png" % "./results", normalize = True)
            vutils.save_image(fake.data, "%s/fake_samples_epoch_%03d.png" % ("./results", epoch), normalize = True)

print ("# Finished.")

Result after 25 epochs (batchsize 256) on CIFAR-10:


Instruction: PyTorch loss function evaluation

Input:
I am trying to optimise a function (fit) using the acceleration of GPU in PyTorch. This is the straight Python code, where I doing the evaluation of fit:

import numpy as np
...
for j in range(P):
    e[:,j] = z - h[:,j];
    fit[j] = 1/(sqrt(2*pi)*sigma*N)*np.sum(exp(-(e[:,j]**2)/(2*sigma**2)));

  The dimension of the variables are:
  z[Nx1], h[NxP], e[NxP], fit[1xP].
  where P is the number of dimension of fit and N is the length of each dimension.

  I am aware that for loops should be avoided, so where it is my attempt to do it using PyTorch through torch.cuda.FloatTensor.

import torch
dtype = torch.cuda.FloatTensor
e     = z - h;
fit   = 1/(torch.sqrt(2*pi)*sigma*N)*torch.sum(torch.exp(-(torch.pw(e,2))/(2*torch.pow(sigma,2))));

  Unfortunately it is not working. What is wrong?
  Thank you!

Output:
I guess you are getting size mismatch error at the following line.

e = z - h

In your example, z is a vector (2d tensor of shape Nx1) and h is a 2d tensor of shape NxP. So, you can't directly subtract h from z.

You can do the following to avoid size mismatch error.

e = z.expand(*h.size()) - h

Here, z.expand(*h.size()) will convert the tensor z from Nx1 to NxP shape by duplicating the column vector P times.


Instruction: How to update a Tensor?

Input:
I've had some troubles updating a tensor using a previous one.
My problem: let's suppose that I have a tensor x1 [Nx1] and a new one calculated through the previous, x2 [Nx1]. Now I want to update the elements of x2 that are less than x1. I'm using dtype=torch.cuda.FloatTensor.

  This is the straight code in Python:

import numpy as np
...
index     = np.where(x1 > x2)
x2[index] = x1[index]

  Why can I do this using PyTorch with dtype=torch.cuda.FloatTensor?
  And if the x1 change to [NxD]

Thank you!

Output:
The code looks really similar to numpy:

idx = (x1 > x2)
x2[idx] = x1[idx]

Using some predefined arrays and printing x2:

x1 = torch.from_numpy(np.array([1, 2, 3, 4, 5])).float().cuda()
x2 = torch.from_numpy(np.array([3, 3, 3, 3, 3])).float().cuda()

  3  3  3  4  5
  [torch.cuda.FloatTensor of size 5 (GPU 0)]

Code would be the same for NxN dimensional tensors. Using:

x1 = torch.from_numpy(np.array([[1, 2, 5], [1, 4, 5]])).float().cuda()
x2 = torch.from_numpy(np.array([[3, 3, 3], [3, 3, 3]])).float().cuda()

  3  3  5
  3  4  5
  [torch.cuda.FloatTensor of size 2x3 (GPU 0)]


Instruction: How to use if statement PyTorch using torch.FloatTensor

Input:
I am trying to use the if statement in my PyTorch code using torch.FloatTensor as data type, to speed it up into the GPU.

This is my code:

import torch
import time
def fitness(x):
     return torch.pow(x, 2)

def velocity(v, gxbest, pxbest, pybest, x, pop):
     return torch.rand(pop).type(dtype)*v + \
            torch.rand(pop).type(dtype)*(pxbest - x) + \
            torch.rand(pop).type(dtype)*(gxbest.expand(x.size(0)) - x)

dtype = torch.cuda.FloatTensor
def main():

    pop, xmax, xmin, niter = 300000, 50, -50, 100
    v                      = torch.rand(pop).type(dtype)
    x                      = (xmax-xmin)*torch.rand(pop).type(dtype)+xmin
    y                      = fitness(x)
    [miny, indexminy]      = y.min(0)
    gxbest                 = x[indexminy]
    pxbest                 = x
    pybest                 = y

    for K in range(niter):

        vnext = velocity(v, gxbest, pxbest, pybest, x, pop)

        xnext = x + vnext
        ynext = fitness(x)
        [minynext, indexminynext]  = ynext.min(0)

        if (minynext < miny):
            miny   = minynext
            gxbest = xnext[indexminynext]

        indexpbest         = (ynext < pybest)
        pxbest[indexpbest] = xnext[indexpbest]
        pybest[indexpbest] = ynext[indexpbest]
        x                  = xnext
        v                  = vnext
main()

Unfortanally it is not working. It is giving me a error message and I can not figure it out what is the problem.

  RuntimeError: bool value of non-empty torch.cuda.ByteTensor objects is ambiguous

How can I use the if in PyTorch? I tried to convert the cuda.Tensor into a numpy array but it did not work also.

  minynext = minynext.cpu().numpy()
  miny = miny.cpu().numpy()

PS: Am I doing the code the efficient/faster way possible ? Or should I change something to achieve faster results?

Output:
If you look into the following simple example:

import torch

a = torch.LongTensor([1])
b = torch.LongTensor([5])

print(a > b)

Output:

 0
[torch.ByteTensor of size 1]

Comparing tensors a and b results in a torch.ByteTensor which is obviously not equivalent to boolean. So, you can do the following.

print(a[0] > b[0]) # False

So, you should change your if condition as follows.

if (minynext[0] < miny[0])


Instruction: Separate optimizer for discriminator and the rest of the model in adversarial setting

Input:
I have a model with the following components.

embedding layer
encoder
generator
discriminator
feed-forward neural network

I want to define two optimizers. One for the discriminator only and one for the rest. I am doing the following.

optimizers = []
model_params = chain(model.embedding.parameters(), model.encoder.parameters(),
                            model.generator.parameters(), model.ffnn.parameters())
optimizers.append(optim.Adam(model_params, args.lr))
optimizers.append(optim.Adam(model.discriminator.parameters(), args.lr))

Is there any better way to do the same? For example, can I take the difference between model.parameters() and model.discriminator.parameters()? If yes, how can I do that?

Output: no, because model.parameters() returns a generator. If you want to modify a generator like you want it you have to convert it to a list anyway.


Instruction: Fine Tuning dimensionality error

Input:
I am trying to use Resnet50 for image classification problem. However it shows error and I could not fix it.

RuntimeError: inconsistent tensor size, expected tensor [120 x 2048] and src [1000 x 2048] to have the same number of elements, but got 245760 and 2048000 elements respectively at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:86

and error happens below.

self.resnet = models.resnet50(num_classes=num_breeds, pretrained='imagenet')

Model is below

class Resnet(nn.Module):
    def __init__(self):
        super(Resnet,self).__init__()
        self.resnet = models.resnet50(num_classes=num_breeds, pretrained='imagenet')
        #self.resnet = nn.Sequential(*list(resnet.children())[:-2])
        #self.fc = nn.Linear(2048,num_breeds)

    def forward(self,x):
        x = self.resnet(x)
        return x

Output:
When you create your models.resnet50 with num_classes=num_breeds, the last layer is a fully connected layer from 2048 to num_classes (which is 120 in your case).

Having pretrained='imagenet' asks pytorch to load all the corresponding weights into your network, but it has for its last layer 1000 classes, not 120. This is the source of the error since the 2048x120 tensor doesn't match the loaded weights 2048x1000.

You should either create your network with 1000 classes and load the weights, then "trim" to the classes you want to keep. Or you could create the network you wished for with 120 classes, but manually load the weights. In the last case, you only need to pay specific attention to the last layer.


Instruction: Pytorch AssertionError: Torch not compiled with CUDA enabled

Input:
I am trying to run code from this repo. I have disabled cuda by changing lines 39/40 in main.py from

parser.add_argument('--type', default='torch.cuda.FloatTensor', help='type of tensor - e.g torch.cuda.HalfTensor')

to

parser.add_argument('--type', default='torch.FloatTensor', help='type of tensor - e.g torch.HalfTensor')

Despite this, running the code gives me the following exception:

Traceback (most recent call last):
  File "main.py", line 190, in <module>
    main()
  File "main.py", line 178, in main
    model, train_data, training=True, optimizer=optimizer)
  File "main.py", line 135, in forward
    for i, (imgs, (captions, lengths)) in enumerate(data):
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
AssertionError: Traceback (most recent call last):
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 62, in _pin_memory_loop
    batch = pin_memory_batch(batch)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 123, in pin_memory_batch
    return [pin_memory_batch(sample) for sample in batch]
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 123, in <listcomp>
    return [pin_memory_batch(sample) for sample in batch]
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 117, in pin_memory_batch
    return batch.pin_memory()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/tensor.py", line 82, in pin_memory
    return type(self)().set_(storage.pin_memory()).view_as(self)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/storage.py", line 83, in pin_memory
    allocator = torch.cuda._host_allocator()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 220, in _host_allocator
    _lazy_init()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 84, in _lazy_init
    _check_driver()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 51, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Spent some time looking through the issues in the Pytorch github, to no avail. Help, please?

Output:
If you look into the data.py file, you can see the function:

def get_iterator(data, batch_size=32, max_length=30, shuffle=True, num_workers=4, pin_memory=True):
    cap, vocab = data
    return torch.utils.data.DataLoader(
        cap,
        batch_size=batch_size, shuffle=shuffle,
        collate_fn=create_batches(vocab, max_length),
        num_workers=num_workers, pin_memory=pin_memory)

which is called twice in main.py file to get an iterator for the train and dev data. If you see the DataLoader class in pytorch, there is a parameter called:

  pin_memory (bool, optional) – If True, the data loader will copy tensors into CUDA pinned memory before returning them.

which is by default True in the get_iterator function. And as a result you are getting this error. You can simply pass the pin_memory param value as False when you are calling get_iterator function as follows.

train_data = get_iterator(get_coco_data(vocab, train=True),
                          batch_size=args.batch_size,
                          ...,
                          ...,
                          ...,
                          pin_memory=False)


Instruction: Tensorflow: Hierarchical Softmax Implementation

Input:
I'm currently having text inputs represented by vector, and I want to classify their categories. Because they are multi-level categories, I meant to use Hierarchical Softmax.

Example:

 - Computer Science
     - Machine Learning
     - NLP
 - Economics
 - Maths
     - Algebra
     - Geometry

I don't know how to implement it in Tensorflow. All examples I've met is using other frameworks.

Thanks

Output:
Finally, I have changed to use Pytorch. It's easier and more straight-forward than Tensorflow.

For anyone further interested in implement HS, you can have a look at my sample instructions: https://gist.github.com/paduvi/588bc95c13e73c1e5110d4308e6291ab
For anyone still want a Tensorflow implementation, this one is for you: https://github.com/tansey/sdp/blob/87e701c9b0ff3eacab29713cb2c9e7181d5c26aa/tfsdp/models.py#L205. But it's a little messy, and the author recommended using Pytorch or other dynamic graph framework


Instruction: ValueError: Floating point image RGB values must be in the 0..1 range. while using matplotlib

Input:
I want to visualize weights of the layer of a neural network. I'm using pytorch.

import torch
import torchvision.models as models
from matplotlib import pyplot as plt

def plot_kernels(tensor, num_cols=6):
    if not tensor.ndim==4:
        raise Exception("assumes a 4D tensor")
    if not tensor.shape[-1]==3:
        raise Exception("last dim needs to be 3 to plot")
    num_kernels = tensor.shape[0]
    num_rows = 1+ num_kernels // num_cols
    fig = plt.figure(figsize=(num_cols,num_rows))
    for i in range(tensor.shape[0]):
        ax1 = fig.add_subplot(num_rows,num_cols,i+1)
        ax1.imshow(tensor[i])
        ax1.axis('off')
        ax1.set_xticklabels([])
        ax1.set_yticklabels([])

    plt.subplots_adjust(wspace=0.1, hspace=0.1)
    plt.show()
vgg = models.vgg16(pretrained=True)
mm = vgg.double()
filters = mm.modules
body_model = [i for i in mm.children()][0]
layer1 = body_model[0]
tensor = layer1.weight.data.numpy()
plot_kernels(tensor)

The above gives this error ValueError: Floating point image RGB values must be in the 0..1 range.

My question is should I normalize and take absolute value of the weights to overcome this error or is there anyother way ?
If I normalize and use absolute value I think the meaning of the graphs change.

[[[[ 0.02240197 -1.22057354 -0.55051649]
   [-0.50310904  0.00891289  0.15427093]
   [ 0.42360783 -0.23392732 -0.56789106]]

  [[ 1.12248898  0.99013627  1.6526649 ]
   [ 1.09936976  2.39608836  1.83921957]
   [ 1.64557672  1.4093554   0.76332706]]

  [[ 0.26969245 -1.2997849  -0.64577204]
   [-1.88377869 -2.0100112  -1.43068039]
   [-0.44531786 -1.67845118 -1.33723605]]]

 [[[ 0.71286005  1.45265901  0.64986968]
   [ 0.75984162  1.8061738   1.06934202]
   [-0.08650422  0.83452386 -0.04468433]]

  [[-1.36591709 -2.01630116 -1.54488969]
   [-1.46221244 -2.5365622  -1.91758668]
   [-0.88827479 -1.59151018 -1.47308767]]

  [[ 0.93600738  0.98174071  1.12213969]
   [ 1.03908169  0.83749604  1.09565806]
   [ 0.71188802  0.85773659  0.86840987]]]

 [[[-0.48592842  0.2971966   1.3365227 ]
   [ 0.47920835 -0.18186836  0.59673625]
   [-0.81358945  1.23862112  0.13635623]]

  [[-0.75361633 -1.074965    0.70477796]
   [ 1.24439156 -1.53563368 -1.03012812]
   [ 0.97597247  0.83084011 -1.81764793]]

  [[-0.80762428 -0.62829626  1.37428832]
   [ 1.01448071 -0.81775147 -0.41943246]
   [ 1.02848887  1.39178836 -1.36779451]]]

 ...,
 [[[ 1.28134537 -0.00482408  0.71610934]
   [ 0.95264435 -0.09291686 -0.28001019]
   [ 1.34494913  0.64477581  0.96984017]]

  [[-0.34442815 -1.40002513  1.66856039]
   [-2.21281362 -3.24513769 -1.17751861]
   [-0.93520379 -1.99811196  0.72937071]]

  [[ 0.63388056 -0.17022935  2.06905985]
   [-0.7285465  -1.24722099  0.30488953]
   [ 0.24900314 -0.19559766  1.45432627]]]

 [[[-0.80684513  2.1764245  -0.73765725]
   [-1.35886598  1.71875226 -1.73327696]
   [-0.75233924  2.14700699 -0.71064663]]

  [[-0.79627383  2.21598244 -0.57396138]
   [-1.81044972  1.88310981 -1.63758397]
   [-0.6589964   2.013237   -0.48532376]]

  [[-0.3710472   1.4949851  -0.30245575]
   [-1.25448656  1.20453358 -1.29454732]
   [-0.56755757  1.30994892 -0.39370224]]]

 [[[-0.67361742 -3.69201088 -1.23768616]
   [ 3.12674141  1.70414758 -1.76272404]
   [-0.22565465  1.66484773  1.38172317]]

  [[ 0.28095332 -2.03035069  0.69989491]
   [ 1.97936332  1.76992691 -1.09842575]
   [-2.22433758  0.52577412  0.18292744]]

  [[ 0.48471382 -1.1984663   1.57565165]
   [ 1.09911084  1.31910467 -0.51982772]
   [-2.76202297 -0.47073677  0.03936549]]]]

Output:
It sounds as if you already know your values are not in that range.  Yes, you must re-scale them to the range 0.0 - 1.0.  I suggest that you want to retain visibility of negative vs positive, but that you let 0.5 be your new "neutral" point.  Scale such that current 0.0 values map to 0.5, and your most extreme value (largest magnitude) scale to 0.0 (if negative) or 1.0 (if positive).

Thanks for the vectors.  It looks like your values are in the range -2.25 to +2.0.  I suggest a rescaling new = (1/(2*2.25)) * old + 0.5


Instruction: Is it possible to check if elements of a tensor are out of boundaries?

Input:
Is it possible to check if elements of a tensor are out of boundaries using torch.cuda.FloatTensor on PyTorch, GPU approach?

Example (check limits):

for i in range(pop):
    if (x[i]>xmax):
        x[i]=xmax
    elif (x[i]<xmin):
        x[i]=xmin

I tried the following, but did not speed up:

idxmax    = (x > xmax) # elements that are bigger that upper limit
idxmim    = (x < min)  # elements that are smaller that upper limit
x[idxmax] = xmax
x[idxmin] = xmin

If not, is it possible to do this check limits part, using only the CPU? How?

Output:
You can get a CPU copy of tensor x, do your operations and then push the tensor to GPU memory again.

x = x.cpu()  # get the CPU copy
# do your operations
x = x.cuda() # move the object back to cuda memory


Instruction: how pytorch nn.module save submodule

Input:
I have some question about how pytorch nn.module works

import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.sub_module = nn.Linear(10, 5)
        self.value = 3

net = Net()
print(net.__dict__)

output

{'_modules': OrderedDict([('sub_module', Linear (10 -> 5))]),  'value': 3, ...}

I know that every attribute of a class should be stored in __dict__, why value(a int value) is in it, but sub_module(a nn.Module) is not, instead, sub_module is stored in _modules

I read the code of nn.Module implementation, but I didn't figure that out. do anyone have any ideas?

thank you !!

Output:
I will try to keep it simple.

Every time you create a new item in the class Net for instance: self.sub_module = nn.Linear(10, 5) it calls the method __setattr__ of its parent class, in this case nn.Module. Then, inside __setattr__ method, the parameters are stored to the dict they belong. In this case since nn.Linear is a module, it is stored to the _modules dict.

Here is the piece of code that does this inside the Module class  https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L389


Instruction: torch.nn.LSTM runtime error

Input:
I'm trying to implement the structure from "Livelinet: A multimodal Deep Recurrent Neural Network to Predict Liveliness in Educational Videos".

For a brief explanation, I split 10-second audio clip into ten 1-second audio clip and get a spectrogram (a picture) from that 1-second audio clip. Then I use CNN to get a representation vector from the picture, finally getting 10 vectors of each 1-second video clips.

Next, I feed those 10 vectors to LSTM, and I got some errors there.
My code and the error traceback are as follows:

class AudioCNN(nn.Module):

def __init__(self):
    super(AudioCNN,self).__init__()
    self.features = alexnet.features
    self.features2 = nn.Sequential(*classifier)
    self.lstm = nn.LSTM(512, 256,2)
    self.classifier = nn.Linear(2*256,2)

def forward(self, x):
    x = self.features(x)
    print x.size()
    x = x.view(x.size(0),256*6*6)
    x = self.features2(x)
    x = x.view(10,1,512)
    h_0,c_0 = self.init_hidden()
    _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
    assert False
    x = x.view(1,1,2*256)
    x = self.classifier(x)

    return x

def init_hidden(self):
    h_0 = torch.randn(2,1,256) #layer * batch * input_dim
    c_0 = torch.randn(2,1,256)
    return h_0, c_0

audiocnn = AudioCNN()
input = torch.randn(10,3,223,223)
input = Variable(input)
audiocnn(input)

Error:

RuntimeErrorTraceback (most recent call last)
<ipython-input-64-2913316dbb34> in <module>()
----> 1 audiocnn(input)

/home//local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--> 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

<ipython-input-60-31881982cca9> in forward(self, x)
     15         x = x.view(10,1,512)
     16         h_0,c_0 = self.init_hidden()
---> 17         _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
     18         assert False
     19         x = x.view(1,1,2*256)

/home/local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--> 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

/home//local/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc in forward(self, input, hx)
    160             flat_weight=flat_weight
    161         )
--> 162         output, hidden = func(input, self.all_weights, hx)
    163         if is_packed:
    164             output = PackedSequence(output, batch_sizes)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, *fargs, **fkwargs)
    349         else:
    350             func = AutogradRNN(*args, **kwargs)
--> 351         return func(input, *fargs, **fkwargs)
    352
    353     return forward

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, weight, hidden)
    242             input = input.transpose(0, 1)
    243
--> 244         nexth, output = func(input, hidden, weight)
    245
    246         if batch_first and batch_sizes is None:

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
     82                 l = i * num_directions + j
     83
---> 84                 hy, output = inner(input, hidden[l], weight[l])
     85                 next_hidden.append(hy)
     86                 all_output.append(output)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
    111         steps = range(input.size(0) - 1, -1, -1) if reverse else range(input.size(0))
    112         for i in steps:
--> 113             hidden = inner(input[i], hidden, *weight)
    114             # hack to handle LSTM
    115             output.append(hidden[0] if isinstance(hidden, tuple) else hidden)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in LSTMCell(input, hidden, w_ih, w_hh, b_ih, b_hh)
     29
     30     hx, cx = hidden
---> 31     gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
     32
     33     ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)

/home//local/lib/python2.7/site-packages/torch/nn/functional.pyc in linear(input, weight, bias)
    551     if input.dim() == 2 and bias is not None:
    552         # fused op is marginally faster
--> 553         return torch.addmm(bias, input, weight.t())
    554
    555     output = input.matmul(weight.t())

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in addmm(cls, *args)
    922         @classmethod
    923         def addmm(cls, *args):
--> 924             return cls._blas(Addmm, args, False)
    925
    926         @classmethod

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in _blas(cls, args, inplace)
    918             else:
    919                 tensors = args
--> 920             return cls.apply(*(tensors + (alpha, beta, inplace)))
    921
    922         @classmethod

RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition

Output:
The error message

  RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition

generally indicates that you are passing a tensor or something else that cannot store history as an input into a module. In your case, your problem is that you return tensors in init_hidden() instead of Variable instances. Therefore, when the LSTM runs it cannot compute the gradient for the hidden layer as its initial input is not a part of the backprop graph.

Solution:

def init_hidden(self):
    h_0 = torch.randn(2,1,256) #layer * batch * input_dim
    c_0 = torch.randn(2,1,256)
    return Variable(h_0), Variable(c_0)

It is also likely that a mean of 0 and a variance of 1 is not helpful as initial values for the LSTM hidden state. Ideally, you would make the initial state trainable as well, e.g.:

h_0 = torch.zeros(2,1,256) # layer * batch * input_dim
c_0 = torch.zeros(2,1,256)
h_0_param = torch.nn.Parameter(h_0)
c_0_param = torch.nn.Parameter(c_0)

def init_hidden(self):
    return h_0_param, c_0_param

In this case the network can learn what initial state suits best. Note that in this case there is no need to wrap h_0_param in a Variable as a Parameter is essentially a Variable with require_grad=True.


Instruction: Rosenbrock function with D dimension using PyTorch

Input:
How can I implement the Rosenbrock function with D dimension, using PyTorch?

  Create the variables, where D is the number of dimensions and N is the number of elements.

x = (xmax - xmin)*torch.rand(N,D).type(dtype) + xmin

  Function :

  Using straight Python I'd do something like this:

fit = 0
for i in range(D-1):
    term1 = x[i + 1] - x[i]**2
    term2 = 1 - x[i]
    fit = fit + 100 * term1**2 + term2**2

  My attempt using Pytorch:

def Rosenbrock(x):
    return torch.sum(100*(x - x**2)**2 + (x-1)**2)

  I just do not know how to do the x[i+1] without using a for loop.
  How can I deal with it?
  Thank you!

Output:
Numpy has the function roll which I think can be really helpful.
Unfortunately, I am not aware of any function similar to numpy.roll for pytorch.

In my attempt, x is a numpy array in the form DxN. First we use roll to move the items in the first dimension (axis=0) one position to the left. Like this, everytime we compare x_1[i] is like if we do x[i+1]. Then, since the function takes only D-1 elements for the sum, we remove the last column slicing the pytorch tensor with [:-1, :]. Then the summation is really similar to the code you posted, just changing x for x_1 at the correct place.

def Rosenbrock(x):
    x_1 = torch.from_numpy(np.roll(x, -1, axis=0)).float()[:-1, :]
    x = torch.from_numpy(x).float()[:-1, :]

    return torch.sum(100 * (x_1 - x ** 2) ** 2 + (x - 1) ** 2, 0)

Similarly, by using roll you could remove you for loop in the numpy version


Instruction: How to construct a 3D Tensor where every 2D sub tensor is a diagonal matrix in PyTorch?

Input:
Consider I have 2D Tensor, index_in_batch * diag_ele.
How can I get a 3D Tensor index_in_batch * Matrix (who is a diagonal matrix, construct by drag_ele)?

The torch.diag() construct diagonal matrix only when input is 1D, and return diagonal element when input is 2D.

Output:
import torch

a = torch.rand(2, 3)
print(a)
b = torch.eye(a.size(1))
c = a.unsqueeze(2).expand(*a.size(), a.size(1))
d = c * b
print(d)

Output

 0.5938  0.5769  0.0555
 0.9629  0.5343  0.2576
[torch.FloatTensor of size 2x3]

(0 ,.,.) =
  0.5938  0.0000  0.0000
  0.0000  0.5769  0.0000
  0.0000  0.0000  0.0555

(1 ,.,.) =
  0.9629  0.0000  0.0000
  0.0000  0.5343  0.0000
  0.0000  0.0000  0.2576
[torch.FloatTensor of size 2x3x3]


Instruction: How to select index over two dimension in PyTorch?

Input:
Given a = torch.randn(3, 2, 4, 5), how I can select sub tensor like  (2, :, 0, :), (1, :, 1, :), (2, :, 2, :), (0, :, 3, :) (a resulting tensor of size (2, 4, 5) or (4, 2, 5)?

While a[2, :, 0, :] gives

 0.5580 -0.0337  1.0048 -0.5044  0.6784
-1.6117  1.0084  1.1886  0.1278  0.3739
[torch.FloatTensor of size 2x5]

however, a[[2, 1, 2, 0], :, [0, 1, 2, 3], :] gives

  TypeError: Performing basic indexing on a tensor and encountered an error indexing dim 0 with an object of type list. The only supported types are integers, slices, numpy scalars, or if indexing with a torch.LongTensor or torch.ByteTensor only a single Tensor may be passed.

though numpy returns a (4, 2, 5) tensor successfully.

Output:
Does it work for you?

import torch

a = torch.randn(3, 2, 4, 5)
print(a.size())

b = [a[2, :, 0, :], a[1, :, 1, :], a[2, :, 2, :], a[0, :, 3, :]]
b = torch.stack(b, 0)

print(b.size()) # torch.Size([4, 2, 5])


Instruction: transfer learning [resnet18] using PyTorch. Dataset: Dog-Breed-Identification

Input:
I am trying to implement a transfer learning approach in PyTorch. This is the dataset that I am using: Dog-Breed

Here's the step that I am following.

1. Load the data and read csv using pandas.
2. Resize (60, 60) the train images and store them as numpy array.
3. Apply stratification and split the train data into 7:1:2 (train:validation:test)
4. use the resnet18 model and train.

Location of dataset

LABELS_LOCATION = './dataset/labels.csv'
TRAIN_LOCATION = './dataset/train/'
TEST_LOCATION = './dataset/test/'
ROOT_PATH = './dataset/'

Reading CSV (labels.csv)

def read_csv(csvf):
    # print(pandas.read_csv(csvf).values)
    data=pandas.read_csv(csvf).values
    labels_dict = dict(data)
    idz=list(labels_dict.keys())
    clazz=list(labels_dict.values())
    return labels_dict,idz,clazz

I did this because of a constraint which I will mention next when I am loading the data using DataLoader.

def class_hashmap(class_arr):
    uniq_clazz = Counter(class_arr)
    class_dict = {}
    for i, j in enumerate(uniq_clazz):
        class_dict[j] = i
    return class_dict

labels, ids, class_names = read_csv(LABELS_LOCATION)
train_images = os.listdir(TRAIN_LOCATION)
class_numbers = class_hashmap(class_names)

Next, I resize the image to 60,60 using opencv, and store the result as numpy array.

resize = []
indexed_labels = []
for t_i in train_images:
    # resize.append(transform.resize(io.imread(TRAIN_LOCATION+t_i), (60, 60, 3)))  # (60,60) is the height and widht; 3 is the number of channels
    resize.append(cv2.resize(cv2.imread(TRAIN_LOCATION+t_i), (60, 60)).reshape(3, 60, 60))
    indexed_labels.append(class_numbers[labels[t_i.split('.')[0]]])

resize = np.asarray(resize)
print(resize.shape)

Here in indexed_labels, I give each label a number.

Next, I split the data into 7:1:2 part

X = resize  # numpy array of images [training data]
y = np.array(indexed_labels)  # indexed labels for images [training labels]

sss = StratifiedShuffleSplit(n_splits=3, test_size=0.2, random_state=0)
sss.get_n_splits(X, y)

for train_index, test_index in sss.split(X, y):
    X_temp, X_test = X[train_index], X[test_index]  # split train into train and test [data]
    y_temp, y_test = y[train_index], y[test_index]  # labels

sss = StratifiedShuffleSplit(n_splits=3, test_size=0.123, random_state=0)
sss.get_n_splits(X_temp, y_temp)

for train_index, test_index in sss.split(X_temp, y_temp):
    print("TRAIN:", train_index, "VAL:", test_index)
    X_train, X_val = X[train_index], X[test_index]  # training and validation data
    y_train, y_val = y[train_index], y[test_index]  # training and validation labels

Next, I loaded the data from the previous step into torch DataLoaders

batch_size = 500
learning_rate = 0.001

train = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=False)

val = torch.utils.data.TensorDataset(torch.from_numpy(X_val), torch.from_numpy(y_val))
val_loader = torch.utils.data.DataLoader(val, batch_size=batch_size, shuffle=False)

test = torch.utils.data.TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))
test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)

# print(train_loader.size)

dataloaders = {
    'train': train_loader,
    'val': val_loader
}

Next, I load the pretrained rensnet model.

model_ft = models.resnet18(pretrained=True)

# freeze all model parameters
# for param in model_ft.parameters():
#     param.requires_grad = False

num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, len(class_numbers))

if use_gpu:
    model_ft = model_ft.cuda()
    model_ft.fc = model_ft.fc.cuda()

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

And then I use the train_model, a method described here in PyTorch's docs.

However, when I run this I get an error.

Traceback (most recent call last):
  File "/Users/nirvair/Sites/pyTorch/TL.py",
    line 244, in <module>
        num_epochs=25)
      File "/Users/nirvair/Sites/pyTorch/TL.py", line 176, in train_model
        outputs = model(inputs)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
        result = self.forward(*input, **kwargs)
      File "/Library/Python/2.7/site-packages/torchvision/models/resnet.py", line 149, in forward
        x = self.avgpool(x)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
        result = self.forward(*input, **kwargs)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/pooling.py", line 505, in forward
        self.padding, self.ceil_mode, self.count_include_pad)
      File "/Library/Python/2.7/site-packages/torch/nn/functional.py", line 264, in avg_pool2d
        ceil_mode, count_include_pad)
      File "/Library/Python/2.7/site-packages/torch/nn/_functions/thnn/pooling.py", line 360, in forward
        ctx.ceil_mode, ctx.count_include_pad)
    RuntimeError: Given input size: (512x2x2). Calculated output size: (512x0x0). Output size is too small at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/THNN/generic/SpatialAveragePooling.c:64

I can't seem to figure out what's going wrong here.

Output:
Your network is too deep for the size of images you are using (60x60). As you know, the CNN layers do produce smaller and smaller feature maps as the input image propagate through the layers. This is because you are not using padding.

The error you have simply says that the next layer is expecting 512 feature maps with a size of 2 pixels by 2 pixels. The actual feature map produced from the forward pass was 512 maps of size 0x0. This mismatch is what triggered the error.

Generally, all stock networks, such as RESNET-18, Inception, etc, require the input images to be of the size 224x224 (at least). You can do this easier using the torchvision transforms[1]. You can also use larger image sizes with one exception for the AlexNet which has the size of feature vector hardcoded as explained in my answer in [2].

Bonus Tip: If you are using the network in pre-tained mode, you will need to whiten the data using the parameters in the pytorch documentation at [3].

Links

http://pytorch.org/docs/master/torchvision/transforms.html
https://stackoverflow.com/a/46865203/7387369
http://pytorch.org/docs/master/torchvision/models.html


Instruction: `for` loop to a multi dimensional array in PyTorch

Input:
I want to implement Q&A systems with attention mechanism. I have two inputs; context and query which shapes are (batch_size, context_seq_len, embd_size) and (batch_size,  query_seq_len, embd_size).
I am following the below paper.
Machine Comprehension Using Match-LSTM and Answer Pointer.   https://arxiv.org/abs/1608.07905

Then, I want to obtain a attention matrix which shape is (batch_size, context_seq_len, query_seq_len, embd_size). In the thesis, they calculate values for each row (it means each context word, G_i, alpha_i in the paper).

My code is below and it is running. But I am not sure my way is good or not. For example, I use for loop for generating sequence data (for i in range(T):). And to obtain each row, I use in-place operator like G[:,i,:,:], embd_context[:,i,:].clone() is a good manner in pytorch? If not, where should I change the code?

And if you notice other points, let me know. I am a new in this field and pytorch. Sorry for my ambiguous question.

class MatchLSTM(nn.Module):
    def __init__(self, args):
        super(MatchLSTM, self).__init__()
        self.embd_size = args.embd_size
        d = self.embd_size
        self.answer_token_len = args.answer_token_len

        self.embd = WordEmbedding(args)
        self.ctx_rnn   = nn.GRU(d, d, dropout = 0.2)
        self.query_rnn = nn.GRU(d, d, dropout = 0.2)

        self.ptr_net = PointerNetwork(d, d, self.answer_token_len) # TBD

        self.w  = nn.Parameter(torch.rand(1, d, 1).type(torch.FloatTensor), requires_grad=True) # (1, 1, d)
        self.Wq = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)
        self.Wp = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)
        self.Wr = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)

        self.match_lstm_cell = nn.LSTMCell(2*d, d)

    def forward(self, context, query):
        # params
        d = self.embd_size
        bs = context.size(0) # batch size
        T = context.size(1)  # context length
        J = query.size(1)    # query length

        # LSTM Preprocessing Layer
        shape = (bs, T, J, d)
        embd_context     = self.embd(context)         # (N, T, d)
        embd_context, _h = self.ctx_rnn(embd_context) # (N, T, d)
        embd_context_ex  = embd_context.unsqueeze(2).expand(shape).contiguous() # (N, T, J, d)
        embd_query       = self.embd(query)           # (N, J, d)
        embd_query, _h   = self.query_rnn(embd_query) # (N, J, d)
        embd_query_ex  = embd_query.unsqueeze(1).expand(shape).contiguous() # (N, T, J, d)

        # Match-LSTM layer
        G = to_var(torch.zeros(bs, T, J, d)) # (N, T, J, d)

        wh_q = torch.bmm(embd_query, self.Wq.expand(bs, d, d)) # (N, J, d) = (N, J, d)(N, d, d)

        hidden     = to_var(torch.randn([bs, d])) # (N, d)
        cell_state = to_var(torch.randn([bs, d])) # (N, d)
        # TODO bidirectional
        H_r = [hidden]
        for i in range(T):
            wh_p_i = torch.bmm(embd_context[:,i,:].clone().unsqueeze(1), self.Wp.expand(bs, d, d)).squeeze() # (N, 1, d) -> (N, d)
            wh_r_i = torch.bmm(hidden.unsqueeze(1), self.Wr.expand(bs, d, d)).squeeze() # (N, 1, d) -> (N, d)
            sec_elm = (wh_p_i + wh_r_i).unsqueeze(1).expand(bs, J, d) # (N, J, d)

            G[:,i,:,:] = F.tanh( (wh_q + sec_elm).view(-1, d) ).view(bs, J, d) # (N, J, d) # TODO bias

            attn_i = torch.bmm(G[:,i,:,:].clone(), self.w.expand(bs, d, 1)).squeeze() # (N, J)
            attn_query = torch.bmm(attn_i.unsqueeze(1), embd_query).squeeze() # (N, d)
            z = torch.cat((embd_context[:,i,:], attn_query), 1) # (N, 2d)

            hidden, cell_state = self.match_lstm_cell(z, (hidden, cell_state)) # (N, d), (N, d)
            H_r.append(hidden)
        H_r = torch.stack(H_r, dim=1) # (N, T, d)

        indices = self.ptr_net(H_r) # (N, M, T) , M means (start, end)
        return indices

Output: I think your code is fine. You can't avoid the loop: for i in range(T): because in equation (2) in the paper (https://openreview.net/pdf?id=B1-q5Pqxl), there is a hidden state coming from Match-LSTM cell which is involved in computing G_i and alpha_i vector and they are used to compute the input for next timestep of the Match-LSTM. So, you need to run the loop for every timestep of the Match-LSTM, I don't see an alternative to avoid the for loop anyway.


Instruction: Taking subsets of a pytorch dataset

Input:
I have a network which I want to train on some dataset (as an example, say CIFAR10). I can create data loader object via

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

My question is as follows: Suppose I want to make several different training iterations. Let's say I want at first to train the network on all images in odd positions, then on all images in even positions and so on. In order to do that, I need to be able to access to those images. Unfortunately, it seems that trainset does not allow such access. That is, trying to do trainset[:1000] or more generally trainset[mask] will throw an error.

I could do instead

trainset.train_data=trainset.train_data[mask]
trainset.train_labels=trainset.train_labels[mask]

and then

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                              shuffle=True, num_workers=2)

However, that will force me to create a new copy of the full dataset in each iteration (as I already changed trainset.train_data so I will need to redefine trainset). Is there some way to avoid it?

Ideally, I would like to have something "equivalent" to

trainloader = torch.utils.data.DataLoader(trainset[mask], batch_size=4,
                                              shuffle=True, num_workers=2)

Output:
You can define a custom sampler for the dataset loader avoiding recreating the dataset (just creating a new loader for each different sampling).

class YourSampler(Sampler):
    def __init__(self, mask):
        self.mask = mask

    def __iter__(self):
        return (self.indices[i] for i in torch.nonzero(self.mask))

    def __len__(self):
        return len(self.mask)

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

sampler1 = YourSampler(your_mask)
sampler2 = YourSampler(your_other_mask)
trainloader_sampler1 = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          sampler = sampler1, shuffle=False, num_workers=2)
trainloader_sampler2 = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          sampler = sampler2, shuffle=False, num_workers=2)

PS: You can find more info here: http://pytorch.org/docs/master/_modules/torch/utils/data/sampler.html#Sampler


Instruction: AttributeError on variable input of custom loss function in PyTorch

Input:
I've made a custom loss function to compute cross-entropy (CE) for a multi-output multi-label problem. Within the class, I want to set the target variable I'm feeding to not require a gradient. I do this within the forward function using a pre-defined function (taken from pytorch source code) outside the class.

    def _assert_no_grad(variable):
        assert not variable.requires_grad

    def forward(self, predicted, target):
        """
        Computes cross entropy between targets and predictions.
        """
        # No gradient over target
        _assert_no_grad(target)

        # Define variables
        p = predicted.clamp(0.01, 0.99)
        t = target.float()

        #Compute cross entropy
        h1 = p.log()*t
        h2 = (1-t)*((1-p).log())
        ce = torch.add(h1, h2)
        ce_out = torch.mean(ce, 1)
        ce_out = torch.mean(ce_out, 0)

        # Save for backward step
        self.save_for_backward(ce_out)

At this point when I run the code in a batched for-loop (see below), I get the following error:

  AttributeError: 'torch.FloatTensor' object has no attribute 'requires_grad'

It seems simple enough as we should be passing a torch.autograd.Variable, however I am already doing this as can be seen in the snippet below.

for t in range(50):

    print('Epoch {}'.format(t))
    if t > 0:
        print('Loss ->', loss)

    for batch_idx, (x_batch, y_batch) in enumerate(train_loader):
        # Wrap in Variable
        x_in, target = Variable(x_batch), Variable(y_batch)

        predicted = model(x_in)

        # Compute and print loss
        loss = criterion(predicted, target)

        # Zero gradients, perform a backward pass, and update the weights.
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

To add a note, my final goal is to generate a class which behaves like BCELoss except for multiple labels and not just binary. I feel like I've already scoured the entire PyTorch docs and primarily been using this and some forum entries.
http://pytorch.org/docs/master/notes/extending.html

So

Output:
The problem is in "target.float()" line which converts your t Variable to Tensor. You can directly use target without any problems in your CE calculation.

Also, I guess you do not really need "self.save_for_backward(ce_out)" as I guess you are defining nn.Module class which will take care of the backward pass internally.


Instruction: How can I feed a batch into LSTM without reordering them by length in PyTorch?

Input:
I am new to Pytorch and I have got into some trouble.
I want to build a rank-model to judge the similarity of question and its answers (including right answers and wrong answers). And I use LSTM as an encoder.

There are two LSTMs in my model and they share the weights. So my model’s input is two sequences (question and answer). But if I use batch, reordering will disrupt the correspondence between questions and answers.
What should I do?

Output:
Maybe a starting point could be to use similar RNN wrapper as here https://github.com/facebookresearch/DrQA/blob/master/drqa/reader/layers.py#L20
You can encode question and asnwer separately (this module will pack and unpack internally and will take care of sorting) and then you can work on the encoded representations


Instruction: torch.addmm received an invalid combination of arguments

Input:
In the official webpage of pytorch I saw the following code with answers:

>> a = torch.randn(4, 4)
>> a

0.0692  0.3142  1.2513 -0.5428
0.9288  0.8552 -0.2073  0.6409
1.0695 -0.0101 -2.4507 -1.2230
0.7426 -0.7666  0.4862 -0.6628
torch.FloatTensor of size 4x4]

>>> torch.max(a, 1)
(
 1.2513
 0.9288
 1.0695
 0.7426
[torch.FloatTensor of size 4]
,
 2
 0
 0
 0
[torch.LongTensor of size 4]
)

I know that the first result corresponds to the maximum number per row, however I do not get the second tensor (LongTensor)

I tried other random example and after a pytorch.max, I came to find these results

0.9477  1.0090  0.8348 -1.3513
-0.4861  1.2581  0.3972  1.5751
-1.2277 -0.6201 -1.0553  0.6069
 0.1688  0.1373  0.6544 -0.7784
[torch.FloatTensor of size 4x4]

(
 1.0090
 1.5751
 0.6069
 0.6544
[torch.FloatTensor of size 4]
,
 1
 3
 3
 2
[torch.LongTensor of size 4]
)

Can anyone tell me what does it really mean these LongTensor data? I thought it was a strange casting between tensors, however after a simple casting of a float tensor, I see that it just cuts decimals

Thanks

Output:
It just tells the index of the max element in your original tensor along the queried dimension.

E.g.

0.9477  1.0090  0.8348 -1.3513
-0.4861  1.2581  0.3972  1.5751
-1.2277 -0.6201 -1.0553  0.6069
 0.1688  0.1373  0.6544 -0.7784
[torch.FloatTensor of size 4x4]

# torch.max(a, 1)
(
 1.0090
 1.5751
 0.6069
 0.6544
[torch.FloatTensor of size 4]
,
 1
 3
 3
 2
[torch.LongTensor of size 4]
)

In the above example in torch.LongTensor,

1 is the index of 1.0090 in your original tensor (torch.FloatTensor)
3 is the index of 1.5751 in your original tensor (torch.FloatTensor)
3 is the index of 0.6069 in your original tensor (torch.FloatTensor)
2 is the index of 0.6544 in your original tensor (torch.FloatTensor)

along dimension 1.

Instead, if you'd have requested torch.max(a, 0), the entries in torch.LongTensor would correspond to the indices of max elements in your original tensor along dimension 0.