Instruction: Advanced indexing in 2d tensor in pytorch

Input:
I have a 2d tensor X. and two lists of indexes that is first index and second call a and b. I want to do

X[a[i],b[i]] = 0 for i in range(len(a))

How can I do this. If i directly do X[a,b] the error is IndexError: The advanced indexing objects could not be broadcast

Output:
Check your lists which contains the indices, some values might be out of range. That's when you will get IndexError like the one below:

  In [43]: X[4,4]

  IndexError                                Traceback (most recent call last)
   in ()
  ----> 1 X[4,4]

  IndexError: index 4 is out of range for dimension 0 (of size 3)

If your indices are in correct range, it should work fine.

Here is an example:

In [35]: X = torch.Tensor([[3, 4, 5, 6], [1, 2, 3, 4], [6, 3, 2, 1]])

In [36]: X
Out[36]:

 3  4  5  6
 1  2  3  4
 6  3  2  1
[torch.FloatTensor of size 3x4]

In [37]: a = [0, 2]

In [38]: b = [1, 2]

In [39]: X[a, b]
Out[39]:

 4
 2
[torch.FloatTensor of size 2]

In [40]: X[a, b] = 0

In [41]: X
Out[41]:

 3  0  5  6
 1  2  3  4
 6  3  0  1
[torch.FloatTensor of size 3x4]


Instruction: I get this error on pytorch "RuntimeError: invalid argument 2: size '[-1 x 400]"

Input:
Can anyone help me how to fix this error?

  RuntimeError: invalid argument 2: size '[-1 x 400]' is invalid for input of with 1597248 elements at /Users/soumith/miniconda2/conda-bld/pytorch_1503975723910/work/torch/lib/TH/THStorage.c:37

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # wrap them in Variable
        inputs, labels = Variable(inputs), Variable(labels)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.data[0]
        if i % 5 == 4:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 5))
            running_loss = 0.0

print('Finished Training')

Output:
When I scaled image size, error stopped occurring.

transform = transforms.Compose(
                   [transforms.Scale((32,32)),
                    transforms.ToTensor(),
                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])


Instruction: How to decode an embedding in PyTorch efficiently?

Input:
I am new to Pytorch and RNN. I am learning how to use RNN to predict numbers as a tutorial from the video: https://www.youtube.com/watch?v=MKA6v99uYKY

In his code, he use python 3 and do the decode like:

out_unembedded = out.view(-1, hidden_size) @ embedding.weight.transpose(0,1)

I am using Python 2 and try the code:

out_unembedded = out.view(-1, hidden_size).dot( embedding.weight.transpose(0,1))

But it seems not right, Then I try to decode like this:

import torch
import torch.nn as nn
from torch.autograd import Variable

word2id = {'hello': 0, 'world': 1, 'I': 2, 'am': 3,'writing': 4,'pytorch': 5}
embeds = nn.Embedding(6, 3)
word_embed = embeds(Variable(torch.LongTensor([word2id['am']])))

id2word = {v: k for k, v in word2id.iteritems()}
index = 0
for row in embeds.weight.split(1):
    if(torch.min( torch.eq(row.data,word_embed.data) ) == 1):
        print index
        print id2word[index]
    index+=1

Is there a more professional way to do this? Thanks!

------------ UPDATE ------------

I find the correct way to substitute @ in Python 2:

out_unembedded = torch.mm( embedded_output.view(-1, hidden_size),embedding.weight.transpose(0, 1))

Output:
I finally figure out the problem. The two decode methods are different.

The first one use

  @

to do the dot product. Instead of searching the exact decoding, it calculates the cosine similarity by dot product and find the most similar word. The value after dot product means the similarity between the target and the word with such index. The equation is:

The second method which build a hash map is to find the index using the exact encoding.


Instruction: Compute gradients w.r.t. the values of embedding vectors in PyTorch

Input:
I am trying to train a dual encoder LSTM model for a chatbot using PyTorch.

I defined two classes: the Encoder class defines the LSTM itself and the Dual_Encoder class applies the Encoder to both context and response utterances that I am trying to train on:

class Encoder(nn.Module):

    def __init__(self,
                 input_size,
                 hidden_size,
                 vocab_size,
                 num_layers = 1,
                 num_directions = 1,
                 dropout = 0,
                 bidirectional = False,
                 rnn_type = 'lstm'):

                 super(Encoder, self).__init__()

                 self.input_size = input_size
                 self.hidden_size = hidden_size
                 self.vocab_size = vocab_size
                 self.num_layers = 1
                 self.num_directions = 1
                 self.dropout = 0,
                 self.bidirectional = False

                 self.embedding = nn.Embedding(vocab_size, input_size, sparse = False, padding_idx = 0)
                 self.lstm = nn.LSTM(self.input_size, self.hidden_size, self.num_layers, batch_first=False, dropout = dropout, bidirectional=False).cuda()

                 self.init_weights()

    def init_weights(self):
        init.orthogonal(self.lstm.weight_ih_l0)

        init.uniform(self.lstm.weight_hh_l0, a=-0.01, b=0.01)

        embedding_weights = torch.FloatTensor(self.vocab_size, self.input_size).cuda()
        init.uniform(embedding_weights, a = -0.25, b= 0.25)

        id_to_vec, emb_dim = create_id_to_vec('/data/train_shuffled_onethousand.csv','/data/glove.6B.100d.txt')

        for id, vec in id_to_vec.items():
            embedding_weights[id] = vec

        del self.embedding.weight
        self.embedding.weight = nn.Parameter(embedding_weights)
        self.embedding.weight.requires_grad = True

        #self.embedding.weight.data.copy_(torch.from_numpy(self.embedding_weights))

    def forward(self, inputs):
        embeddings = self.embedding(inputs)
        outputs, hiddens = self.lstm(embeddings)
        return outputs, hiddens

#%%

class DualEncoder(nn.Module):

    def __init__(self, encoder):
         super(DualEncoder, self).__init__()
         self.encoder = encoder
         self.number_of_layers = 1
         #h_0 (num_layers * num_directions, batch, hidden_size):
         #tensor containing the initial hidden state for each element in the batch.
         #dual_hidden_size = self.encoder.hidden_size * self.encoder.num_directions

         M = torch.FloatTensor(self.encoder.hidden_size, self.encoder.hidden_size).cuda()

         init.normal(M)

         self.M = nn.Parameter(M, requires_grad = True)

    def forward(self, contexts, responses):
        #output (seq_len, batch, hidden_size * num_directions):
        #tensor containing the output features (h_t) from the last layer
        #of the RNN, for each t.

        #h_n (num_layers * num_directions, batch, hidden_size):
        #tensor containing the hidden state for t=seq_len
        context_out, context_hn = self.encoder(contexts)

        response_out, response_hn = self.encoder(responses)

        scores_list = []

        y_preds = None

        for e in range(999):
            context_h = context_out[e][-1].view(1, self.encoder.hidden_size)
            response_h = response_out[e][-1].view(self.encoder.hidden_size,1)

            dot_var = torch.mm(torch.mm(context_h, self.M), response_h)[0][0]

            dot_tensor = dot_var.data
            dot_tensor.cuda()

            score = torch.sigmoid(dot_tensor)
            scores_list.append(score)

        y_preds_tensor = torch.stack(scores_list).cuda()
        y_preds = autograd.Variable(y_preds_tensor).cuda()

        return y_preds

#%% TRAINING

torch.backends.cudnn.enabled = False
#%%
vocab = create_vocab('/data/train_shuffled_onethousand.csv')
vocab_len = len(vocab)
emb_dim = get_emb_dim('/data/glove.6B.100d.txt')
#%%

encoder_model = Encoder(
        input_size = emb_dim,
        hidden_size = 300,
        vocab_size = vocab_len)

encoder_model.cuda()
#%%
dual_encoder = DualEncoder(encoder_model)

dual_encoder.cuda()
#%%
loss_func = torch.nn.BCELoss()

loss_func.cuda()

learning_rate = 0.001
epochs = 100
#batch_size = 50

optimizer = optim.Adam(dual_encoder.parameters(),
                       lr = learning_rate)
#%%
for i in range(epochs):

    context_matrix, response_matrix, y = make_matrices('/data/train_shuffled_onethousand.csv')

    context_matrix = autograd.Variable(context_matrix, requires_grad=True).cuda()

    response_matrix = autograd.Variable(response_matrix, requires_grad=True).cuda()

    y_label = y.cuda()

    y_preds = dual_encoder(context_matrix, response_matrix)

    loss = loss_func(y_preds, y_label)

    if i % 10 == 0:
        print("Epoch: ", i, ", Loss: ", loss.data[0])

    #evaluation metrics...

    dual_encoder.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm(dual_encoder.parameters(), 10)

    optimizer.step()

The following error occurs:

2018-01-06 06:07:02,148 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,148 INFO - File "all_scripts.py", line 258, in forward
2018-01-06 06:07:02,148 INFO - context_out, context_hn = self.encoder(contexts)
2018-01-06 06:07:02,149 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
2018-01-06 06:07:02,149 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,149 INFO - File "all_scripts.py", line 229, in forward
2018-01-06 06:07:02,149 INFO - embeddings = self.embedding(inputs)
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
2018-01-06 06:07:02,150 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 103, in forward
2018-01-06 06:07:02,150 INFO - self.scale_grad_by_freq, self.sparse
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 40, in forward
2018-01-06 06:07:02,151 INFO - assert not ctx.needs_input_grad[0], "Embedding doesn't " \
2018-01-06 06:07:02,151 INFO - AssertionError: Embedding doesn't compute the gradient w.r.t. the indices

I do understand why the problem occurs (surely it makes no sense to compute the gradient w.r.t. the indices).
But I do not understand how to adjust the code so that it computes the gradients w.r.t. the content values of the embedding vectors.

All help highly appreciated!

(Also see the thread in the PyTorch forum)

Output: After some extensive adjustments, the code works now. The problem was not only the embedding initialization. See my github repo for the improved code.


Instruction: What's the meaning of function eval() in torch.nn module

Input: Official comment shows that "This has any effect only on modules such as Dropout or BatchNorm." But I don't understand its implementation.

Output:
Dropout and BatchNorm (and maybe some custom modules) behave differently during training and evaluation. You must let the model know when to switch to eval mode by calling .eval() on the model.

This sets self.training to False for every module in the model. If you are implementing your own module that must behave differently during training and evaluation, you can check the value of self.training while doing so.


Instruction: Cropping a minibatch of images in Pytorch -- each image differently

Input:
I have a tensor named input with dimensions 64x21x21. It is a minibatch of 64 images, each 21x21 pixels. I'd like to crop each image down to 11x11 pixels. So the output tensor I want would have dimensions 64x11x11.

I'd like to crop each image around a different "center pixel." The center pixels are given by a 2-dimensional long tensor named center with dimensions 64x2. For image i, center[i][0] gives the row index and center[i][1] gives the column index for the pixel that should be at the center in the output. We can assume that the center pixel is always at least 5 pixels away from the border.

Is there an efficient way to do this in pytorch (on the gpu)?

UPDATE: Let me clarify that the center tensor is formed by a deep neural network. It acts as a "hard attention mechanism," to use the reinforcement learning term for it. After I "crop" an image, that subimage becomes the input to another neural network. That's why I want to do the cropping in Pytorch: because the operations before and after the cropping are in Pytorch. I'd like to avoid having to transfer anything from the GPU back to the CPU.

Output:
I raised the question over on the pytorch forums, and got an answer there from smth. The grid_sample function should totally solve the problem.

https://discuss.pytorch.org/t/cropping-a-minibatch-of-images-each-image-a-bit-differently/12247


Instruction: save predictions from pytorch model

Input:
Im following the pytorch transfer learning tutorial and applying it to the kaggle seed classification task,Im just not sure how to save the predictions in a csv file so that i can make the submission,
Any suggestion would be helpful,This is what i have ,

use_gpu = torch.cuda.is_available()
 model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

num_ftrs = model.fc.in_features
model.fc = torch.nn.Linear(num_ftrs, len(classes))
if use_gpu:
    model = model.cuda()

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

loaders = {'train':train_loader, 'valid':valid_loader, 'test': test_loader}

model = train_model(loaders, model, criterion, optimizer, exp_lr_scheduler, num_epochs=50)

Output:
Once you have trained your model, you can evaluate it on your testing data. This gives you a Variable, probably on the GPU. From there, you'll want to copy its tensor to the CPU with cpu() and convert it into a numpy array with numpy(). You can then use numpy's CSV functionality or use e.g. pandas' DataFrame.to_csv. In the first case, you'd have something like this:

# evaluate on Variable x with testing data
y = model(x)
# access Variable's tensor, copy back to CPU, convert to numpy
arr = y.data.cpu().numpy()
# write CSV
np.savetxt('output.csv', arr)


Instruction: Out of memory error during evaluation but training works fine

Input:
I have recently upgraded PyTorch from 0.2 to 0.3. Surprisingly my old programs are throwing an out of memory error during evaluation (in eval() mode) but training works just fine. I am using the same batch size for training and evaluation. I am totally clueless what is happening? Did anyone face similar issue? Is there any possible solution?

I tried using volatile=True param on the variables and it didn't help. Please note, I am not doing anything special to use cuDNN. I am using the default setting.

def validate(self, dev_corpus):
    # Turn on evaluation mode which disables dropout.
    self.model.eval()

    dev_batches = helper.batchify(dev_corpus.data, self.config.batch_size)
    print('number of dev batches = ', len(dev_batches))

    dev_loss = 0
    num_batches = len(dev_batches)
    for batch_no in range(1, num_batches + 1):
        session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels = helper.session_to_tensor(
            dev_batches[batch_no - 1], self.dictionary)
        if self.config.cuda:
            session_queries = session_queries.cuda()
            session_query_length = session_query_length.cuda()
            rel_docs = rel_docs.cuda()
            rel_docs_length = rel_docs_length.cuda()
            doc_labels = doc_labels.cuda()

        loss = self.model(session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels)
        if loss.size(0) > 1:
            loss = loss.mean()
        dev_loss += loss.data[0]

    return dev_loss / num_batches

I am using the above function for evaluation. Here, session_queries, session_query_length, .... rest variables are created by enabling volatile=True.

Please help!!

Output:
I think it fails during validation because the volatile flag is now deprecated and has no effect. Starting from 0.4.0, to avoid the gradient being computed for all variables during validation, you should use a context manager. Code example:

with torch.no_grad():
    # Your validation code

With that, the operation history and gradients are not stored. Thus this will save memory.

Refer to 0.4.0 migration guide for more details.

Furthermore, you could delete references to those variables after you finished the evaluation like this:

someVar = Variable(someVar)
del someVar


Instruction: Problems of Pytorch installation on Ubuntu 17.10 (GPU)

Input:
I would like to use PyTorch and its GPU computations on my computer.

I have a computer running with Ubuntu 17.10. The computer (Alienware m17x) has two graphic cards:

An integrated Intel Ivybridge Mobile
A Nvidia Geforce 675M.

In order to install PyTorch, I followed the instructions on the PyTorch website pytorch.org

1) I installed CUDA 9 with the deb file: https://developer.nvidia.com/cuda-downloads

=> Linux/x86_64/Ubuntu/17.04/deb (local)

2) I installed Pytorch using the conda command line: conda install pytorch torchvision cuda90 -c pytorch

None of these two steps returned me any type of errors.

I restarted my computer. Apparently the two cards are detected:

$ lspci | grep -i vga
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core    processor Graphics Controller (rev 09)
01:00.0 VGA compatible controller: NVIDIA Corporation GF114M [GeForce GTX 675M] (rev a1)

But apparently there is something wrong with the drivers or CUDA itself. nvidia-detector does not return me anything:

$ nvidia-detector
none

And pytorch can not use cuda:

[1]: import torch
In [2]: torch.cuda.is_available()
Out[2]: False

Could you help me? I can provide additional informations if necessary, but I am not sure what could be relevant.

Output:
You do not need to install cuda to use a GPU with pytorch, if you install pytorch like this: pytorch binaries include all the necessary cuda libraries.

Therefore it also does not matter which cuda version flavour you choose when installing pytorch. Usually one will probably want the latest version, but in cases where an old GPU needs to get used, the pytorch binary that comes with the older cuda version may be the only one still supporting that GPU.

If no GPU is detected then this is probably not related to the CUDA library but to your kernel driver. Make sure that your system has the latest tested NVIDIA proprietary kernel driver installed.

What is maybe a bit confusing is that one can install pytorch binaries with cuda support on any system, including one without a GPU or with a GPU but without the system driver installed. This works fine until you try to actually use the GPU and invoke .cuda()


Instruction: Pytorch module error in Jupyter Notebook

Input:
I installed pytorch using conda command when the virtual env was activated.

But, there are some problems when I import torch modules in Jupyter Notebook.

I checked the sys.path both in the prompt and in Jupyter Notebook.

Well.. in the prompt, the result of sys.path is

['', '/home/usrname/anaconda3/lib/python36.zip',
     '/home/usrname/anaconda3/lib/python3.6',
     '/home/usrname/anaconda3/lib/python3.6/lib-dynload',
     '/home/usrname/anaconda3/lib/python3.6/site-packages']

and there are no errors when I import torch modules.

But, in the jupyter notebook(executed in chrome), the sys.path is

['',
 '/home/usrname/anaconda3/lib/python36.zip',
 '/home/usrname/anaconda3/lib/python3.6',
 '/home/usrname/anaconda3/lib/python3.6/lib-dynload',
 '/home/usrname/anaconda3/lib/python3.6/site-packages',
 '/home/usrname/anaconda3/lib/python3.6/site-packages/IPython/extensions',
 '/home/usrname/.ipython']

and I see an error: No module named 'torch'

I can't solve this problem...

Output:
I had the same issues with this but managed to solve the problem. I think PyTorch has an 'issue' with the regular Jupyter application in Anaconda, so I urge you to first install the numpy jupyter notebook that supports PyTorch, then you can launch you notebook again.
Use (while in your virtual env path):

conda install numpy jupyter notebook

Hope this helps.


Instruction: Invalid syntax error when passing a list of modules in Pytorch

Input:
I have two blocks in my deep models which are defined as follows:

def make_conv_bn_relu(in_channels, out_channels, kernel_size=3, stride=1, padding=1):
    return [
        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=False),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True),
    ]

def make_conv_relu(in_channels, out_channels, kernel_size=3, stride=1, padding=1):
    return [
        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=True),
        nn.ReLU(inplace=True),
    ]

Now, I want to pass it in nn.Sequential.

self.down1 = nn.Sequential(*make_conv_bn_relu(in_channels, 16, kernel_size=3, stride=1, padding=1 ), *make_conv_bn_relu(16, 32, kernel_size=3, stride=2, padding=1 ),)

But I am getting the following error:

Traceback (most recent call last):
  File "train_unet.py", line 17, in <module>
    from net.model.unet1 import UNet256_3x3 as Net
  File "/home/avijit.d/Kaggle/Pytorch/source/dummy-01/net/model/unet1.py", line 40
    self.down1 = nn.Sequential(*make_conv_bn_relu(in_channels, 16, kernel_size=3, stride=1, padding=1 ), *make_conv_bn_relu(16, 32, kernel_size=3, stride=2, padding=1 ),)
                                                                                ^
SyntaxError: invalid syntax

How to get rid of this? I am using Python 2.7

Output:
You can't use multiple unpacks in python2. But if you really want to use it then just concatenate lists:

nn.Squential(*(make_foo() + make_bar()))


Instruction: Pytorch network parameter calculation

Input:
Can someone tell me please about how the network parameter (10) is calculated? Thanks in advance.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(x.size()[0], -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
print(net)
print(len(list(net.parameters())))

Output:

Net(
  (conv1): Conv2d (1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d (6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120)
  (fc2): Linear(in_features=120, out_features=84)
  (fc3): Linear(in_features=84, out_features=10)
)
10

Best,
Zack

Output:
Most layer modules in PyTorch (e.g. Linear, Conv2d, etc.) group parameters into specific categories, such as weights and biases. Each of the five layer instances in your network has a "weight" and a "bias" parameter. This is why "10" is printed.

Of course, all of these "weight" and "bias" fields contain many parameters. For example, your first fully connected layer self.fc1 contains 16 * 5 * 5 * 120 = 48000 parameters. So len(params) doesn't tell you the number of parameters in the network--it gives you just the total number of "groupings" of parameters in the network.


Instruction: pytorch: can't load CNN model and do prediction TypeError: 'collections.OrderedDict' object is not callable

Input:
I trained a CNN model using MNIST dataset and now want to predict a classification of the image, which contains a number 3.

But when I tried to use this CNN to predict, pytorch gives me this error:

TypeError: 'collections.OrderedDict' object is not callable

And here's what I write:

cnn = torch.load("/usr/prakt/w153/Desktop/score_detector.pkl")
img = scipy.ndimage.imread("/usr/prakt/w153/Desktop/resize_num_three.png")
test_x = Variable(torch.unsqueeze(torch.FloatTensor(img), dim=1), volatile=True).type(torch.FloatTensor).cuda()
test_output, last_layer = cnn(test_x)
pred = torch.max(test_output, 1)[1].cuda().data.squeeze()
print(pred)

here's some explaination:
img is the to be predicted image with size 28*28 score_detector.pkl is the trained CNN model

any help will be appreciated!

Output:
Indeed, you are loading a state_dict rather than the model itself.

Saving the model is as follows:

torch.save(model.state_dict(), 'model_state.pth')

Whereas to load the model state you first need to init the model and then load the state

model = Model()
model.load_state_dict(torch.load('model_state.pth'))

If you trained your model on GPU but would like to load the model on a laptop which doesn't have CUDA, then you would need to add one more argument

model.load_state_dict(torch.load('model_state.pth', map_location='cpu'))


Instruction: coding patterns for efficient minibatch loop using custom pytorch dataset

Input:
Are there any general recommendations to handle data efficiently in a custom Dataset so that it plays nicely with the minibatch eval/train loop? To illustrate what I mean more concretely, let's say I define this synthetic toy dataset that maps x to x+1:

import torch.utils.data as data

class Dataset(data.Dataset):
    def __init__(self):
        super(Dataset, self).__init__()
        # list of [x, y]
        self.dataset = [
            [1, 2],
            [2, 3],
            [3, 4],
            [4, 5]
        ]

    def __getitem__(self, index):
        item = self.dataset[index]
        return item[0], item[1]

    def __len__(self):
        return len(self.dataset)

In practice, this would be wrapped in a DataLoader and accessed inside an eval/train loop, something like this:

dataset = Dataset()
data_loader = data.DataLoader(dataset=dataset, batch_size=2, shuffle=True)
epochs = 100
for i_epoch in range(epochs):
    for i_minibatch, minibatch in enumerate(data_loader):
        x, y = minibatch
        # predict and train

The dataset object might return primitive Python objects like numbers or lists, like in my example implementation, but in the "predict and train" part of the last code snippet, we need some specific data types to compute stuff, like a torch.FloatTensor (it seems the data loader can do this implicitly), probably even wrapped as a torch.autograd.Variable, and some calls to .cuda() might also be necessary. My question is about general advice for when to make these data transformations and function calls.

For example, one option would be to have everything already saved as a torch.FloatTensor inside the dataset, and in the data_loader loop we could add the Variable wrapper and call .cuda(). We could also have all or part of the data on the GPU already by calling .cuda() in the Dataset constructor or in the getitem method. I think there might be pros and cons to all of these approaches. If I'm training a model for several epochs, I don't want to introduce unnecessary overhead each epoch or minibatch iteration that could have been avoided by precomputing stuff in the dataset. Probably someone with more knowledge about the internals of pytorch (maybe related to some caching or jit compilation happening under the hood) might be able to point to more specific reasons to choose one approach over another.

Output:
Generally, datasets are stored in files in a format that is more friendly to storage on disk. When you load the dataset, you want the datatypes to be more friendly to PyTorch. This is accomplished by transformations interface of torchvision library. For example, for MNIST below is standard transformations:

datasets.MNIST('../data', train=True, download=True,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ])

Here ToTensor divides all values in tensor by 255 so that if the data is RGB image then values in tensor would be 0.0 to 1.0.

The key thing is here is that your data in disk should ideally be agnostic of what you might want to do with it (training, visualization, calculate stats etc) as well as agnostic of frameworks being used. You apply transformations after you load data that are relevant to stuff you are doing.

One other thing I want to mention is handling very large datasets like ImageNet. There are few important things:

You should avoid using separate image files as your dataset because this doesn't work well in clusters. Instead you can pack all files in format like LMDB or uncompressed zip (use Python ZipFile module) and then only access these files sequentially. The random access in large files will slow you down tremendously.
You should avoid using shuffle option in DataLoader class for large datasets. If you do that then again you are accessing large file with random access and performance will tank. Instead, what you can do is to sequentially read K = C * total_epochs * batch_size records where C is some constant of your choice >= 1. Then shuffle K records in memory and then divide them in batches. Unfortunately you have to do this manually for now.


Instruction: Getting CUDA out of memory

Input:
Im trying to train a network but i get,
I set my batch-size as 300 and i get this error,but even if i reduce this to 100 i still get this error,and more frustratingly for running 10 epoch on ~1200 images it takes about 40 minutes,any suggestions what is going wrong and how may i speed the process!
Any tips will be extremely helpful,Thanks in advance.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-3b43ff4eea72> in <module>()
      5         labels = Variable(labels).cuda()
      6
----> 7         optimizer.zero_grad()
      8         outputs = cnn(images)
      9         loss = criterion(outputs, labels)

/usr/local/lib/python3.5/dist-packages/torch/optim/optimizer.py in     zero_grad(self)
    114                 if p.grad is not None:
    115                     if p.grad.volatile:
--> 116                         p.grad.data.zero_()
    117                     else:
    118                         data = p.grad.data

RuntimeError: cuda runtime error (2) : out of memory at /pytorch  /torch/lib/THC/generic/THCTensorMath.cu:35`

Even though my GPU's are free

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.111                Driver Version: 384.111                       |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                   N/A |
| 23%   18C    P8    15W / 250W |  10864MiB / 11172MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+

|   1  GeForce GTX 108...  Off  | 00000000:08:00.0 Off |                  N/A |
 | 23%   20C    P8    15W / 250W |     10MiB / 11172MiB |          0%      Default
+-------------------------------+----------------------+---------------

Output:
Fairly general question. Here is how I would think on this problem.

Try to set batch size (number of batches) to 1. If this fixed the problem you may try to find optimal batch size.

If even for bs=1 you get "RuntimeError: cuda runtime error (2) : out of memory" :

Do not use linear layers that are too large.
A linear layer nn.Linear(m, n) uses O(nm)O(nm)O(nm) memory: that is to say, the memory requirements of the weights scales quadratically with the number of features considering also the gradients.
Do not accumulate history across your training loop.
If you sum the loss recursively inside a loop 10000 or more your back-propagation evaluation will be huge; taking lot of memory.
Delete tensors you don't need with del explicitly.
Run ps -elf | grep python and python processes on your GPU kill -9 [pid] if you have doubts some other Python process is eating your memory.


Instruction: PyTorch memory model: "torch.from_numpy()" vs "torch.Tensor()"

Input:
I'm trying to have an in-depth understanding of how PyTorch Tensor memory model works.

# input numpy array
In [91]: arr = np.arange(10, dtype=float32).reshape(5, 2)

# input tensors in two different ways
In [92]: t1, t2 = torch.Tensor(arr), torch.from_numpy(arr)

# their types
In [93]: type(arr), type(t1), type(t2)
Out[93]: (numpy.ndarray, torch.FloatTensor, torch.FloatTensor)

# ndarray
In [94]: arr
Out[94]:
array([[ 0.,  1.],
       [ 2.,  3.],
       [ 4.,  5.],
       [ 6.,  7.],
       [ 8.,  9.]], dtype=float32)

I know that PyTorch tensors share the memory buffer of NumPy ndarrays. Thus, changing one will be reflected in the other. So, here I'm slicing and updating some values in the Tensor t2

In [98]: t2[:, 1] = 23.0

And as expected, it's updated in t2 and arr since they share the same memory buffer.

In [99]: t2
Out[99]:

  0  23
  2  23
  4  23
  6  23
  8  23
[torch.FloatTensor of size 5x2]

In [101]: arr
Out[101]:
array([[  0.,  23.],
       [  2.,  23.],
       [  4.,  23.],
       [  6.,  23.],
       [  8.,  23.]], dtype=float32)

But, t1 is also updated. Remember that t1 was constructed using torch.Tensor() whereas t2 was constructed using torch.from_numpy()

In [100]: t1
Out[100]:

  0  23
  2  23
  4  23
  6  23
  8  23
[torch.FloatTensor of size 5x2]

So, no matter whether we use torch.from_numpy() or torch.Tensor() to construct a tensor from an ndarray, all such tensors and ndarrays share the same memory buffer.

Based on this understanding, my question is why does a dedicated function torch.from_numpy() exists when simply torch.Tensor() can do the job?

I looked at the PyTorch documentation but it doesn't mention anything about this? Any ideas/suggestions?

Output:
from_numpy() automatically inherits input array dtype. On the other hand, torch.Tensor is an alias for torch.FloatTensor.

Therefore, if you pass int64 array to torch.Tensor, output tensor is float tensor and they wouldn't share the storage. torch.from_numpy gives you torch.LongTensor as expected.

a = np.arange(10)
ft = torch.Tensor(a)  # same as torch.FloatTensor
it = torch.from_numpy(a)

a.dtype  # == dtype('int64')
ft.dtype  # == torch.float32
it.dtype  # == torch.int64


Instruction: In google-cloud-ml, setup.py error during setting up PYTORCH

Input:
i write setup.py like below to setup pytorch in google-cloud-ml engine

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['torchvision']
DEPENDENCY_LINKS =['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl']
setup(
    name='trainer',
    version='0.1',
    dependency_links=DEPENDENCY_LINKS,
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My pytorch trainer application package.'
)

error message

"Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_ZQ7aQ/torch/"

i don't know what it happen....
when i searched about that problem... the answers are just upgrade the setuptools... but i don't know how to upgrade setuptools in ml-engine...

so please help me...!
i want to run pytorch code in ml engine

Output:
Seems like DEPENDENCY_LINKS has been ignored by pip.
Instead, I copied the whl file to a GCS bucket, and used the flag '--package gs://my-bucket/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl' in gcloud to install the whl file before executing 'pip install torchvision' and it worked.

You also need to remove DEPENDENCY_LINKS from setup.py

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['torchvision']
setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My pytorch trainer application package.'
)


Instruction: In PyTorch how are layer weights and biases initialized by default?

Input:
I was wondering how are layer weights and biases initialized by default? E.g. if I create the linear layer
     torch.nn.Linear(5,100)
How are weights and biases for this layer initialized by default?

Output:
PyTorch 1.0

Most layers are initialized using Kaiming Uniform method. Example layers include Linear, Conv2d, RNN etc. If you are using other layers, you should look up that layer on this doc. If it says weights are initialized using U(...) then its Kaiming Uniform method. Bias is initialized using LeCunn init, i.e., uniform(-std, std) where standard deviation std is 1/sqrt(fan_in) (code).

PyTorch 0.4.1, 0.3.1

Weights and biases are initialized using LeCunn init (see sec 4.6) for conv layers (code: 0.3.1, 0.4.1).

If you want to override default initialization then see this answer.


Instruction: Is NVIDIA GeForce GT 635M suitable for Deep Learning?

Input: My GPU model is NVIDIA GeForce GT 635M and on NVIDIA site it is said that this GPU  is CUDA-enabled. Can I use TensorFlow or PyTorch or any other kind of Deep Learning platforms with this GPU or it is not suitable?

Output:
Any GPU can be used for Deep Learning training. You can even train only on your CPU. This is mostly true for using TensorFlow and PyTorch as well, but there are caveats as discussed later.

As for their suitability for the task, it is true that some GPUs are better than others, for various reasons. One would clearly be just the performance aspect - the more powerful the GPU, the quicker the training. Another is the CUDA capability of the given GPU, which you have mentioned in your question. The GPU you mention, NVIDIA GeForce GT 653M, is CUDA enabled, but has the Compute Capability of only 2.1 which is on the low end. This also means that you won't be able to use TensorFlow or PyTorch because they require >= 3.0 Compute Capability (thanks to janneb for pointing that out). You will still be able to use them, but not with your GPU, only with CPU.

It is not a bad GPU to train on, mainly if you are just beginning with Deep Learning, but it is not the best either. You do not need to worry about not being able to use some software or not being able to train some networks because of that though.


Instruction: Very weird behaviour when running the same deep learning code in two different GPUs

Input:
I am training networks using pytorch framework. I had K40 GPU in my computer. Last week, I added 1080 to the same computer.

In my first experiment, I observed identical results in both GPU. Then, I tried my second code on both GPUs. In this case, I "constantly" got good results in K40 while getting "constantly" awful results in 1080 for "exactly the same code".

First, I thought the only reason for getting such diverse outputs would be the random seeds in the codes. So, I fixed the seeds like this:

torch.manual_seed(3)
torch.cuda.manual_seed_all(3)
numpy.random.seed(3)

But, this did not solve the issue. I believe issue cannot be randomness because I was "constantly" getting good results in K40 and "constantly" getting bad results in 1080. Moreover, I tried exactly the same code in 2 other computers and 4 other 1080 GPUs and always achieved good results. So, problem has to be about the 1080 I recently plugged in.

I suspect problem might be about driver, or the way I installed pytorch. But, it is still weird that I only get bad results for "some" of the experiments. For the other experiments, I had the identical results.

Can anyone help me on this?

Output:
I had the same problem. I solved the problem by simply changing

sum

to

torch.sum

. Please try to change all the build-in functions to GPU one.


Instruction: Unable to Learn XOR Representation using 2 layers of Multi-Layered Perceptron (MLP)

Input:
Using PyTorch nn.Sequential model, I'm unable to learn all four representation of the XOR booleans:

import numpy as np

import torch
from torch import nn
from torch.autograd import Variable
from torch import FloatTensor
from torch import optim

use_cuda = torch.cuda.is_available()

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

# Converting the X to PyTorch-able data structure.
X_pt = Variable(FloatTensor(X))
X_pt = X_pt.cuda() if use_cuda else X_pt
# Converting the Y to PyTorch-able data structure.
Y_pt = Variable(FloatTensor(Y), requires_grad=False)
Y_pt = Y_pt.cuda() if use_cuda else Y_pt

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())
criterion = nn.L1Loss()
learning_rate = 0.03
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
num_epochs = 10000

for _ in range(num_epochs):
    predictions = model(X_pt)
    loss_this_epoch = criterion(predictions, Y_pt)
    loss_this_epoch.backward()
    optimizer.step()
    print([int(_pred > 0.5) for _pred in predictions], list(map(int, Y_pt)), loss_this_epoch.data[0])

After learning:

for _x, _y in zip(X_pt, Y_pt):
    prediction = model(_x)
    print('Input:\t', list(map(int, _x)))
    print('Pred:\t', int(prediction))
    print('Ouput:\t', int(_y))
    print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    0
Ouput:   1
######
Input:   [1, 1]
Pred:    0
Ouput:   0
######

I've tried running the same code over a couple of random seeds but it didn't manage to learn all for XOR representation.

Without PyTorch, I could easily train a model with self-defined derivative functions and manually perform the backpropagation, see https://www.kaggle.io/svf/2342536/635025ecf1de59b71ea4fa03eb84f9f9/results.html#After-some-enlightenment

Why is it that the 2-layered MLP using PyTorch didn't learn the XOR representation?

How is the model in PyTorch:

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

different from the one that is hand-written with the derivatives and the manually written backpropagation and optimizer step from https://www.kaggle.com/alvations/xor-with-mlp ?

Are the same the one hidden layered perceptron network?

Updated

Strangely, adding a nn.Sigmoid() between the nn.Linear layers didn't work:

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Sigmoid(),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())
criterion = nn.L1Loss()
learning_rate = 0.03
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
num_epochs = 10000

for _ in range(num_epochs):
    predictions = model(X_pt)
    loss_this_epoch = criterion(predictions, Y_pt)
    loss_this_epoch.backward()
    optimizer.step()

for _x, _y in zip(X_pt, Y_pt):
    prediction = model(_x)
    print('Input:\t', list(map(int, _x)))
    print('Pred:\t', int(prediction))
    print('Ouput:\t', int(_y))
    print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    1
Ouput:   1
######
Input:   [1, 1]
Pred:    1
Ouput:   0
######

But adding nn.ReLU() did:

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.ReLU(),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

...
for _x, _y in zip(X_pt, Y_pt):
prediction = model(_x)
print('Input:\t', list(map(int, _x)))
print('Pred:\t', int(prediction))
print('Ouput:\t', int(_y))
print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    1
Ouput:   1
######
Input:   [1, 1]
Pred:    1
Ouput:   0
######

Isn't a sigmoid enough for the non-linear activation?

I understand that the ReLU fits the task of boolean output but shouldn't the Sigmoid function produce the same/similar effect?

UPDATED 2

Running the same training 100 times:

from collections import Counter
import random
random.seed(100)

import torch
from torch import nn
from torch.autograd import Variable
from torch import FloatTensor
from torch import optim
use_cuda = torch.cuda.is_available()

all_results=[]

for _ in range(100):
    hidden_dim = 2

    model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                          nn.ReLU(), # Does the sigmoid has a build in biased?
                          nn.Linear(hidden_dim, output_dim),
                          nn.Sigmoid())

    criterion = nn.MSELoss()
    learning_rate = 0.03
    optimizer = optim.SGD(model.parameters(), lr=learning_rate)
    num_epochs = 3000

    for _ in range(num_epochs):
        predictions = model(X_pt)
        loss_this_epoch = criterion(predictions, Y_pt)
        loss_this_epoch.backward()
        optimizer.step()
        ##print([float(_pred) for _pred in predictions], list(map(int, Y_pt)), loss_this_epoch.data[0])

    x_pred = [int(model(_x)) for _x in X_pt]
    y_truth = list([int(_y[0]) for _y in Y_pt])
    all_results.append([x_pred == y_truth, x_pred, loss_this_epoch.data[0]])

tf, outputsss, losses__ = zip(*all_results)
print(Counter(tf))

It only managed to learn the XOR representation 18 out of 100 times... -_-|||

Output:
It's because nn.Linear has no activation built in, so your model is effectively a linear classifier, and XOR is the canonical example of a problem that can't be solved using linear classifiers.

Change this:

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

to that:

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Sigmoid(),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

and only then will your model be equivalent to the one from the linked Kaggle notebook.


Instruction: PyTorch: How to print output blob size of each layer in network?

Input:
I can print network structure like (also how to print positional index of each 'simple' layer? because in this example we have 3 for Fire module and it's content ('simple' layers) don't have index):

net = models.squeezenet1_1(pretrained=True)
print(net)

SqueezeNet(
  (features): Sequential(
    (0): Conv2d (3, 64, kernel_size=(3, 3), stride=(2, 2))
    (1): ReLU(inplace)
    (2): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (3): Fire(
      (squeeze): Conv2d (64, 16, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (16, 64, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (16, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (4): Fire(
      (squeeze): Conv2d (128, 16, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (16, 64, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (16, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (5): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (6): Fire(
      (squeeze): Conv2d (128, 32, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (32, 128, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (32, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (7): Fire(
      (squeeze): Conv2d (256, 32, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (32, 128, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (32, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (8): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (9): Fire(
      (squeeze): Conv2d (256, 48, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (48, 192, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (48, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (10): Fire(
      (squeeze): Conv2d (384, 48, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (48, 192, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (48, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (11): Fire(
      (squeeze): Conv2d (384, 64, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (64, 256, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (64, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (12): Fire(
      (squeeze): Conv2d (512, 64, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (64, 256, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (64, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
  )
  (classifier): Sequential(
    (0): Dropout(p=0.5)
    (1): Conv2d (512, 1000, kernel_size=(1, 1), stride=(1, 1))
    (2): ReLU(inplace)
    (3): AvgPool2d(kernel_size=13, stride=1, padding=0, ceil_mode=False, count_include_pad=True)
  )
)

And I can print weights size like:

for i, weights in enumerate(list(net.parameters())):
    print('i:',i,'weights:',weights.size())

i: 0 weights: torch.Size([64, 3, 3, 3])
i: 1 weights: torch.Size([64])
i: 2 weights: torch.Size([16, 64, 1, 1])
i: 3 weights: torch.Size([16])
i: 4 weights: torch.Size([64, 16, 1, 1])
i: 5 weights: torch.Size([64])
i: 6 weights: torch.Size([64, 16, 3, 3])
i: 7 weights: torch.Size([64])
i: 8 weights: torch.Size([16, 128, 1, 1])
i: 9 weights: torch.Size([16])
i: 10 weights: torch.Size([64, 16, 1, 1])
i: 11 weights: torch.Size([64])
i: 12 weights: torch.Size([64, 16, 3, 3])
i: 13 weights: torch.Size([64])
i: 14 weights: torch.Size([32, 128, 1, 1])
i: 15 weights: torch.Size([32])
i: 16 weights: torch.Size([128, 32, 1, 1])
i: 17 weights: torch.Size([128])
i: 18 weights: torch.Size([128, 32, 3, 3])
i: 19 weights: torch.Size([128])
i: 20 weights: torch.Size([32, 256, 1, 1])
i: 21 weights: torch.Size([32])
i: 22 weights: torch.Size([128, 32, 1, 1])
i: 23 weights: torch.Size([128])
i: 24 weights: torch.Size([128, 32, 3, 3])
i: 25 weights: torch.Size([128])
i: 26 weights: torch.Size([48, 256, 1, 1])
i: 27 weights: torch.Size([48])
i: 28 weights: torch.Size([192, 48, 1, 1])
i: 29 weights: torch.Size([192])
i: 30 weights: torch.Size([192, 48, 3, 3])
i: 31 weights: torch.Size([192])
i: 32 weights: torch.Size([48, 384, 1, 1])
i: 33 weights: torch.Size([48])
i: 34 weights: torch.Size([192, 48, 1, 1])
i: 35 weights: torch.Size([192])
i: 36 weights: torch.Size([192, 48, 3, 3])
i: 37 weights: torch.Size([192])
i: 38 weights: torch.Size([64, 384, 1, 1])
i: 39 weights: torch.Size([64])
i: 40 weights: torch.Size([256, 64, 1, 1])
i: 41 weights: torch.Size([256])
i: 42 weights: torch.Size([256, 64, 3, 3])
i: 43 weights: torch.Size([256])
i: 44 weights: torch.Size([64, 512, 1, 1])
i: 45 weights: torch.Size([64])
i: 46 weights: torch.Size([256, 64, 1, 1])
i: 47 weights: torch.Size([256])
i: 48 weights: torch.Size([256, 64, 3, 3])
i: 49 weights: torch.Size([256])
i: 50 weights: torch.Size([1000, 512, 1, 1])
i: 51 weights: torch.Size([1000])

How to print output blob size of each layer in network?

Output:
You can register a hook (callback function) which will print out shapes of input and output tensors like described in the manual: Forward and Backward Function Hooks
Example:

net.register_forward_hook(your_print_blobs_function)

After this you need to do one forward pass against some input tensor.

expected_image_shape = (3, 224, 224)
input_tensor = torch.autograd.Variable(torch.rand(1, *expected_image_shape))
# this call will invoke all registered forward hooks
output_tensor = net(input_tensor)


Instruction: Slicing a numpy 3-d matrix into 2-d matrix

Input:
I have a 3d numpy matrix t as follows, generated randomly:

t = np.random.rand(2,2,2)
array([[[ 0.80351862,  0.25631294],
        [ 0.7971346 ,  0.29468456]],

       [[ 0.33771957,  0.91776256],
        [ 0.6018604 ,  0.55290615]]])

I want to extract a 2-d matrix such that the result is sliced along the columns of the 3-d matrix. Something like:

array([[ 0.25631294  ,  0.91776256],
       [ 0.29468456,  0.55290615]])

How can I slice in such a way?

Thanks for the help.

Output:
That's just taking the last dim, with a transpose:

>>> t[:,:,1].T
array([[ 0.25631294,  0.91776256],
       [ 0.29468456,  0.55290615]])


Instruction: AttentionDecoderRNN without MAX_LENGTH

Input:
From the PyTorch Seq2Seq tutorial, http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder

We see that the attention mechanism is heavily reliant on the MAX_LENGTH parameter to determine the output dimensions of the attn -> attn_softmax -> attn_weights, i.e.

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

More specifically

self.attn = nn.Linear(self.hidden_size * 2, self.max_length)

I understand that the MAX_LENGTH variable is the mechanism to reduce the no. of parameters that needs to be trained in the AttentionDecoderRNN.

If we don't have a MAX_LENGTH pre-determined. What values should we initialize the attn layer with?

Would it be the output_size? If so, then that'll be learning the attention with respect to the full vocabulary in the target language. Isn't that the real intention of the Bahdanau (2015) attention paper?

Output: Attention modulates the input to the decoder. That is attention modulates the encoded sequence which is of the same length as the input sequence. Thus, MAX_LENGTH should be the maximum sequence length of all your input sequences.


Instruction: vgg16 in pytorch got size error

Input:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
from torchvision import datasets, transforms, models
import time
import argparse
import os

batch_size = 64

train_dataset = datasets.CIFAR10(root='./data/cifar10/',
                                 train=True,
                                 transform=transforms.ToTensor(),
                                 download=True)
test_dataset = datasets.CIFAR10(root='./data/cifar10/',
                                train=False,
                                transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1)
        )
        self.classifier = nn.Sequential(
            nn.Linear(25088, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return F.softmax(x)

model = Vgg16()

# print(model)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss().cuda()

if torch.cuda.device_count() > 0:
    # os.environ["CUDA_VISIBLE_DEVICES"]= '0'
    print("USE", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
else:
    print("USE ONLY CPU!")

if torch.cuda.is_available():
    model.cuda()

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        if torch.cuda.is_available():
            data, target = Variable(data.cuda()), Variable(target.cuda())
        else:
            data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        # loss = F.nll_loss(output, target)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        if torch.cuda.is_available():
            data, target = Variable(data.cuda(), volatile=True), Variable(target.cuda())
        else:
            data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data[0]
        pred = output.data.max(1, keepdim=True)[1]  # [0] : value, [1]: index
        correct += pred.eq(target.data.view_as(pred)).sum()
        test_loss /= len(test_loader.dataset)
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))

for epoch in range(0, 200):
    train(epoch)
    test()

When I run this codes.. this error occured.

  -> RuntimeError: size mismatch at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:243

When I print(model) in other vgg codes in pytorch, There is a 25088 input size of FC layer...
So, I tried to set this parameter 25088,There is size mismatch error.
When I change this input_size from 25088 to 512, there is no error but training is not works well.(Never changed loss in training process and Always have 10% accuracy in test process)
So I think this input size of FC layer is the problem.. What can I do in this situation?
Thanks in advance;

Output:
I identified the problem. When you define the classifier, you are defining a fully-connected layer nn.Linear(25088, 4096) while the output from the convolutional part after doing x = x.view(x.size(0), -1) is (batch_size, 512). To match the size of the output of the convolutional part with the beginning of the classifier, you should change the classifier definition to:.

self.classifier = nn.Sequential(
        nn.Linear(512, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, num_classes)
)

Like this, it should work.

Note that I put 128 out_features in each fully-connected but you can change those numbers as you prefer.

Also adding the BatchNorm layer after the convolution might help the training to converge

class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1)
        )
        self.classifier = nn.Linear(512, 10)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x


Instruction: Why VGG-16 takes input size 512 * 7 * 7?

Input:
According to https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py

I don`t understand why VGG models take 512 * 7 * 7 input_size of fully-connected layer.
Last convolution layer is

nn.Conv2d(512, 512, kernel_size=3, padding=1),
nn.ReLU(True),
nn.MaxPool2d(kernel_size=2, stride=2, dilation=1)

Codes in above link.

class VGG(nn.Module):

    def __init__(self, features, num_classes=1000, init_weights=True):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )

Output:
The VGG neural net has two sections of layers: the "feature" layer and the "classifier" layer. The input to the feature layer is always an image of size 224 x 224 pixels.

The feature layer has 5 nn.MaxPool2d(kernel_size=2, stride=2) convolutions. See referenced source code line 76: each 'M' character in the configurations sets up one MaxPool2d convolution.

A MaxPool2d convolution with these specific parameters reduces the tensor size in half. So we have 224 --> 112 --> 56 --> 28 --> 14 --> 7 which means that the output of the feature layer is a 512 channels * 7 * 7 tensor. This is the input to the "classifier" layer.


Instruction: Error in loading model in PyTorch

Input:
I Have the following code snippet

from train import predict
import random
import torch

ann=torch.load('ann.pt') #importing trained model

while True:
      k=raw_input("User:")
      intent,top_value,top_index = predict(str(k),ann)
      print(intent)

when I run the script it is throwing the error as below:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    ann=torch.load('ann.pt') #importing trained model
  File "/home/local/ZOHOCORP/raghav-5305/miniconda2/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/home/local/ZOHOCORP/raghav-5305/miniconda2/lib/python2.7/site-packages/torch/serialization.py", line 409, in _load
    result = unpickler.load()
AttributeError: 'module' object has no attribute 'ANN'

I have ann.pt file in the same folder as my script is.
Kindly help me identify fix the error and load the model.
Thanks in advance.

Output:
When trying to save both parameters and model, pytorch pickles the parameters but only store path the model Class. For instance, changing tree structure or refactoring can break loading.
Therefore as the documentation points out, it is not recommended, prefer only save/load parameters:

  ...the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors.

For more help, it'll be useful to show your saving code.


Instruction: CMake on Linux CentOS 7, how to force the system to use cmake3?

Input:
I tried to install PyTorch on my Linux CentOS 7.3. I downloaded its package, ran this command and got this error:

sudo python setup.py install

running install
running build_deps
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
  CMake 3.0 or higher is required.  You are running version 2.8.12.2

-- Configuring incomplete, errors occurred!

So I tried to install CMake 3 by using the command

sudo yum -y install cmake3

The installation went alright, but the system still uses cmake2.8 as default.
If I type the yum info comnmand, I get this:

sudo yum info cmake

Installed Packages
Name        : cmake
Arch        : x86_64
Version     : 2.8.12.2
Release     : 2.el7
Size        : 27 M
Repo        : installed
From repo   : base
Summary     : Cross-platform make system
URL         : http://www.cmake.org
License     : BSD and MIT and zlib
Description : CMake is used to control the software compilation process using simple
            : platform and compiler independent configuration files. CMake generates
            : native makefiles and workspaces that can be used in the compiler
            : environment of your choice. CMake is quite sophisticated: it is possible
            : to support complex environments requiring system configuration, preprocessor
            : generation, code generation, and template instantiation.

So, the problem is clear: the system still sees cmake2.8 as default, and therefore Python does not use cmake3 for its PyTorch installation.
How can I solve this problem?

Thanks

Output:
Once you have both the cmake and the cmake3 package installed on your machine, you can use update-alternatives to switch between both packages.
Use the alternatives command to register both installations:
$ sudo alternatives --install /usr/local/bin/cmake cmake /usr/bin/cmake 10 \
--slave /usr/local/bin/ctest ctest /usr/bin/ctest \
--slave /usr/local/bin/cpack cpack /usr/bin/cpack \
--slave /usr/local/bin/ccmake ccmake /usr/bin/ccmake \
--family cmake

$ sudo alternatives --install /usr/local/bin/cmake cmake /usr/bin/cmake3 20 \
--slave /usr/local/bin/ctest ctest /usr/bin/ctest3 \
--slave /usr/local/bin/cpack cpack /usr/bin/cpack3 \
--slave /usr/local/bin/ccmake ccmake /usr/bin/ccmake3 \
--family cmake

After these two commands, cmake3 will be invoked by default, when you enter cmake from a bash prompt or start a bash script. The commands also take care of registering a few secondary commands like ctest which need to be switched along with cmake.
If you need to switch back to cmake 2.8 as the default, run the following command:
$ sudo alternatives --config cmake

There are 2 programs which provide 'cmake'.

  Selection    Command
-----------------------------------------------
   1           cmake (/usr/bin/cmake)
*+ 2           cmake (/usr/bin/cmake3)

Enter to keep the current selection[+], or type selection number: 1


Instruction: How to find functions imported from torch._C in source code

Input:
I'm trying to track down the implementation of torch.nn.NLLLoss in the source code. I got as far as a call to torch._C.nll_loss in the function nll_loss in the file torch.nn.functional. But I can't find a place where _C is created.

Anyone have any info on this?

Output:
Take a look at A Tour of PyTorch Internals on the PyTorch blog. Relevant excerpt:

PyTorch defines a new package torch. In this post we will consider the ._C module. This module is known as an “extension module” - a Python module written in C. Such modules allow us to define new built-in object types (e.g. the Tensor) and to call C/C++ functions.
The ._C module is defined in torch/csrc/Module.cpp. The init_C() / PyInit__C() function creates the module and adds the method definitions as appropriate. This module is passed around to a number of different __init() functions that add further objects to the module, register new types, etc.

Part II to that post goes into detail about the build system. In the section on NN modules, it says

Briefly, let’s touch on the last part of the build_deps command: generate_nn_wrappers(). We bind into the backend libraries using PyTorch’s custom cwrap tooling, which we touched upon in a previous post. For binding TH and THC we manually write the YAML declarations for each function. However, due to the relative simplicity of the THNN and THCUNN libraries, we auto-generate both the cwrap declarations and the resulting C++ code.
The reason we copy the THNN.h and THCUNN.h header files into torch/lib is that this is where the generate_nn_wrappers() code expects these files to be located. generate_nn_wrappers() does a few things:

Parses the header files, generating cwrap YAML declarations and writing them to output .cwrap files
Calls cwrap with the appropriate plugins on these .cwrap files to generate source code for each
Parses the headers a second time to generate THNN_generic.h - a library that takes THPP Tensors, PyTorch’s “generic” C++ Tensor Library, and calls into the appropriate THNN/THCUNN library function based on the dynamic type of the Tensor

Perhaps not that helpful without the context, but I don't think I should copy the entire post here.
When I tried to track down the definition of NLLLoss without having read those posts, I ended up at aten/src/THNN/generic/ClassNLLCriterion.c, via aten/src/ATen/nn.yaml. The latter is probably the YAML the second post talks about, but I haven't checked.


Instruction: binary image digit convolution neural network has a very high confidence when nothing is in the image

Input:
I've been training my convolutional neural network on closely cropped printed digits, similar in style to the Mnist dataset. It works perfectly, with close to 100 percent on both the training and test data.

At first I trained it on  4 channel binary images where white was '255' and black was '0'. It has 10 outputs, one for each digit, which i then softmax in order to get a probability for each category.

However, I want to use a sliding window technique in order to analyse a page of digits. This is impossible because, for a fully white input, it returns almost %100 confidence that it is a 4, while pretty much nothing for everything else.

I thought it might be the fact the neural net was training on the white space in the image rather than the black space, as the black pixels had value 0, so I inverted the images and trained the network again.  again, it simply returns really  close to 100% confidence on a fully white image.

For both, it returns low percentages for each class when the image is completely black,as it should, although 4 is still the highest

I don't understand the intuition behind this, so any help would be great, even if you could just say its not usual behaviour. Is this to be expected? should i create another class for things that arent digits aswell and train it on that?

heres my neural network:
its fully convolutional so that i can implement fast sliding window with it, but the last convolutions are equivalent to fully connected layers

class fully_convolutional_1channel(nn.Module):
    def __init__(self):

        super(fully_convolutional_1channel, self).__init__()

        self.conv1 = nn.Conv2d(1, 6, 5)

        self.pool = nn.MaxPool2d(2, 2)

        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fconv1 = nn.Conv2d(16, 120, (4,2))

        self.fconv2 = nn.Conv2d(120, 84,  1)
        self.fconv3 = nn.Conv2d(84, 10, 1)

    def forward(self, x):

        x = self.conv1(x)

        x = F.relu(x)
        #relu does not change size

        x = self.pool(x)

        x = self.pool(F.relu(self.conv2(x)))

        #x = x.view(-1, 16 * 4 * 2)

        x = F.relu(self.fconv1(x))

        x = F.relu(self.fconv2(x))

        x = self.fconv3(x)

        #print(list(x.size))

        return x

Output:
  should i create another class for things that arent digits aswell and train it on that?

Yes. Your network has been trained to recognize digits, not white spaces. You could either re-train the network with this strategy or preprocess the page of digits to avoid feeding the network with white spaces.


Instruction: UserWarning: Implicit dimension choice for log_softmax has been deprecated

Input:
I´m using Mac OS el capitán and I am trying to follow the quick start tutorial for OpenNMT pytorch version.  In the training step I get the following warning message:

OpenNMT-py/onmt/modules/GlobalAttention.py:177: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.

align_vectors = self.sm(align.view(batch*targetL, sourceL))
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/modules/container.py:67: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)

Step 1: Preprocess data (works as expected)

python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo

Step 2: Train model (produces the warning message)

python train.py -data data/demo -save_model demo-model

Has anyone come across this warning or have any pointers to solve it?

Output:
It is almost always you will need the last dimension when you compute the cross-entropy so your line may look like:

torch.nn.functional.log_softmax(x, -1)


Instruction: why do we need to call `detach_` in `zero_grad`?

Input:
I am reading the source code of optimizer.zero_grad()

http://pytorch.org/docs/master/_modules/torch/optim/optimizer.html#Optimizer.zero_grad

def zero_grad(self):
    """Clears the gradients of all optimized :class:`Variable` s."""
    for group in self.param_groups:
        for p in group['params']:
            if p.grad is not None:
                p.grad.detach_()
                p.grad.zero_()

I wonder why detach_() is necessary? What does it even mean to detach a gradient variable instead of a normal variable?

Why isn't zero_() enough?

Output: You do not want previous gradient updates to affect future gradient updates so you are detaching the previous gradients from the graph.


Instruction: PyTorch and Convolutional Neural Networks

Input:
I have a image input 340px*340px and I want to classify it to 2 classes.
I want to create convolution neural network (PyTorch framework). I have a problem with input and output of layer.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 3 channels (RGB), kernel=5, but i don't understand why 6.
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        #why 16?
        self.conv2 = nn.Conv2d(6, 16, 5)
        #why 107584 = 328*328
        self.fc1 = nn.Linear(107584, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        # i dont understand this line
        x = x.view(x.size(0),  -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Is it network is correct?

Output:
  # 3 channels (RGB), kernel=5, but i don't understand why 6.

The second parameter of Conv2d is out_channels. In a convolutional layer you can arbitrarily define a number of out channels. So it's set to 6 because someone set it to 6.

  # why 16?

Same as above.

  #why 107584 = 328*328

and

  \ # i dont understand this line

Tensor.view() returns a new tensor with the same data as the self tensor but of a different size.
x = x.view(x.size(0),  -1): -1 means "infer from other dimensions" so, you are forcing the Tensor to be [1, 15*164*164] => [1, 403440].

403440 is also the correct value for self.fc1 = nn.Linear(107584, 120), instead of 107584.


Instruction: LSTM with Attention

Input:
I am trying to add attention mechanism to stacked LSTMs implementation https://github.com/salesforce/awd-lstm-lm

All examples online use encoder-decoder architecture, which I do not want to use (do I have to for the attention mechanism?).

Basically, I have used https://webcache.googleusercontent.com/search?q=cache:81Q7u36DRPIJ:https://github.com/zhedongzheng/finch/blob/master/nlp-models/pytorch/rnn_attn_text_clf.py+&cd=2&hl=en&ct=clnk&gl=uk

def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5, dropouth=0.5, dropouti=0.5, dropoute=0.1, wdrop=0, tie_weights=False):
    super(RNNModel, self).__init__()
    self.encoder = nn.Embedding(ntoken, ninp)
    self.rnns = [torch.nn.LSTM(ninp if l == 0 else nhid, nhid if l != nlayers - 1 else (ninp if tie_weights else nhid), 1, dropout=0) for l in range(nlayers)]
    for rnn in self.rnns:
        rnn.linear = WeightDrop(rnn.linear, ['weight'], dropout=wdrop)
    self.rnns = torch.nn.ModuleList(self.rnns)
    self.attn_fc = torch.nn.Linear(ninp, 1)
    self.decoder = nn.Linear(nhid, ntoken)

    self.init_weights()

def attention(self, rnn_out, state):
    state = torch.transpose(state, 1,2)
    weights = torch.bmm(rnn_out, state)# torch.bmm(rnn_out, state)
    weights = torch.nn.functional.softmax(weights)#.squeeze(2)).unsqueeze(2)
    rnn_out_t = torch.transpose(rnn_out, 1, 2)
    bmmed = torch.bmm(rnn_out_t, weights)
    bmmed = bmmed.squeeze(2)
    return bmmed

def forward(self, input, hidden, return_h=False, decoder=False, encoder_outputs=None):
    emb = embedded_dropout(self.encoder, input, dropout=self.dropoute if self.training else 0)
    emb = self.lockdrop(emb, self.dropouti)

    new_hidden = []
    raw_outputs = []
    outputs = []
    for l, rnn in enumerate(self.rnns):
        temp = []
        for item in emb:
            item = item.unsqueeze(0)
            raw_output, new_h = rnn(item, hidden[l])

            raw_output = self.attention(raw_output, new_h[0])

            temp.append(raw_output)
        raw_output = torch.stack(temp)
        raw_output = raw_output.squeeze(1)

        new_hidden.append(new_h)
        raw_outputs.append(raw_output)
        if l != self.nlayers - 1:
            raw_output = self.lockdrop(raw_output, self.dropouth)
            outputs.append(raw_output)
    hidden = new_hidden

    output = self.lockdrop(raw_output, self.dropout)
    outputs.append(output)

    outputs = torch.stack(outputs).squeeze(0)
    outputs = torch.transpose(outputs, 2,1)
    output = output.transpose(2,1)
    output = output.contiguous()
    decoded = self.decoder(output.view(output.size(0)*output.size(1), output.size(2)))
    result = decoded.view(output.size(0), output.size(1), decoded.size(1))
    if return_h:
        return result, hidden, raw_outputs, outputs
    return result, hidden

This model is training, but my loss is quite high as compared to the model without the attention model.

Output:
I understood your question but it is a bit tough to follow your code and find the reason why the loss is not decreasing. Also, it is not clear why you want to compare the last hidden state of the RNN with all the hidden states at every time step.

Please note, a particular trick/mechanism is useful if you use it in the correct way.
The way you are trying to use attention mechanism, I am not sure if it is the correct way. So, don't expect that since you are using attention trick in your model, you will get good results!! You should think, why attention mechanism will bring advantage to your desired task?

You didn't clearly mention what is that task you are targetting? Since you have pointed to a repo which contains code on language modeling, I am guessing the task is: given a sequence of tokens, predict the next token.

One possible problem I can see in your code is: in the for item in emb: loop, you will always use the embedddings as input to each LSTM layer, so having a stacked LSTM doesn't make sense to me.

Now, let me first answer your question and then show step-by-step how can you build your desired NN architecture.

  Do I need to use encoder-decoder architecture to use attention mechanism?

The encoder-decoder architecture is better known as sequence-to-sequence to learning and it is widely used in many generation task, for example, machine translation. The answer to your question is no, you are not required to use any specific neural network architecture to use attention mechanism.

The structure you presented in the figure is little ambiguous but should be easy to implement. Since your implementation is not clear to me, I am trying to guide you to a better way of implementing it. For the following discussion, I am assuming we are dealing with text inputs.

Let's say, we have an input of shape 16 x 10 where 16 is batch_size and 10 is seq_len. We can assume we have 16 sentences in a mini-batch and each sentence length is 10.

batch_size, vocab_size = 16, 100
mat = np.random.randint(vocab_size, size=(batch_size, 10))
input_var = Variable(torch.from_numpy(mat))

Here, 100 can be considered as the vocabulary size. It is important to note that throughout the example I am providing, I am assuming batch_size as the first dimension in all respective tensors/variables.

Now, let's embed the input variable.

embedding = nn.Embedding(100, 50)
embed = embedding(input_var)

After embedding, we got a variable of shape 16 x 10 x 50 where 50 is the embedding size.

Now, let's define a 2-layer unidirectional LSTM with 100 hidden units at each layer.

rnns = nn.ModuleList()
nlayers, input_size, hidden_size = 2, 50, 100
for i in range(nlayers):
    input_size = input_size if i == 0 else hidden_size
    rnns.append(nn.LSTM(input_size, hidden_size, 1, batch_first=True))

Then, we can feed our input to this 2-layer LSTM to get the output.

sent_variable = embed
outputs, hid = [], []
for i in range(nlayers):
    if i != 0:
        sent_variable = F.dropout(sent_variable, p=0.3, training=True)
    output, hidden = rnns[i](sent_variable)
    outputs.append(output)
    hid.append(hidden[0].squeeze(0))
    sent_variable = output

rnn_out = torch.cat(outputs, 2)
hid = torch.cat(hid, 1)

Now, you can simply use the hid to predict the next word. I would suggest you do that. Here, shape of hid is batch_size x (num_layers*hidden_size).

But since you want to use attention to compute soft alignment score between last hidden states with each hidden states produced by LSTM layers, let's do this.

sent_variable = embed
hid, con = [], []
for i in range(nlayers):
    if i != 0:
        sent_variable = F.dropout(sent_variable, p=0.3, training=True)
    output, hidden = rnns[i](sent_variable)
    sent_variable = output

    hidden = hidden[0].squeeze(0) # batch_size x hidden_size
    hid.append(hidden)
    weights = torch.bmm(output[:, 0:-1, :], hidden.unsqueeze(2)).squeeze(2)
    soft_weights = F.softmax(weights, 1)  # batch_size x seq_len
    context = torch.bmm(output[:, 0:-1, :].transpose(1, 2), soft_weights.unsqueeze(2)).squeeze(2)
    con.append(context)

hid, con = torch.cat(hid, 1), torch.cat(con, 1)
combined = torch.cat((hid, con), 1)

Here, we compute soft alignment score between the last state with all the states of each time step. Then we compute a context vector which is just a linear combination of all the hidden states. We combine them to form a single representation.

Please note, I have removed the last hidden states from output: output[:, 0:-1, :] since you are comparing with last hidden state itself.

The final combined representation stores the last hidden states and context vectors produced at each layer. You can directly use this representation to predict the next word.

Predicting the next word is straight-forward and as you are using a simple linear layer is just fine.

Edit: We can do the following to predict the next word.

decoder = nn.Linear(nlayers * hidden_size * 2, vocab_size)
dec_out = decoder(combined)

Here, the shape of dec_out is batch_size x vocab_size. Now, we can compute negative log-likelihood loss which will be used to backpropagate later.

Before computing the negative log-likelihood loss, we need to apply log_softmax to the output of the decoder.

dec_out = F.log_softmax(dec_out, 1)
target = np.random.randint(vocab_size, size=(batch_size))
target = Variable(torch.from_numpy(target))

And, we also defined the target which is required to compute the loss. See NLLLoss for details. So, now we can compute the loss as follows.

criterion = nn.NLLLoss()
loss = criterion(dec_out, target)
print(loss)

The printed loss value is:

Variable containing:
 4.6278
[torch.FloatTensor of size 1]

Hope the entire explanation helps you!!


Instruction: optim.lr_scheduler.ReduceLROnPlateau gives error value cannot be converted to type float without overflow: inf

Input:
I am using pytorch with this install command: pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.1-cp35-cp35m-linux_x86_64.whl.

I have a model that trains without issues, but when I add a learning rate scheduler, I get an error

My scheduler:

# In init
self.optimizer = optim.Adam(self.model.parameters(), lr=0.01)
self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    self.optimizer, 'min', factor=0.1, patience=5, verbose=True
)

# after each epoch
self.scheduler.step(loss)

The error:

...
my_project/.env/lib/python3.5/site-packages/torch/optim/lr_scheduler.py in <lambda>(a, best)
    330         if mode == 'min' and threshold_mode == 'rel':
    331             rel_epsilon = 1. - threshold
--> 332             self.is_better = lambda a, best: a < best * rel_epsilon
    333             self.mode_worse = float('Inf')
    334         elif mode == 'min' and threshold_mode == 'abs':

RuntimeError: value cannot be converted to type float without overflow: inf

Doc: http://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau
Related thread: https://discuss.pytorch.org/t/value-cannot-be-converted-to-type-double-without-overflow-inf/11752/7

Output:
I'm using gpu tensors, eg:

Variable(torch.from_numpy(X).type(torch.FloatTensor).cuda(), requires_grad=False)

If I cast it to the cpu like that, the error goes away

# after each epoch
self.scheduler.step(loss.cpu().data.numpy())

Still, I would like a cleaner solution.


Instruction: Differentiate gradients

Input:
Is there a way to differentiate gradients in PyTorch?

For example, I can do this in TensorFlow:

from pylab import *
import tensorflow as tf
tf.reset_default_graph()
sess = tf.InteractiveSession()

def gradient_descent( loss_fnc, w, max_its, lr):
    '''a gradient descent "RNN" '''
    for k in range(max_its):
        w = w - lr * tf.gradients( loss_fnc(w), w )[0]
    return w

lr = tf.Variable( 0.0, dtype=tf.float32)
w = tf.Variable( tf.zeros(10), dtype=tf.float32)
reg = tf.Variable( 1.0, dtype=tf.float32 )

def loss_fnc(w):
    return tf.reduce_sum((tf.ones(10) - w)**2) + reg * tf.reduce_sum( w**2 )

w_n = gradient_descent( loss_fnc, w, 10, lr )

sess.run( tf.initialize_all_variables())

# differentiate through the gradient_descent RNN with respnect to the initial weight
print(tf.gradients( w_n, w))

# differentiate through the gradient_descent RNN with respnect to the learning rate
print(tf.gradients( w_n, lr))

and the output is

[<tf.Tensor 'gradients_10/AddN_9:0' shape=(10,) dtype=float32>]

[<tf.Tensor 'gradients_11/AddN_9:0' shape=() dtype=float32>]

How would I do something similar in PyTorch?

Output:
You just need to use the function torch.autograd.grad it does exactly the same as tf.gradients.

So in pytorch this would be:

from torch.autograd import Variable, grad
import torch

def gradient_descent( loss_fnc, w, max_its, lr):
    '''a gradient descent "RNN" '''
    for k in range(max_its):
        w = w - lr * grad( loss_fnc(w), w )
    return w

lr = Variable(torch.zeros(1), , requires_grad=True)
w = Variable( torch.zeros(10), requires_grad=True)
reg = Variable( torch.ones(1) , requires_grad=True)

def loss_fnc(w):
    return torch.sum((Variable(torch.ones(10)) - w)**2) + reg * torch.sum( w**2 )

w_n = gradient_descent( loss_fnc, w, 10, lr )

# differentiate through the gradient_descent RNN with respnect to the initial weight
print(grad( w_n, w))

# differentiate through the gradient_descent RNN with respnect to the learning rate
print(grad( w_n, lr))


Instruction: Can I assign the coordinate that convolutional filter works on?

Input:
I do not want the conv kernel to slide every pixel in the image, I only need to to do conv operation on the pixels I assigned. For example, on a 224 x 224 image, normally the conv operation will slide all 224 x 224 = 50176 pixels. However, if I only want it to do conv on (15,67), (34, 90), (143,201)...., how can I do it?
BTW, I want these operations all run on GPU.
Thanks a lot!

Output:
If you have the kernel K (of shape H_k x W_k x C_in x C_out) and the image I (of shape MB x H x W x C_in) and you have a position where you want to apply the kernel h, w then you don't need to use a convolution, you can just apply the kernel directly with a broadcasting matrix multiplication:

Out = torch.matmul(I[:, h:h+H_k, w:w+W_k, None, :], K.unsqueeze(0))[:, :, :, 0, :]


Instruction: Using a pytorch model for inference

Input:
I am using the fastai library (fast.ai) to train an image classifier. The model created by fastai is actually a pytorch model.

type(model)
<class 'torch.nn.modules.container.Sequential'>

Now, I want to use this model from pytorch for inference. Here is my code so far:

torch.save(model,"./torch_model_v1")
the_model = torch.load("./torch_model_v1")
the_model.eval() # shows the entire network architecture

Based on the example shown here: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#sphx-glr-beginner-data-loading-tutorial-py, I understand that I need to write my own data loading class which will override some of the functions in the Dataset class. But what is not clear to me is the transformations that I need to apply at test time? In particular, how do I normalize the images at test time?

Another question: is my approach of saving and loading the model in pytorch fine? I read in the tutorial here: http://pytorch.org/docs/master/notes/serialization.html that the approach that I have used is not recommended. The reason is not clear though.

Output:
Just to clarify: the_model.eval() not only prints the architecture, but sets the model to evaluation mode.

  In particular, how do I normalize the images at test time?

It depends on the model you have. For instance, for torchvision modules, you have to normalize the inputs this way.

Regarding on how to save / load models, torch.save/torch.load "saves/loads an object to a disk file."

So, if you save the_model, it will save the entire model object, including its architecture definition and some other internal aspects. If  you save the_model.state_dict(), it will save a dictionary containing the model state (i.e. parameters and buffers) only. Saving the model can break the code in various ways, so the preferred method is to save and load only the model state. However, I'm not sure if fast.ai "model file" is actually a full model or the state of a model. You have to check this so you can correctly load it.


Instruction: Pytorch pretrained model (VGG-19) same image gives slightly different class score in the final FC layer

Input:
I am using pretrained vgg-19 from torch.vision module
I have the pre-processing of image data like below:

normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

preprocess = transforms.Compose([
   transforms.Scale(224),
   transforms.ToTensor(),
   normalize
])

The problem is if I pass an image like a tennis ball through the network and save all the 1000 class score from the final FC layer and pass the same image again after some time, the final FC layer i.e the class score is slightly changed. Although the image category detected by the network is correct. (its a tennis ball)

Is it normal to have the slightly different class score for the same image? I mean, does this suppose to happen? Or for the correctly implemented pre-trained model, the network should give you the exact same class score every time for the same image.

Output:
Possible reasons why the class score could be different:

You are using GPUs: The GPU evaluation is slightly stochastic, so if you are using GPUs in your feedforward evaluation, it could give slightly different scores.
You are using the model in training mode model.train() and didn't activate the evaluation mode using model.eval() and your model contains some stochastic parts like dropout or adding noise to input.  Then those stochastic parts will still be active.


Instruction: Titan XP vs Quadro P400 GPU in Pytorch

Input:
I gave the the two GPUs on my machine a try and I expected the Titan-XP to be faster than the Quadro-P400. However, both gave almost the same execution time.

I need to know if PyTorch will dynamically choose one GPU over another, or, I myself will have to specify which one PyTorch will use, during run-time.

Here is the code snippet used in the test:

import torch
import time

def do_something(gpu_device):
    torch.cuda.set_device(gpu_device)  # torch.cuda.set_device(device_num)
    print("current GPU device ", torch.cuda.current_device())
    strt = time.time()
    a = torch.randn(100000000).cuda()
    xx = time.time() - strt
    print("execution time, to create 1E8 random numbers, is ", xx)
    # print(a)
    # print(a + 2)

no_of_GPUs= torch.cuda.device_count()
print("how many GPUs are there:", no_of_GPUs)
for i  in range(0, no_of_GPUs):
    print(i, "th GPU is", torch.cuda.get_device_name(i))
    do_something(i)

Sample output:

how many GPUs are there: 2
0 th GPU is TITAN Xp COLLECTORS EDITION
current GPU device  0
execution time, to create 1E8 random numbers, is  5.527713775634766

1 th GPU is Quadro P400
current GPU device  1
execution time, to create 1E8 random numbers, is  5.511776685714722

Output:
Despite what you might believe, the lack of performance difference which you see is because the random number generation is being run on your host CPU not the GPU. If I modify your do_something routine like this:

def do_something(gpu_device, ongpu=False, N=100000000):
    torch.cuda.set_device(gpu_device)
    print("current GPU device ", torch.cuda.current_device())
    strt = time.time()
    if ongpu:
        a = torch.cuda.FloatTensor(N).normal_()
    else:
        a = torch.randn(N).cuda()
    print("execution time, to create 1E8 random no, is ", time.time() - strt)
    return a

and run it two ways, I get very different execution times:

In [4]: do_something(0)
current GPU device  0
execution time, to create 1E8 random no, is  7.736972808837891
Out[4]:

-9.3955e-01
-1.9721e-01
-1.1502e+00
     ......
-1.2428e+00
 3.1547e-01
-2.1870e+00
[torch.cuda.FloatTensor of size 100000000 (GPU 0)]

In [5]: do_something(0,True)
current GPU device  0
execution time, to create 1E8 random no, is  0.001735687255859375
Out[5]:

 4.1403e+06
 5.7016e+06
 1.2710e+07
     ......
 8.9790e+06
 1.3779e+07
 8.0731e+06
[torch.cuda.FloatTensor of size 100000000 (GPU 0)]

i.e. your version takes 7 seconds and mine takes 1.7ms.  I think it is obvious which one ran on the GPU....