Instruction: Error while converting pytorch model to core-ml

Input:
C = torch.cat((A,B),1)

shape of tensors:

A is (1, 128, 128, 256)
B is (1, 1, 128, 256)

Expected C value is (1, 129, 128, 256)

This code is working on pytorch, but while converting to core-ml it gives me below error:

"Error while converting op of type: {}. Error message: {}\n".format(node.op_type, err_message, )
TypeError: Error while converting op of type: Concat. Error message: unable to translate constant array shape to CoreML shape"

Output:
It was coremltools version related issue. Tried with latest beta coremltools 3.0b2.

Following works without any error with latest beta.

import torch

class cat_model(torch.nn.Module):
    def __init__(self):
        super(cat_model, self).__init__()

    def forward(self, a, b):
        c = torch.cat((a, b), 1)
        # print(c.shape)
        return c

a = torch.randn((1, 128, 128, 256))
b = torch.randn((1, 1, 128, 256))

model = cat_model()
torch.onnx.export(model, (a, b), 'cat_model.onnx')

import onnx
model = onnx.load('cat_model.onnx')
onnx.checker.check_model(model)
print(onnx.helper.printable_graph(model.graph))

from onnx_coreml import convert
mlmodel = convert(model)


Instruction: How to properly convert pytorch LSTM to keras CuDNNLSTM?

Input:
I am trying to hand-convert a Pytorch model to Tensorflow for deployment. ONNX doesn't seem to natively go from Pytorch LSTMs to Tensorflow CuDNNLSTMs so that's why I'm writing it by hand.

I've tried the code below:
This is running in an Anaconda environment running Python 2.7, Pytorch 1.0, tensorflow 1.12, cuda9. I'm running this with no bias in the Pytorch layer as it follows a batchnorm, but since Keras does not provide that option I'm simply assigning a 0 bias.

import torch
import tensorflow as tf
import numpy as np
from tensorflow.keras.layers import CuDNNLSTM, Bidirectional
from tensorflow.keras.models import Sequential, Model

input_size = 80
hidden_size = 512
with torch.no_grad():
    rnn1 = torch.nn.LSTM(input_size=input_size, hidden_size=hidden_size, bidirectional=True, bias=False, batch_first=True).cuda()

model = Sequential()
model.add(Bidirectional(CuDNNLSTM(hidden_size, return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1] * 2
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                  ]

model.layers[0].set_weights(keras_format_weights)

random_test = np.random.rand(1, 1, 80)

res1, _ = rnn1.forward(torch.FloatTensor(random_test).cuda())
res1 = res1.detach().cpu().numpy()
res2 = model.predict(random_test)

print(np.allclose(res1, res2, atol=1e-2))
print(res1)
print(res2)

False
[[[ 0.01265562  0.07478553  0.0470101  ... -0.02260824  0.0243004
   -0.0261014 ]]]
[[[-0.05316251 -0.00230848  0.03070898 ...  0.01497027  0.00976444
   -0.01095549]]]

Now, this does work with the generic Keras LSTM:

model = Sequential()
model.add(Bidirectional(LSTM(hidden_size, recurrent_activation='sigmoid', return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1]
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros((bias_size,)),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros((bias_size,))
                  ]

But I need the speed advantages of the CuDNNLSTM, and Pytorch is using the same backend anyway.

Output:
Update: the solution was to convert the torch model into a keras base LSTM model, then call

base_lstm_model.save_weights('1.h5')
cudnn_lstm_model.load_weights('1.h5')


Instruction: how to increase number of images with data augmentation

Input:
I'm trying to apply data augmentation with pytorch. In particular, I have a dataset of 150 images and I want to apply 5 transformations (horizontal flip, 3 random rotation ad vertical flip) to every single image to have 750 images, but with my code I always have 150 images.

'train': transforms.Compose([
    transforms.Resize(224),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(degrees = (90,90)),
    transforms.RandomRotation(degrees = (180,180)),
    transforms.RandomRotation(degrees = (270,270)),
    transforms.RandomVerticalFlip(p=1),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

Output:
You're misunderstanding the API. When you add some transform to your dataset, it is essentially a function which is being applied to every sample from that dataset and then returned. transforms.Compose applies sub-transforms sequentially, rather than returning multiple results (with each translation either being applied or not). So

transforms.Compose([
    transforms.RandomRotation(degrees = (90, -90)),
    transforms.RandomRotation(degrees = (180, -180)),
])

will just rotate your image once at a random angle between 90 and 90 degrees (in other words, by exactly 90 degrees) and then again by 180. This is equivalent to a single RandomRotation(degrees=(270, 270)) (it is actually worse because it leads to more data corruption in the process).

So, most transforms are as above - "linear" - one input, one output. There are some "forking" transforms which produce more outputs than inputs. An example is FiveCrop. Please pay attention to its note on how to deal with that. Even with "forking" transforms, you will still get the same number of items in your dataset, it's just that your batches will be bigger.

If you specifically want to have a dataset which contains 4 differently rotated copies of each item and yields them randomly (ie. possibly each rotated variant comes in a different batch), you will have to write some custom data loading logic. For that, you may want to base your work on source of DatasetFolder.

Why is the API made the way it is? In practice, most people are fine with the transforms as they are currently - in your place, they would just write a transform which randomly flips by 0, 90, 180 or 270 degrees and then train their network for 4 times more epochs than you would, on average getting one sample of each.


Instruction: Is there a `tensor` operation or function in Pytorch that works like cv2.dilate in OpenCV?

Input:
I built several masks through a network. These masks are stored in a torch.tensor variable. I would like to do a cv2.dilate like operation on every channel of the tensor.

I know there is a way that convert the tensor to numpy.ndarray and then apply cv2.dilate to every channel using a for loop. But since there are about 32 channels, this method might slow down the forward operation in the network.

Output:
I think dilate is essentially conv2d operation in torch. See the code below

import cv2
import numpy as np
import torch

im = np.array([ [0, 0, 0, 0, 0],
                [0, 1, 0, 0, 0],
                [0, 1, 1, 0, 0],
                [0, 0, 0, 1, 0],
                [0, 0, 0, 0, 0] ], dtype=np.float32)
kernel = np.array([ [1, 1, 1],
                    [1, 1, 1],
                    [1, 1, 1] ], dtype=np.float32)
print(cv2.dilate(im, kernel))
# [[1. 1. 1. 0. 0.]
#  [1. 1. 1. 1. 0.]
#  [1. 1. 1. 1. 1.]
#  [1. 1. 1. 1. 1.]
#  [0. 0. 1. 1. 1.]]
im_tensor = torch.Tensor(np.expand_dims(np.expand_dims(im, 0), 0)) # size:(1, 1, 5, 5)
kernel_tensor = torch.Tensor(np.expand_dims(np.expand_dims(kernel, 0), 0)) # size: (1, 1, 3, 3)
torch_result = torch.clamp(torch.nn.functional.conv2d(im_tensor, kernel_tensor, padding=(1, 1)), 0, 1)
print(torch_result)
# tensor([[[[1., 1., 1., 0., 0.],
#           [1., 1., 1., 1., 0.],
#           [1., 1., 1., 1., 1.],
#           [1., 1., 1., 1., 1.],
#           [0., 0., 1., 1., 1.]]]])


Instruction: How to compute gradient of the error with respect to the model input?

Input:
Given a simple 2 layer neural network, the traditional idea is to compute the gradient w.r.t. the weights/model parameters. For an experiment, I want to compute the gradient of the error w.r.t the input. Are there existing Pytorch methods that can allow me to do this?

More concretely, consider the following neural network:

import torch.nn as nn
import torch.nn.functional as F

class NeuralNet(nn.Module):
    def __init__(self, n_features, n_hidden, n_classes, dropout):
        super(NeuralNet, self).__init__()

        self.fc1 = nn.Linear(n_features, n_hidden)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(n_hidden, n_classes)
        self.dropout = dropout

    def forward(self, x):
        x = self.sigmoid(self.fc1(x))
        x = F.dropout(x, self.dropout, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

I instantiate the model and an optimizer for the weights as follows:

import torch.optim as optim
model = NeuralNet(n_features=args.n_features,
            n_hidden=args.n_hidden,
            n_classes=args.n_classes,
            dropout=args.dropout)
optimizer_w = optim.SGD(model.parameters(), lr=0.001)

While training, I update the weights as usual. Now, given that I have values for the weights, I should be able to use them to compute the gradient w.r.t. the input. I am unable to figure out how.

def train(epoch):
    t = time.time()
    model.train()
    optimizer.zero_grad()
    output = model(features)
    loss_train = F.nll_loss(output[idx_train], labels[idx_train])
    acc_train = accuracy(output[idx_train], labels[idx_train])
    loss_train.backward()
    optimizer_w.step()

    # grad_features = loss_train.backward() w.r.t to features
    # features -= 0.001 * grad_features

for epoch in range(args.epochs):
    train(epoch)

Output:
It is possible, just set input.requires_grad = True for each input batch you're feeding in, and then after loss.backward() you should see that input.grad holds the expected gradient. In other words, if your input to the model (which you call features in your code) is some M x N x ... tensor, features.grad will be a tensor of the same shape, where each element of grad holds the gradient with respect to the corresponding element of features. In my comments below, I use i as a generalized index - if your parameters has for instance 3 dimensions, replace it with features.grad[i, j, k], etc.

Regarding the error you're getting: PyTorch operations build a tree representing the mathematical operation they are describing, which is then used for differentiation. For instance c = a + b will create a tree where a and b are leaf nodes and c is not a leaf (since it results from other expressions). Your model is the expression, and its inputs as well as parameters are the leaves, whereas all intermediate and final outputs are not leaves. You can think of leaves as "constants" or "parameters" and of all other variables as of functions of those. This message tells you that you can only set requires_grad of leaf variables.

Your problem is that at the first iteration, features is random (or however else you initialize) and is therefore a valid leaf. After your first iteration, features is no longer a leaf, since it becomes an expression calculated based on the previous ones. In pseudocode, you have

f_1 = initial_value # valid leaf
f_2 = f_1 + your_grad_stuff # not a leaf: f_2 is a function of f_1

to deal with that you need to use detach, which breaks the links in the tree, and makes the autograd treat a tensor as if it was constant, no matter how it was created. In particular, no gradient calculations will be backpropagated through detach. So you need something like

features = features.detach() - 0.01 * features.grad

Note: perhaps you need to sprinkle a couple more detaches here and there, which is hard to say without seeing your whole code and knowing the exact purpose.


Instruction: Loading custom dataset in pytorch

Input:
Normally, when we are loading data in pytorch, we do the followings

for x, y in dataloaders:
    # Do something

However, in this dataset called MusicNet, they declare their own dataset and dataloader like this

train_set = musicnet.MusicNet(root=root, train=True, download=True, window=window)#, pitch_shift=5, jitter=.1)
test_set = musicnet.MusicNet(root=root, train=False, window=window, epoch_size=50000)

train_loader = torch.utils.data.DataLoader(dataset=train_set,batch_size=batch_size,**kwargs)
test_loader = torch.utils.data.DataLoader(dataset=test_set,batch_size=batch_size,**kwargs)

Then they load the data like this

with train_set, test_set:
    for i, (x, y) in enumerate(train_loader):
        # Do something

Question 1

I don't understand why the code doesn't work without the line with train_set, test_set.

Question 2

Also, how do I access the data?

I tried

train_set.access(2560,0)

and

with train_set, test_set:
    x, y = train_set.access(2560,0)

They either give me an error message like

  KeyError Traceback (most recent call last) in
  ----> 1 train_set.access(2560,0)

  /workspace/raven_data/AMT/MusicNet/pytorch_musicnet/musicnet.py in
  access(self, rec_id, s, shift, jitter) 106 107 if self.mmap:
  --> 108 x = np.frombuffer(self.records[rec_id][0][ssz_float:int(s+scaleself.window)*sz_float],
  dtype=np.float32).copy() 109 else: 110 fid,_ = self.records[rec_id]

  KeyError: 2560

or giving me an empty x and y

Output:
  Question 1

  I don't understand why the code doesn't work without the line with train_set, test_set.

For you to be able to use the torch.utils.data.DataLoader with a custom dataset design, you must create a class of your dataset which subclasses torch.utils.data.Dataset (and implementing specific functions) and pass it to the dataloader, even they say so:

  All other datasets should subclass it. All subclasses should override __len__, that provides the size of the dataset, and __getitem__, supporting integer indexing in range from 0 to len(self) exclusive.

This is what happens in:

train_set = musicnet.MusicNet(root=root, train=True, download=True, window=window)#, pitch_shift=5, jitter=.1)

test_set = musicnet.MusicNet(root=root, train=False, window=window, epoch_size=50000)

train_loader = torch.utils.data.DataLoader(dataset=train_set,batch_size=batch_size,**kwargs)
test_loader = torch.utils.data.DataLoader(dataset=test_set,batch_size=batch_size,**k

If you check their musicnet.MusicNet, you will find that they do so.

  Question 2

  Also, how do I access the data?

There are possible ways:

To get only a batch from the dataset, you can do:

batch = next(iter(train_loader))

To access the whole dataset (especially, in your example):

dataset = train_loader.dataset.records

(The .records is the part which may vary from dataset to another, I said .records because this is what I found in here)


Instruction: BucketIterator throws 'Field' object has no attribute 'vocab'

Input:
It's not a new question, references I found without any solution working for me first and second.
I'm a newbie to PyTorch, facing AttributeError: 'Field' object has no attribute 'vocab' while creating batches of the text data in PyTorch using torchtext.

Following up the book Deep Learning with PyTorch I wrote the same example as explained in the book.

Here's the snippet:

from torchtext import data
from torchtext import datasets
from torchtext.vocab import GloVe

TEXT = data.Field(lower=True, batch_first=True, fix_length=20)
LABEL = data.Field(sequential=False)
train, test = datasets.IMDB.splits(TEXT, LABEL)

print("train.fields:", train.fields)
print()
print(vars(train[0]))  # prints the object

TEXT.build_vocab(train, vectors=GloVe(name="6B", dim=300),
                 max_size=10000, min_freq=10)

# VOCABULARY
# print(TEXT.vocab.freqs)  # freq
# print(TEXT.vocab.vectors)  # vectors
# print(TEXT.vocab.stoi)  # Index

train_iter, test_iter = data.BucketIterator.splits(
    (train, test), batch_size=128, device=-1, shuffle=True, repeat=False)  # -1 for cpu, None for gpu

# Not working (FROM BOOK)
# batch = next(iter(train_iter))

# print(batch.text)
# print()
# print(batch.label)

# This also not working (FROM Second solution)
for i in train_iter:
    print (i.text)
    print (i.label)

Here's the stacktrace:

AttributeError                            Traceback (most recent call last)
<ipython-input-33-433ec3a2ca3c> in <module>()
      7
      8
----> 9 for i in train_iter:
     10     print (i.text)
     11     print (i.label)

/anaconda3/lib/python3.6/site-packages/torchtext/data/iterator.py in __iter__(self)
    155                     else:
    156                         minibatch.sort(key=self.sort_key, reverse=True)
--> 157                 yield Batch(minibatch, self.dataset, self.device)
    158             if not self.repeat:
    159                 return

/anaconda3/lib/python3.6/site-packages/torchtext/data/batch.py in __init__(self, data, dataset, device)
     32                 if field is not None:
     33                     batch = [getattr(x, name) for x in data]
---> 34                     setattr(self, name, field.process(batch, device=device))
     35
     36     @classmethod

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in process(self, batch, device)
    199         """
    200         padded = self.pad(batch)
--> 201         tensor = self.numericalize(padded, device=device)
    202         return tensor
    203

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in numericalize(self, arr, device)
    300                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    301             else:
--> 302                 arr = [self.vocab.stoi[x] for x in arr]
    303
    304             if self.postprocessing is not None:

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in <listcomp>(.0)
    300                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    301             else:
--> 302                 arr = [self.vocab.stoi[x] for x in arr]
    303
    304             if self.postprocessing is not None:

AttributeError: 'Field' object has no attribute 'vocab'

  If not using BucketIterator, what else I can use to get a similar
  output?

Output:
You haven't built the vocab for the LABEL field.
After TEXT.build_vocab(train, ...), run LABEL.build_vocab(train), and the rest will run.


Instruction: Pytorch - Inferring linear layer in_features

Input:
I am building a toy model to take in some images and give me a classification. My model looks like:

conv2d -> pool -> conv2d -> linear -> linear.

My issue is that when we create the model, we have to calculate the size of the first linear layer in_features based on the size of the input image. If we get new images of different sizes, we have to recalculate in_features for our linear layer. Why do we have to do this? Can't it just be inferred?

Output:
As of 1.8, PyTorch now has LazyLinear which infers the input dimension:

A torch.nn.Linear module where in_features is inferred.


Instruction: Is it possible to use Keras module in Pytorch script?

Input:
I'm creating a network similar to lpcnet introduced by mozilla but in PyTorch. At the one point I need the module src/mdense.py in my script but it is written with keras. Is it possible to import keras module to torch framework? What could be the easiest way? I don't want to re-write the module with PyTorch.

There exists some converters but as I see, these are in early stages. Is there any I can depend on?

Output: No, not really, you can import it (as in python import), but Keras code won't work with PyTorch as they use different differentiation methods and they are completely different libraries. The only way is to rewrite the module using PyTorch's API.


Instruction: Pytorch RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0

Input:
I'm building a simple content based recommendations system. In order to compute the cosine similarity in a GPU accelerated way, i'm using Pytorch.

At the time of creating the tfidf vocabulary tensor from a csr_matrix, it promts the following RuntimeErrorr

RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0

I'm doing it in this way:

coo = tfidf_matrix.tocoo()
values = coo.data
indices = np.vstack( (coo.row, coo.col ))
i = torch.LongTensor(indices)
v = torch.FloatTensor(values)
tfidf_matrix_tensor = torch.sparse.FloatTensor(i, v, torch.Size(coo1.shape)).to_dense()
# Prompts the error

I tried with a small test (tfidf matrix size = 10,296) dataset and it works.
The tfidf matrix size from the real dataset is (27639, 226957)

Output: I tried the same piece of code that was throwing this error with the older version of PyTorch. It said that I need to have more RAM. Therefore, it's not a PyTorch bug. The only solution is to reduce matrix size somehow.


Instruction: How can I get predictions from these pretrained models?

Input:
I've been trying to generate human pose estimations, I came across many pretrained models (ex. Pose2Seg, deep-high-resolution-net ), however these models only include scripts for training and testing, this seems to be the norm in code written to implement models from research papers ,in deep-high-resolution-net I have tried to write a script to load the pretrained model and feed it my images, but the output I got was a bunch of tensors and I have no idea how to convert them to the .json annotations that I need.

total newbie here, sorry for my poor English in advance, ANY tips are appreciated.

I would include my script but its over 100 lines.

PS: is it polite to contact the authors and ask them if they can help?

because it seems a little distasteful.

Output:
Im not doing skeleton detection research, but your problem seems to be general.

(1) I dont think other people should teaching you from begining on how to load data and run their code from begining.

(2) For running other peoples code,  just modify their test script which is provided e.g

https://github.com/leoxiaobin/deep-high-resolution-net.pytorch/blob/master/tools/test.py

They already helps you loaded the model

 model = eval('models.'+cfg.MODEL.NAME+'.get_pose_net')(
        cfg, is_train=False
    )

    if cfg.TEST.MODEL_FILE:
        logger.info('=> loading model from {}'.format(cfg.TEST.MODEL_FILE))
        model.load_state_dict(torch.load(cfg.TEST.MODEL_FILE), strict=False)
    else:
        model_state_file = os.path.join(
            final_output_dir, 'final_state.pth'
        )
        logger.info('=> loading model from {}'.format(model_state_file))
        model.load_state_dict(torch.load(model_state_file))

    model = torch.nn.DataParallel(model, device_ids=cfg.GPUS).cuda()

Just call

# evaluate on Variable x with testing data
y = model(x)
# access Variable's tensor, copy back to CPU, convert to numpy
arr = y.data.cpu().numpy()
# write CSV
np.savetxt('output.csv', arr)

You should be able to open it in excel

(3) "convert them to the .json annotations that I need".

That's the problem nobody can help. We don't know what format you want. For their format, it can be obtained either by their paper. Or looking at their training data by

X, y = torch.load('some_training_set_with_labels.pt')

By correlating the x and y. Then you should have a pretty good idea.


Instruction: Unexpected data types when trying to train a pytorch model

Input:
I'm putting together a basic neural network to learn pytorch. Attempting to train it always fails with the message "Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'". I suspect I'm doing something wrong with putting the data together, but I don't know what.

The data in question is a couple of one-dimensional lists of numbers that I've generated, which should be linearly separable.

I've pasted my code below.

class MyDataset(Dataset):
    def __init__(self, xs, ys):
        assert len(xs) == len(ys), "Input and output tensors must be the same length"
        self.xs = np.array(xs, dtype=np.double)
        self.ys = np.array(ys, dtype=np.double)

    def __getitem__(self, idx):
        return (self.xs[idx], self.ys[idx])

    def __len__(self):
        return len(self.xs)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(1, 1)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        return x

def train(data, validation, net, epochs=100):
    learning_rate = 0.01
    optimizer = optim.SGD(net.parameters(), lr=learning_rate)
    criterion = nn.MSELoss()

    for epoch in range(0, epochs):
        print('Beginning epoch ', epoch+1)
        training_losses = []
        validation_losses = []
        for x_batch, y_batch in data:
            optimizer.zero_grad()
            yhat = net(x_batch)
            loss = criterion(y_batch, yhat)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            training_losses.append(loss)

        with torch.no_grad():
            for x_batch, y_batch in validation:
                net.eval()
                yhat = net(x_batch)
                loss = criterion(y_batch, yhat)
                validation_losses.append(loss)

        print('Ending epoch ', epoch+1, 'Training loss: ', np.mean(training_losses), 'Validation loss: ', np.mean(validation_losses))

And this is how I'm generating the data and attempting to train it:

num_samples = 10000
foos = [100 + np.random.normal(scale=20) for x in range(0, num_samples)]
bars = [200 + np.random.normal(scale=20) for x in range(0, num_samples)]

xs = foos + bars
xs = torch.tensor([[x] for x in xs])
ys = np.concatenate([np.zeros(num_samples), np.ones(num_samples)])
ys = torch.tensor([[y] for y in ys])

dataset = MyDataset(xs, ys)

train_dataset, val_dataset = random_split(dataset, [16000, 4000])

train_loader = DataLoader(dataset=train_dataset, batch_size=16)
val_loader = DataLoader(dataset=val_dataset, batch_size=20)

net = Net()
train(train_loader, val_loader, net)

Finally, here's the stack trace:

<ipython-input-114-ab674ae015a5> in train(data, validation, net, epochs)
     13             print('x_batch: ', type(x_batch[0].item()))
     14             print('y_batch: ', type(y_batch[0].item()))
---> 15             yhat = net(x_batch)
     16             loss = criterion(y_batch, yhat)
     17             loss.backward()

/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-58-ec2e6d981760> in forward(self, x)
      5
      6     def forward(self, x):
----> 7         x = F.relu(self.layer1(x))
      8         return x

/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     65     @weak_script_method
     66     def forward(self, input):
---> 67         return F.linear(input, self.weight, self.bias)
     68
     69     def extra_repr(self):

/usr/local/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1350     if input.dim() == 2 and bias is not None:
   1351         # fused op is marginally faster
-> 1352         ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
   1353     else:
   1354         output = input.matmul(weight.t())

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

I've attempted to debug by logging the types of x_batch and y_batch from within the train method, but they're both showing as float, so I'm stumped as to where the Double is coming from.

Any suggestions?

Output:
PyTorch uses single-precision floats by default.

In the lines:

        self.xs = np.array(xs, dtype=np.double)
        self.ys = np.array(ys, dtype=np.double)

Replace np.double with np.float32.


Instruction: How can I convert numpy.ndarray having type object to torch.tensor?

Input:
I'm trying to work on lstm in pytorch. It takes only tensors as the input. The data that I have is in the form of a numpy.object_ and if I convert this to a numpy.float, then it can be converted to tensor.

I checked the data type using print(type(array)) it gives class 'numpy.ndarray' as output and print(arr.dtype.type) gives class 'numpy.object_' as output.

Or is there any way to convert tuple directly to torch.tensor?

Output:
The pytorch LSTM returns a tuple. So you get this error as your second LSTM layer self.seq2 can not handle this tuple. So,
change

prefix1=self.seq1(input1)
suffix1=self.seq1(input2)

to something like this:

prefix1_out, prefix1_states = self.seq1(input1)
suffix1_out, suffix1_states = self.seq1(input2)

and then pass prefix1_out and suffix1_out tensors to the next LSTM layers as

prefix2_out, prefix2_states = self.seq2(prefix1_out)
suffix2_out, suffix2_states = self.seq2(suffix1_out)

And, concat prefix1_out and suffix1_out tensors like this

result = torch.cat([out1,out2],1)

Also, change

r1=F.sigmoid(self.fc1(result))
r2=self.fc2(r1)

to something like this:

out_ll = self.fc1(result)
r1 = nn.Sigmoid()
r2 = self.fc2(r1(out_ll))


Instruction: Pyinstaller executable fails importing torchvision

Input:
This is my main.py:

import torchvision
input("Press key")

It runs correctly in the command line: python main.py

I need an executable for windows. So I did : pyinstaller main.py

But when I launched the main.exe, inside /dist/main I got this error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>

  ... (omitted)

  File "site-packages\torchvision\ops\misc.py", line 135, in <module>
  File "site-packages\torchvision\ops\misc.py", line 148, in FrozenBatchNorm2d
  File "site-packages\torch\jit\__init__.py", line 850, in script_method
  File "site-packages\torch\jit\frontend.py", line 152, in get_jit_def
  File "inspect.py", line 973, in getsource
  File "inspect.py", line 955, in getsourcelines
  File "inspect.py", line 786, in findsource
OSError: could not get source code
[2836] Failed to execute script main

It seems that some source code is not correctly imported from pyinstaller. I am not sure if the problems is the torch module or torchvision.

Additional info:

I recently installed Visual Studio 2019

System info:

Window 10
Python 3.7
torch-1.1.0
torchvision-0.3.0

[EDIT]

I found that the problem is in the definition of the class FrozenBatchNorm2d inside torchvision. The following script produce the same error as the one before posted:

main.py

import torch

class FrozenBatchNorm2d(torch.jit.ScriptModule):

    def __init__(self, n):
        super(FrozenBatchNorm2d, self).__init__()

    @torch.jit.script_method

    def forward(self):
        pass

I copied all the torch source file. But I still got the error...

Output:
Downgrade torchvision to the previous version fix the error.

pip uninstall torchvision
pip install torchvision==0.2.2.post3


Instruction: How does python map works with torch.tensor?

Input:
I am now in python so I am trying to understand this line from pytorch tutorial.

x_train, y_train, x_valid, y_valid = map(
    torch.tensor, (x_train, y_train, x_valid, y_valid)
)

I understand how map works on a single element

def sqr(a):
    return a * a

a = [1, 2, 3, 4]

a = map(sqr, a)
print(list(a))

And here I need to use list(a) to convert map object back to list.

But what I don't understand, is how does it work on multiple variables?

If I try to do this

def sqr(a):
    return a * a

a = [1, 2, 3, 4]
b = [1, 3, 5, 7]

a, b = map(sqr, (a, b))
print(list(a))
print(list(b))

I get an error: TypeError: can't multiply sequence by non-int of type 'list'

Please clarify this for me
Thank you

Output:
map works on a single the same way it works on list/tuple of lists, it fetches an element of the given input regardless what is it.

The reason why torch.tensor works, is that it accepts a list as input.

If you unfold the following line you provided:

x_train, y_train, x_valid, y_valid = map(
    torch.tensor, (x_train, y_train, x_valid, y_valid)
)

it's the same as doing:

x_train, y_train, x_valid, y_valid = [torch.tensor(x_train), torch.tensor(y_train), torch.tensor(x_valid), torch.tensor(y_valid)]

On other hand, your sqr function does not accept lists. It expects a scalar type to square, which is not the case for your a an b, they are lists.

However, if you change sqr to:

def sqr(a):
    return [s * s for s in a]

a = [1, 2, 3, 4]
b = [1, 3, 5, 7]

a, b = map(sqr, (a, b))

or as suggested by @Jean, a, b = map(sqr, x) for x in (a, b)

It will work.


Instruction: pytorch masked_fill: why can't I mask all zeros?

Input:
I want to mask the all the zeros in the score matrix with -np.inf, but I can only get part of zeros masked, looked like

you see in the upper right corner there are still zeros that didn't get masked with -np.inf

Here's my codes:

q = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
k = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
scores = torch.matmul(q, k.transpose(0,1)) / math.sqrt(10)
mask = torch.Tensor([1,1,1,1,0,0])
mask = mask.unsqueeze(1)
scores = scores.masked_fill(mask==0, -np.inf)

Maybe the mask is wrong?

Output:
Your mask is wrong. Try

scores = scores.masked_fill(scores == 0, -np.inf)

scores now looks like

tensor([[1.4796, 1.2361, 1.2137, 0.9487,   -inf,   -inf],
        [0.6889, 0.4428, 0.6302, 0.4388,   -inf,   -inf],
        [0.8842, 0.7614, 0.8311, 0.6431,   -inf,   -inf],
        [0.9884, 0.8430, 0.7982, 0.7323,   -inf,   -inf],
        [  -inf,   -inf,   -inf,   -inf,   -inf,   -inf],
        [  -inf,   -inf,   -inf,   -inf,   -inf,   -inf]])


Instruction: Why does PyTorch gather function require index argument to be of type LongTensor?

Input: I'm writing some code in PyTorch and I came across the gather function. Checking the documentation I saw that the index argument takes in a LongTensor, why is that? Why does it need to take in a LongTensor instead of another type such as IntTensor? What are the benefits?

Output: By default all indices in pytorch are represented as long tensors - allowing for indexing very large tensors beyond just 4GB elements (maximal value of "regular" int).


Instruction: How to properly display Pytorch's math notation inside docstring?

Input:
When looking at the docstring of Pytorch functions, the math notations are not properly displayed, e.g.:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html

.. math::
    \text{loss}(x, class) = weight[class] \left(-x[class] + \log\left(\sum_j \exp(x[j])\right)\right)

Only if I use my IDE to display the documentation, it renders the latex notation properly.

Is there any switch for displaying math on the website?

Output:
The docs in source code won't render unless you try some script injection via Greasemonkey or so.
But first have a look at the standard docs where you can find the rendered formula.


Instruction: Matrix multiplication (element-wise) from numpy to Pytorch

Input:
I got two numpy arrays (image and and environment map),

MatA
MatB

Both with shapes (256, 512, 3)

When I did the multiplication (element-wise) with numpy:

prod = np.multiply(MatA,MatB)

I got the wanted result (visualize via Pillow when turning back to Image)

But when I did it using pytorch, I got a really strange result(not even close to the aforementioned).

I did it with the following code:

MatATensor = transforms.ToTensor()(MatA)
MatBTensor = transforms.ToTensor()(MatB)

prodTensor = MatATensor * MatBTensor

For some reasons, the shape for both MatATensor and MatBtensor is

torch.Size([3, 256, 512])

Same for the prodTensor too.
When I tried to reshape to (256,512,3), I got an error.

Is there a way to get the same result?

I am new to pytorch, so any help would be appreciated.

Output:
If you read the documentation of transforms.ToTensor() you'll see this transformation does not only convert a numpy array to torch.FloatTensor, but also transpose its dimensions from HxWx3 to 3xHxW.
To "undo" this you'll need to

 prodasNp = (prodTensor.permute(2, 0, 1) * 255).to(torch.uint8).numpy()

See permute for more information.


Instruction: free(): invalid pointer Aborted (core dumped)

Input:
I-m trying to run my python program it seems that it should run smoothly however I encounter an error that I haven't seen before it says:

free(): invalid pointer
Aborted (core dumped)

However I'm not sure how to try and fix error since it doesn't give me too much information about the problem itself.

At first I thought it should be a problem with the sizes of the tensor in my network however they are completely fine. I've google the problem a little and found that I can see that is a problem with allocating memory where I shouldn't, but I don't know how to fix this problem

My code is divided in two different files, and I use two libraries to be able to use Sinkhorn loss function and make sample randomly a mesh.

import argparse
import point_cloud_utils as pcu
import time

import numpy as np
import torch
import torch.nn as nn
from fml.nn import SinkhornLoss

import common
def main():
    # x is a tensor of shape [n, 3] containing the positions of the vertices that
    x = torch._C.from_numpy(common.loadpointcloud("sphere.txt"))
    # t is a tensor of shape [n, 3] containing a set of nicely distributed samples in the unit cube
    v, f = common.unit_cube()
    t = torch._C.sample_mesh_lloyd(pcu.lloyd(v,f,x.shape[0]).astype(np.float32)) # sample randomly a point cloud (cube for now?)

    # The model is a simple fully connected network mapping a 3D parameter point to 3D
    phi = common.MLP(in_dim=3, out_dim=3)

    # Eps is 1/lambda and max_iters is the maximum number of Sinkhorn iterations to do
    emd_loss_fun = SinkhornLoss(eps=1e-3, max_iters=20,
                                stop_thresh=1e-3, return_transport_matrix=True)

    mse_loss_fun = torch.nn.MSELoss()

    # Adam optimizer at first
    optimizer = torch.optim.Adam(phi.parameters(), lr= 10e-3)

    fit_start_time = time.time()

    for epoch in range(100):
        optimizer.zero_grad()

        # Do the forward pass of the neural net, evaluating the function at the parametric points
        y = phi(t)

        # Compute the Sinkhorn divergence between the reconstruction*(using the francis library) and the target
        # NOTE: The Sinkhorn function expects a batch of b point sets (i.e. tensors of shape [b, n, 3])
        # since we only have 1, we unsqueeze so x and y have dimension [1, n, 3]
        with torch.no_grad():
            _, P = emd_loss_fun(phi(t).unsqueeze(0), x.unsqueeze(0))

        # Project the transport matrix onto the space of permutation matrices and compute the L-2 loss
        # between the permuted points
        loss = mse_loss_fun(y[P.squeeze().max(0)[1], :], x)
        # loss = mse_loss_fun(P.squeeze() @ y,  x)  # Use the transport matrix directly

        # Take an optimizer step
        loss.backward()
        optimizer.step()

        print("Epoch %d, loss = %f" % (epoch, loss.item()))

    fit_end_time = time.time()

    print("Total time = %f" % (fit_end_time - fit_start_time))
    # Plot the ground truth, reconstructed points, and a mesh representing the fitted function, phi
    common.visualitation(x,t,phi)

if __name__ == "__main__":
    main()

The error message is:
free(): invalid pointer
Aborted (core dumped)

That again doesn't help me that much. I'll appreciate it a lot if someone has any idea what is happening or if you know more about this error.

Output:
Edit: The cause is actually known. The recommended solution is to build both packages from source.

There is a known issue with importing both open3d and PyTorch. The cause is unknown. https://github.com/pytorch/pytorch/issues/19739

A few possible workarounds exist:

(1) Some people have found that changing the order in which you import the two packages can resolve the issue, though in my personal testing both ways crash.

(2) Other people have found compiling both packages from source to help.

(3) Still others have found that moving open3d and PyTorch to be called from separate scripts resolves the issue.


Instruction: What is the difference between model.training = False and model.param.require_grad = False

Input:
What is the difference between these two:

model.training = False

and

for param in model.parameters():
    param.require_grad = False

Output:
model.training = False sets the module in evaluation mode, i.e.,

if model.training == True:
    # Train mode
if model.training == False:
    # Evaluation mode

So, effectively layers like dropout, batchnorm etc. which behave different on the train and test procedures know what is going on and hence can behave accordingly.

while

for param in model.parameters():
    param.require_grad = False

freeze the layers so that these layers are not trainable.

The basic idea is that all models have a function model.children() which returns it’s layers. Within each layer, there are parameters (or weights), which can be obtained using .param() on any children (i.e. layer). Now, every parameter has an attribute called requires_grad which is by default True. True means it will be backpropagrated and hence to freeze a layer you need to set requires_grad to False for all parameters of a layer.


Instruction: Reproducibility and performance in PyTorch

Input:
The documentation states:

  Deterministic mode can have a performance impact, depending on your model.

My question is, what is meant by performance here. Processing speed or model quality (i.e. minimal loss)? In other words, when setting manual seeds and making the model perform in a deterministic way, does that cause longer training time until minimal loss is found, or is that minimal loss worse than when the model is non-deterministic?

For completeness' sake, I manually make the model deterministic by setting all of these properties:

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)

Output:
Performance refers to the run time; CuDNN has several ways of implementations,  when cudnn.deterministic is set to true, you're telling CuDNN that you only need the deterministic implementations (or what we believe they are). In a nutshell, when you are doing this, you should expect the same results on the CPU or the GPU on the same system when feeding the same inputs. Why would it affect the performance? CuDNN uses heuristics for the choice of the implementation. So, it actually depends on your model how CuDNN will behave; choosing it to be deterministic may affect the runtime because their could have been, let's say, faster way of choosing them at the same point of running.

Concerning your snippet, I do the exact seeding, it has been working good (in terms of reproducibility) for 100+ DL experiments.


Instruction: PyTorch: Sigmoid of weights?

Input:
I'm new to neural networks/PyTorch. I'm trying to make a net that takes in a vector x, first layer is h_j = w_j^T * x + b_j, output is max_j{h_j}. The only thing is that I want the w_j to be restricted between 0 and 1, by having w_j = S(k*a_j), where S is the sigmoid function, k is some constant, and a_j are the actual weight variables (w_j is just a function of a_j). How do I do this in PyTorch? I can't just use a torch.nn.Linear layer, there has to be something else/additional to add in the sigmoid function on the weights?

Side question, for that last output layer, can I just use torch.max to get the max of the previous layer's outputs? Does that behave nicely, or is there some torch.nn.Max or some pooling stuff that I don't understand that needs to happen?

Thanks!

Output:
I am really not sure why would you do that but you can declare a custom layer as below to apply sigmoid to weights.

class NewLayer(nn.Module):
    def __init__ (self, input_size, output_size):
        super().__init__()
        self.W = nn.Parameter(torch.zeros(input_size, output_size))
        # kaiming initialization (use any other if you like)
        self.W = nn.init.kaiming_normal_(self.W)
        self.b = nn.Parameter(torch.ones(output_size))
    def forward(self, x):
        # applying sigmoid to weights and getting results
        ret = torch.addmm(self.b, x, torch.sigmoid(self.W))
        return ret

Once you do this, you can use this as you would use linear layer in your code.


Instruction: Pytorch nn Module generalization

Input:
Let us take a look at the simple class:

class Temp1(nn.Module):

    def __init__(self, stateSize, actionSize, layers=[10, 5], activations=[F.tanh, F.tanh] ):

        super(Temp1, self).__init__()
        self.layer1 = nn.Linear(stateSize, layers[0])
        self.layer2 = nn.Linear(layers[0], layers[1])
        self.fcFinal = nn.Linear( layers[1], actionSize )
        return

This is a fairly straight forward pytorch module. It creates a simple sequential dense network. If we check its hidden parameters, we see the following:

t1 = Temp1(2, 2)
list(t1.parameters())

This is the expected result ...

[Parameter containing:
 tensor([[-0.0311, -0.5513],
         [-0.0634, -0.3783],
         [-0.2514,  0.6139],
         [ 0.4711, -0.0241],
         [-0.1739,  0.2208],
         [-0.1533,  0.3838],
         [-0.6490, -0.5784],
         [ 0.5312,  0.6703],
         [ 0.3506,  0.3652],
         [ 0.1768, -0.4158]], requires_grad=True), Parameter containing:
 tensor([-0.3199, -0.4154, -0.5530, -0.6738, -0.4411,  0.2641, -0.3576,  0.0447,
          0.0254,  0.0965], requires_grad=True), Parameter containing:
 tensor([[-2.8257e-01,  6.7583e-02,  9.0356e-02,  1.0868e-01,  4.0876e-02,
           4.0616e-02,  4.4419e-02, -8.1544e-02,  2.5244e-01,  3.8777e-03],
         [-8.0950e-03, -1.4175e-01, -2.9492e-01,  3.1439e-01, -2.3065e-01,
          -6.6631e-02,  3.0047e-01,  2.8353e-01,  2.3457e-01, -3.1399e-03],
         [-5.2522e-02, -2.2183e-01, -1.5485e-01,  2.6317e-01,  2.8273e-01,
          -7.4823e-02, -5.3704e-02,  9.3526e-02, -1.7916e-01, -3.1132e-04],
         [ 8.9063e-02,  2.9263e-01, -1.0052e-01,  8.7005e-02, -1.1246e-01,
          -2.7968e-01,  4.1411e-02, -1.6776e-01,  1.2363e-01, -2.2808e-01],
         [ 2.9244e-02,  5.8296e-02, -2.9729e-01, -3.1437e-01, -9.3182e-02,
          -7.5236e-03,  5.6159e-02, -2.2075e-02,  1.0337e-01,  8.1123e-02]],
        requires_grad=True), Parameter containing:
 tensor([ 0.2240,  0.0997, -0.0047, -0.1784, -0.0369], requires_grad=True), Parameter containing:
 tensor([[ 0.3546, -0.2180,  0.1723, -0.0463,  0.2572],
         [-0.1669, -0.1364, -0.0398,  0.2233, -0.1805]], requires_grad=True), Parameter containing:
 tensor([ 0.0871, -0.1698], requires_grad=True)]

Now, let us try to generalize this a bit:

class Temp(nn.Module):

    def __init__(self, stateSize, actionSize, layers=[10, 5], activations=[F.tanh, F.tanh] ):

        super(Temp, self).__init__()

        # Generate the fullly connected layer functions
        self.fcLayers = []

        oldN = stateSize
        for i, layer in enumerate(layers):
            self.fcLayers.append( nn.Linear(oldN, layer) )
            oldN = layer
        self.fcFinal = nn.Linear( oldN, actionSize )
        return

It turns out that the number of parameters within this module is no longer the same ...

t = Temp(2, 3)
list(t.parameters())

[Parameter containing:
 tensor([[-0.3342,  0.4111,  0.0418,  0.4457,  0.0648],
         [ 0.4364, -0.0360, -0.2239,  0.4025,  0.1661],
         [ 0.1932, -0.0896,  0.3269, -0.2179,  0.1035]], requires_grad=True),
 Parameter containing:
 tensor([-0.2867, -0.1354, -0.0026], requires_grad=True)]

I believe understand why this is happening. The bigger question is, how do we overcome this problem? The second, generalized method for example will not be sent to the GPU properly, and will not be trained by an optimizer.

Output:
The problem is that most of the nn.Linear layers in the "generalized" version are stored in a regular pythonic list (self.fcLayers). pytorch does not know to look for nn.Paramters inside regular pythonic members of nn.Module.

Solution:
If you wish to store nn.Modules in a way that pytorch can manage them, you need to use specialized pytorch containers.
For instance, if you use nn.ModuleList instead of a regular pythonic list:

self.fcLayers = nn.ModuleList([])

your example should work fine.

BTW,
you need pytorch to know that members of your nn.Module are modules themselves not only to get their parameters, but also for other functions, such as moving them to gpu/cpu, setting their mode to eval/training etc.


Instruction: Can torchtext's BucketIterator pad all batches to the same length?

Input:
I recently started using torchtext to replace my glue code and I'm running into an issue where I'd like to use an attention layer in my architecture. In order to do this, I need to know the maximum sequence length of my training data.

The problem is that torchtext.data.BucketIterator does padding on a per-batch basis:

# All 4 examples in the batch will be padded to maxlen in the batch
train_iter = torchtext.data.BucketIterator(dataset=train, batch_size=4)

Is there some way to ensure that all training examples are padded to the same length; i.e., the maxlen in training?

Output: When instantiating a torchtext.data.Field, there's an optional keyword argument called fix_length which, when set, defines the length to which all samples will be padded; by default it is not set which implies flexible padding.


Instruction: Pytorch : Results of vector multiplications are different for same input

Input:
I did not understand why these multiplication outputs are different.

print(features*weights)
print('------------')
print(features*weights.view(5,1))
print('------------')
print(torch.mm(features,weights.view(5,1)))

outputs:

tensor([[ 0.1314, -0.2796,  1.1668, -0.1540, -2.8442]])
------------
tensor([[ 0.1314, -0.7035, -0.8472,  0.9971, -1.5130],
        [ 0.0522, -0.2796, -0.3367,  0.3963, -0.6013],
        [-0.1809,  0.9688,  1.1668, -1.3733,  2.0837],
        [-0.0203,  0.1086,  0.1308, -0.1540,  0.2336],
        [ 0.2469, -1.3224, -1.5927,  1.8745, -2.8442]])
------------
tensor([[-1.9796]])

Output:
If I'm not wrong, what you are trying to understand is:

features = torch.rand(1, 5)
weights = torch.Tensor([1, 2, 3, 4, 5])
print(features)
print(weights)

# Element-wise multiplication of shape (1 x 5)
# out = [f1*w1, f2*w2, f3*w3, f4*w4, f5*w5]
print(features*weights)

# weights has been reshaped to (5, 1)
# Element-wise multiplication of shape (5 x 5)
# out =   [f1*w1, f2*w1, f3*w1, f4*w1, f5*w1]
#         [f1*w2, f2*w2, f3*w2, f4*w2, f5*w2]
#         [f1*w3, f2*w3, f3*w3, f4*w3, f5*w3]
#         [f1*w4, f2*w4, f3*w4, f4*w4, f5*w4]
#         [f1*w5, f2*w5, f3*w5, f4*w5, f5*w5]
print(features*weights.view(5, 1))

# Matrix-multiplication
# (1, 5) * (5, 1) -> (1, 1)
# out = [f1*w1 + f2*w2 + f3*w3 + f4*w4 + f5*w5]
print(torch.mm(features, weights.view(5, 1)))

Output:

tensor([[0.1467, 0.6925, 0.0987, 0.5244, 0.6491]])  # features
tensor([1., 2., 3., 4., 5.])                        # weights

tensor([[0.1467, 1.3851, 0.2961, 2.0976, 3.2455]])  # features*weights
tensor([[0.1467, 0.6925, 0.0987, 0.5244, 0.6491],
        [0.2934, 1.3851, 0.1974, 1.0488, 1.2982],
        [0.4400, 2.0776, 0.2961, 1.5732, 1.9473],
        [0.5867, 2.7701, 0.3947, 2.0976, 2.5964],
        [0.7334, 3.4627, 0.4934, 2.6220, 3.2455]])  # features*weights.view(5,1)
tensor([[7.1709]])                                  # torch.mm(features, weights.view(5, 1))


Instruction: Conv2d not accepting tensor as input, saying its not tensor

Input:
I want to pass a tensor through a convolutional 2 layer. I am not able to execute it as I am getting a type error even though I have converted my numpy array to a tensor.

I tried using tf.convert_to_tensor() to solve this problem. Didn't work

import numpy as np
import tensorflow as tf

class Generator():

  def __init__(self):

    self.conv1 = nn.Conv2d(1, 28, kernel_size=3, stride=1, padding=1)
    self.pool1 = nn.MaxPool2d(kernel_size=3, stride=0, padding=1)

    self.fc1 = nn.Linear(100, 10)
    self.fc2 = nn.Linear(10, 5)

  def forward_pass(self, x):                                                                       #Why do we pass the object itself in every method?

    x = self.conv1(x)
    print(x)
    x = self.pool1(x)
    print(x)

    x = self.fc1(x)
    print(x)
    x = self.fc2(x)
    print(x)

    return x

arr = tf.convert_to_tensor(np.random.random((3,28,28)))

gen = Generator()
gen.forward_pass(arr)

Error message -

TypeError                                 Traceback (most recent call last)

<ipython-input-31-9fa8e764dcdb> in <module>()
      1 gen = Generator()
----> 2 gen.forward_pass(arr)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339
    340

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not Tensor

Output:
You are trying to pass a TensorFlow tensor to a PyTorch function. TensorFlow and PyTorch are separate projects with different data structures which, in general, cannot be used interchangeably in this way.

To convert a NumPy array to a PyTorch tensor, you can use:

import torch
arr = torch.from_numpy(np.random.random((3,28,28)))


Instruction: Pytorch expected type Long but got type int

Input:
I revived an error

 Expected object of scalar type Long but got scalar type Int for argument #3 'index'

This is from this line.

targets = torch.zeros(log_probs.size()).scatter_(1, targets.unsqueeze(1).data.cpu(), 1)

I am not sure what to do as I tried to convert this to a long using several places. I tried putting a

.long

at the end as well as setting the dtype to be torch.long which still didn't work.

Very similar to this but he didn't do anything to get the answer
"Expected Long but got Int" while running PyTorch script

I have change a lot of the code and here is my last rendition, but is now giving me the same issue.

    def forward(self, inputs, targets):
            """
            Args:
                inputs: prediction matrix (before softmax) with shape (batch_size, num_classes)
                targets: ground truth labels with shape (num_classes)
            """
            log_probs = self.logsoftmax(inputs)
            targets = torch.zeros(log_probs.size()).scatter_(1, targets.unsqueeze(1).data.cpu(), 1)
            if self.use_gpu: targets = targets.to(torch.device('cuda'))
            targets = (1 - self.epsilon) * targets + self.epsilon / self.num_classes
            loss = (- targets * log_probs).mean(0).sum()
            return loss

Output:
The dtype of your index argument (i.e., targets.unsqueeze(1).data.cpu()) needs to be torch.int64.

(The error message is a bit confusing: torch.long doesn't exist. But "Long" in PyTorch internals means int64).


Instruction: Max pooling layers have weights. interesting work

Input:
In the pytorch tutorial step of " Deep Learning with PyTorch: A 60 Minute Blitz > Neural Networks "

I have a question that what dose mean params[1] in the networks?

The reason why i have this think is because of max polling dose not have any weight values.

for example.

If you write some codes like that
'
def init(self) :
    self.conv1 = nn.Conv2d(1 , 6 , 5)
'

this means input has 1 channel, 6 output channel, conv(5,5)

So i understood that params[0] has  6 channel, 5 by 5 matrix random mapping values when init.

for the same reason

params[2] has like same form, but 16 channel. i understood this too.

but params[1], what dose mean?

Maybe it is just presentation method of existence for max polling.

but at the end of this tutorial, in step of the " update the weights "
It's likely updated by this code below.

learning_rate = 0.01
for f in net.parameters():
    f.data.sub_(f.grad.data * learning_rate)

this is code for construct a network

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

params = list(net.parameters())
print(params[1])

Parameter containing:
tensor([-0.0614, -0.0778,  0.0968, -0.0420,  0.1779, -0.0843],
       requires_grad=True)

please visit this pytorch tutorial site.
https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py

Summary

I have a one question.

that is why max pooling layers has four weights which can be updated ?

I think they shouldn't have any weights right?

Am I wrong?

Please help me. I'm a korean.

Output:
You are wrong about that. It has nothing to do with max_pooling.
As you can read in your "linked" Tutorial, is the "nn.paramter"-Tensor automatically registered as parameter when it gets assigned to a Module.

Which, in your case basically, means that everything listed within the __init__ is a module and parameter can be assigned to.

What the values mean inside parameter, well its the parameter your model needs to calculate its steps. to picture it

params[0] -> self.conf1 -> Layer-Input
params[1] -> self.conf1 -> Layer-Output
params[2] -> self.conf2 -> Layer-Input
params[3] -> self.conf2 -> Layer-Output
params[4] -> self.fc1   -> Layer-Input
params[5] -> self.fc1   -> Layer-Output

and so on until you reach params[9], which is the end of your whole parameter list.

EDIT: forgot about the weights
These values are indicator of what your Net has learned.
Therefore you have the ability to alter these values in order to fine-tune your Net to fit your needs.

And if you ask why then 2 rows for each layer?
Well, when you do backpropagation you need these values to locate issues within your Layers.
That's why it stored before passed into a Layer and then after returning from that layer.

hope things are a little clearer now.


Instruction: Pytorch neural network (probably) does not learn

Input:
my homework is to train a network on a given data set of 3000 frogs, cats and dogs. The network I built doesn't seem to improve at all. Why is that?

The training data x_train is a numpy ndarray of shape (3000,32,32,3).

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(3,28,5)
        self.conv2 = nn.Conv2d(28,100,5)
        self.fc1 = nn.Linear(2500,120)
        self.fc2 = nn.Linear(120,3)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = x.view(-1,2500)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x)

model = Netz()
optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)

def train(epoch):
    model.train()
    avg_loss = 0
    correct = 0
    criterion = F.nll_loss
    for i in range(len(x_train)):
        optimizer.zero_grad()
        x = torch.tensor(x_train[i])
        x = x.permute(2, 0, 1)
        x = Variable(x)
        x = x.unsqueeze(0)
        target = Variable(torch.Tensor([y_train[i]]).type(torch.LongTensor))
        out = model(x)
        loss = criterion(out, target)
        avg_loss += loss
        pred = out.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()
        loss.backward()
        optimizer.step()
        if i%64==0:
            print("epoch ", epoch, " [", i, "/", len(x_train), "] average loss: ", avg_loss.item() / 64, " correct: ", correct, "/64")
            avg_loss = 0
            correct = 0

I expect the mean error to decrease over time, but it seems to keep fluctuating around the same number...

Output:
Your loss is fluctuating means your network is not powerful enough to extract meaningful embeddings. I can recommend trying one of these few things.

Add more layers.
Use a smaller learning rate.
Use a larger dataset or use a pre-trained model if you only have a small dataset.
Normalize your dataset.
Shuffle training set.
Play with hyperparameters.


Instruction: How to properly Forward the dropout layer

Input:
I created the following deep network with dropout layers like below:

class QNet_dropout(nn.Module):

    """
        A MLP with 2 hidden layer and dropout

        observation_dim (int): number of observation features
        action_dim (int): Dimension of each action
        seed (int): Random seed
    """

    def __init__(self, observation_dim, action_dim, seed):
        super(QNet_dropout, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(observation_dim, 128)
        self.fc2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Dropout(0.5)
        self.fc5 = nn.Linear(64, action_dim)

    def forward(self, observations):
        """
           Forward propagation of neural network

        """

        x = F.relu(self.fc1(observations))
        x = F.linear(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.linear(self.fc4(x))
        x = self.fc5(x)
        return x

However, when I tried to run the code, I got the following errors:

/home/workspace/QNetworks.py in forward(self, observations)
     90
     91         x = F.relu(self.fc1(observations))
---> 92         x = F.linear(self.fc2(x))
     93         x = F.relu(self.fc3(x))
     94         x = F.linear(self.fc4(x))

TypeError: linear() missing 1 required positional argument: 'weight'

It seems like I didn't properly use/forward the dropout layer. What should be the correct way to do the Forward for the dropout layer? Thanks!

Output:
The F.linear() function used incorrectly. You should use your stated linear function instead of torch.nn.functional. The dropout layer should be after Relu. You can call Relu function from torch.nn.functional.

import torch
import torch.nn.functional as F

class QNet_dropout(nn.Module):

    """
        A MLP with 2 hidden layer and dropout

        observation_dim (int): number of observation features
        action_dim (int): Dimension of each action
        seed (int): Random seed
    """

    def __init__(self, observation_dim, action_dim, seed):
        super(QNet_dropout, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(observation_dim, 128)
        self.fc2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Dropout(0.5)
        self.fc5 = nn.Linear(64, action_dim)

    def forward(self, observations):
        """
           Forward propagation of neural network

        """
        x = self.fc2(F.relu(self.fc1(observations)))
        x = self.fc4(F.relu(self.fc3(x)))
        x = self.fc5(x)
        return x

observation_dim = 512
model = QNet_dropout(observation_dim, 10, 512)
batch_size = 8
inpt  = torch.rand(batch_size, observation_dim)
output = model(inpt)
print ("output shape: ", output.shape)


Instruction: Autoencoder model either oscillates or doesn't converge on MNIST dataset

Input:
Already ran the code 3 months ago with intended results. Changed nothing. Tried troubleshooting by using codes from (several) earlier versions, including among the earliest (which definitely worked). The problem persists.

# 4 - Constructing the undercomplete architecture
class autoenc(nn.Module):
    def __init__(self, nodes = 100):
        super(autoenc, self).__init__() # inheritence
        self.full_connection0 = nn.Linear(784, nodes) # encoding weights
        self.full_connection1 = nn.Linear(nodes, 784) # decoding weights
        self.activation = nn.Sigmoid()

    def forward(self, x):
        x = self.activation(self.full_connection0(x)) # input encoding
        x = self.full_connection1(x) # output decoding
        return x

# 5 - Initializing autoencoder, squared L2 norm, and optimization algorithm
model = autoenc().cuda()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(),
                          lr = 1e-3, weight_decay = 1/2)

# 6 - Training the undercomplete autoencoder model
num_epochs = 500
batch_size = 32
length = int(len(trn_data) / batch_size)

loss_epoch1 = []

for epoch in range(num_epochs):
    train_loss = 0
    score = 0.

    for num_data in range(length - 2):
        batch_ind = (batch_size * num_data)
        input = Variable(trn_data[batch_ind : batch_ind + batch_size]).cuda()

        # === forward propagation ===
        output = model(input)
        loss = criterion(output, trn_data[batch_ind : batch_ind + batch_size])

        # === backward propagation ===
        loss.backward()

        # === calculating epoch loss ===
        train_loss += np.sqrt(loss.item())
        score += 1. #<- add for average loss error instead of total
        optimizer.step()

    loss_calculated = train_loss/score
    print('epoch: ' + str(epoch + 1) + '   loss: ' + str(loss_calculated))
    loss_epoch1.append(loss_calculated)

When plotting the loss now, it oscillates oscillates wildly (at lr = 1e-3). Whereas 3 months ago, it was steadily converging (at lr = 1e-3).

Can't upload pictures yet due to recently created account.

How it looks like now.

Though this is when I reduce the learning rate to 1e-5. When it's at 1e-3, it's just all over the places.

How it should look like, and used to look like at lr = 1e-3.

Output:
You should do optimizer.zero_grad() before you do loss.backward() since the gradients accumulate. This is most likely causing the issue.

The general order to be followed during training phase :

optimizer.zero_grad()
output = model(input)
loss = criterion(output, label)
loss.backward()
optimizer.step()

Also, the value of weight decay used (1 / 2) was causing an issue.


Instruction: Why do we pass nn.Module as an argument to class definition for neural nets?

Input:
I want to understand why we pass torch.nn.Module as a argument when we define the class for a neural network like GAN's

import torch
import torch.nn as nn

class Generator(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, f):
        super(Generator, self).__init__()
        self.map1 = nn.Linear(input_size, hidden_size)
        self.map2 = nn.Linear(hidden_size, hidden_size)
        self.map3 = nn.Linear(hidden_size, output_size)
        self.f = f

Output:
This line

class Generator(nn.Module):

simple means the Generator class will inherit the nn.Module class, it is not an argument.

However, the dunder init method:

def __init__(self, input_size, hidden_size, output_size, f):

Has self which is why you may consider this as an argument.

Well this is Python class instance self. There were tinkering battles should it stay or should it go, but Guido, explained in his blog why it has to stay.


Instruction: Implementing FFT with Pytorch

Input:
I am trying to implement FFT by using the conv1d function provided in Pytorch.

Generating artifical signal

import numpy as np
import torch
from torch.autograd import Variable
from torch.nn.functional import conv1d

from scipy import fft, fftpack

import matplotlib.pyplot as plt

%matplotlib inline

# Creating filters

d = 4096 # size of windows

def create_filters(d):
    x = np.arange(0, d, 1)
    wsin = np.empty((d,1,d), dtype=np.float32)
    wcos = np.empty((d,1,d), dtype=np.float32)
    window_mask = 1.0-1.0*np.cos(x)
    for ind in range(d):
        wsin[ind,0,:] = np.sin(2*np.pi*((ind+1)/d)*x)
        wcos[ind,0,:] = np.cos(2*np.pi*((ind+1)/d)*x)

    return wsin,wcos

wsin, wcos = create_filters(d)
wsin_var = Variable(torch.from_numpy(wsin), requires_grad=False)
wcos_var = Variable(torch.from_numpy(wcos),requires_grad=False)

# Creating signal

t = np.linspace(0,1,4096)
x = np.sin(2*np.pi*100*t)+np.sin(2*np.pi*200*t)+np.random.normal(scale=5,size=(4096))

plt.plot(x)

FFT with Pytorch

signal_input = torch.from_numpy(x.reshape(1,-1),)[:,None,:4096]

signal_input = signal_input.float()

zx = conv1d(signal_input, wsin_var, stride=1).pow(2)+conv1d(signal_input, wcos_var, stride=1).pow(2)

FFT with Scipy

fig = plt.figure(figsize=(20,5))
plt.plot(np.abs(fft(x).reshape(-1))[:500])

My Question

As you can see the two outputs are quite similar in terms of the peaks characteristics. That means my implementation is not totally wrong.
However, there are also some subtleties, such as the scale of the spectrum, and the signal to noise ratio. I am unable to figure out what's missing here to get the exact same result.

Output:
You calculated the power rather than the amplitude.
You simply need to add the line zx = zx.pow(0.5) to take the square root to get the amplitude.


Instruction: In PyTorch/Numpy, how to multiply rows of a matrix with "matrices" in a 3-D tensor?

Input:
For example, a = torch.Tensor([[1,2],[3,4]]) (for numpy it is just a = np.array([[1,2],[3,4]])), and b = torch.ones((2,2,2)),

I would like to product every row of a with the two 2x2 matrices, and get a new matrix [[3,3],[7,7]] (i.e. [1,2]*[[1,1],[1,1]]=[3,3], [3,4]*[[1,1],[1,1]]=[7,7]). Is it possible to achieve this? Thanks!

Output:
I consider this an ugly solution, but perhaps this is what you want to achieve:

a = torch.Tensor([[1,2],[3,4]])
b = torch.ones((2,2,2))

A = torch.mm(a[0].view(-1, 2), b[0])
B = torch.mm(a[1].view(-1, 2), b[1])
res = torch.cat([A, B], dim=0)
print(res)

output:

tensor([[3., 3.],
        [7., 7.]])


Instruction: Pytorch's packed_sequence/pad_sequence pads tensors vertically for list of tensors

Input:
I am trying to pad sequence of tensors for LSTM mini-batching, where each timestep in the sequence contains a sub-list of tensors (representing multiple features in a single timestep).

For example, sequence 1 would have 3 timesteps and within each timestep there are 2 features. An example below would be:

Sequence 1 = [[1,2],[2,2],[3,3],[3,2],[3,2]]

Sequence 2 = [[4,2],[5,1],[4,4]]

Sequence 3 = [[6,9]]

I run pytorch's pad_sequence function (this goes for pack_sequence too) like below:

import torch
import torch.nn.utils.rnn as rnn_utils

a = torch.tensor([[1,2],[2,2],[3,3],[3,2],[3,2]])
b = torch.tensor([[4,2],[5,1],[4,4]])
c = torch.tensor([[6,9]])
result = rnn_utils.pad_sequence([a, b, c])

My expected output is as follows:

Sequence 1 = [[1,2],[2,2],[3,3],[3,2],[3,2]]

Sequence 2 = [[4,2],[5,1],[4,4],[0,0],[0,0]]

Sequence 3 = [[6,9],[0,0],[0,0],[0,0],[0,0]]

However, the output I got is as follows:

tensor([[[1, 2],
         [4, 2],
         [6, 9]],

        [[2, 2],
         [5, 1],
         [0, 0]],

        [[3, 3],
         [4, 4],
         [0, 0]],

        [[3, 2],
         [0, 0],
         [0, 0]],

        [[3, 2],
         [0, 0],
         [0, 0]]])

The padding seems to go vertically rather than what I expect. How do I go about getting the correct padding that I need?

Output:
Simply change

result = rnn_utils.pad_sequence([a, b, c])

to

result = rnn_utils.pad_sequence([a, b, c], batch_first=True)
seq1 = result[0]
seq2 = result[1]
seq3 = result[2]

By default, batch_first is False. Output will be in B x T x * if True, or in T x B x * otherwise, where

B is batch size. It is equal to the number of elements in sequences,

T is length of the longest sequence, and

* is any number of trailing dimensions, including none.

output:

tensor([[1, 2],
        [2, 2],
        [3, 3],
        [3, 2],
        [3, 2]]) # sequence 1
tensor([[4, 2],
        [5, 1],
        [4, 4],
        [0, 0],
        [0, 0]]) # sequence 2
tensor([[6, 9],
        [0, 0],
        [0, 0],
        [0, 0],
        [0, 0]]) # sequence 3


Instruction: RuntimeError: input must have 2 dimensions, got 3

Input:
I am trying to run a sequence of bigrams into an LSTM. This involves:

2.Padding the sequences using pad_sequence

3.Inputting the padded sequences in the embedding layer

4.Packing the output of the embedding layer

5.Inserting the pack into the LSTM.

class LSTMClassifier(nn.Module):
    # LSTM initialization
    def __init__(self, embedding_dim=32, hidden_dim=50, vocab_size=7138, label_size=2, static_size, batch_size=32):
        super(LSTMClassifier, self).__init__()
        # Initializing batch size
        self.batch_size = batch_size
        # Setting the hidden layer dimension of the LSTM
        self.hidden_dim = hidden_dim
        # Initializing the embedding layer
        self.embeddings = nn.Embedding(vocab_size, embedding_dim-2)
        # Initializing the LSTM layer with one hidden layer
        self.lstm = nn.LSTM(((embedding_dim*vocab_size)+static_size), hidden_dim, num_layers=1, batch_first=True)
        # Initializing linear linear that takes the hidden layer output
        self.hidden2label = nn.Linear(hidden_dim, label_size)
        # Initializing the hidden layer
        self.hidden = self.init_hidden()

    # Defining the hidding state of the LSTM
    def init_hidden(self):
        # the first is the hidden h
        # the second is the cell  c
        return (autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim).cuda()))

    # Defining the feed forward logic of the LSTM. It contains:
    # 1. The embedding layer
    # 2. The LSTM layer with one hidden layer
    # 3. The software layer
    def forward(self, seq, freq, time, static):
        # reset the LSTM hidden state. Must be done before you run a new batch. Otherwise the LSTM will treat
        # a new batch as a continuation of a sequence
        self.hidden = self.init_hidden()

        # Get sequence lengths
        seq_lengths = torch.LongTensor(list(map(len, seq))) # length of 59

        # Pad the sequences
        seq = rnn_utils.pad_sequence(seq, batch_first = True)
        freq = rnn_utils.pad_sequence(freq, batch_first = True)
        time = rnn_utils.pad_sequence(time, batch_first = True)
        static = rnn_utils.pad_sequence(static, batch_first = True)

        seq = autograd.Variable(seq)
        freq = autograd.Variable(freq)
        time = autograd.Variable(time)
        static = autograd.Variable(static)

        # This is the pass to the embedding layer.
        # The sequence is of dimension N and the output is N x Demb
        embeds = self.embeddings(seq)
        embeds = torch.cat((embeds,freq), dim=3)
        embeds = torch.cat((embeds,time), dim=3)
        print(embeds.size()) #torch.Size([32, 59, 7138, 32])

        x = embeds.view(self.batch_size, seq.size()[1], 1,-1)
        print(x.size()) #torch.Size([32, 59, 1, 228416])

        static = static.view(self.batch_size, -1,1,3)
        x = torch.cat([x, static], dim=3)
        print(x.size()) #torch.Size([32, 59, 1, 228419])

        # pack the padded sequence so that paddings are ignored
        x = torch.nn.utils.rnn.pack_padded_sequence(x, seq_lengths, batch_first=True)

        lstm_out, self.hidden = self.lstm(x, self.hidden)

        # unpack the packed padded sequence so that it is ready for prediction
        lstm_out = torch.nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)

        y = self.hidden2label(lstm_out[-1])
        log_probs = F.log_softmax(y)
        return log_probs

However, I get an error that is the following:

---> 66         lstm_out, self.hidden = self.lstm(x, self.hidden)
     67
     68         # unpack the packed padded sequence so that it is ready for prediction

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    173                 hx = (hx, hx)
    174
--> 175         self.check_forward_args(input, hx, batch_sizes)
    176         _impl = _rnn_impls[self.mode]
    177         if batch_sizes is None:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    129             raise RuntimeError(
    130                 'input must have {} dimensions, got {}'.format(
--> 131                     expected_input_dim, input.dim()))
    132         if self.input_size != input.size(-1):
    133             raise RuntimeError(

RuntimeError: input must have 2 dimensions, got 3
x = rnn_utils.pad_sequence(x, batch_first=True)
import torch

I thought LSTM model required 3 dimensional input? I am very confused as to why it requires 2 dimensions. How should I fix it?

Output:
Finally found solution after 5 hours of browsing...
def forward(self, input):
    input = input.unsqueeze(0).unsqueeze(0)
    # Initializing hidden state for first input with zeros
    h0 = torch.zeros(1, input.size(0), 128)

    # Initializing cell state for first input with zeros
    c0 = torch.zeros(1, input.size(0), 128)
    hidden = (h0.detach(), c0.detach())

    out1, _ = self.lstm(input, hidden)
    out2 = F.leaky_relu(self.fc1(out1))
    qvalue = self.fc2(out2)
    return qvalue


Instruction: what is the right usage of _extra_files arg in torch.jit.save

Input:
one option I tried is pickling vocab and saving with extrafiles arg

import torch
import pickle

class Vocab(object):
    pass

vocab = Vocab()
pickle.dump(open('path/to/vocab.pkl','w'))

m = torch.jit.ScriptModule()

## I am not sure about the usage of this arg, the docs didn't help me
extra_files = torch._C.ExtraFilesMap()
extra_files['vocab.pkl'] = 'path/to/vocab.pkl'
# I also tried  pickle.dumps(vocab), and directly vocab

torch.jit.save(m, 'scriptmodule.pt', _extra_files=extra_files)

## Load with extra files.
files = {'vocab.pkl': ''}
torch.jit.load('scriptmodule.pt', _extra_files = files)

this gives

TypeError: import_ir_module(): incompatible function arguments. The following argument types are supported:
    1. (arg0: Callable[[List[str]], torch._C.ScriptModule], arg1: str, arg2: object, arg3: torch._C.ExtraFilesMap) -> None

other option is obviously to load the pickle separately, but I was looking for single file option.

it would be nice if one could just add vocab to to the torchscript ... it would also be nice to know if there is some reason for not doing this that I am obviously not aware of.

Output:
I believe that the documentation for torch.jit.load is incorrect. You need to create an ExtraFilesmap() object to load the saved files.

The following is an example of how I got things to work:
Step 1: Save model

extra_files = torch._C.ExtraFilesMap()
extra_files['foo.txt'] = 'bar'
traced_script_module.save(serialized_model_path, _extra_files=extra_files)

Step 2: Load model

files = torch._C.ExtraFilesMap()
files['foo.txt'] = ''
loaded_model = torch.jit.load(serialized_model_path, _extra_files=files)
print(files)


Instruction: When using Conv2D in PyTorch, does padding or dilation happen first?

Input:
Consider the following bit of code:

torch.nn.Conv2d(1, 1, 2, padding = 1, dilation = 2)

Which of the following two cases is a correct interpretation?

Output:
If you look at the bottom of the nn.Conv2d documentation you'll see the formula used to compute the output size of the conv layer:

Notice how padding is not affected by the value of dilation. I suppose this indicates "pad first" approach.


Instruction: Import from Github : How to fix ImportError

Input:
I want to use the open source person re-identification library in Python

on Ubuntu 19.04
with Anaconda
no CUDA
in the terminal PyCharm (or not)
Python version 3.7.3
PyTorch version 1.1.0

For that I have to follow instruction like on their deposite git :

git clone https://github.com/Cysu/open-reid.git
cd open-reid
python setup.py install

python examples/softmax_loss.py -d viper -b 64 -j 2 -a resnet50 --logs-dir logs/softmax-loss/viper-resnet50

I receive the following error:

from sklearn.utils.extmath
import pinvh

  ImportError: cannot import name 'pinvh'

I have tried to create virtual environments with previous versions of PyTorch (0.4.1, 0.4.0 and 1.0.1) but I always got:

  File "examples/softmax_loss.py", line 12, in <module>
    from reid import datasets
ModuleNotFoundError: No module named 'reid'

I do not know how to fix it.

EDIT :

Hi thanks for the answer, the problem is that the import are like :

from reid import datasets
from reid import models
from reid.dist_metric import DistanceMetric
from reid.trainers import Trainer
from reid.evaluators import Evaluator
from reid.utils.data import transforms as T
from reid.utils.data.preprocessor import Preprocessor
from reid.utils.logging import Logger
from reid.utils.serialization import load_checkpoint, save_checkpoint

I tried :

from ../reid import datasets

But I got a

File "examples/softmax_loss.py", line 12
from ../reid import datasets
   ^
SyntaxError: invalid syntax

EDIT 2 :
After re-installing Python 3.7.3 and pytorch 1.1.0 the problem persist with pinvh... I still got this message :

ImportError: cannot import name 'pinvh' from 'sklearn.utils.extmath'

If you can tell me how to fix it or try to tell me if it works please

Output:
Since the directory structure is as below:

/(root)-->|
          |
          |-->reid |--> (contents inside reid)
          |
          |
          |-->examples | -->softmax_loss.py
          |
          |-->(Other  contents in root directory)

It can be observed that reid is not in the same directory as softmax_loss.py, but instead in the parent directory.

So, in the file softmax_loss.py, at line number 12 and below, replace reid with ../reid, this looks for the directory reid in the parent directory.

The other method is to use: import ../reid as R or any other variable; Then use from R import datasets, and so on


Instruction: Assigning values to torch tensors

Input:
I'm trying to assign some values to a torch tensor. In the sample code below, I initialized a tensor U and try to assign a tensor b to its last 2 dimensions. In reality, this is a loop over i and j that solves some relation for a number of training data (here 10) and assigns it to its corresponding location.

import torch

U = torch.zeros([10, 1, 4, 4])
b = torch.rand([10, 1, 1, 1])

i = 2
j = 2
U[:, :, i, j] = b

I was expecting vector b to be assigned for dimensions i and j of corresponding training data (shape of training data being (10,1)) but it gives me an error. The error that I get is the following

RuntimeError: expand(torch.FloatTensor{[10, 1, 1, 1]}, size=[10, 1]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (4)

Any suggestions on how to fix it would be appreciated.

As an example, you can think of this as if '[10, 1]' is the shape of my data. Imagine it is 10 images, each of which has one channel. Then imagine each image is of shape '[4, 4]'. In each iteration of the loop, pixel '[i, j]' for all images and channels is being calculated.

Output:
Your b tensor has too much dimensions.

U[:, :, i, j] has a [10, 1] shape (try U[:, :, i, j].shape)

Use b = torch.rand([10, 1]) instead.


Instruction: How to access the network weights while using PyTorch 'nn.Sequential'?

Input:
I'm building a neural network and I don't know how to access the model weights for each layer.

I've tried

model.input_size.weight

Code:

input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))

I expected to get the weights but I got

  'Sequential' object has no attribute 'input_size'

Output:
I've tried many ways, and it seems that the only way is by naming each layer by passing OrderedDict

from collections import OrderedDict
model = nn.Sequential(OrderedDict([
                  ('fc1', nn.Linear(input_size, hidden_sizes[0])),
                  ('relu1', nn.ReLU()),
                  ('fc2', nn.Linear(hidden_sizes[0], hidden_sizes[1])),
                  ('relu2', nn.ReLU()),
                  ('output', nn.Linear(hidden_sizes[1], output_size)),
                  ('softmax', nn.Softmax(dim=1))]))

So to access the weights of each layer, we need to call it by its own unique layer name.

For example to access weights of layer 1  model.fc1.weight

Parameter containing:
tensor([[-7.3584e-03, -2.3753e-02, -2.2565e-02,  ...,  2.1965e-02,
      1.0699e-02, -2.8968e-02],
    [ 2.2930e-02, -2.4317e-02,  2.9939e-02,  ...,  1.1536e-02,
      1.9830e-02, -1.4294e-02],
    [ 3.0891e-02,  2.5781e-02, -2.5248e-02,  ..., -1.5813e-02,
      6.1708e-03, -1.8673e-02],
    ...,
    [-1.2596e-03, -1.2320e-05,  1.9106e-02,  ...,  2.1987e-02,
     -3.3817e-02, -9.4880e-03],
    [ 1.4234e-02,  2.1246e-02, -1.0369e-02,  ..., -1.2366e-02,
     -4.7024e-04, -2.5259e-02],
    [ 7.5356e-03,  3.4400e-02, -1.0673e-02,  ...,  2.8880e-02,
     -1.0365e-02, -1.2916e-02]], requires_grad=True)


Instruction: Getting alignment/attention during translation in OpenNMT-py

Input: Does anyone know how to get the alignments weights when translating in Opennmt-py? Usually the only output are the resulting sentences and I have tried to find a debugging flag or similar for the attention weights. So far, I have been unsuccessful.

Output:
You can get the attention matrices. Note that it is not the same as alignment which is a term from statistical (not neural) machine translation.

There is a thread on github discussing it. Here is a snippet from the discussion. When you get the translations from the mode, the attentions are in the attn field.

import onmt
import onmt.io
import onmt.translate
import onmt.ModelConstructor
from collections import namedtuple

# Load the model.
Opt = namedtuple('Opt', ['model', 'data_type', 'reuse_copy_attn', "gpu"])

opt = Opt("PATH_TO_SAVED_MODEL", "text", False, 0)
fields, model, model_opt =  onmt.ModelConstructor.load_test_model(
    opt, {"reuse_copy_attn" : False})

# Test data
data = onmt.io.build_dataset(
    fields, "text", "PATH_TO_DATA", None, use_filter_pred=False)
data_iter = onmt.io.OrderedIterator(
    dataset=data, device=0,
    batch_size=1, train=False, sort=False,
    sort_within_batch=True, shuffle=False)

# Translator
translator = onmt.translate.Translator(
    model, fields, beam_size=5, n_best=1,
    global_scorer=None, cuda=True)

builder = onmt.translate.TranslationBuilder(
   data, translator.fields, 1, False, None)

batch = next(data_iter)
batch_data = translator.translate_batch(batch, data)
translations = builder.from_batch(batch_data)
translations[0].attn # <--- here are the attentions


Instruction: Getting the gradients of a model trained in OpenNMT-py

Input:
When training a model using OpenNMT-py, we get a dict as output, containing the weights and biases of the network. However, these tensors have requires_grad = False, and so, do not have a gradient. For example. with one layer, we might have the following tensors, denoting embeddings as well as weights and biases in the encoder and decoder. None of them have a gradient attribute.

encoder.embeddings.emb_luts.0.weight

decoder.embeddings.emb_luts.0.weight

encoder.rnn.weight_ih_l0

encoder.rnn.weight_hh_l0

encoder.rnn.bias_ih_l0

encoder.rnn.bias_hh_l0

decoder.rnn.layers.0.weight_ih

decoder.rnn.layers.0.weight_hh

decoder.rnn.layers.0.bias_ih

decoder.rnn.layers.0.bias_hh

Can OpenNMT-py be made to set requires_gradient = True with some option I have not found or is there some other way to obtain the gradient of these tensors?

Output:
The gradients are accessible only inside the training loop, where optim.step() is called. If you want to log the gradients (or norm of gradients or whatever) to TensorBoard, you can probably best get them before the optimizer step is called. It happens in the _gradient_accumulation method of the Trainer object.

Be aware that there are two places where optim.step() is called. Which one is used depends on whether you do the update after every batch or whether you accumulate gradient from multiple batches and do the update afterward.


Instruction: How to calculate output sizes after a convolution layer in a configuration file?

Input:
I'm new to convolutional neural networks and wanted to know how to calculate or figure out the output sizes between layers of a model given a configuration file for pytorch similar to those following instructions in this link.

Most of the stuff I've already looked at hasn't been very clear and concise. How am I supposed to calculate the sizes through each layer?
Below is a snippet of a configuration file that would be parsed.

# (3, 640, 640)
[convolutional]
batch_normalize=1
filters=16
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

# (16, 320, 320)

Output:
In short, there is a common formula for output dims calculation:

You can find explanation in A guide to receptive field arithmetic for Convolutional Neural Networks.

In addition, I'd like to recommend amazing article A guide to convolution arithmetic for deep learning.

And this repo conv_arithmetic with convolution animations.


Instruction: TypeError: can't assign a str to a torch.LongTensor pytoch

Input:
I'm trying to convert a list of strings into tensors but I get this error

lengths = [len(cap) for cap in captions]
    targets = torch.zeros(len(captions), max(lengths)).long()
    for i, cap in enumerate(captions):
        end = lengths[i]
        targets[i, :end] = cap[:end]

Output:
You can use python's ord to convert characters to their unicode:

targets[i, :end] = torch.from_numpy(np.array(list(map(ord, cap[:end])))).to(torch.long)


Instruction: Use only certain layers of pretrained torchvision network

Input:
I'm trying to use only certain layers in a pretrained torchvision Faster-RCNN network initialized by:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

This works. However, passing model.modules() or model.children() into an nn.Sequential yields an error. Even passing the whole model leads to errors, e.g.

model = torch.nn.Sequential(*model.modules())
model.eval()
# x is a [C, H, W] image
y = model(x)

leads to

AttributeError: 'dict' object has no attribute 'dim'

and

model = torch.nn.Sequential(*model.children())
model.eval()
# x is a [C, H, W] image
y = model(x)

leads to

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple

This confuses me because I have modified other PyTorch pretrained models like that in the past. How can I use the FasterRCNN pretrained model to create a new (pretrained) model that uses only certain layers, e.g. all layers but the last one?

Output:
Unlike other simple CNN models, it is not trivial to convert an R-CNN based detector to a simple nn.Sequential model. If you look at the functionality of R-CNN ('generalized_rcnn.py') you'll see that the output features (computed by the FCN backbone) are not just passed to the RPN component, but rather combined with the input image and even with the targets (during training).

Therefore, I suppose if you want to change the way faster R-CNN behaves, you'll have to use the base class torchvision.models.detection.FasterRCNN() and provide it with different roi pooling parameters.


Instruction: How convert a list with None in it to torch.tensor()?

Input:
For example I have a variable z = [1, 3, None, 5, 6]

I would like to do: torch.tensor(z)

and get something like the following:

torch.tensor([1,3, None, 5,6], dtype=torch.float)

Howhever, such attempt raises an error

  TypeError: must be real number, not NoneType

Is there a way to convert such list to a torch.tensor?

I don't want to impute this None value with something else. Numpy arrays are capable of converting such lists np.array([1, 3, None, 5, 6]), but I'd prefer not to convert back and forth variable either.

Output:
It depends on what you do. Probable the best is to convert None to 0.

Converting things to numpy arrays and then to Torch tensors is a very good path since it will convert None to np.nan. Then you can create the Torch tensor even holding np.nan.

import torch
import numpy as np

a = [1,3, None, 5,6]
b = np.array(a,dtype=float) # you will have np.nan from None
print(b) #[ 1.  3. nan  5.  6.]
np.nan_to_num(b, copy=False)
print(b) #[1. 3. 0. 5. 6.]
torch.tensor(b, dtype=torch.float) #tensor([1., 3., 0., 5., 6.])

Try also copy=True inside np.nan_to_num and you will get nan inside your tensor instead of 0.


Instruction: DataLoader num_workers vs torch.set_num_threads

Input: Is there a difference between the parallelization that takes place between these two options? I’m assuming num_workers is solely concerned with the parallelizing the data loading. But is setting torch.set_num_threads for training in general? Trying to understand the difference between these options. Thanks!

Output:
The num_workers for the DataLoader specifies how many parallel workers to use to load the data and run all the transformations. If you are loading large images or have expensive transformations then you can be in situation where GPU is fast to process your data and your DataLoader is too slow to continuously feed the GPU. In that case setting higher number of workers helps. I typically increase this number until my epoch step is fast enough. Also, a side tip: if you are using docker, usually you want to set shm to 1X to 2X number of workers in GB for large dataset like ImageNet.

The torch.set_num_threads specifies how many threads to use for parallelizing CPU-bound tensor operations. If you are using GPU for most of your tensor operations then this setting doesn't matter too much. However, if you have tensors that you keep on cpu and you are doing lot of operations on them then you might benefit from setting this. Pytorch docs, unfortunately, don't specify which operations will benefit from this so see your CPU utilization and adjust this number until you can max it out.


Instruction: Making a Prediction Sagemaker Pytorch

Input:
I have trained and deployed a model in Pytorch with Sagemaker. I am able to call the endpoint and get a prediction. I am using the default input_fn() function (i.e. not defined in my serve.py).

model = PyTorchModel(model_data=trained_model_location,
                     role=role,
                     framework_version='1.0.0',
                     entry_point='serve.py',
                     source_dir='source')
predictor = model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

A prediction can be made as follows:

input ="0.12787057,  1.0612601,  -1.1081504"
predictor.predict(np.genfromtxt(StringIO(input), delimiter=",").reshape(1,3) )

I want to be able to serve the model with REST API and am HTTP POST using lambda and API gateway. I was able to use invoke_endpoint() for this with an XGBOOST model in Sagemaker this way. I am not sure what to send into the body for Pytorch.

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT  ,
ContentType='text/csv',
Body=???)

I believe I need to understand how to write the customer input_fn to accept and process the type of data I am able to send through invoke_client. Am I on the right track and if so, how could the input_fn be written to accept a csv from invoke_endpoint?

Output:
Yes you are on the right track. You can send csv-serialized input to the endpoint without using the predictor from the SageMaker SDK, and using other SDKs such as boto3 which is installed in lambda:

import boto3
runtime = boto3.client('sagemaker-runtime')

payload = '0.12787057,  1.0612601,  -1.1081504'

response = runtime.invoke_endpoint(
    EndpointName=ENDPOINT_NAME,
    ContentType='text/csv',
    Body=payload.encode('utf-8'))

result = json.loads(response['Body'].read().decode())

This will pass to the endpoint a csv-formatted input, that you may need to reshape back in the input_fn to put in the appropriate dimension expected by the model.

for example:

def input_fn(request_body, request_content_type):
    if request_content_type == 'text/csv':
        return torch.from_numpy(
            np.genfromtxt(StringIO(request_body), delimiter=',').reshape(1,3))

Note: I wasn't able to test the specific input_fn above with your input content and shape but I used the approach on Sklearn RandomForest couple times, and looking at the Pytorch SageMaker serving doc the above rationale should work.

Don't hesitate to use endpoint logs in Cloudwatch to diagnose any inference error (available from the endpoint UI in the console), those logs are usually much more verbose that the high-level logs returned by the inference SDKs