Instruction: Pytorch: backpropagating from sum of matrix elements to leaf variable

Input:
I'm trying to understand backpropagation in pytorch a bit better. I have a code snippet that successfully does backpropagation from the output d to the leaf variable a, but then if I add in a reshape step, the backpropagation no longer gives the input a gradient.

I know reshape is out-of-place, but I'm still not sure how to contextualize this.

Any thoughts?

Thanks.

#Works
a = torch.tensor([1.])
a.requires_grad = True
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

print('a gradient is')
print(a.grad) #=> Tensor([1.])

#Doesn't work
a = torch.tensor([1.])
a.requires_grad = True
a = a.reshape(a.shape)
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

print('a gradient is')
print(a.grad) #=> None

Output:
Edit:

Here is a detailed explanation of what's going on ("this isn't a bug per se, but it is definitely a source of confusion"): https://github.com/pytorch/pytorch/issues/19778

So one solution is to specifically ask to retain grad for now non-leaf a:

a = torch.tensor([1.])
a.requires_grad = True
a = a.reshape(a.shape)
a.retain_grad()
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

Old answer:

If you move a.requires_grad = True after the reshape, it works:

a = torch.tensor([1.])
a = a.reshape(a.shape)
a.requires_grad = True
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

Seems like a bug in PyTorch, because after this a.requires_grad is still true.

a = torch.tensor([1.])
a.requires_grad = True
a = a.reshape(a.shape)

This seems to be related to the fact the a is no longer a leaf in your "Doesn't work" example, but still a leaf in other cases (print a.is_leaf to check).


Instruction: PyTorch transformations change dimensions

Input:
I do not understand why pytorch transformations from 100x100 pic make 3x100 pic.

print("Original shape ", x.shape)

x = transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor()
])(x)

print("After transformation shape ", x.shape)

outputs

Original shape  torch.Size([100, 100, 3])
After transformation shape  torch.Size([3, 100, 3])

What is happening?

Output:
According to the docs https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToPILImage, if the input is a Torch tensor, the shape is C x H x W. So 100 is considered to be number of channels. Because there is no mode that corresponds to 100 channels, it interpreted as RGB (3 channels).

So you need input shape torch.Size([3, 100, 100]) to make it work as you want.


Instruction: In PyTorch, how to I make certain module `Parameters` static during training?

Input:
Context:

In pytorch, any Parameter is a special kind of Tensor. A Parameter is automatically registered with a module's parameters() method when it is assigned as an attribute.

During training, I will pass m.parameters() to the Optimizer instance so they can be updated.

Question: For a built-in pytorch module, how to I prevent certain parameters from being modified by the optimizer?

s = Sequential(
        nn.Linear(2,2),
        nn.Linear(2,3),   # I want this one's .weight and .bias to be constant
        nn.Linear(3,1)
    )

Can I make it so they don't appear in s.parameters()?
Can I make the parameters read-only so any attempted changes are ignored?

Output:
Parameters can be made static by setting their attribute requires_grad=False.

In my example case:

params = list(s.parameters())  # .parameters() returns a generator
# Each linear layer has 2 parameters (.weight and .bias),
# Skipping first layer's parameters (indices 0, 1):
params[2].requires_grad = False
params[3].requires_grad = False

When a mix of requires_grad=True and requires_grad=False tensors are used to make a calculation, the result inherits requires_grad=True.

According to the PyTorch autograd mechanics documentation:

  If there’s a single input to an operation that requires gradient, its output will also require gradient. Conversely, only if all inputs don’t require gradient, the output also won’t require it. Backward computation is never performed in the subgraphs, where all Tensors didn’t require gradients.

My concern was that if I disabled gradient tracking for the middle layer, the first layer wouldn't receive backpropagated gradients. This was faulty understanding.

Edge Case: If I disable gradients for all parameters in a module and try to train, the optimizer will raise an exception. Because there is not a single tensor to apply the backward() pass to.

This edge case is why I was getting errors. I was trying to test requires_grad=False on parameters for module with a single nn.Linear layer. That meant I disabled tracking for all parameters, which caused the optimizer to complain.


Instruction: Pytorch: How plot the prediction output from segmentation task when batch size is larger than 1?

Input:
I have a doubt and a question about plot output of different batches in segmentation subject.

The snippet below plot the probability of each class and the prediction output.

I am sure the prob plots is plotting one batch, but not sure about prediction when I got the torch.argmax(outputs, 1). Am I plotted the argmax of one batch while the output of the network has the size of [10,4,256,256].

Also, I am wondering how can I plot the prediction of all batches while my batch size is 10.

outputs = model(t_image)

fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(nrows=1, ncols=5, sharex=True, sharey=True, figsize=(6,6))

img1 = ax1.imshow(torch.exp(outputs[0,0,:,:]).detach().cpu(), cmap = 'jet')
ax1.set_title("prob class 0")

img2 = ax2.imshow(torch.exp(outputs[0,1,:,:]).detach().cpu(), cmap = 'jet')
ax2.set_title("prob class 1")

img3 = ax3.imshow(torch.exp(outputs[0,2,:,:]).detach().cpu(), cmap = 'jet')
ax3.set_title("prob class 2")

img4 = ax4.imshow(torch.exp(outputs[0,3,:,:]).detach().cpu(), cmap = 'jet')
ax4.set_title("prob class 3")

img5 = ax5.imshow(torch.argmax(outputs, 1).detach().cpu().squeeze(), cmap = 'jet')
ax5.set_title("predicted")

Output:
Not sure about what you are asking. Assuming you are using the NCHW data layout, your output is 10 samples per batch, 4 channels (each channel for a different class), and 256x256 resolution, then the first 4 graphs are plotting the class scores of the four classes.

For the 5th plot, your torch.argmax(outputs, 1).detach().cpu().squeeze() would give you a 10x256x256 image, which is the class prediction results for all 10 images in the batch, and matplotlib cannot properly plot it directly. So you would want to do torch.argmax(outputs[0,:,:,:], 0).detach().cpu().squeeze() which would get you a 256x256 map, which you can plot.

Since the result would range from 0 to 3 which represents the 4 classes, (and may be displayed as a very dim image), usually people would use a palette to color the plots. An example is provided here and looks like the cityscapes_map[p] line in the example.

For plotting all 10, why not write a for loop:

for i in range(outputs.size(0)):
    # do whatever you do with outputs[i, ...]
    # ...
    plt.show()

and go over each result in the batch one by one. There is also the option to have 10 rows in your subplot, if your screen is big enough.


Instruction: translate from pyrotch to tensorflow

Input:
What does this line of code in PyTorch do?

normA = A.mul(A).sum(dim=1).sum(dim=1).sqrt()
Y = A.div(normA.view(batchSize, 1, 1).expand_as(A))

Normally it should be a second term like this:

torch.div(input, value, out=None) → Tensor

Output:
Your question is a little unclear because you didn't mention what is the shape of tensor A and what is normA. But I guess the following:

A is a tensor of shape (batchSize, X, Y)
normA is a tensor of norms of all batch elements of A and its' shape is (batchSize).

So, you normalize the tensor A with the following statement.

A.div(normA.view(batchSize, 1, 1).expand_as(A))

Where, normA.view(batchSize, 1, 1).expand_as(A) is first converted into a tensor of shape (batchSize, X, Y) and then you divide A by the resulting tensor.

An example (created from my guess):

batchSize = 8

A = torch.randn(batchSize, 5, 5)
normA = A.norm(dim=-1).norm(dim=-1)
print(normA.size()) # torch.Size([8])

normA = normA.view(batchSize, 1, 1).expand_as(A)
print(normA.size()) # torch.Size([8, 5, 5])

A = A.div(normA)


Instruction: PyTorch - RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

Input:
I load my previously trained model and want to classify a single (test) image from the disk through this model. All the operations in my model are carried out on my GPU. Hence, I move the numpy array of the test image to GPU by calling cuda() function. When I call the forward() function of my model with the numpy array of the test image, I get the RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'.

Here is the code I use to load the image from disk and call the forward() function:

test_img = imageio.imread('C:\\Users\\talha\\Desktop\\dog.png')
test_img = imresize(test_img, (28, 28))
test_tensor = torch.from_numpy(test_img)
test_tensor = test_tensor.cuda()
test_tensor = test_tensor.type(torch.FloatTensor)
log_results = model.forward(test_tensor)

Software Environment:

torch: 1.0.1

GPU: Nvidia GeForce GTX 1070

OS: Windows 10 64-bit

Python: 3.7.1

Output:
Convert to FloatTensor before sending it over the GPU.

So, the order of operations will be:

test_tensor = torch.from_numpy(test_img)

# Convert to FloatTensor first
test_tensor = test_tensor.type(torch.FloatTensor)

# Then call cuda() on test_tensor
test_tensor = test_tensor.cuda()

log_results = model.forward(test_tensor)


Instruction: I am having trouble calculating the accuracy, recall, precision and f1-score for my model

Input:
I have got my confusion matrix working correctly, just having some trouble producing the scores. A little help would go a long way. I am currently getting the error. "Tensor object is not callable".

def get_confused(model_ft):
    nb_classes = 120
    from sklearn.metrics import precision_recall_fscore_support as score
    confusion_matrix = torch.zeros(nb_classes, nb_classes)
    with torch.no_grad():
        for i, (inputs, classes) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            classes = classes.to(device)
            outputs = model_ft(inputs)
            _, preds = torch.max(outputs, 1)
            for t, p in zip(classes.view(-1), preds.view(-1)):
                    confusion_matrix[t.long(), p.long()] += 1

            cm = confusion_matrix(classes, preds)
            recall = np.diag(cm) / np.sum(cm, axis = 1)
            precision = np.diag(cm) / np.sum(cm, axis = 0)
    print(confusion_matrix)
    print(confusion_matrix.diag()/confusion_matrix.sum(1))

Output:
The problem is with this line.

cm = confusion_matrix(classes, preds)

confusion_matrix is a tensor and you can't call it like a function. Hence Tensor is not callable. I am also, not sure why you need this line. Instead, I think you would want to write cm= confusion_matrix.cpu().data.numpy() to make it a numpy array I think. From your code, it seems cm is np.array.


Instruction: Pytorch sum tensors doing an operation within each set of numbers

Input:
I have the following Pytorch tensor:

V1 = torch.tensor([[2, 4], [6, 4], [5, 3]])

I want to do the sum of the difference of each pair of numbers (applying absolute value), something like the code below;

result.sum(abs(2-4), abs(6-4), abs(5-3))

I can do this using a for statement:

total = 0
for i in range(0,vector.size(0)):
    total = total + torch.abs(vector.data[i][1] - vector.data[i][0])

But I want to do it without using a for.

Is there a way to do that?

Output:
You can do

torch.abs(V1[:, 1]- V1[:, 0])

and to sum over it

torch.sum(torch.abs(V1[:, 1]- V1[:, 0]))


Instruction: Sample each image from dataset N times in single batch

Input:
I'm currently working on task of learning representation (deep embeddings). The dataset I use have only one example image per object. I also use augmentation.

During training, each batch must contain N different augmented versions of single image in dataset (dataset[index] always returns new random transformation).

Is there some standart solution or library with DataLoader for this purpose, that will work with torch.utils.data.distributed.DistributedSampler?
If not, will any DataLoader, inherited from torch.utils.data.DataLoader (and calling super().__init__(...)), will work in distributed training?

Output:
As far as I know, this is not a standard way of doing things — even if you have only one sample per object, one would still sample different images from different object per batch, and in different epochs the sampled images would be transformed differently.

That said, if you truly want to do what you are doing, why not simply write a wrapper of you dataset?

class Wrapper(Dataset):
    N = 16
    def __getitem__(self, index):
        sample = [ super().__getitem__(index) for _ in N ]
        sample = torch.stack(sample, dim=0)
        return sample

Then each of your batch would be BxNxCxHxW where B is the batch size, N is your repetition. You can reshape your batch after you get it from the dataloader.


Instruction: TypeError: argument 0 is not a Variable

Input:
I am trying to learn CNN with my own data. The shape of the data is (1224, 15, 23). 1224 is the number of data, and each data is (15, 23). CNN is built with PyTorch.

I think there is no logical error because conv2D needs 4-D tensor, and I feed (batch, channel, x, y).

when I build an instance of the Net class I got this error.

TypeError: argument 0 is not a Variable

I have been using PyTroch for half of a year but this error is the first time and I am still confused.

Here is my code.

class Net(nn.Module):
    def __init__(self, n):
        super(Net,self).__init__()

        self.conv = nn.Sequential(nn.Conv2d(1, 32, kernel_size=3, stride=1),
                                 nn.ReLU(),
                                 nn.Conv2d(32, 64, kernel_size=3, stride=1),
                                 nn.ReLU(),
                                 nn.Conv2d(64, 64, kernel_size=3, stride=1),  # 64 x 9 x 17
                                 nn.ReLU()
                                )

        conv_out_size = self._get_conv_out(input_shape)
        self.fc = nn.Sequential(nn.Linear(64 * 9 * 17, 128),
                               nn.ReLU(),
                               nn.Linear(128, n)
                               )

    def _get_conv_out(self, shape):
        o = self.conv(torch.zeros(1, *shape))
        return int(np.prod(o.size()))

    def forward(self, x):
        conv_out = self.conv(x).view(x.size()[0], -1)
        return sefl.fc(conv_out)

if __name__=='__main__':
    num_epochs = 1
    num_classes = 2
    input_shape = train_img[0].shape  # 1, 15, 23

    net = Net(num_classes)
    iteration = 51
    BATCH_SIZE = 24
    LEARNING_RATE = 0.0001

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

    loss_list= []
    batch_index = 0

    # train
    for epoch in range(num_epochs):
        for i in range(iteration):
            input_img = torch.FloatTensor(train_img[batch_index: batch_index + BATCH_SIZE])
            print(input_img.size())  # 24, 1, 15, 23

            outputs = net(input_img)
            loss = criterion(outputs, labels)
            loss_list.append(loss.item())

            # Backprop
            opimizer.zero_grad()
            loss.backward()
            optimizer.step()

And the error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-179-0f6bc7588c29> in <module>
      4     input_shape = train_img[0].shape  # 1, 15, 23
      5
----> 6     net = Net(num_classes)
      7     iteration = 51
      8     BATCH_SIZE = 24

<ipython-input-178-8a68d4a0dc4a> in __init__(self, n)
     11                                 )
     12
---> 13         conv_out_size = self._get_conv_out(input_shape)
     14         self.fc = nn.Sequential(nn.Linear(64 * 9 * 17, 128),
     15                                nn.ReLU(),

<ipython-input-178-8a68d4a0dc4a> in _get_conv_out(self, shape)
     18
     19     def _get_conv_out(self, shape):
---> 20         o = self.conv(torch.zeros(1, *shape))
     21         return int(np.prod(o.size()))
     22

C:\DTools\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--> 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

C:\DTools\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     65     def forward(self, input):
     66         for module in self._modules.values():
---> 67             input = module(input)
     68         return input
     69

C:\DTools\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--> 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

C:\DTools\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    280     def forward(self, input):
    281         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 282                         self.padding, self.dilation, self.groups)
    283
    284

C:\DTools\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\functional.py in conv2d(input, weight, bias, stride, padding, dilation, groups)
     88                 _pair(0), groups, torch.backends.cudnn.benchmark,
     89                 torch.backends.cudnn.deterministic, torch.backends.cudnn.enabled)
---> 90     return f(input, weight, bias)
     91
     92

TypeError: argument 0 is not a Variable

Output:
Your code actually works for PyTorch >= 0.4.1. I guess your PyTorch version is < 0.4 and so you need to pass a Variable in the following line.

o = conv(torch.autograd.Variable(torch.zeros(1, *x.shape)))

In PyTorch >= 0.4.1, the concept of Variable no longer exists. So, torch.FloatTensor can be directly passed to NN layers.


Instruction: pytorch model loading and prediction, AttributeError: 'dict' object has no attribute 'predict'

Input:
model = torch.load('/home/ofsdms/san_mrc/checkpoint/best_v1_checkpoint.pt', map_location='cpu')
results, labels = predict_function(model, dev_data, version)

> /home/ofsdms/san_mrc/my_utils/data_utils.py(34)predict_squad()
-> phrase, spans, scores = model.predict(batch)
(Pdb) n
AttributeError: 'dict' object has no attribute 'predict'

How do I load  a saved checkpoint of pytorch model, and use the same for prediction. I have the model saved in .pt extension

Output:
the checkpoint you save is usually a state_dict:  a dictionary containing the values of the trained weights - but not the actual architecture of the net. The actual computational graph/architecture of the net is described as a python class (derived from nn.Module).
To use a trained model you need:

Instantiate a model from the class implementing the computational graph.
Load the saved state_dict to that instance:

model.load_state_dict(torch.load('/home/ofsdms/san_mrc/checkpoint/best_v1_checkpoint.pt', map_location='cpu')


Instruction: PyTorch: inconsistent pretrained VGG output

Input:
When loading a pretrained VGG network with the torchvision.models module and using it to classify an arbitrary RGB image, the network's output differs noticeably from invocation to invocation. Why does this happen? From my understanding no part of the VGG forward pass should be non-deterministic.

Here's an MCVE:

import torch
from torchvision.models import vgg16

vgg = vgg16(pretrained=True)

img = torch.randn(1, 3, 256, 256)

torch.all(torch.eq(vgg(img), vgg(img))) # result is 0, but why?

Output:
vgg16 has a nn.Dropout layer that, during training, randomly drops 50% of its inputs. During test time you should "switch off" this behavior by setting the mode of the net to "eval" mode:

vgg.eval()
torch.all(torch.eq(vgg(img), vgg(img)))

Out[73]: tensor(1, dtype=torch.uint8)

Note that there are other layers with random behavior and different behavior for training and evaluation (e.g., BatchNorm). Therefore, it is important to switch to eval() mode before evaluating a trained model.


Instruction: How can I calculate the network gradients w.r.t weights for all inputs in PyTorch?

Input:
I'm trying to figure out how I can calculate the gradient of the network for each input. And I'm a bit lost. Essentially, what I would want, is to calculate d self.output/d weight1 and d self.output/d weight2 for all values of input x. So, I would have a matrix of size (1000, 5) for example. Where the 1000 is for the size of the input x, and 5 is the number of weights in the layer.

The example I've included below returns weights as size (1,5). What exactly is being calculated here? Is this d self.output/ d weight1 for 1 input of x, or an average of all inputs?

Secondly, would a matmul of features.grad and weight1.grad be the same as what I'm asking? A matrix of all the gradients of weight1 for all values of x.

class Network(torch.nn.Module):

    def __init__(self, iNode, hNode, oNode):
        super(Network, self).__init__()

        print("Building Model...")

        iNode = int(iNode) ; self.iNode = iNode
        hNode = int(hNode) ; self.hNode = hNode
        oNode = int(oNode) ; self.oNode = oNode

        self.fc1 = nn.Linear(iNode, hNode, bias=False)
        self.fc2 = nn.Linear(hNode, oNode, bias=False)

    def forward(self, x):
        self.hidden_probs = self.fc1(x)
        self.hidden = self.actFunc1(self.hidden_probs)
        self.output_probs = self.fc2(self.hidden)
        self.output = self.actFunc2(self.output_probs)
        return self.output

    def actFunc1(self, x):
        return 1.0/(1.0+torch.exp(-x))

    def actFunc2(self, x):
        return x

    def trainData(self, features, labels, epochs, alpha, optimisation, verbose=False):

        for epoch in range(0,epochs):
            net_pred = self.forward(features)
            net_pred.backward(gradient=torch.ones(features.size())) #calc. dout/dw for all w
print(features.grad.size()) #returns (1000,1)

            with torch.no_grad():
                for name, param in self.named_parameters():
                    if(param.requires_grad):
                        param -= alpha*param.grad

                for name, param in self.named_parameters():
                    if(param.requires_grad):
                        param.grad.zero_()

            sys.stdout.write("Epoch: %06i\r" % (epoch))
            sys.stdout.flush()
        sys.stdout.write("\n")

Output:
I am not sure what exactly you are trying to achieve because normally you only work with the sum of gradients of (d output)/(d parameter) and not with any other gradients in between as autograd takes care that, but let me try to answer.

Question 1

  The example I've included below returns weights as size (1,5). What exactly is being calculated here? Is this d self.output/ d weight1 for 1 input of x, or an average of all inputs?

You get size (1,5) because training is done in mini batches, meaning the gradients for each data point with respect to the (5) weights are calculated and summed over the mini batch.
According to the docs:

  This attribute is None by default and becomes a Tensor the first time a call to backward() computes gradients for self. The attribute will then contain the gradients computed and future calls to backward() will accumulate (add) gradients into it.

If you explicitly want the gradient for each data point, then make your mini batch size one. Normally we train in mini batches because updating after each data point can be unstable, image jumping in a different direction each time, where with a batch this would average out.
On the other extreme, many data sets are simply too large to calculate the gradient in one go.

Question 2

An example might give more insight:

    import torch
    x = torch.tensor([1.5], requires_grad=True)
    a = torch.nn.Parameter(torch.tensor([2.]))
    b = torch.nn.Parameter(torch.tensor([10.]))
    y = x*a
    z = y+0.5*b
    temp = z.backward()
    print('gradients of a: %0.2f and b: %0.2f' % (a.grad.item(), b.grad.item()))

I start with two parameters, a and b, and calculate z=a*x+0.5*b.
No gradients are calculated yet, pytorch only keeps track of the history of operations, so all .grad attributes are empty.
When z.backward() is called, the gradients of the output with respect to the parameters are calculated, which you can view by calling grad on the parameters.

Updating the parameters can then be done like you are already doing a -= alpha*a.grad.


Instruction: pytorch embedding index out of range

Input:
I'm following this tutorial here https://cs230-stanford.github.io/pytorch-nlp.html. In there a neural model is created, using nn.Module, with an embedding layer, which is initialized here

self.embedding = nn.Embedding(params['vocab_size'], params['embedding_dim'])

vocab_size is the total number of training samples, which is 4000.  embedding_dim is 50.  The relevant piece of the forward method is below

def forward(self, s):
        # apply the embedding layer that maps each token to its embedding
        s = self.embedding(s)   # dim: batch_size x batch_max_len x embedding_dim

I get this exception when passing a batch to the model like so
model(train_batch)
train_batch is a numpy array of dimension batch_sizexbatch_max_len.  Each sample is a sentence, and each sentence is padded so that it has the length of the longest sentence in the batch.

  File
  "/Users/liam_adams/Documents/cs512/research_project/custom/model.py",
  line 34, in forward
      s = self.embedding(s)   # dim: batch_size x batch_max_len x embedding_dim   File
  "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/modules/module.py",
  line 493, in call
      result = self.forward(*input, **kwargs)   File "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/modules/sparse.py",
  line 117, in forward
      self.norm_type, self.scale_grad_by_freq, self.sparse)   File "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/functional.py",
  line 1506, in embedding
      return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: index out of range at
  ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:193

Is the problem here that the embedding is initialized with different dimensions than those of my batch array?  My batch_size will be constant but batch_max_len will change with every batch. This is how its done in the tutorial.

Output:
Found the answer here https://discuss.pytorch.org/t/embeddings-index-out-of-range-error/12582

I'm converting words to indexes, but I had the indexes based off the total number of words, not vocab_size which is a smaller set of the most frequent words.


Instruction: Numpy/Pytorch dtype conversion / compatibility

Input:
I'm trying to find some documentation understand how dtypes are combined. For example:

x : np.int32 = ...
y : np.float64 = ...

What is going to be the type of x + y ?
does it depend on the operator (here +) ?
does it depend where it is stored (z = x + y vs z[...] = x + y) ?

I'm looking for part of the documentation that describe these kind of scenario but so far I'm empty-handed.

Output:
If data types don't match, then NumPy will upcast the data to the higher precision data types if possible. And it doesn't depend on the type of (arithmetic) operation that we do or to the variables that we assign to, unless that variable already has some other dtype. Here is a small illustration:

In [14]: x = np.arange(3, dtype=np.int32)
In [15]: y = np.arange(3, dtype=np.float64)

# `+` is equivalent to `numpy.add()`
In [16]: summed = x + y

In [17]: summed.dtype
Out[17]: dtype('float64')

In [18]: np.add(x, y).dtype
Out[18]: dtype('float64')

If you don't explicitly assign a datatype, then the result will be upcasted to the higher of data types of the given inputs. For example, numpy.add() accepts a dtype kwarg where you can specify the datatype of the resultant array.

And, one can check whether two different datatypes can be safely casted according to casting rules by using numpy.can_cast()

For the sake of completeness, I add the following numpy.can_cast() matrix:

>>> def print_casting_matrix(ntypes):
...     ntypes_ex = ["X"] + ntypes.split()
...     print("".join(ntypes_ex))
...     for row in ntypes:
...         print(row, sep='\t', end=''),
...         for col in ntypes:
...             print(int(np.can_cast(row, col)), sep='\t', end='')
...         print()

>>> print_casting_matrix(np.typecodes['All'])

And the output would be the following matrix which shows what dtypes can be safely casted (indicated by 1) and what dtypes cannot be casted (indicated by 0), following the order of from casting (along axis-0) to to casting (axis-1) :

# to casting -----> ----->
X?bhilqpBHILQPefdgFDGSUVOMm
?11111111111111111111111101
b01111110000001111111111101
h00111110000000111111111101
i00011110000000011011111101
l00001110000000011011111101
q00001110000000011011111101
p00001110000000011011111101
B00111111111111111111111101
H00011110111110111111111101
I00001110011110011011111101
L00000000001110011011111101
Q00000000001110011011111101
P00000000001110011011111101
e00000000000001111111111100
f00000000000000111111111100
d00000000000000011011111100
g00000000000000001001111100
F00000000000000000111111100
D00000000000000000011111100
G00000000000000000001111100
S00000000000000000000111100
U00000000000000000000011100
V00000000000000000000001100
O00000000000000000000001100
M00000000000000000000001110
m00000000000000000000001101

Since the characters are cryptic, we can use the following for better understanding of the above casting matrix:

In [74]: for char in np.typecodes['All']:
    ...:     print(char, " --> ", np.typeDict[char])

And the output would be:

?  -->  <class 'numpy.bool_'>
b  -->  <class 'numpy.int8'>
h  -->  <class 'numpy.int16'>
i  -->  <class 'numpy.int32'>
l  -->  <class 'numpy.int64'>
q  -->  <class 'numpy.int64'>
p  -->  <class 'numpy.int64'>
B  -->  <class 'numpy.uint8'>
H  -->  <class 'numpy.uint16'>
I  -->  <class 'numpy.uint32'>
L  -->  <class 'numpy.uint64'>
Q  -->  <class 'numpy.uint64'>
P  -->  <class 'numpy.uint64'>
e  -->  <class 'numpy.float16'>
f  -->  <class 'numpy.float32'>
d  -->  <class 'numpy.float64'>
g  -->  <class 'numpy.float128'>
F  -->  <class 'numpy.complex64'>
D  -->  <class 'numpy.complex128'>
G  -->  <class 'numpy.complex256'>
S  -->  <class 'numpy.bytes_'>
U  -->  <class 'numpy.str_'>
V  -->  <class 'numpy.void'>
O  -->  <class 'numpy.object_'>
M  -->  <class 'numpy.datetime64'>
m  -->  <class 'numpy.timedelta64'>


Instruction: Setting device for pytorch version 0.3.1.post2

Input: In newer versions of pytorch, you can set the device by using torch.device. I need to use torch version 0.3.1.post2 for some legacy code. How do I set the device for this version?

Output:
As far as I know, you can use the set_device function. But this is not encouraged. Please see the reference.

The suggested method is, you just set the CUDA_VISIBLE_DEVICES environmental variable. You can run your script as follows.

CUDA_VISIBLE_DEVICES=GPU_ID python script_name.py

In your program, you can just simply use .cuda() to use GPUs. (e.g., model=model.cuda())


Instruction: How to define specific number of convolutional kernels/filters in pytorch?

Input:
On pytorch website they have the following model in their tutorial

class BasicCNN(nn.Module):
    def __init__(self):
        super(BasicCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = x.permute(0, 3, 1, 2)
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How many kernels/filters this model has? Is it two - e.g conv1 and conv2. How do I easily create many filters by specifying the number of them? For example 100 filters.

Thanks!

Output:
Your question is a little ambiguous but let me try to answer it.

Usually, in a convolutional layer, we set the number of filters as the number of out_channels. But this is not straight-forward. Let's discuss based on the example that you provided.

  What are the convolutional layer parameters?

model = BasicCNN()
for name, params in model.named_parameters():
    if 'conv' in name:
        print(name, params.size())

Output:

conv1.weight torch.Size([6, 3, 5, 5])
conv1.bias torch.Size([6])
conv2.weight torch.Size([16, 6, 5, 5])
conv2.bias torch.Size([16])

  Explanation

Let's consider conv1 layer in the above model. We can say, there are 6 filters of shape 5 x 5 because we have chosen 2d Convolution. Since the number of input channels is 3, so there are in total 6 x 3 = 18 kernels.

Here, the inputs of this model are 3d like images. You can consider, we have images with shape W x H and there are 3 channels (RGB) for images. So, we can feed 3d tensors representing images to this model.

Now coming back to your question, "How do I easily create many filters by specifying the number of them? For example 100 filters.". If you want to simply use 100 filters per input channel, then just set 100 in conv1 instead of 6. This is typically what people do in computer vision!

But you can definitely modify the architecture as per your need and identify the best setting.


Instruction: PyTorch and Chainer implementations of the Linear layer- are they equivalent?

Input:
I want to use a Linear, Fully-Connected Layer as one of the input layers in my network. The input has shape (batch_size, in_channels, num_samples). It is based on the Tacotron paper: https://arxiv.org/pdf/1703.10135.pdf, the Enocder prenet part.
It feels to me as if Chainer and PyTorch have different implementations of the Linear layer - are they really performing the same operations or am I misunderstanding something?
In PyTorch, behavior of the Linear layer follows the documentations: https://pytorch.org/docs/0.3.1/nn.html#torch.nn.Linear
according to which, the shape of the input and output data are as follows:

Input: (N,∗,in_features) where * means any number of additional dimensions
Output: (N,∗,out_features) where all but the last dimension are the same shape as the input.

Now, let's try creating a linear layer in pytorch and performing the operation. I want an output with 8 channels, and the input data will have 3 channels.
import numpy as np
import torch
from torch import nn
linear_layer_pytorch = nn.Linear(3, 8)

Let's create some dummy input data of shape (1, 4, 3) - (batch_size, num_samples, in_channels:
data = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4], dtype=np.float32).reshape(1, 4, 3)
data_pytorch = torch.from_numpy(data)

and finally, perform the operation:
results_pytorch = linear_layer_pytorch(data_pytorch)
results_pytorch.shape

the shape of the output is as follows: Out[27]: torch.Size([1, 4, 8])
Taking a look at the source of the PyTorch implementation:
def linear(input, weight, bias=None):
    # type: (Tensor, Tensor, Optional[Tensor]) -> Tensor
    r"""
    Applies a linear transformation to the incoming data: :math:`y = xA^T + b`.

    Shape:

        - Input: :math:`(N, *, in\_features)` where `*` means any number of
          additional dimensions
        - Weight: :math:`(out\_features, in\_features)`
        - Bias: :math:`(out\_features)`
        - Output: :math:`(N, *, out\_features)`
    """
    if input.dim() == 2 and bias is not None:
        # fused op is marginally faster
        ret = torch.addmm(bias, input, weight.t())
    else:
        output = input.matmul(weight.t())
        if bias is not None:
            output += bias
        ret = output
    return ret

It transposes the weight matrix that is passed to it, broadcasts it along the batch_size axis and performs a matrix multiplications. Having in mind how a linear layer works, I imagine it as 8 nodes, connected through a synapse, holding a weight, with every channel in an input sample, thus in my case it has 3*8 weights. And that is exactly the shape I see in debugger (8, 3).
Now, let's jump to Chainer. The Chainer's linear layer documentation is available here: https://docs.chainer.org/en/stable/reference/generated/chainer.links.Linear.html#chainer.links.Linear. According to this documentation, the Linear layer wraps the function linear, which according to the docs, flattens the input along the non-batch dimensions and the shape of it's weight matrix is (output_size, flattend_input_size)
import chainer
linear_layer_chainer = chainer.links.Linear(8)
results_chainer = linear_layer_chainer(data)
results_chainer.shape
Out[21]: (1, 8)

Creating the layer as linear_layer_chainer = chainer.links.Linear(3, 8) and calling it causes a size mismatch. So in case of chainer, I have gotten a totally different results, because this time around I have a weight matrix that is of shape (8, 12) and my results have a shape of (1, 8). So now, here is my question : since the results are clearly different,both the weight matrices and the outputs have different shapes, how can I make them equivalent and what should be the desired output? In the PyTorch implementation of Tacotron it seems that the PyTorch approach is used as is (https://github.com/mozilla/TTS/blob/master/layers/tacotron.py) - Prenet. If that is the case, how can I make the Chainer produce the same results (I have to implement this in Chainer). I will be grateful for any inshight, sorry that the post has gotten this long.

Output:
Chainer Linear layer (a bit frustratingly) does not apply the transformation to the last axis. Chainer flattens the rest of the axes. Instead you need to provide how many batch axes there are, documentation which is 2 in your case:

# data.shape == (1, 4, 3)
results_chainer = linear_layer_chainer(data, n_batch_axes=2)
# 2 batch axes (1,4) means you apply linear to (..., 3)
# results_chainer.shape == (1, 4, 8)

You can also use l(data, n_batch_axes=len(data.shape)-1) to always apply to the last dimension which is the default behaviour in PyTorch, Keras etc.


Instruction: Slicing uneven columns from tensor array

Input:
I have an array like so:

([[[ 0,  1,  2],
 [ 3,  4,  5]],

[[ 6,  7,  8],
[ 9, 10, 11]],

[[12, 13, 14],
[15, 16, 17]]])

If i want to slice the numbers 12 to 17 i would use:

arr[2, 0:2, 0:3]

but how would i go about slicing the array to get 12 to 16?

Output:
You'll need to "flatten" the last two dimensions first. Only then will you be able to extract the elements you want:

xf = x.view(x.size(0), -1)  # flatten the last dimensions
xf[2, 0:5]

Out[87]: tensor([12, 13, 14, 15, 16])


Instruction: Considerations of model definitions when moving from Tensorflow to PyTorch

Input:
I've just recently switched to PyTorch after getting frustrated in debugging tf and understand that it is equivalent to coding in numpy almost completely. My question is what are the permitted python aspects we can use in a PyTorch model (to be put completely on GPU) eg. if-else has to be implemented as follows in tensorflow

a = tf.Variable([1,2,3,4,5], dtype=tf.float32)
b = tf.Variable([6,7,8,9,10], dtype=tf.float32)
p = tf.placeholder(dtype=tf.float32)
ps = tf.placeholder(dtype=tf.bool)

li = [None]*5
li_switch = [True, False, False, True, True]

for i in range(5):
    li[i] = tf.Variable(tf.random.normal([5]))

sess = tf.Session()
sess.run(tf.global_variables_initializer())

def func_0():
    return tf.add(a, p)
def func_1():
    return tf.subtract(b, p)

with tf.device('GPU:0'):
    my_op = tf.cond(ps, func_1, func_0)

for i in range(5):
    print(sess.run(my_op, feed_dict={p:li[i], ps:li_switch[i]}))

How would the structure change in pytorch for the above code? How to place the variables and ops above on GPU and parallelize the list inputs to our graph in pytorch?

Output:
In pytorch, the code can be written like the way normal python code is written.

CPU

import torch
a = torch.FloatTensor([1,2,3,4,5])
b = torch.FloatTensor([6,7,8,9,10])
cond = torch.randn(5)

for ci in cond:
    if ci > 0:
        print(torch.add(a, 1))
    else:
        print(torch.sub(b, 1))

GPU

Move the tensors to GPU like this:

a = torch.FloatTensor([1,2,3,4,5]).to('cuda')
b = torch.FloatTensor([6,7,8,9,10]).to('cuda')
cond = torch.randn(5).to('cuda')

import torch.nn as nn

class Cond(nn.Module):
    def __init__(self):
        super(Cond, self).__init__()

    def forward(self, cond, a, b):
        result =  torch.empty(cond.shape[0], a.shape[0]).cuda()
        for i, ci in enumerate(cond):
            if ci > 0:
                result[i] = torch.add(a, 1)
            else:
                result[i] = torch.sub(b, 1)

        return result

cond_model = Cond().to('cuda')
output = cond_model(cond, a, b)

https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#cuda-tensors


Instruction: How to add a 2d tensor to every 2d tensor from a 3d tensor

Input:
I'm trying to add a 2d tensor to every 2d tensor from a 3d tensor.
Let's say i have a  tensor a with (2,3,2) shape and a tensor b with (2,2) shape.

a = [[[1,2],
      [1,2],
      [1,2]],
     [[3,4],
      [3,4],
      [3,4]]]

b = [[1,2], [3,4]]

#the result i want to get
a[:, 0, :] + b
a[:, 1, :] + b
a[:, 2, :] + b

I wanna know if there is a method in pytorch that can do this.

Output:
The most efficient way of doing this would be to add an extra second dimension to b and use broadcasting to add:

a = torch.Tensor([[[1,2],[1,2],[1,2]],[[3,4],[3,4],[3,4]]])
b = torch.Tensor([[1,2],[3,4]])
a += b.unsqueeze(1)


Instruction: Pytorch: How can I find indices of first nonzero element in each row of a 2D tensor?

Input:
I have a 2D tensor with some nonzero element in each row like this:

import torch
tmp = torch.tensor([[0, 0, 1, 0, 1, 0, 0],
                    [0, 0, 0, 1, 1, 0, 0]], dtype=torch.float)

I want a tensor containing the index of first nonzero element in each row:

indices = tensor([2],
                 [3])

How can I calculate it in Pytorch?

Output:
I could find a tricky answer for my question:

  tmp = torch.tensor([[0, 0, 1, 0, 1, 0, 0],
                     [0, 0, 0, 1, 1, 0, 0]], dtype=torch.float)
  idx = reversed(torch.Tensor(range(1,8)))
  print(idx)

  tmp2= torch.einsum("ab,b->ab", (tmp, idx))

  print(tmp2)

  indices = torch.argmax(tmp2, 1, keepdim=True)
  print(indeces)

The result is:

tensor([7., 6., 5., 4., 3., 2., 1.])
tensor([[0., 0., 5., 0., 3., 0., 0.],
       [0., 0., 0., 4., 3., 0., 0.]])
tensor([[2],
        [3]])


Instruction: Get Euclidian and infinite distance in Pytorch

Input:
I'm trying to get the Euclidian Distance in Pytorch, using torch.dist, as shown below:

torch.dist(vector1, vector2, 1)

If I use "1" as the third Parameter, I'm getting the Manhattan distance, and the result is correct, but I'm trying to get the Euclidian and Infinite distances and the result is not right. I tried with a lot of different numbers on the third parameter, but I'm not able to get the desired distances.

How can I get the Euclidian and Infinite distances using Pytorch?

Output:
You should use the .norm() instead of .dist().

vector1 = torch.FloatTensor([3, 4, 5])
vector2 = torch.FloatTensor([1, 1, 1])

dist = torch.norm(vector1 - vector2, 1)
print(dist) # tensor(9.)
dist = torch.norm(vector1 - vector2, 2)
print(dist) # tensor(5.3852)
dist = torch.norm(vector1 - vector2, float("inf"))
print(dist) # tensor(4.)

dist = torch.dist(vector1, vector2, 1)
print(dist) # tensor(9.)
dist = torch.dist(vector1, vector2, 2)
print(dist) # tensor(5.3852)
dist = torch.dist(vector1, vector2, float("inf"))
print(dist) # tensor(1.)

As we can see for the infinity distance, .norm() returns the correct answer.


Instruction: IndexError - Implementing the test of CBOW with pytorch

Input:
I am not very used to python & machine learning code. I am testing on pytorch CBOW test, but it says something about Index Error. Can anyone help?

# model class
class CBOW(nn.Module):

    ...
    def get_word_embedding(self, word):
    word = torch.cuda.LongTensor([word_to_ix[word]])
    return self.embeddings(word).view(1, -1)

# test method
def test_cbow(model, train_words, word_to_ix):
    # test word similarity
    word_1 = train_words[2] #randomly chosen word
    word_2 = train_words[3] #randomly chosen word

    word_1_vec = model.get_word_embedding(word_1)[0].cpu()
    word_2_vec = model.get_word_embedding(word_2)[0].cpu()

    print(word_1_vec)
    print(word_2_vec)

    word_similarity = (word_1_vec.dot(word_2_vec) / (torch.norm(word_1_vec) * torch.norm(word_2_vec))).data.numpy()[0]
    print("Similarity between '{}' & '{}' : {:0.4f}".format(word_1, word_2, word_similarity))

# executing the test
test_cbow(model, train_words, word_to_ix)

BELOW IS THE RESULT:

tensor([ 0.8978,  1.0713, -1.6856, -1.0967, -0.0114,  0.4107, -0.4293, -0.7351,
         0.4410, -1.5937, -1.3773,  0.7744,  0.0739, -0.3263,  1.0342,  1.0420,
        -1.1333,  0.4158,  1.1316, -0.0141, -0.8383,  0.2544, -2.2409, -1.1858,
         0.2652, -0.3232,  0.1287, -1.5274,  0.3199, -2.1822,  0.9464, -0.6619,
         1.1549,  0.5276,  0.0849, -0.1594, -1.7922,  1.3567, -0.4376, -0.9093,
         1.0701,  1.5373, -1.3277, -1.1833,  1.8070, -0.0551, -0.8439,  1.5236,
        -0.3890, -0.2306, -0.7392, -1.6435,  0.4485,  0.8988, -0.5958, -0.6989,
         1.6123, -1.6668,  0.0583,  0.6698, -0.6998,  1.1942,  0.6355,  0.7437,
        -1.0006, -0.5398,  1.3197,  1.3696, -0.3221,  0.9004,  0.6268,  0.0221,
         0.0269, -1.7966, -1.6153, -0.1695, -0.0339, -0.5145,  1.5744, -0.3388,
        -0.9617,  0.6750, -1.1334,  0.0377,  1.1123,  1.1002, -0.3605,  0.2105,
        -1.6570,  1.3818,  0.9183,  0.0274,  0.9072,  0.8414,  0.3424,  0.2199,
         1.6546, -0.1357,  1.1291, -0.5309], grad_fn=<CopyBackwards>)
tensor([-0.6263, -0.5639,  2.1590, -0.3659,  0.2862, -0.4542, -0.4825, -0.1776,
        -0.4242,  0.9525,  0.7138, -0.3107,  1.8733, -0.3406,  0.0277,  1.6775,
         2.1893,  2.0332,  0.7185,  0.0050, -0.1627, -0.1113,  1.0444,  1.4057,
         0.2183,  0.3405,  0.0930,  1.2428, -0.0740,  0.3991, -0.2722,  1.4980,
         0.9207,  0.5008, -1.9297,  0.5600,  1.6416,  1.1550,  0.1440,  0.0739,
        -0.7465, -0.2458,  0.9217,  0.7156, -1.2558, -0.9891, -0.7313,  0.8501,
        -1.2851, -0.3068, -0.0796,  0.9361,  0.0927, -1.2988,  0.7422,  0.1388,
         1.3895, -0.7935,  0.4008, -0.1338,  1.5563,  0.5864,  0.6606, -0.2341,
         0.1218, -0.7313,  0.5073, -0.2941,  0.0316, -2.5356, -0.0885,  2.5765,
         0.2090,  0.2819, -0.0386,  0.7986,  2.1165, -0.0271, -0.2987,  0.2905,
         0.0149,  0.2403,  0.0752, -1.5535,  0.3794,  2.0638,  1.0603,  0.0703,
        -0.3643, -1.5671, -0.4736, -1.3035,  0.6583,  0.2531,  0.9829, -0.6025,
        -0.8148, -0.3457, -0.7339,  0.6758], grad_fn=<CopyBackwards>)

What I am also confused is that I have to convert cuda datatype to numpy, since I used cuda in get_word_embedding method. Is adding .cpu() to convert the datatype correct?

IndexError                                Traceback (most recent call last)
<ipython-input-68-39d73aa6e0de> in <module>()
     17     print("Similarity between '{}' & '{}' : {:0.4f}".format(word_1, word_2, word_similarity))
     18
---> 19 test_cbow(model, train_words, word_to_ix)

<ipython-input-68-39d73aa6e0de> in test_cbow(model, train_words, word_to_ix)
     14     print(type(word_1_vec))
     15
---> 16     word_similarity = (word_1_vec.dot(word_2_vec) / (torch.norm(word_1_vec) * torch.norm(word_2_vec))).data.numpy()[0]
     17     print("Similarity between '{}' & '{}' : {:0.4f}".format(word_1, word_2, word_similarity))
     18

**IndexError: too many indices for array**

Output:
In your code, word_similarity is not an array, so you can't access it's 0th element. Just modify your code as:

word_similarity = (word_1_vec.dot(word_2_vec) / (torch.norm(word_1_vec) * torch.norm(word_2_vec))).data.numpy()

You can also use:

word_similarity = (word_1_vec.dot(word_2_vec) / (torch.norm(word_1_vec) * torch.norm(word_2_vec))).item()

Here, the .item() in PyTorch would directly give you a float value since the similarity value is a scalar.

  When to use .cpu()?

To convert a cuda tensor to cpu tensor, you need to use .cpu(). You cannot directly convert a cuda tensor to numpy. You have to convert to cpu tesnor first and then call .numpy().


Instruction: How to create embeddedings for a column that is a list of categorical values

Input:
I am having some trouble deciding how to create embeddings for a categorical feature for my DNN model. The feature consists of a non fixed set of tags.

The feature is like:

column = [['Adventure','Animation','Comedy'],
          ['Adventure','Comedy'],
          ['Adventure','Children','Comedy']

I would like to do this with tensorflow so I know the tf.feature_column module should work, I just don't know which version to use.

Thanks!

Output:
First you need to fill in your features to the same length.

import itertools
import numpy as np

column = np.array(list(itertools.zip_longest(*column, fillvalue='UNK'))).T
print(column)

[['Adventure' 'Animation' 'Comedy']
 ['Adventure' 'Comedy' 'UNK']
 ['Adventure' 'Children' 'Comedy']]

Then you can use tf.feature_column.embedding_column to create embeddings for a categorical feature. The inputs of embedding_column must be a CategoricalColumn created by any of the categorical_column_* function.

# if you have big vocabulary list in files, you can use tf.feature_column.categorical_column_with_vocabulary_file
cat_fc = tf.feature_column.categorical_column_with_vocabulary_list(
    'cat_data', # identifying the input feature
    ['Adventure', 'Animation', 'Comedy', 'Children'], # vocabulary list
    dtype=tf.string,
    default_value=-1)

cat_column = tf.feature_column.embedding_column(
    categorical_column =cat_fc,
    dimension = 5,
    combiner='mean')

categorical_column_with_vocabulary_list will ignore the 'UNK' since there is no 'UNK' in vocabulary list. dimension specifying dimension of the embedding and combiner specifying how to reduce if there are multiple entries in a single row with 'mean' the default in embedding_column.

The result:

tensor = tf.feature_column.input_layer({'cat_data':column}, [cat_column])

with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    session.run(tf.tables_initializer())
    print(session.run(tensor))

[[-0.694761   -0.0711766   0.05720187  0.01770079 -0.09884425]
 [-0.8362482   0.11640486 -0.01767573 -0.00548441 -0.05738768]
 [-0.71162754 -0.03012567  0.15568805  0.00752804 -0.1422816 ]]


Instruction: How to convert binary classifier output/loss tensor in LSTM to multiclass

Input:
I am trying to train an LSTM model to predict what year a song was written given its lyrics using word-level association in Pytorch. There are 51 potential classes/labels (1965-2015) - however I was working off of a template that used a binary classifier for a different problem. I have been trying to figure out how to change the model to predict multiple classes (1965, 1966, etc).

I understand that you are supposed to provide a tensor of size C=num_classes as the output. However, I did that by making output_size=51 but I am getting an error, which makes me think there is something related to defining or operating on the criterion class I am defining that I am not doing correctly.

Here is the model:

class LyricLSTM(nn.Module):
    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        super().__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                            dropout=drop_prob, batch_first=True)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()
        #self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x, hidden):
        batch_size = x.size(0)

        # embeddings and lstm_out
        embeds = self.embedding(x)
        lstm_out, hidden = self.lstm(embeds, hidden)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        # sigmoid function
        sig_out = self.sig(out)
        #sig_out = self.softmax(out)

        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1)
        sig_out = sig_out[:, -1]  # get last batch of labels

        # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

        return hidden

And the training loop:

n_epochs = 10
batch_size = 16 #100  # 11 batches of size 337 so iters = 11 (11 * 337 = 3707)

# Split into training, validation, testing - train= 80% | valid = 10% | test = 10%
split_frac = 0.8
train_x = encoded_lyrics[0:int(split_frac * len(encoded_lyrics))] # 3707 training samples
train_y = encoded_years[0:int(split_frac * len(encoded_lyrics))]  # 3707 training samples

# Dataloaders and batching
# create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))

# make sure to SHUFFLE your data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)

output_size = 51
embedding_dim = 400
hidden_dim = 128 #256
n_layers = 2
lstmc = lstm.LyricLSTM(vocab_len, output_size, embedding_dim, hidden_dim, n_layers)

# Loss function + accuracy reporting
current_loss = 0
losses = np.zeros(n_epochs)  # For plotting
accuracy = np.zeros(n_epochs)

lr = 0.001
criterion = nn.CrossEntropyLoss() #nn.BCELoss()
optimizer = torch.optim.Adam(lstmc.parameters(), lr=lr)
counter = 0
print_every = 1
clip = 5  # gradient clipping

# Main training loop
start = time.time()
lstmc.train()
for epoch in range(0, n_epochs):
    # initialize hidden state
    h = lstmc.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        lstmc.zero_grad()

        # get the output from the model
        inputs = inputs.type(torch.LongTensor)
        output, h = lstmc(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()

        nn.utils.clip_grad_norm_(lstmc.parameters(), clip)
        optimizer.step()

I am getting this error when I run the code

File "main.py", line 182, in main
    loss = criterion(output.squeeze(), labels.float())
/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
/venv/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1295, in log_softmax
    ret = input.log_softmax(dim)
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

This is the output I am getting and the labels (for batch size 16):

Output: tensor([0.4962, 0.5025, 0.4963, 0.4936, 0.5058, 0.4872, 0.4995, 0.4852, 0.4840,
        0.4791, 0.4984, 0.5034, 0.4796, 0.4826, 0.4811, 0.4859],
       grad_fn=<SqueezeBackward0>)

Labels: tensor([1994., 1965., 1981., 1986., 1973., 1981., 1975., 1968., 1981., 1968.,
        1989., 1981., 1988., 1991., 1983., 1982.])

I was expecting for the output to be a tensor of length 51 where each element contained the likelihood that that year was the correct answer (ex: output[0] = first year / 1965, output[1] = 1966, etc).

Output:
You must provide input as (N, C) and target as (N) to CrossEntropyLoss. I suspect in your model's foward() method, the following code segment is wrong.

sig_out = self.sig(out) # shape: batch_size*seq_len x output_size

# reshape to be batch_size first
sig_out = sig_out.view(batch_size, -1) # shape: batch_size x seq_len*output_size
sig_out = sig_out[:, -1] # shape: batch_size

What you are trying to do with your last statement? Also, what you want to do with the seq_len dimension of the LSTM output?

Try to think about what you are doing here.

Although I think the shape of the output tensor is wrong, make sure output is a 2d tensor of shape (N, C) and labels is a 1d tensor of shape (N).

Also, I see a few problems in your code.

Usually, it is a good practice to apply zero_grad to the optimizer, not on the model. Don't do the following.

# zero accumulated gradients
lstmc.zero_grad()

Instead, do: optimizer.zero_grad()

You shouldn't use Sigmoid with 51 classes. Rather use fully connected layers followed by a softmax layer. And you don't need view() operations before the fc and softmax layers.

self.fc = nn.Linear(hidden_dim, output_size)
self.softmax = nn.LogSoftmax(dim=-1) # use -1 to apply in the last axis

...

out = self.dropout(lstm_out)
out = self.softmax(self.fc(out))

So, don't use the following code segment.

# stack up lstm outputs
lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim) # DON'T DO THIS


Instruction: Gradient calculation not disabled in no_grad() PyTorch

Input:
Why is the gradient calculation of y not disabled in the following piece of code?

x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
y = x**2
print(y.requires_grad)
with torch.no_grad():
    print((x ** 2).requires_grad)
    print(y.requires_grad)

Which gives the following output:

True
True
True
False
True

Output:
Going through the official documentation says that the results would have require_grad=False even though the inputs have required_grad=True

  Disabling gradient calculation is useful for inference, when you are sure
      that you will not call :meth:Tensor.backward(). It will reduce memory
      consumption for computations that would otherwise have requires_grad=True.
      In this mode, the result of every computation will have
      requires_grad=False, even when the inputs have requires_grad=True.


Instruction: KeyError: 'predictions' When use SimpleSeq2SeqPredictor to predict string

Input:
Please first search our GitHub repository for similar questions.  If you don't find a similar example you can use the following template:

System (please complete the following information):
 - OS: Ubunti 18.04
 - Python version: 3.6.7
 - AllenNLP version: v0.8.3
 - PyTorch version: 1.1.0

Question
When I Try to predict string using SimpleSeq2SeqPredictor, It always show that

Traceback (most recent call last):
  File "predict.py", line 96, in <module>
    p = predictor.predict(i)
  File "venv/lib/python3.6/site-packages/allennlp/predictors/seq2seq.py", line 17, in predict
    return self.predict_json({"source" : source})
  File "/venv/lib/python3.6/site-packages/allennlp/predictors/predictor.py", line 56, in predict_json
    return self.predict_instance(instance)
  File "/venv/lib/python3.6/site-packages/allennlp/predictors/predictor.py", line 93, in predict_instance
    outputs = self._model.forward_on_instance(instance)
  File "/venv/lib/python3.6/site-packages/allennlp/models/model.py", line 124, in forward_on_instance
    return self.forward_on_instances([instance])[0]
  File "/venv/lib/python3.6/site-packages/allennlp/models/model.py", line 153, in forward_on_instances
    outputs = self.decode(self(**model_input))
  File "/venv/lib/python3.6/site-packages/allennlp/models/encoder_decoders/simple_seq2seq.py", line 247, in decode
    predicted_indices = output_dict["predictions"]
KeyError: 'predictions'

I try to do a translate system, but I am newbie, most of code come from
https://github.com/mhagiwara/realworldnlp/blob/master/examples/mt/mt.py
http://www.realworldnlpbook.com/blog/building-seq2seq-machine-translation-models-using-allennlp.html

this is my training code

EN_EMBEDDING_DIM = 256
ZH_EMBEDDING_DIM = 256
HIDDEN_DIM = 256
CUDA_DEVICE = 0
prefix = 'small'

reader = Seq2SeqDatasetReader(
    source_tokenizer=WordTokenizer(),
    target_tokenizer=CharacterTokenizer(),
    source_token_indexers={'tokens': SingleIdTokenIndexer()},
    target_token_indexers={'tokens': SingleIdTokenIndexer(namespace='target_tokens')},
    lazy = True)
train_dataset = reader.read(f'./{prefix}-data/train.tsv')
validation_dataset = reader.read(f'./{prefix}-data/val.tsv')

vocab = Vocabulary.from_instances(train_dataset,
                                    min_count={'tokens': 3, 'target_tokens': 3})

en_embedding = Embedding(num_embeddings=vocab.get_vocab_size('tokens'),
                            embedding_dim=EN_EMBEDDING_DIM)
# encoder = PytorchSeq2SeqWrapper(
#     torch.nn.LSTM(EN_EMBEDDING_DIM, HIDDEN_DIM, batch_first=True))
encoder = StackedSelfAttentionEncoder(input_dim=EN_EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, projection_dim=128, feedforward_hidden_dim=128, num_layers=1, num_attention_heads=8)

source_embedder = BasicTextFieldEmbedder({"tokens": en_embedding})

# attention = LinearAttention(HIDDEN_DIM, HIDDEN_DIM, activation=Activation.by_name('tanh')())
# attention = BilinearAttention(HIDDEN_DIM, HIDDEN_DIM)
attention = DotProductAttention()

max_decoding_steps = 20   # TODO: make this variable
model = SimpleSeq2Seq(vocab, source_embedder, encoder, max_decoding_steps,
                        target_embedding_dim=ZH_EMBEDDING_DIM,
                        target_namespace='target_tokens',
                        attention=attention,
                        beam_size=8,
                        use_bleu=True)
optimizer = optim.Adam(model.parameters())
iterator = BucketIterator(batch_size=32, sorting_keys=[("source_tokens", "num_tokens")])

iterator.index_with(vocab)
if torch.cuda.is_available():
    cuda_device = 0
    model = model.cuda(cuda_device)
else:
    cuda_device = -1
trainer = Trainer(model=model,
                    optimizer=optimizer,
                    iterator=iterator,
                    train_dataset=train_dataset,
                    validation_dataset=validation_dataset,
                    num_epochs=50,
                    serialization_dir=f'ck/{prefix}/',
                    cuda_device=cuda_device)

# for i in range(50):
    # print('Epoch: {}'.format(i))
trainer.train()

predictor = SimpleSeq2SeqPredictor(model, reader)

for instance in itertools.islice(validation_dataset, 10):
    print('SOURCE:', instance.fields['source_tokens'].tokens)
    print('GOLD:', instance.fields['target_tokens'].tokens)
    print('PRED:', predictor.predict_instance(instance)['predicted_tokens'])

# Here's how to save the model.
with open(f"ck/{prefix}/manually_save_model.th", 'wb') as f:
    torch.save(model.state_dict(), f)
vocab.save_to_files(f"ck/{prefix}/vocabulary")

and this is my predict code

EN_EMBEDDING_DIM = 256
ZH_EMBEDDING_DIM = 256
HIDDEN_DIM = 256
CUDA_DEVICE = 0
prefix = 'big'

reader = Seq2SeqDatasetReader(
    source_tokenizer=WordTokenizer(),
    target_tokenizer=CharacterTokenizer(),
    source_token_indexers={'tokens': SingleIdTokenIndexer()},
    target_token_indexers={'tokens': SingleIdTokenIndexer(namespace='target_tokens')},
    lazy = True)
# train_dataset = reader.read(f'./{prefix}-data/train.tsv')
# validation_dataset = reader.read(f'./{prefix}-data/val.tsv')

# vocab = Vocabulary.from_instances(train_dataset,
#                                     min_count={'tokens': 3, 'target_tokens': 3})
vocab = Vocabulary.from_files("ck/small/vocabulary")

en_embedding = Embedding(num_embeddings=vocab.get_vocab_size('tokens'),
                            embedding_dim=EN_EMBEDDING_DIM)
# encoder = PytorchSeq2SeqWrapper(
#     torch.nn.LSTM(EN_EMBEDDING_DIM, HIDDEN_DIM, batch_first=True))
encoder = StackedSelfAttentionEncoder(input_dim=EN_EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, projection_dim=128, feedforward_hidden_dim=128, num_layers=1, num_attention_heads=8)

source_embedder = BasicTextFieldEmbedder({"tokens": en_embedding})

# attention = LinearAttention(HIDDEN_DIM, HIDDEN_DIM, activation=Activation.by_name('tanh')())
# attention = BilinearAttention(HIDDEN_DIM, HIDDEN_DIM)
attention = DotProductAttention()

max_decoding_steps = 20   # TODO: make this variable
model = SimpleSeq2Seq(vocab, source_embedder, encoder, max_decoding_steps,
                        target_embedding_dim=ZH_EMBEDDING_DIM,
                        target_namespace='target_tokens',
                        attention=attention,
                        beam_size=8,
                        use_bleu=True)

# And here's how to reload the model.
with open("./ck/small/best.th", 'rb') as f:
    model.load_state_dict(torch.load(f))

predictor = Seq2SeqPredictor(model, dataset_reader=reader)
# print(predictor.predict("The dog ate the apple"))

test = [
    'Surely ,he has no power over those who believe and put their trust in their Lord ;',
    'And assuredly We have destroyed the generations before you when they did wrong ,while their apostles came unto them with the evidences ,and they were not such as to believe . In this wise We requite the sinning people .',
    'And warn your tribe ( O Muhammad SAW ) of near kindred .',
    'And to the Noble Messengers whom We have mentioned to you before ,and to the Noble Messengers We have not mentioned to you ; and Allah really did speak to Moosa .',
    'It is He who gave you hearing ,sight ,and hearts ,but only few of you give thanks .',
    'spreading in them much corruption ?',
    'That will envelop the people . This will be a painful punishment .',
    'When you received it with your tongues and spoke with your mouths what you had no knowledge of ,and you deemed it an easy matter while with Allah it was grievous .',
    'of which you are disregardful .',
    'Whoever disbelieves ,then the calamity of his disbelief is only on him ; and those who do good deeds ,are preparing for themselves .'
]

for i in test:
    p = predictor.predict(i) # <------------------- ERROR !!!!!!!
    print(p)

Am I do something wrong ?

Output:
solved
I forgot add  model.eval() after load model.
Sorry


Instruction: How to use Adam optim considering of its adaptive learning rate?

Input:
In the Adam optimization algorithm, the learning speed is adjusted according to the number of iterations. I don't quite understand Adam's design, especially when using batch training. When using batch training, if there are 19,200 pictures, each time 64 pictures are trained, it is equivalent to 300 iterations. If our epoch has 200 times, there are a total of 60,000 iterations. I don't know if such multiple iterations will reduce the learning speed to a very small size. So when we are training, shall we initialize the optim after each epoch, or do nothing throughout the process?

Using pytorch. I have tried to initialize the optim after each epoch if I use batch train, and I do nothing when the number of data is small.

For expample, I don't know whether the two pieces of code is correct:

optimizer = optim.Adam(model.parameters(), lr=0.1)
for epoch in range(100):
    ###Some code
    optim.step()

Another piece of code:

for epoch in range(100):
    optimizer = optim.Adam(model.parameters(), lr=0.1)
    ###Some code
    optim.step()

Output:
You can read the official paper here https://arxiv.org/pdf/1412.6980.pdf

Your update looks somewhat like this (for brevity, sake I have omitted the warmup-phase):

new_theta = old_theta-learning_rate*momentum/(velocity+eps)

The intuition here is that if momentum>velocity, then the optimizer is in a plateau, so the the learning_rate is increased because momentum/velocity > 1. on the other hand if momentum<velocity, then the optimizer is in a steep slope or noisy region, so the learning_rate is decreased.

The learning_rate isn't necessarily decreased throughout the training, as you have mentioned in you question.


Instruction: What are the differences between type of layer and its activation function in PyTorch?

Input:
I am trying to write some simple neural network using pytorch. I am new to this library. I faced with two ways of implementing the same idea: a layer with some fixed activation function (e.g. tanh).

The first way to implement it:

l1 = nn.Tanh(n_in, n_out)

The second way:

l2 = nn.Linear(n_in, n_out) # linear layer, that do nothing with its input except summation

but in forward propagation use:

import torch.nn.functional as F
x = F.tanh(l2(x)) # x - value that propagates from layer to layer

What are the differences between those mechanisms? Which one is better for which purposes?

Output:
An activation function is just a non-linear function and it doesn't have any parameters. So, your first approach doesn't make any sense!

However, you can use a sequential wrapper to combine a linear layer with tanh activation.

model = nn.Sequential(
    nn.Linear(n_in, n_out),
    nn.Tanh()
)
output = model(input)


Instruction: Deep learning : How to build character level embedding?

Input:
I am trying to use character level embedding in my model but I have few doubts regarding character level embedding.

So for word level embedding :

Sentence = 'this is a example sentence'

create the vocab :

vocab = {'this' : 0 , 'is' :1 , 'a': 2 'example' : 3, 'sentence' : 4 }

encode the sentence :

encoded_sentence = [ 0, 1 , 2 , 3 , 4 ]

now send it to any pre-trained embedding like word2vec or glove :

each id will be replaced with 300 or embedding dim :

embedding_sentence = [ [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] ]

and if we are dealing with batches then we pad the sentences

So the shape goes like this :

[ batch_size , max_sentence_length , embedding_dim ]

Now for character level embedding I have few doubts :

so for char level embedding :

Sentence = 'this is a example sentence'

create the char_vocab :

char_vocab = [' ', 'a', 'c', 'e', 'h', 'i', 'l', 'm', 'n', 'p', 's', 't', 'x']

int_to_vocab = {n:m for m,n in enumerate(char_vocab)}

encoded the sentence by char level :

Now here is my confusion , so in word embedding we first tokenise the sentence and then encode each token with vocab id ( word_id)

but for char embedding if I am tokenzing the sentence and then encoding with character level then shape will be 4 dim and I can't feed this to LSTM.

But if i am not tokenising and directly encoding raw text then it's 3 dim and I can feed it to LSTM

for example :

with tokenization :

 token_sentence = ['this','is','a','example','sentence']

encoded_char_level = []

for words in token_sentence:
    char_lvel = [int_to_vocab[char] for char in words]
    encoded_char_level.append(char_lvel)

it's look like this:

[[0, 1, 2, 3],
 [2, 3],
 [5],
 [6, 7, 5, 8, 9, 10, 6],
 [3, 6, 11, 0, 6, 11, 12, 6]]

Now we have to pad this for two level , one is char_level padding and second is sentence level padding:

char_level_padding:

[[0, 1, 2, 3, 0, 0, 0,0],
 [2, 3, 0, 0, 0, 0, 0, 0],
 [5, 0, 0, 0, 0, 0, 0, 0],
 [6, 7, 5, 8, 9, 10, 6, 0],
 [3, 6, 11, 0, 6, 11, 12, 6]]

Now if we have 4 sentences then we have to pad each sentence with max sentence len so shape will be :

[batch_size , max_sentence_length , max_char_length ]

Now if we pass this to embedding layer then:

[ batch_size , max_sentence_length, max_char_length , embedding_dim ]

Which is 4 dim.

How to encode sentences with character level and use it with tensorflow LSTM layer?

Because lstm takes 3 dim input [ batch_size , max_sequence_length , embedding_dim ]

Can I use it something like :

[ Batch_size , ( max_sentence_length x max_char_length ) , dim ]

so for example :

[ 12 , [ 3 x 4 ] , 300 ]

Output:
You can concatenate the character level features with a fixed length.

For example:

``[[0, 1, 2, 3, 0, 0, 0,0],
  [2, 3, 0, 0, 0, 0, 0, 0],
  [5, 0, 0, 0, 0, 0, 0, 0],
  [6, 7, 5, 8, 9, 10, 6, 0],
  [3, 6, 11, 0, 6, 11, 12, 6]]``

can be changed to:
    [[0, 1, 2, 3, 0, 0, 0,0,2, 3, 0, 0, 0, 0, 0, 0,5, 0, 0, 0, 0, 0, 0, 0,6, 7, 5, 8, 9, 10, 6, 0,3, 6, 11, 0, 6, 11, 12, 6]]


Instruction: Pytorch how to multiply tensors of variable size except the first dimention

Input:
I have one tensor which is A = 40x1.

i need to multiply this one with 3 other tensors: B = 40x100x384, C = 40x10, D=40x10.

for example in tensor B, we got 40 100x384 matrixes and i need each one of these matrixes to be multiplied with its corresponding element from A

what is the best way to do this in pytorch? Suppose that we could have more matrixes like B,C,D they will always be in the style 40xKxL or 40xJ

Output:
If I understand correctly, you want to multiply every i-th matrix K x L by the corresponding i-th scalar in A.

One possible way is:

(A * B.view(len(A), -1)).view(B.shape)

Or you can use the power of broadcasting:

A = A.reshape(len(A), 1, 1)
# now A is (40, 1, 1) and you can do
A*B
A*C
A*D

essentially each trailing dimension equal to 1 in A is stretched and copied to match the other matrix.


Instruction: RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'

Input:
I'm failing to run my GAN on GPU. I call to(device) for all both models and all tensors but still keep getting the following error:

Traceback (most recent call last):
  File "code/a3_gan_template.py", line 185, in <module>
    main(args)
  File "code/a3_gan_template.py", line 162, in main
    train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device)
  File "code/a3_gan_template.py", line 100, in train
    d_x = discriminator.forward(imgs)
  File "code/a3_gan_template.py", line 80, in forward
    out = self.model(img)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1352, in linear
    ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'

Source code:

import argparse
import os

import torch
import torch.nn as nn
import torchvision.transforms as transforms
from torchvision.utils import save_image
from torchvision import datasets
from torch.nn.functional import binary_cross_entropy

class Generator(nn.Module):
    def __init__(self, latent_dim):
        super(Generator, self).__init__()

        # Construct generator. You are free to experiment with your model,
        # but the following is a good start:
        #   Linear args.latent_dim -> 128
        #   LeakyReLU(0.2)
        #   Linear 128 -> 256
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 256 -> 512
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 512 -> 1024
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 1024 -> 784
        #   Output non-linearity

        self.latent_dim = latent_dim
        self.model = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.LeakyReLU(0.2),
            nn.Linear(128, 256),
            nn.BatchNorm1d(256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 1024),
            nn.BatchNorm1d(1024),
            nn.LeakyReLU(0.2),
            nn.Linear(1024, 784),
            nn.Sigmoid()
        )

    def forward(self, z):
        # Generate images from z
        out = self.model(z)

        return out

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        # Construct distriminator. You are free to experiment with your model,
        # but the following is a good start:
        #   Linear 784 -> 512
        #   LeakyReLU(0.2)
        #   Linear 512 -> 256
        #   LeakyReLU(0.2)
        #   Linear 256 -> 1
        #   Output non-linearity

        self.model = nn.Sequential(
            nn.Linear(784, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, img):
        # return discriminator score for img
        out = self.model(img)

        return out

def train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device):
    for epoch in range(args.n_epochs):
        for i, (imgs, _) in enumerate(dataloader):
            batch_count = epoch * len(dataloader) + i

            imgs.to(device)
            batch_size = imgs.shape[0]
            imgs = imgs.reshape(batch_size, -1)

            z = torch.rand(batch_size, generator.latent_dim, device=device)
            gen_imgs = generator(z)

            discriminator.to(device)
            d_x = discriminator(imgs)
            d_g_z = discriminator(gen_imgs)

            ones = torch.ones(d_g_z.shape, device=device)

            # Train Generator
            # ---------------
            loss_G = binary_cross_entropy(d_g_z, ones)

            optimizer_G.zero_grad()
            loss_G.backward(retain_graph=True)
            optimizer_G.step()

            # Train Discriminator
            # -------------------
            if batch_count % args.d_train_interval == 0:
                loss_D = binary_cross_entropy(d_x, 0.9 * ones) + binary_cross_entropy(d_g_z, 0. * ones)

                optimizer_D.zero_grad()
                loss_D.backward()
                optimizer_D.step()

            # Save Images
            # -----------
            if batch_count % args.save_interval == 0:
                print(f'epoch: {epoch} batches: {batch_count} L_G: {loss_G.item():0.3f} L_D: {loss_D.item():0.3f}')
                # You can use the function save_image(Tensor (shape Bx1x28x28),
                # filename, number of rows, normalize) to save the generated
                # images, e.g.:
                save_image(gen_imgs[:25],
                           f'images/{batch_count}.png',
                           nrow=5, normalize=True)

def main(args):
    # Create output image directory
    os.makedirs('images', exist_ok=True)

    # Set device
    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    # load data
    dataloader = torch.utils.data.DataLoader(
        datasets.MNIST('./data/mnist', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize( (0.5,), (0.5,) )
                           ])),
        batch_size=args.batch_size, shuffle=True)

    # Initialize models and optimizers
    generator = Generator(args.latent_dim)
    generator.to(device)
    discriminator = Discriminator()
    discriminator.to(device)
    optimizer_G = torch.optim.Adam(generator.parameters(), lr=args.lr)
    optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=args.lr)

    # Start training
    train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device)

    # You can save your generator here to re-use it to generate images for your
    # report, e.g.:
    torch.save(generator.state_dict(), "mnist_generator.pt")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--n_epochs', type=int, default=200,
                        help='number of epochs')
    parser.add_argument('--batch_size', type=int, default=64,
                        help='batch size')
    parser.add_argument('--lr', type=float, default=0.0002,
                        help='learning rate')
    parser.add_argument('--latent_dim', type=int, default=100,
                        help='dimensionality of the latent space')
    parser.add_argument('--save_interval', type=int, default=500,
                        help='save every SAVE_INTERVAL iterations')
    parser.add_argument('--d_train_interval', type=int, default=25,
                        help='train discriminator (only) every D_TRAIN_INTERVAL iterations')
    args = parser.parse_args()

    main(args)

Any ideas on how to figure out what is missing? Thanks!

Output:
Found the solution. Turns out .to(device) does not work in place for tensors.

# wrong
imgs.to(device)

# correct
imgs = imgs.to(device)


Instruction: groupby aggregate mean in pytorch

Input:
I have a 2D tensor:
samples = torch.Tensor([
    [0.1, 0.1],    #-> group / class 1
    [0.2, 0.2],    #-> group / class 2
    [0.4, 0.4],    #-> group / class 2
    [0.0, 0.0]     #-> group / class 0
])

and a label for each sample corresponding to a class:
labels = torch.LongTensor([1, 2, 2, 0])

so len(samples) == len(labels). Now I want to calculate the mean for each class / label. Because there are 3 classes (0, 1 and 2) the final vector should have dimension [n_classes, samples.shape[1]] So the expected solution should be:
result == torch.Tensor([
    [0.1, 0.1],
    [0.3, 0.3], # -> mean of [0.2, 0.2] and [0.4, 0.4]
    [0.0, 0.0]
])

Question: How can this be done in pure pytorch (i.e. no numpy so that I can autograd) and ideally without for loops?

Output:
All you need to do is form an  mxn matrix (m=num classes, n=num samples) which will select the appropriate weights, and scale the mean appropriately. Then you can perform a matrix multiplication between your newly formed matrix and the samples matrix.

Given your labels, your matrix should be (each row is a class number, each class a sample number and its weight):

[[0.0000, 0.0000, 0.0000, 1.0000],
 [1.0000, 0.0000, 0.0000, 0.0000],
 [0.0000, 0.5000, 0.5000, 0.0000]]

Which you can form as follows:

M = torch.zeros(labels.max()+1, len(samples))
M[labels, torch.arange(len(samples)] = 1
M = torch.nn.functional.normalize(M, p=1, dim=1)
torch.mm(M, samples)

Output:

tensor([[0.0000, 0.0000],
        [0.1000, 0.1000],
        [0.3000, 0.3000]])

Note that the output means are correctly sorted in class order.

Why does M[labels, torch.arange(len(samples))] = 1 work?

This is performing a broadcast operation between the labels and the number of samples. Essentially, we are generating a 2D index for every element in labels: the first specifies which of the m classes it belongs to, and the second simply specifies its index position (from 1 to N). Another way would be top explicitly generate all the 2D indices:

twoD_indices = []
for count, label in enumerate(labels):
  twoD_indices.append((label, count))


Instruction: Why do I get RuntimeError: CUDA error: invalid argument in pytorch?

Input:
Recently I've frequently been getting RuntimeError: CUDA error: invalid argument when calling functions like torch.cholesky e.g.:

import torch
a = torch.randn(3, 3, device="cuda:0")
a = torch.mm(a, a.t()) # make symmetric positive-definite
torch.cholesky(a)

This works fine if I use device="cpu" instead. This error isn't very descriptive, so I'm not sure what's wrong here.

Output: I discovered that this error was because the machine I'm running things on has CUDA 10 installed now, but I just installed pytorch as pip install torch. From their website, the proper way to install with pip and CUDA 10 is pip install https://download.pytorch.org/whl/cu100/torch-1.1.0-cp37-cp37m-linux_x86_64.whl.


Instruction: Pytorch: How to assign a default value for look-up table using torch tensor

Input:
Say that I have two tensors as follows:

a = torch.tensor([[1, 2, 3], [1, 2, 3]])
b = torch.tensor([0, 2, 3, 4])

where b is the lookup value for a such as:

b[a]

will return the value of:

tensor([[2, 3, 4], [2, 3, 4]])

My problem is, what if I only have a look-up table of:

c = torch.tensor([0, 2, 3])

In which, for every out-of-index, I would like it to be assigned to index 0, such as c[a] will return

tensor([[2, 3, 0], [2, 3, 0]])

If I run c[a], of course, I will get this result:

RuntimeError: index 3 is out of bounds for dim with size 3

Thanks for your help.

Output:
  Code

# replace values greater than a certain number
def custom_replace(tensor, value, on_value):
    # we create a copy of the original tensor,
    # because of the way we are replacing them.
    res = tensor.clone()
    res[tensor>=value] = on_value
    return res

a = torch.tensor([[1, 2, 3], [1, 2, 3]])
c = torch.tensor([0, 2, 3])

a_ = custom_replace(a, c.size(0), 0)
print(c[a_])

  Output

tensor([[2, 3, 0],
        [2, 3, 0]])


Instruction: Transform List to Tensor more accurat

Input:
I want to return in the Dataloader a list.
But to return it, its need to be a tensor right?
So I transform it but in this process information is lost, is there another way to do this?

pt_tensor_from_list = torch.tensor(pose_transform)
pt_tensor_from_list = torch.FloatTensor(pose_transform)

I excpect the output:

  ([[-0.0003000000142492354, -0.0008999999845400453,
  0.00039999998989515007, 0], [0.0010000000474974513, -0.00019999999494757503, 0.0003000000142492354, 0], [0.00019999999494757503, -0.0005000000237487257,
  -0.0008999999845400453, 0], [5.484399795532227, -24.28619956970215, 117.5000991821289, 1])

But it is:

  ([[ -0.0003, -0.0009, 0.0004, 0.0000],
      [ 0.0010, -0.0002, 0.0003, 0.0000],
      [ 0.0002, -0.0005, -0.0009, 0.0000],
      [ 5.4844, -24.2862, 117.5001, 1.0000]])

Output:
You are not losing any information during such conversion. The reason why it looks like that, more compactly, is as when you are printing a tensor, it invokes __str__() or __repr__() methods and it makes your tensor looks more pretty. As you can find here torch.Tensor uses a kind of internal tensor formatter called _tensor_str. If you look inside the code link you will find that by default parameter precision is set to 4:

  precision: Number of digits of precision for floating point output          (default = 4).

That's why you have only 4 digits for tensor values when printing the tensor. But actually, values stored in a tensor are the same as in original list.

Here is small example to get an idea:

Code:

import torch

test_list = ([[-0.0003000000142492354, -0.0008999999845400453, 0.00039999998989515007, 0],
              [0.0010000000474974513, -0.00019999999494757503, 0.0003000000142492354, 0],
              [0.00019999999494757503, -0.0005000000237487257, -0.0008999999845400453, 0],
              [5.484399795532227, -24.28619956970215, 117.5000991821289, 1]])

print('Original values:')
for i in test_list:
    for j in i:
        print(j)

pt_tensor_from_list = torch.FloatTensor(test_list)

print('When printing FloatTensor:')
print(pt_tensor_from_list.dtype, pt_tensor_from_list, sep='\n')

print('When printing each value separately:')
for i in pt_tensor_from_list:
    for j in i:
        print(j.item())

Output:

Original values:
-0.0003000000142492354
-0.0008999999845400453
0.00039999998989515007
0
0.0010000000474974513
-0.00019999999494757503
0.0003000000142492354
0
0.00019999999494757503
-0.0005000000237487257
-0.0008999999845400453
0
5.484399795532227
-24.28619956970215
117.5000991821289
1
When printing FloatTensor:
torch.float32
tensor([[-3.0000e-04, -9.0000e-04,  4.0000e-04,  0.0000e+00],
        [ 1.0000e-03, -2.0000e-04,  3.0000e-04,  0.0000e+00],
        [ 2.0000e-04, -5.0000e-04, -9.0000e-04,  0.0000e+00],
        [ 5.4844e+00, -2.4286e+01,  1.1750e+02,  1.0000e+00]])
When printing each value separately:
-0.0003000000142492354
-0.0008999999845400453
0.00039999998989515007
0.0
0.0010000000474974513
-0.00019999999494757503
0.0003000000142492354
0.0
0.00019999999494757503
-0.0005000000237487257
-0.0008999999845400453
0.0
5.484399795532227
-24.28619956970215
117.5000991821289
1.0

As you can see, we are getting the same values when printing each value separately.

BUT you can lose some info if you choose the wrong tensor types, for example HalfTensor instead of FloatTensor. Here is an example:

Code:

pt_tensor_from_list = torch.HalfTensor(test_list)

print('When printing HalfTensor:')
print(pt_tensor_from_list.dtype, pt_tensor_from_list, sep='\n')

print('When printing each value separately:')
for i in pt_tensor_from_list:
    for j in i:
        print(j.item())

Output:

When printing HalfTensor:
torch.float16
tensor([[-2.9993e-04, -8.9979e-04,  4.0007e-04,  0.0000e+00],
        [ 1.0004e-03, -2.0003e-04,  2.9993e-04,  0.0000e+00],
        [ 2.0003e-04, -5.0020e-04, -8.9979e-04,  0.0000e+00],
        [ 5.4844e+00, -2.4281e+01,  1.1750e+02,  1.0000e+00]],
       dtype=torch.float16)
When printing each value separately:
-0.0002999305725097656
-0.0008997917175292969
0.0004000663757324219
0.0
0.0010004043579101562
-0.00020003318786621094
0.0002999305725097656
0.0
0.00020003318786621094
-0.0005002021789550781
-0.0008997917175292969
0.0
5.484375
-24.28125
117.5
1.0

As you can notice now the values are (a little bit) different. Visit pytorch tensor docs to learn more about different types of torch.tensor.


Instruction: Pytorch argsort ordered, with duplicate elements in the tensor

Input:
I have a vector A = [0,1,2,3,0,0,1,1,2,2,3,3]. I need to sort it in an increasing matter such that it is listed in an ordered fashion and from that extract the argsort. To better explain this I need to sort A to such that it returns B = [0,4,5,1,6,7,2,8,9,3,10,11]. However, when I use pyotrch's torch.argsort(A) it returns B = [4,5,0,1,6,7,2,8,9,3,10,11].

I'm assuming the algorithm that does so cannot be controlled on my end. Is there anyway to approach this without introducing for loops? Such operation are part of my NN model and will cause performance issues if not done efficiently. Thanks!

Output:
Here is a pure PyTorch based solution leveraging broadcasting, torch.unique(), and torch.nonzero(). This would be of great boost particularly for a GPU based implementation/run, which is not possible if we have to switch back to NumPy, argsort it and then transfer back to PyTorch (as suggested in other approaches).

# our input tensor
In [50]: A = torch.tensor([0,1,2,3,0,0,1,1,2,2,3,3])

# construct an intermediate boolean tensor
In [51]: boolean = A[:, None] == torch.unique(A)

In [52]: boolean
Out[52]:
tensor([[1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1],
        [1, 0, 0, 0],
        [1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 1, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1],
        [0, 0, 0, 1]], dtype=torch.uint8)

Once we have this boolean tensor, we can find the desired indices by checking for positions where there is an 1 after transposing the boolean tensor.

That would give us both sorted input and the indices. Since we want only the indices, we can just grab those by indexing for the last column (1 or -1)

In [53]: torch.nonzero(boolean.t())[:, -1]
Out[53]: tensor([ 0,  4,  5,  1,  6,  7,  2,  8,  9,  3, 10, 11])

Here's the result for one more example provided by OP in the comments:

In [55]: A_large = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9])

In [56]: boolean_large = A_large[:, None] == torch.unique(A_large)

In [57]: torch.nonzero(boolean_large.t())[:, -1]
Out[57]:
tensor([ 0, 10, 11,  1, 12, 13,  2, 14, 15,  3, 16, 17,  4, 18, 19,  5, 20, 21,
         6, 22, 23,  7, 24, 25,  8, 26, 27,  9, 28, 29])

Note: Unlike with NumPy-based solution proposed in other answers, here we don't have to worry about what kind of sorting algorithm we've to use because we are not using any sorting at all.


Instruction: How to reproduce RNN results on several runs?

Input:
I call same model on same input twice in a row and I don't get the same result, this model have nn.GRU layers so I suspect that it have some internal state that should be release before second run?

How to reset RNN hidden state to make it the same as if model was initially loaded?

UPDATE:

Some context:

I'm trying to run model from here:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L93

I'm calling generate:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Here it's actually have some code using random generator in pytorch:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L110

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L129

I have placed (I'm running code on CPU):

torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)

in

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py

after all imports.

I have checked GRU weights between runs and they are the same:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L153

Also I have checked logits and sample between runs and logits are the same but sample are not, so @Andrew Naguib seems were right about random seeding, but I'm not sure where the code that fixes random seed should be placed?

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

UPDATE 2:

I have placed seed init inside generate and now results are consistent:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Output:
I believe this may be highly related to Random Seeding. To ensure reproducible results (as stated by them) you have to seed torch as in this:

import torch
torch.manual_seed(0)

And also, the CuDNN module.

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

If you're using numpy, you could also do:

import numpy as np
np.random.seed(0)

However, they warn you:

  Deterministic mode can have a performance impact, depending on your model.

A suggested script I regularly use which has been working very good to reproduce results is:

# imports
import numpy as np
import random
import torch
# ...
""" Set Random Seed """
if args.random_seed is not None:
    """Following seeding lines of code are to ensure reproducible results
       Seeding the two pseudorandom number generators involved in PyTorch"""
    random.seed(args.random_seed)
    np.random.seed(args.random_seed)
    torch.manual_seed(args.random_seed)
    # https://pytorch.org/docs/master/notes/randomness.html#cudnn
    if not args.cpu_only:
        torch.cuda.manual_seed(args.random_seed)
        cudnn.deterministic = True
        cudnn.benchmark = False


Instruction: How to access the predictions of pytorch classification model? (BERT)

Input:
I'm running this file:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py

This is the prediction code for one input batch:

  input_ids = input_ids.to(device)
  input_mask = input_mask.to(device)
  segment_ids = segment_ids.to(device)
  label_ids = label_ids.to(device)

  with torch.no_grad():
       logits = model(input_ids, segment_ids, input_mask, labels=None)

       loss_fct = CrossEntropyLoss()
       tmp_eval_loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))

       eval_loss += tmp_eval_loss.mean().item()
       nb_eval_steps += 1
       if len(preds) == 0:
           preds.append(logits.detach().cpu().numpy())
       else:
           preds[0] = np.append(preds[0], logits.detach().cpu().numpy(), axis=0)

The task is a binary classification.
I want to access the binary output.

I've tried this:

  curr_pred = logits.detach().cpu()

  if len(preds) == 0:
      preds.append(curr_pred.numpy())
  else:
      preds[0] = np.append(preds[0], curr_pred.numpy(), axis=0)

  probablities = curr_pred.softmax(1).numpy()[:, 1]

But the results seem weird. So I'm not sure if it's the correct way.

My hypothesis - I'm receiving the output of the last layer, therefore after softmax, it's the true probabilities (vector of dim 2 - the probability to the 1st and probability to the 2nd class.)

Output:
After looking at this part of the run_classifier.py code:

    # copied from the run_classifier.py code
    eval_loss = eval_loss / nb_eval_steps
    preds = preds[0]
    if output_mode == "classification":
        preds = np.argmax(preds, axis=1)
    elif output_mode == "regression":
        preds = np.squeeze(preds)
    result = compute_metrics(task_name, preds, all_label_ids.numpy())

You are just missing:

    preds = preds[0]
    preds = np.argmax(preds, axis=1)

Then they just use preds to compute the accuracy as:

    def simple_accuracy(preds, labels):
         return (preds == labels).mean()


Instruction: Will installing Nvidia CUDA ver10 break CUDA ver9 on the same machine?

Input:
I am currently using pytorch and tensorflow with cuda ver9.0. I am pondering whether to install the latest cuda version 10. Will installing cuda v10 break cuda v9? Can both co-exist on the same desktop PC? Is it advisable to uninstall cuda v9 after installing cuda v10 or is it better to leave both versions installed?

I am using Windows 10.

Output:
I will answer my own question. Installing CUDA v10 will not break CUDA v9 on the same machine. Both can co-exist.

I installed CUDA v10 successfully. Pytorch has been tested to work successfully with CUDA v10.