Instruction: What wrong when i load state_dict of resnet50.pth with pytorch

Input:
i load the resnet50.pth and KeyError of 'state_dict'
pytorch version is 0.4.1

i tried delete/add torch.nn.parallel but it didn't help
and resnet50.pth loaded from pytorch API

related code

model = ResNet(len(CLASSES), pretrained=args.use_imagenet_weights)
if cuda_is_available:
    model = nn.DataParallel(model,  device_ids=[2]).cuda()
if args.model:
    print("Loading model " + args.model)
    state_dict = torch.load(args.model)['state_dict']
    model.load_state_dict(state_dict)

Traceback

Loading model resnet50-19c8e357.pth
Traceback (most recent call last):
  File "train.py", line 67, in <module>
    state_dict = torch.load(args.model)['state_dict']
KeyError: 'state_dict'

when print(torch.load(args.model).keys())

odict_keys(['conv1.weight', 'bn1.running_mean', 'bn1.running_var', 'bn1.weight', 'bn1.bias', 'layer1.0.conv1.weight', 'layer1.0.bn1.running_mean', 'layer1.0.bn1.running_var', 'layer1.0.bn1.weight', 'layer1.0.bn1.bias', 'layer1.0.conv2.weight', 'layer1.0.bn2.running_mean', 'layer1.0.bn2.running_var', 'layer1.0.bn2.weight', 'layer1.0.bn2.bias', 'layer1.0.conv3.weight', 'layer1.0.bn3.running_mean', 'layer1.0.bn3.running_var', 'layer1.0.bn3.weight', 'layer1.0.bn3.bias', 'layer1.0.downsample.0.weight', 'layer1.0.downsample.1.running_mean', 'layer1.0.downsample.1.running_var', 'layer1.0.downsample.1.weight', 'layer1.0.downsample.1.bias', 'layer1.1.conv1.weight', 'layer1.1.bn1.running_mean', 'layer1.1.bn1.running_var', 'layer1.1.bn1.weight', 'layer1.1.bn1.bias', 'layer1.1.conv2.weight', 'layer1.1.bn2.running_mean', 'layer1.1.bn2.running_var', 'layer1.1.bn2.weight', 'layer1.1.bn2.bias', 'layer1.1.conv3.weight', 'layer1.1.bn3.running_mean', 'layer1.1.bn3.running_var', 'layer1.1.bn3.weight', 'layer1.1.bn3.bias', 'layer1.2.conv1.weight', 'layer1.2.bn1.running_mean', 'layer1.2.bn1.running_var', 'layer1.2.bn1.weight', 'layer1.2.bn1.bias', 'layer1.2.conv2.weight', 'layer1.2.bn2.running_mean', 'layer1.2.bn2.running_var', 'layer1.2.bn2.weight', 'layer1.2.bn2.bias', 'layer1.2.conv3.weight', 'layer1.2.bn3.running_mean', 'layer1.2.bn3.running_var', 'layer1.2.bn3.weight', 'layer1.2.bn3.bias', 'layer2.0.conv1.weight', 'layer2.0.bn1.running_mean', 'layer2.0.bn1.running_var', 'layer2.0.bn1.weight', 'layer2.0.bn1.bias', 'layer2.0.conv2.weight', 'layer2.0.bn2.running_mean', 'layer2.0.bn2.running_var', 'layer2.0.bn2.weight', 'layer2.0.bn2.bias', 'layer2.0.conv3.weight', 'layer2.0.bn3.running_mean', 'layer2.0.bn3.running_var', 'layer2.0.bn3.weight', 'layer2.0.bn3.bias', 'layer2.0.downsample.0.weight', 'layer2.0.downsample.1.running_mean', 'layer2.0.downsample.1.running_var', 'layer2.0.downsample.1.weight', 'layer2.0.downsample.1.bias', 'layer2.1.conv1.weight', 'layer2.1.bn1.running_mean', 'layer2.1.bn1.running_var', 'layer2.1.bn1.weight', 'layer2.1.bn1.bias', 'layer2.1.conv2.weight', 'layer2.1.bn2.running_mean', 'layer2.1.bn2.running_var', 'layer2.1.bn2.weight', 'layer2.1.bn2.bias', 'layer2.1.conv3.weight', 'layer2.1.bn3.running_mean', 'layer2.1.bn3.running_var', 'layer2.1.bn3.weight', 'layer2.1.bn3.bias', 'layer2.2.conv1.weight', 'layer2.2.bn1.running_mean', 'layer2.2.bn1.running_var', 'layer2.2.bn1.weight', 'layer2.2.bn1.bias', 'layer2.2.conv2.weight', 'layer2.2.bn2.running_mean', 'layer2.2.bn2.running_var', 'layer2.2.bn2.weight', 'layer2.2.bn2.bias', 'layer2.2.conv3.weight', 'layer2.2.bn3.running_mean', 'layer2.2.bn3.running_var', 'layer2.2.bn3.weight', 'layer2.2.bn3.bias', 'layer2.3.conv1.weight', 'layer2.3.bn1.running_mean', 'layer2.3.bn1.running_var', 'layer2.3.bn1.weight', 'layer2.3.bn1.bias', 'layer2.3.conv2.weight', 'layer2.3.bn2.running_mean', 'layer2.3.bn2.running_var', 'layer2.3.bn2.weight', 'layer2.3.bn2.bias', 'layer2.3.conv3.weight', 'layer2.3.bn3.running_mean', 'layer2.3.bn3.running_var', 'layer2.3.bn3.weight', 'layer2.3.bn3.bias', 'layer3.0.conv1.weight', 'layer3.0.bn1.running_mean', 'layer3.0.bn1.running_var', 'layer3.0.bn1.weight', 'layer3.0.bn1.bias', 'layer3.0.conv2.weight', 'layer3.0.bn2.running_mean', 'layer3.0.bn2.running_var', 'layer3.0.bn2.weight', 'layer3.0.bn2.bias', 'layer3.0.conv3.weight', 'layer3.0.bn3.running_mean', 'layer3.0.bn3.running_var', 'layer3.0.bn3.weight', 'layer3.0.bn3.bias', 'layer3.0.downsample.0.weight', 'layer3.0.downsample.1.running_mean', 'layer3.0.downsample.1.running_var', 'layer3.0.downsample.1.weight', 'layer3.0.downsample.1.bias', 'layer3.1.conv1.weight', 'layer3.1.bn1.running_mean', 'layer3.1.bn1.running_var', 'layer3.1.bn1.weight', 'layer3.1.bn1.bias', 'layer3.1.conv2.weight', 'layer3.1.bn2.running_mean', 'layer3.1.bn2.running_var', 'layer3.1.bn2.weight', 'layer3.1.bn2.bias', 'layer3.1.conv3.weight', 'layer3.1.bn3.running_mean', 'layer3.1.bn3.running_var', 'layer3.1.bn3.weight', 'layer3.1.bn3.bias', 'layer3.2.conv1.weight', 'layer3.2.bn1.running_mean', 'layer3.2.bn1.running_var', 'layer3.2.bn1.weight', 'layer3.2.bn1.bias', 'layer3.2.conv2.weight', 'layer3.2.bn2.running_mean', 'layer3.2.bn2.running_var', 'layer3.2.bn2.weight', 'layer3.2.bn2.bias', 'layer3.2.conv3.weight', 'layer3.2.bn3.running_mean', 'layer3.2.bn3.running_var', 'layer3.2.bn3.weight', 'layer3.2.bn3.bias', 'layer3.3.conv1.weight', 'layer3.3.bn1.running_mean', 'layer3.3.bn1.running_var', 'layer3.3.bn1.weight', 'layer3.3.bn1.bias', 'layer3.3.conv2.weight', 'layer3.3.bn2.running_mean', 'layer3.3.bn2.running_var', 'layer3.3.bn2.weight', 'layer3.3.bn2.bias', 'layer3.3.conv3.weight', 'layer3.3.bn3.running_mean', 'layer3.3.bn3.running_var', 'layer3.3.bn3.weight', 'layer3.3.bn3.bias', 'layer3.4.conv1.weight', 'layer3.4.bn1.running_mean', 'layer3.4.bn1.running_var', 'layer3.4.bn1.weight', 'layer3.4.bn1.bias', 'layer3.4.conv2.weight', 'layer3.4.bn2.running_mean', 'layer3.4.bn2.running_var', 'layer3.4.bn2.weight', 'layer3.4.bn2.bias', 'layer3.4.conv3.weight', 'layer3.4.bn3.running_mean', 'layer3.4.bn3.running_var', 'layer3.4.bn3.weight', 'layer3.4.bn3.bias', 'layer3.5.conv1.weight', 'layer3.5.bn1.running_mean', 'layer3.5.bn1.running_var', 'layer3.5.bn1.weight', 'layer3.5.bn1.bias', 'layer3.5.conv2.weight', 'layer3.5.bn2.running_mean', 'layer3.5.bn2.running_var', 'layer3.5.bn2.weight', 'layer3.5.bn2.bias', 'layer3.5.conv3.weight', 'layer3.5.bn3.running_mean', 'layer3.5.bn3.running_var', 'layer3.5.bn3.weight', 'layer3.5.bn3.bias', 'layer4.0.conv1.weight', 'layer4.0.bn1.running_mean', 'layer4.0.bn1.running_var', 'layer4.0.bn1.weight', 'layer4.0.bn1.bias', 'layer4.0.conv2.weight', 'layer4.0.bn2.running_mean', 'layer4.0.bn2.running_var', 'layer4.0.bn2.weight', 'layer4.0.bn2.bias', 'layer4.0.conv3.weight', 'layer4.0.bn3.running_mean', 'layer4.0.bn3.running_var', 'layer4.0.bn3.weight', 'layer4.0.bn3.bias', 'layer4.0.downsample.0.weight', 'layer4.0.downsample.1.running_mean', 'layer4.0.downsample.1.running_var', 'layer4.0.downsample.1.weight', 'layer4.0.downsample.1.bias', 'layer4.1.conv1.weight', 'layer4.1.bn1.running_mean', 'layer4.1.bn1.running_var', 'layer4.1.bn1.weight', 'layer4.1.bn1.bias', 'layer4.1.conv2.weight', 'layer4.1.bn2.running_mean', 'layer4.1.bn2.running_var', 'layer4.1.bn2.weight', 'layer4.1.bn2.bias', 'layer4.1.conv3.weight', 'layer4.1.bn3.running_mean', 'layer4.1.bn3.running_var', 'layer4.1.bn3.weight', 'layer4.1.bn3.bias', 'layer4.2.conv1.weight', 'layer4.2.bn1.running_mean', 'layer4.2.bn1.running_var', 'layer4.2.bn1.weight', 'layer4.2.bn1.bias', 'layer4.2.conv2.weight', 'layer4.2.bn2.running_mean', 'layer4.2.bn2.running_var', 'layer4.2.bn2.weight', 'layer4.2.bn2.bias', 'layer4.2.conv3.weight', 'layer4.2.bn3.running_mean', 'layer4.2.bn3.running_var', 'layer4.2.bn3.weight', 'layer4.2.bn3.bias', 'fc.weight', 'fc.bias'])

just want to run plz

Output:
Did you perhaps mean the following?

state_dict = torch.load(args.model['state_dict'])

From your edit, it seems that your model is the model itself. There is no state_dict. So just use

state_dict = torch.load(args.model)


Instruction: Adding parameters to a pretrained model

Input:
In Pytorch, we load the pretrained model as follows:

net.load_state_dict(torch.load(path)['model_state_dict'])

Then the network structure and the loaded model have to be exactly the same. However, is it possible to load the weights but then modify the network/add an extra parameter?

Note:
If we add an extra parameter to the model earlier before loading the weights, e.g.

self.parameter = Parameter(torch.ones(5),requires_grad=True)

we will get Missing key(s) in state_dict: error when loading the weights.

Output:
Let's create a model and save its' state.

class Model1(nn.Module):
    def __init__(self):
        super(Model1, self).__init__()

        self.encoder = nn.LSTM(100, 50)

    def forward(self):
        pass

model1 = Model1()
torch.save(model1.state_dict(), 'filename.pt') # saving model

Then create a second model which has a few layers common to the first model. Load the states of the first model and load it to the common layers of the second model.

class Model2(nn.Module):
    def __init__(self):
        super(Model2, self).__init__()

        self.encoder = nn.LSTM(100, 50)
        self.linear = nn.Linear(50, 200)

    def forward(self):
        pass

model1_dict = torch.load('filename.pt')
model2 = Model2()
model2_dict = model2.state_dict()

# 1. filter out unnecessary keys
filtered_dict = {k: v for k, v in model1_dict.items() if k in model2_dict}
# 2. overwrite entries in the existing state dict
model2_dict.update(filtered_dict)
# 3. load the new state dict
model2.load_state_dict(model2_dict)


Instruction: How to cover a label list under the multi-label classification context into one-hot encoding with pytorch?

Input:
I have a list of one batch data with multi-label for every sample. So how to covert it into torch.Tensor in one-hot encoding?
For example, with batch_size=5 and class_num=6,

label =[
[1,2,3],
[4,6],
[1],
[1,4,5],
[4]
]

how to make it into one-hot encoding in pytorch?

label_tensor=tensor([
[1,1,1,0,0,0],
[0,0,0,1,0,1],
[1,0,0,0,0,0],
[1,0,0,1,1,0],
[0,0,0,1,0,0]
])

Output:
If the batch size can be derived from len(labels):

def to_onehot(labels, n_categories, dtype=torch.float32):
    batch_size = len(labels)
    one_hot_labels = torch.zeros(size=(batch_size, n_categories), dtype=dtype)
    for i, label in enumerate(labels):
        # Subtract 1 from each LongTensor because your
        # indexing starts at 1 and tensor indexing starts at 0
        label = torch.LongTensor(label) - 1
        one_hot_labels[i] = one_hot_labels[i].scatter_(dim=0, index=label, value=1.)
    return one_hot_labels

and you have 6 categories and want the output to be a tensor of integers:

to_onehot(labels, n_categories=6, dtype=torch.int64)

tensor([[1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 0, 1],
        [1, 0, 0, 0, 0, 0],
        [1, 0, 0, 1, 1, 0],
        [0, 0, 0, 1, 0, 0]])

I would stick to torch.float32 in case you want to use label smoothing, mix-up or something along those lines later.


Instruction: Saving Pytorch model.state_dict() to s3

Input:
I am trying to save a trained Pytorch model to S3. However, the torch.save(model.state_dict(), file_name) seems to support only local files. How can the state dict be saved to an S3 file?

I'm using Torch 0.4.0

Output:
As discussed by Soumith Chintala, Pytorch doesn't have custom APIs to do this job. However you can use boto3 or Petastorm library to solve the problem.

Here's a concrete example to write to an S3 object directly:

import boto3

# Convert your existing model to JSON
saved_model = model.to_json()

# Write JSON object to S3 as "model.json"
client = boto3.client('s3')
client.put_object(Body=saved_model,
                  Bucket='BUCKET_NAME',
                  Key='model.json')


Instruction: Strange behavior of Inception_v3

Input:
I am trying to create a generative network based on the pre-trained Inception_v3.

1)  I fix all the weights in the model

2)  create a Variable whose size is (2, 3, 299, 299)

3)  create targets of size (2, 1000) that I want my final layer activations to become as close as possible to by optimizing the Variable.
(I do not set the batchsize of 1, because unlike VGG16, Inception_v3 doesn't take batchsize=1, but that's not the point).

The following code should work, but gives me the error: «RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation».

# minimalist code with Inception_v3 that throws the error:

import torch
from torch.autograd import Variable
import torch.optim as optim
import torch.nn as nn
import torchvision

torch.set_default_tensor_type('torch.FloatTensor')
Iv3 = torchvision.models.inception_v3(pretrained=True)
for i in Iv3.parameters():
    i.requires_grad = False

criterion = nn.CrossEntropyLoss()

x = Variable(torch.randn(2, 3, 299, 299), requires_grad=True)
target = torch.empty(2, dtype=torch.long).random_(1000)

output = Iv3(x)
loss = criterion(output[0], target)
loss.backward()

print(x.grad)

This is very strange, because if I do the same thing with VGG16, everything works fine:

# minimalist working code with VGG16:

import torch
from torch.autograd import Variable
import torch.optim as optim
import torch.nn as nn
import torchvision

# torch.cuda.empty_cache()
# vgg16 = torchvision.models.vgg16(pretrained=True).cuda()
# torch.set_default_tensor_type('torch.cuda.FloatTensor')

torch.set_default_tensor_type('torch.FloatTensor')
vgg16 = torchvision.models.vgg16(pretrained=True)
for i in vgg16.parameters():
    i.requires_grad = False

criterion = nn.CrossEntropyLoss()

x = Variable(torch.randn(2, 3, 229, 229), requires_grad=True)
target = torch.empty(2, dtype=torch.long).random_(1000)

output = vgg16(x)
loss = criterion(output, target)
loss.backward()

print(x.grad)

Please help.

Output: Thanks to @iacolippo the issue is solved. Turns out the problem was due to  Pytorch 1.0.0. No problem with Pytorch 0.4.1. though.


Instruction: How to use torch to speed up some common computations?

Input:
I am trying make some common computations, like matrix multiplication, but without gradient computation. An example of my computation is like

import numpy as np
from scipy.special import logsumexp

var = 1e-8
a = np.random.randint(0,10,(128,20))
result = np.logsumexp(a, axis=1) / 2. + np.log(np.pi * var)

I want to use torch (gpu) to speed up the computation. Here is the code

import numpy as np
import torch

var = 1e-8
a = np.random.randint(0,10,(128,20))
a = torch.numpy_from(a).cuda()

result = torch.logsumexp(a, dim=1)/ 2. + np.log(np.pi*var)

but i have some questions:

Could the above code speed up the computation? I don't know if it works.
Do I need to convert all values into torch.tensor, like from var to torch.tensor(var).cuda() and from np.log(np.pi*var) to a torch.tensor?
Do I need to convert all tensors into gpu by myself, especially for some intermediate variable?
If the above code doesn't work, how can I speed up the computation with gpu?

Output:
You could use torch only to do the computations.

   import torch
   # optimization by passing device argument, tensor is created on gpu and hence move operation is saved
   # convert to float to use with logsumexp
   a = torch.randint(0,10, (128,20), device="cuda").float()
   result = torch.logsumexp(a, dim=1)/ 2.

Answers to your some of your questions:

  Could the above code speed up the computation?

It depends. If you have too many matrix multiplication, using gpu can give speed up.

  Do I need to convert all values into torch.tensor, like from var to torch.tensor(var).cuda() and from np.log(np.pi*var) to a torch.tensor?

Yes

  Do I need to convert all tensors into gpu by myself, especially for some intermediate variable?

Only leaf variables need to converted, intermediate variable will be placed on device on which the operations are done. For ex: if a and b are on gpu, then as a result of operation c=a+b, c will also be on gpu.


Instruction: How to convert some tensorflow code into pytorch version

Input:
I recently started using pytorch. I've been using tensorflow framework before. I have a piece of code that I implemented with tensorflow, which I now want to convert to the pytorch version.

I'm new to pytorch and I'm not familiar with its functions and the transformation process is not smooth, so I'd like to consult.

Here's the code I want to convert：

def kl_loss_compute(logits1, logits2):
    """ KL loss
    """
    pred1 = tf.nn.softmax(logits1)
    pred2 = tf.nn.softmax(logits2)
    loss = tf.reduce_mean(tf.reduce_sum(pred2 * tf.log(1e-8 + pred2 / (pred1 + 1e-8)), 1))

    return loss

python: 3.6, ubuntu: 16.04

logits1 and logits2 are FC layer's outputs. Their shape is [batch, n]

Output:
Here is my Implementation (I am taking an example of logits of dimension [3,5]):

Tensorflow Version:

import tensorflow as tf
def kl_loss_compute(logits1, logits2):
    """ KL loss
    """
    pred1 = tf.nn.softmax(logits1)
    print(pred1.eval())
    pred2 = tf.nn.softmax(logits2)
    print(pred2.eval())
    loss = tf.reduce_mean(tf.reduce_sum(pred2 * tf.log(1e-8 + pred2 / (pred1 + 1e-8)), 1))
    return loss

x1 = tf.random.normal([3, 5], dtype=tf.float32)
x2 = tf.random.normal([3, 5], dtype=tf.float32)
with tf.Session() as sess:
    x1 = sess.run(x1)
    print(x1)
    x2 = sess.run(x2)
    print(x2)
    print(30*'=')
    print(sess.run(kl_loss_compute(x1, x2)))

Output:

[[ 0.9801388  -0.2514422  -0.28299806  0.85130763  0.4565948 ]
 [-1.0744809   0.20301117  0.21026622  1.0385195   0.41147012]
 [ 1.2385081   1.1003486  -2.0818367  -1.0446491   1.8817908 ]]
[[ 0.04036871  0.82306993  0.82962424  0.5209219  -0.10473887]
 [ 1.7777447  -0.6257034  -0.68985045 -1.1191329  -0.2600192 ]
 [ 0.03387258  0.44405013  0.08010675  0.9131149   0.6422863 ]]
==============================
[[0.32828477 0.09580362 0.09282765 0.2886025  0.19448158]
 [0.04786159 0.17170973 0.17296004 0.39596024 0.21150835]
 [0.2556382  0.22265059 0.00923886 0.02606533 0.48640704]]
[[0.12704821 0.27790183 0.27972925 0.20543297 0.10988771]
 [0.7349108  0.06644011 0.062312   0.04056362 0.09577343]
 [0.12818882 0.19319147 0.13425465 0.30881628 0.23554876]]
0.96658206

PyTorch Version:

def kl_loss_compute(logits1, logits2):
    """ KL loss
    """
    pred1 = torch.softmax(logits1, dim=-1, dtype=torch.float32)
    print(pred1)
    pred2 = torch.softmax(logits2, dim=-1, dtype=torch.float32)
    print(pred2)
    loss = torch.mean(torch.sum(pred2 * torch.log(1e-8 + pred2 / (pred1 + 1e-8)), -1))
    return loss
# same inputs are used here as above(see the inputs used in tensorflow code in the output)
x = torch.Tensor([[ 0.9801388,  -0.2514422 , -0.28299806 , 0.85130763,  0.4565948 ],
                 [-1.0744809 ,  0.20301117,  0.21026622,  1.0385195,   0.41147012],
                 [ 1.2385081  , 1.1003486,  -2.0818367,  -1.0446491,   1.8817908 ]])
y = torch.Tensor([[ 0.04036871 , 0.82306993,  0.82962424,  0.5209219,  -0.10473887],
                 [ 1.7777447  ,-0.6257034,  -0.68985045, -1.1191329,  -0.2600192 ],
                 [ 0.03387258 , 0.44405013 , 0.08010675,  0.9131149,   0.6422863 ]])
print(kl_loss_compute(x, y))

Output:

tensor([[0.3283, 0.0958, 0.0928, 0.2886, 0.1945],
        [0.0479, 0.1717, 0.1730, 0.3960, 0.2115],
        [0.2556, 0.2227, 0.0092, 0.0261, 0.4864]])
tensor([[0.1270, 0.2779, 0.2797, 0.2054, 0.1099],
        [0.7349, 0.0664, 0.0623, 0.0406, 0.0958],
        [0.1282, 0.1932, 0.1343, 0.3088, 0.2355]])
tensor(0.9666)


Instruction: How to use multiprocessing in PyTorch?

Input:
I'm trying to use PyTorch with complex loss function. In order to accelerate the code, I hope that I can use the PyTorch multiprocessing package.

The first trial, I put 10x1 features into the NN and get 10x4 output.

After that, I want to pass 10x4 parameters into a function to do some calculation. (The calculation will be complex in the future.)

After calculating, the function will return a 10x1 array in total. This array will be set as NN_energy and calculate loss function.

Besides, I also want to know if there is another method to create a backward-able array to store the NN_energy array, instead of using

NN_energy = net(Data_in)[0:10,0]

Thanks a lot.

Full Code:

import torch
import numpy as np
from torch.autograd import Variable
from torch import multiprocessing

def func(msg,BOP):
    ans = (BOP[msg][0]+BOP[msg][1]/BOP[msg][2])*BOP[msg][3]
    return ans

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden_1, n_hidden_2, n_output):
        super(Net, self).__init__()
        self.hidden_1 = torch.nn.Linear(n_feature , n_hidden_1)  # hidden layer
        self.hidden_2 = torch.nn.Linear(n_hidden_1, n_hidden_2)  # hidden layer
        self.predict  = torch.nn.Linear(n_hidden_2, n_output  )  # output layer

    def forward(self, x):
        x = torch.tanh(self.hidden_1(x))      # activation function for hidden layer
        x = torch.tanh(self.hidden_2(x))      # activation function for hidden layer
        x = self.predict(x)                   # linear output
        return x

if __name__ == '__main__': # apply_async
    Data_in      = Variable( torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() )
    Ground_truth = Variable( torch.from_numpy( np.asarray(list(range(20,30))).reshape(10,1) ).float() )

    net = Net( n_feature=1 , n_hidden_1=15 , n_hidden_2=15 , n_output=4 )     # define the network
    optimizer = torch.optim.Rprop( net.parameters() )
    loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss
    NN_output = net(Data_in)
    args = range(0,10)
    pool = multiprocessing.Pool()
    return_data = pool.map( func, zip(args, NN_output) )
    pool.close()
    pool.join()

    NN_energy = net(Data_in)[0:10,0]
    for i in range(0,10):
        NN_energy[i] = return_data[i]

    loss = torch.sqrt( loss_func( NN_energy , Ground_truth ) )     # must be (1. nn output, 2. target)
    print(loss)

Error messages:

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\torch\multiprocessing\reductions.py",
  line 126, in reduce_tensor
      raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "

  RuntimeError: Cowardly refusing to serialize non-leaf tensor which
  requires_grad, since autograd does not support crossing process
  boundaries.  If you just want to transfer the data, call detach() on
  the tensor before serializing (e.g., putting it on the queue).

Output:
First of all, Torch Variable API is deprecated since a very long time, just don't use it.
Next, torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() is wrong at many levels: np.asarray of list is useless since a copy will be performed anyway, and np.array takes list as input by design. Then, np.arange is available to return a range as numpy array, and it is also available on Torch. Next, specifying both dimension for reshape is useless and error prone, you could simply do reshape((-1, 1)), or even better unsqueeze(-1).
Here is the simplified expression torch.arange(10, dtype=torch.float32, requires_grad=True).unsqueeze(-1).
Using multiprocessing pool is a bad practice if using batch processing is possible. It will be both way more efficient and readable. Indeed, performing N small algebraic operations in parallel is always slower and a larger single algebraic operation, and even more on GPU. More importantly, computing the gradient is not supported by multiprocessing, hence the error that you get. Yet, this is partially true, because it is supports for tensors on cpu since 1.6.0. Have a lok, to the official release changelog.
Could you post a more representative example of what func method could be to make sure you really need it ?
NB: Distributed autograd as you are looking is now available in Pytorch as an experimental feature available in beta since 1.6.0. Have a look to the official documentation.


Instruction: DataParallel multi-gpu RuntimeError: chunk expects at least a 1-dimensional tensor

Input:
I am trying to run my model on multiple gpus using DataParallel by setting model = nn.DataParallel(model).cuda(), but everytime getting this error -

  RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at
  /pytorch/aten/src/ATen/native/TensorShape.cpp:184).

My code is correct. Does anyone know what's wrong?

I have tried setting device_ids=[0,1] parameter and also CUDA_VISIBLE_DEVICES on the terminal. Also tried different batch sizes.

Output:
To identify the problem, you should check the shape of your input data for each mini-batch. The documentation says, nn.DataParallel splits the input tensor in dim0 and sends each chunk to the specified GPUs. From the error message, it seems you are trying to pass a 0-dimensional tensor.

One possible reason can be if you have a mini-batch with n examples and you are running your program on more than n GPUs, then you will get this error.

Let's consider the following scenario.

Total training examples = 161, Batch size = 80, total mini-batches = 3
Number of GPUs specified for DataParallel = 3

Now, in the above scenario, in the 3rd mini-batch, there will be 1 example. So, it is not possible to send chunks to all the specific GPUs and you will receive the error message. So, please check if you are not a victim of this issue.


Instruction: How to calculate correct batch size for LSTM?

Input:
I have a daily time series data like below.

            CashIn  CashOut
Date
2016-01-01  0.0     6500.0
2016-01-02  0.0     23110.0
2016-01-03  0.0     7070.0
2016-01-04  0.0     18520.0
2016-01-05  20840.0 22200.0
.
.
.
2019-03-25  59880.0 25500.0
2019-03-26  49270.0 17860.0
2019-03-27  45160.0 48600.0
2019-03-28  39480.0 22840.0
2019-03-29  70260.0 25950.0
2019-03-30  19250.0 24350.0
2019-03-31  46870.0 14400.0

My total data size is 1186. I want to forecast the CashIn and CashOut values between 2019-04-01 and 2019-04-30 by using LSTM.

I wrote a batch calculator like below.

def get_batches(arr, batch_size, seq_length):

    batch_size_total = batch_size * seq_length

    n_batches = len(arr)//batch_size_total

    arr = arr[:n_batches * batch_size_total]
    arr = arr.reshape((batch_size, -1))

    for n in range(0, arr.shape[1], seq_length):
        x = arr[:, n:n+seq_length]
        y = np.zeros_like(x)
        try:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, n+seq_length]
        except IndexError:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, 0]
        yield x, y

I am trying to divide this dataset by using get_batches function to batches which have 30 sequence length because I have daily time series and I want to predict the next 30 days.

batches = get_batches(np.array(data_cashIn), 40, 30)

If I write 39 instead of 40 as a parameter in get_bathces function then I am losing the last 16 daily data but I do not want to lose these data.

How can I do this correctly?

Output:
I think that you'll always end up with a number that is not working. Because it's not the best practice. I suggest that you use the DataLoader which will easily load batches for you (and here's how you can have a custom dataset fed to the dataloder). By giving the batch_size to the Dataloader it will split your dataset into the maximum possible batches of batch_size with the last batch being <=batch_size.

In regard to the LSTM, use batch_first=True and have your batch in this shape (batch, seq, feature). This will allow you to drop the headache of giving a specific size, and the input_size must be equal to feature.


Instruction: Can’t convert np.ndarray of type numpy.bool_

Input:
I am wondering why I am getting this error when I used :

Y_train_class = torch.tensor(Y_train_class.values)

  TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

I have tried to convert my data to float but seems failed

X_train = pd.read_csv('c:/Data/x_train_set_yu.csv', header= None)
Y_train = pd.read_csv('c:/Data/y_train_set_yu.1.csv', header= None)

Y_train_class = (Y_train >= 550)
print (Y_train_class)
X_test = pd.read_csv('c:/Data/X_test.csv',header= None)

X_train = torch.tensor(X_train.values)

Y_train.astype(np.float32)
Y_train_class.astype(np.float32)

Y_train_class = torch.tensor(Y_train_class.values)

  TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

Output: I got this type of error when my Y_train had a string value instead of integers. After replacing strings with integers my error got resolved.


Instruction: Getting the last layer from a pretrained pytorch for transfer learning?

Input:
This is what I did:

list(tmp.state_dict().keys())[-1].split('.')[0]

What is the proper way?
My goal is to replace the last layer for the purpose of transfer learning.

Output:
You can simple follow these steps to get the last layer from a pretrained pytorch model:

We can get the layers by using model.children().
Convert this into a list by using a list() command on it.
Remove the last layer by indexing the list.
Finally, use the PyTorch function nn.Sequential() to stack this modified list together into a new model.

nn.Sequential(*list(model.children())[:-1])

You can read more about this from here.


Instruction: how to upload and read a zip file containing training and testing images data from google colab from my pc

Input:
I am new to google colab. I am implementing a pretrained vgg16 and resnet50 model using pytorch, but I am unable to load my file and read it as it returns an error of no directory found

I have uploaded the data through file also I have used to upload it using

from google.colab import files
uploaded = files.upload()

The file got uploaded but when I tried to unzip it because it is a zip file using
!unzip content/cropped_months
then it says

  no file found

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision.transforms import *
from torch.optim import lr_scheduler
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy

from google.colab import files
uploaded = files.upload()

!unzip content/cropped_months

data_dir = 'content/cropped_months'

#Define transforms for the training data and testing data
train_transforms = transforms.Compose([transforms.RandomRotation(30),transforms.RandomResizedCrop(224),transforms.RandomHorizontalFlip(),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

test_transforms = transforms.Compose([transforms.Resize(256),transforms.CenterCrop(224),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])

#pass transform here-in
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

#data loaders
trainloader = torch.utils.data.DataLoader(train_data, batch_size=8, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=8, shuffle=True)

print("Classes: ")
class_names = train_data.classes
print(class_names)

first error

  unzip:  cannot find or open content/cropped_months,
  content/cropped_months.zip or content/cropped_months.ZIP.

second error

  --------------------------------------------------------------------------- FileNotFoundError                         Traceback (most recent call
  last)  in ()
       16
       17 #pass transform here-in
  ---> 18 train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
       19 test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
       20

  2 frames
  /usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py
  in _find_classes(self, dir)
      114         if sys.version_info >= (3, 5):
      115             # Faster and available in Python 3.5 and above
  --> 116             classes = [d.name for d in os.scandir(dir) if d.is_dir()]
      117         else:
      118             classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]

  FileNotFoundError: [Errno 2] No such file or directory:
  'content/cropped_months (1)/train'

Output:
You are probably trying to access the wrong path. In my notebook, the file was uploaded to the working directory.

Use google.colab.files to upload the zip.

from google.colab import files
files.upload()

Upload your file. Google Colab will display where it was saved:

Saving dummy.zip to dummy.zip

Then just run !unzip:

!unzip dummy.zip


Instruction: RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor

Input:
I keep getting the error message below. I cannot seem to pinpoint to the tensor mentioned. Below you'll find the trainer.py and main.py modules. The model I am developing is GAN on CelebA dataset. I am running the code on a remote server so have spent a handful amount of time debugging my model.

This is the full error message:

Traceback (most recent call last):
  File "main.py", line 52, in <module>
    main(opt)
  File "main.py", line 47, in main
    trainer.train(train_loader)
  File "/home/path/trainer.py", line 45, in train
    d_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_org, size_average=False) / out_cls.size(0)
  File "/home/path/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2077, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor

trainer.py

from tqdm import tqdm
import torch
import torch.nn.functional as F
from model import Discriminator, Generator
from tensorboardX import SummaryWriter

class Trainer():
    def __init__(self, opt):
        # Generator
        self.G = Generator(64, 5, 6)
        # Discriminator
        self.D = Discriminator(128, 64, 5, 6)
        # Generator optimizer
        self.g_optimizer = torch.optim.Adam(self.G.parameters(), opt.lr)
        self.d_optimizer = torch.optim.Adam(self.D.parameters(), opt.lr)
        self.opt = opt
        if self.opt.cuda:
            self.G = self.G.cuda()
            self.D = self.D.cuda()

    def train(self, data_loader):
        """Function to train the model
        """
        print('Training model')
        writer_d = SummaryWriter('runs/disc') # discriminator writer
        writer_g = SummaryWriter('runs/gen') # generator writer
        print('Start training...')
        for epoch in tqdm(range(self.opt.epochs)):
            for x_real, label_org in tqdm(data_loader):
                pass
                # Generate target domain labels randomly.
                rand_idx = torch.randperm(label_org.size(0))
                label_trg = label_org[rand_idx]
                c_org = label_org.clone()
                c_trg = label_org.clone()
                if self.opt.cuda:
                    x_real = x_real.cuda()           # Input images
                    c_org = c_org.cuda()             # Original domain labels
                    c_trg = c_trg.cuda()             # Target domain labels
                    label_org = label_org.cuda()     # Labels for computing classification loss
                    label_trg = label_trg.cuda()     # Labels for computing classification loss

                out_src, out_cls = self.D(x_real)
                d_loss_real = - torch.mean(out_src)
                d_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_org, size_average=False) / out_cls.size(0)

                # Compute loss with fake images
                x_fake = self.G(x_real, c_trg)
                out_src, out_cls = self.D(x_fake.detach())
                d_loss_fake = torch.mean(out_src)

                # Compute loss for gradient penalty
                alpha = torch.rand(x_real.size(0), 1, 1, 1).cuda()
                x_hat = (alpha * x_real.data + (1 - alpha) * x_fake.data).requires_grad_(True)
                out_src, _ = self.D(x_hat)

                # Backward and optimize
                d_loss = d_loss_real + d_loss_fake + d_loss_cls
                self.g_optimizer.zero_grad()
                self.d_optimizer.zero_grad()
                d_loss.backward()
                self.d_optimizer.step()

                if (i + 1) % 2 == 0:
                    # Original-to-target domain
                    x_fake = self.G(x_real, c_trg)
                    out_src, out_cls = self.D(x_fake)
                    g_loss_fake = - torch.mean(out_src)
                    g_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_trg, size_average=False) / out_cls.size(0)

                    # Target-to-original domain
                    x_reconst = self.G(x_fake, c_org)
                    g_loss_rec = torch.mean(torch.abs(x_real - x_reconst))

                    # Backward and optimize
                    g_loss = g_loss_fake + g_loss_rec
                    self.g_optimizer.zero_grad()
                    self.d_optimizer.zero_grad()
                    g_loss.backward()
                    self.g_optimizer.step()

                # write loss to tensorboard
                writer_d.add_scalar('data/loss', d_loss, epoch)
                writer_d.add_scalar('data/loss', g_loss, epoch)
            print('Finished Training')

    def test(self, data_loader):
        with torch.no_grad():
            for i, (x_real, c_org) in enumerate(data_loader):

                # Prepare input images and target domain labels.
                if self.opt.cuda:
                    x_real = x_real.cuda()

                # Translate images.
                x_fake_list = [x_real]
                for c_trg in c_trg_list:
                    x_fake_list.append(self.G(x_real, c_trg))

main.py

import argparse
import random
import torch
from torch import nn
import torch.optim as optim
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
from preprocess import pre_process
from celeb_dataset import CelebDataset
from trainer import Trainer

# Setting up the argument parser
parser = argparse.ArgumentParser()
parser.add_argument('--workers', type=int, help='number of data loading workers', default=4)
parser.add_argument('--batchSize', type=int, default=8, help='input batch size')
parser.add_argument('--epochs', type=int, default=20, help='number of epochs to train')
parser.add_argument('--lr', type=float, default=0.0002, help='learning rate')
parser.add_argument('--cuda', action='store_true', help='enables cuda')
parser.add_argument('--manualSeed', type=int, help='manual seed')
parser.add_argument('--dataset_path', type=str, default='./data/celeba', help='dataset path')

opt = parser.parse_args()
print(opt)

if opt.manualSeed is None:
    opt.manualSeed = random.randint(1, 10000)
print("Random Seed: ", opt.manualSeed)

def main(opt):
    # Setup  the parameters for the training/testing
    params = {
        'batch_size': opt.batchSize,
        'shuffle': True,
        'num_workers': opt.workers
    }

    # preprocess and setup dataset and datalader
    processed_data = pre_process(opt.dataset_path)
    train_dataset = CelebDataset(processed_data[:-2000])
    test_dataset = CelebDataset(processed_data[2000:])
    train_loader = DataLoader(train_dataset, **params)
    test_loader = DataLoader(test_dataset, **params)

    trainer = Trainer(opt)
    trainer.train(train_loader)
    trainer.test(test_loader)

if __name__ == "__main__":
    main(opt)

Output:
You are getting that error because one of out_cls, label_org is not on the GPU.

Where does your code enact the parser.add_argument('--cuda', action='store_true', help='enables cuda') option?

Perhaps something like:

trainer = Trainer(opt)
if opt.cuda:
    trainer = trainer.cuda()


Instruction: AttributeError: module 'torch' has no attribute '_six'. Bert model in Pytorch

Input:
I tried to load pre-trained model by using BertModel class in pytorch.

I have _six.py under torch, but it still shows module 'torch' has no attribute '_six'

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
    551                                     .format(torch.typename(value), name))
    552                 modules[name] = value
--> 553             else:
    554                 buffers = self.__dict__.get('_buffers')
    555                 if buffers is not None and name in buffers:

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in register_parameter(self, name, param)
    140             raise KeyError("parameter name can't be empty string \"\"")
    141         elif hasattr(self, name) and name not in self._parameters:
--> 142             raise KeyError("attribute '{}' already exists".format(name))
    143
    144         if param is None:

AttributeError: module 'torch' has no attribute '_six'

Output: In jupyter notebook simply restarting kernel works fine


Instruction: How can I force torch.jit.trace to compule my module by ignoring hooks?

Input:
I have a module containing hook, and I would like to compile it with jit's trace:

compiled_model = torch.jit.trace(model,  torch.rand(1, 3, 256, 256))

But I get the error:

ValueError: Modules that have hooks assigned can't be compiled

How can I force trace to ignore the hooks ?

Output:
If you want to bypass trace's check, you can recursively remove all the hooks from your model.

This can be done by iterating over the children:

from collections import OrderedDict
def remove_hooks(model):
    model._backward_hooks = OrderedDict()
    model._forward_hooks = OrderedDict()
    model._forward_pre_hooks = OrderedDict()
    for child in model.children():
        remove_hooks(child)

Then you can force the compilation:

remove_hooks(model)
compiled_model = torch.jit.trace(model,  torch.rand(1, 3, 256, 256))

But if the hook are actually doing real work and you want to keep them in the trace (which was my case) you can just comment torch's raise in torch/jit/__init__.py the lines :

if orig._backward_hooks or orig._forward_hooks or orig._forward_pre_hooks:
    raise ValueError("Modules that have hooks assigned can't be compiled")

It worked for me and I managed to compile a fastai model.


Instruction: Expected target size (50, 88), got torch.Size([50, 288, 88])

Input:
I am trying to train my neuronal network. Train in the model is correct, but I can't calculate loss. The output and the target have the same dimension.

I had tried to use torch.stack, but I can't because the size of the each input is (252, x) where x is the same in the 252 elements, but is different for the others inputs.

I use a custom Dataset:

class MusicDataSet(Dataset):
def __init__(self, transform=None):
    self.ms, self.target, self.tam = sd.cargarDatos()
    self.mean, self.std = self.NormalizationValues()
def __len__(self):
    return self.tam

def __getitem__(self, idx):
    #Normalize
    inp = (self.ms[idx]-self.mean)/self.std
    inp = torch.from_numpy(inp).float()
    inp = inp.t()
    inp = inp.to('cuda')

    target= torch.from_numpy(self.target[idx])
    target = target.long()
    target = target.t()
    target = target.to('cuda')

    return inp, target

I must say that list can't be cast with something like: target = torch.Tensor() or torch.stack() because this (252, x), as I have already said.

def music_collate_fn(batch):
    data = [item[0] for item in batch]
    data = pad_sequence(data, batch_first=True)
    target = [item[0] for item in batch]
    target = pad_sequence(target, batch_first=True)
    return data, target

musicSet = mds.MusicDataSet()
train_loader = torch.utils.data.DataLoader(musicSet,batch_size=50, collate_fn = music_collate_fn, shuffle=False)

input_dim = 252
hidden_dim = (512,1024,512)
output_dim = 88
mlp = rn.MLP(input_dim, hidden_dim, output_dim).to(device)

optimizer = torch.optim.RMSprop(mlp.parameters(), lr = learning_rate)
criterion = nn.CrossEntropyLoss()
for batch_idx, (x,y) in enumerate(train_loader):
    outputs = mlp(x.to(device))
    loss = criterion(outputs, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The size of output and target is the same,

output: torch.Size([50, 288, 88])
target:  torch.Size([50, 288, 88])

But the next error apears when I try to calculate loss:

  File "<ipython-input-205-3c47d7aa11a4>", line 32, in <module>
    loss = criterion(outputs, y)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch
\nn\functional.py", line 1800, in nll_loss
    out_size, target.size()))

ValueError: Expected target size (50, 88), got torch.Size([50, 288, 88])

Output:
I think you are using CrossEntropyLoss incorrectly. See the documentation here.

In particular, if the input is of shape [NxCxd] then target should be of shape [Nxd], and value in target are integer between 0 and C-1 i.e you can just provide the class labels and it is not required to one-hot encode the target variable. Error message also states that same.


Instruction: Reshape and pad a tensor given a list of lengths

Input:
I have given a 2d tensor in of shape a x b like the following (where a = 9 and each of A1, A2, ..., C2 represents a b-dimensional vector):

Furthermore, I have an array of lengths, where sum(lengths) = a and each entry is a positive integer:

Then I would like to obtain a 3d output tensor out, where the first lengths[0] entries of in form the first row, the next lengths[1] entries of in form the second row, and so on. That is, the output tensor should have the shape len(lengths) x max(lengths) x b, and be padded with zeros (each 0 in the below picture represents a b-dimensional zero vector):

As this is part of a neural network that is trained using backpropagation, all operations used must be differentiable. How can this be achieved (ideally, with good performance) using PyTorch?

Output:
Here is my implementation using torch.nn.utils.rnn.pad_sequence():

in_tensor = torch.rand((9, 3))
print(in_tensor)
print(36*'=')
lengths = torch.tensor([3, 4, 2])
cum_len = 0
y = []
for idx, val in enumerate(lengths):
    y.append(in_tensor[cum_len : cum_len+val])
    cum_len += val
print(torch.nn.utils.rnn.pad_sequence(y, batch_first=True)))

output:

# in_tensor of shape (9 x 3)
tensor([[0.9169, 0.3549, 0.6211],
        [0.4832, 0.5475, 0.8862],
        [0.8708, 0.5462, 0.9374],
        [0.4605, 0.1167, 0.5842],
        [0.1670, 0.2862, 0.0378],
        [0.2438, 0.5742, 0.4907],
        [0.1045, 0.5294, 0.5262],
        [0.0805, 0.2065, 0.2080],
        [0.6417, 0.4479, 0.0688]])
====================================
# out tensor of shape (len(lengths) x max(lengths) x b), in this case b is 3
tensor([[[0.9169, 0.3549, 0.6211],
         [0.4832, 0.5475, 0.8862],
         [0.8708, 0.5462, 0.9374],
         [0.0000, 0.0000, 0.0000]],

        [[0.4605, 0.1167, 0.5842],
         [0.1670, 0.2862, 0.0378],
         [0.2438, 0.5742, 0.4907],
         [0.1045, 0.5294, 0.5262]],

        [[0.0805, 0.2065, 0.2080],
         [0.6417, 0.4479, 0.0688],
         [0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000]]])


Instruction: How to understand this on pytorch website?

Input:
I notice this on pytorch official website: https://pytorch.org/docs/stable/nn.html

If the following conditions are satisfied:

1) cudnn is enabled,

2) input data is on the GPU,

3) input data has dtype torch.float16,

4) V100 GPU is used, and

5) input data is not in PackedSequence format.

Then, persistent algorithm can be selected to improve performance.

Could anyone explain it? Thanks.

Output: This refers a to very low level performance optimization of GPU cache usage, which is explained more in-depth here (note: this is not a PyTorch material, but I believe it does a good enough job at explaining). In other words, if all the bullets are satisfied, PyTorch will default to a different algorithm under the hood, hopefully providing higher RNN performance.


Instruction: Pytorch: no CUDA-capable device is detected on Linux

Input:
When trying to run some Pytorch code I get this error:

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=74 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
File "demo.py", line 173, in test
pca = torch.FloatTensor( np.load('../basics/U_lrw1.npy')[:,:6]).cuda()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:74

I a running a cloud virtual machine using the 'Google Deep Learning VM'
Version: tf-gpu.1-13.m25
Based on: Debian GNU/Linux 9.9 (stretch) (GNU/Linux 4.9.0-9-amd64 x86_64\n)
Linux tf-gpu-interruptible 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64

Environment info:

$ nvidia-smi
Sun May 26 05:32:33 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.72       Driver Version: 410.72       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   42C    P0    74W / 149W |      0MiB / 11441MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

 $ echo $CUDA_PATH

 $ echo $LD_LIBRARY_PATH
/usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64

$ env | grep CUDA
CUDA_VISIBLE_DEVICES=0

$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.
7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
audioread==2.1.7
backports.functools-lru-cache==1.5
certifi==2019.3.9
chardet==3.0.4
cloudpickle==1.1.1
cycler==0.10.0
dask==1.2.2
decorator==4.4.0
dlib==19.17.0
enum34==1.1.6
filelock==3.0.12
funcsigs==1.0.2
future==0.17.1
gdown==3.8.1
idna==2.8
joblib==0.13.2
kiwisolver==1.1.0
librosa==0.6.3
llvmlite==0.28.0

Output:
I didn't get the main reason for your problem. But I noticed one thing, GPU-Util 100%, while there are no processes running behind.

You can try out in the following directions.

sudo nvidia-smi -pm 1

which enables in persistence mode. This might solve your problem. The combination of ECC with non persistence mode can lead to 100% Utilization of GPU.

You can also disable ECC with the command nvidia -smi -e 0
Or best will be restart once again the whole process from the starting i.e reboot the Operating System once again.

Note: I'm not sure whether it will work for you or not. I had faced similar issue earlier so I am just telling based on my experience.
Hope this will help you.


Instruction: How to initialize the weights of different layers of nn.Sequential block in different styles in pytorch?

Input:
Let's suppose I have a nn.Sequential block, it has 2 linear layers. I want to initialize the weights of first layer by uniform distribution but want to initialize the weights of second layer as constant 2.0.

net = nn.Sequential()
net.add_module('Linear_1', nn.Linear(2, 5, bias = False))
net.add_module('Linear_2', nn.Linear(5, 5, bias = False)

Output:
Here is one way of doing so:

import torch
import torch.nn as nn

net = nn.Sequential()

ll1 = nn.Linear(2, 5, bias = False)
torch.nn.init.uniform_(ll1.weight, a=0, b=1) # a: lower_bound, b: upper_bound
net.add_module('Linear_1', ll1)
print(ll1.weight)

ll2 = nn.Linear(5, 5, bias = False)
torch.nn.init.constant_(ll2.weight, 2.0)
net.add_module('Linear_2', ll2)
print(ll2.weight)

print(net)

Output:

Parameter containing:
tensor([[0.2549, 0.7823],
        [0.3439, 0.4721],
        [0.0709, 0.6447],
        [0.3969, 0.7849],
        [0.7631, 0.5465]], requires_grad=True)

Parameter containing:
tensor([[2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.]], requires_grad=True)

Sequential(
(Linear_1): Linear(in_features=2, out_features=5, bias=False)
(Linear_2): Linear(in_features=5, out_features=5, bias=False)
)


Instruction: how to deploy pytorch neural network for production code in Unity

Input: I have a neural network trained in pytorch that I'd like to deploy into a Unity app. What's the best way to do it? I'm also interested in allowing the user to further train the neural network in the Unity app, which I guess would require to integrate some part of pytorch into Unity (maybe there's a way to integrate pytorch's C++ / torchscript API with Unity?). If anybody has experience with this, I'd like to know what the best alternatives are.

Output:
Check out the new features in Unity ML Agents. There is an inference engine within Unity ML Agents (called Barracuda) that allows you to use pretrained models within your app. AFAIK, you can convert Tensorflow and ONNX models into Barracuda. It should not be a problem as Pytorch models can be converted to the ONNX format. You may need to retrain your model if it is directly affected by the app (for example, if it is an RL agent).
EDIT: To answer your second question, you can continue to train the model but not in real time. What you may be able to do is collect data from the user, and use that to further train the model (that is how TensorFlow Serving works). You can do that by converting the PyTorch model into a TensorFlow model via ONNX.
EDIT 2: Barracuda is now a standalone and production ready inference engine that runs exclusively on the ONNX format. Any framework that can be converted into the format (e.g. Keras, Pytorch, MXNet) will work as long as they contain the supported operators.


Instruction: CUDA runtime unknown error, maybe a driver problem? CUDA can't see my gpu

Input:
My code is very simple for now:

import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.cuda.current_device()

    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-3380d2c12118> in <module>
----> 1 torch.cuda.current_device()

~/.conda/envs/tensorflow/lib/python3.6/site-packages/torch/cuda/__init__.py in current_device()
    349 def current_device():
    350     r"""Returns the index of a currently selected device."""
--> 351     _lazy_init()
    352     return torch._C._cuda_getDevice()
    353

~/.conda/envs/tensorflow/lib/python3.6/site-packages/torch/cuda/__init__.py in _lazy_init()
    161             "Cannot re-initialize CUDA in forked subprocess. " + msg)
    162     _check_driver()
--> 163     torch._C._cuda_init()
    164     _cudart = _load_cudart()
    165     _cudart.cudaGetErrorName.restype = ctypes.c_char_p

RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp:51

Looking in the internet it looks like it is a version problem, but I swear I tried all combinations of drivers from CUDA 10.0, 10.1, tensorflow-gpu 13, 12, etc. and nothing seems to work.

NVIDIA driver: nvidia-smi:

    +-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.14       Driver Version: 430.14       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 930MX       Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   36C    P8    N/A /  N/A |    139MiB /  2004MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       986      G   /usr/lib/xorg/Xorg                            64MiB |
|    0      1242      G   /usr/bin/gnome-shell                          72MiB |
+-----------------------------------------------------------------------------+

CUDA VERSION nvcc --version :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

tensorflow-gpu version: pip list | grep tensorflow:

tensorflow                         1.13.1
tensorflow-estimator               1.13.0

pytorch version pip list | grep torch

pytorch-pretrained-bert            0.6.2
torch                              1.1.0
torchvision                        0.3.0

Can anyone see a problem of compatibility and explain why and how I can fix it?

Output:
Did you test your cuda installation ? If not you can use (which will take a while):

 $ cd ~/NVIDIA_CUDA-10.0_Samples
 $ make

And then:

$ cd ~/NVIDIA_CUDA-10.0_Samples/bin/x86_64/linux/release
$./deviceQuery

You should get "Test passed!" as result.

Source


Instruction: Why does multi layer perceprons outperform RNN in CartPole?

Input:
Recently, I compared two models for a DQN on CartPole-v0 environment. One of them is a multilayer perceptron with 3 layers and the other is an RNN built up from an LSTM and 1 fully connected layer. I have an experience replay buffer of size 200000 and the training doesn't start until it is filled up.
Although MLP has solved the problem under a reasonable amount of training steps  (this means to achieve a mean reward of 195 for the last 100 episodes), the RNN model could not converge as quickly and its maximum mean reward did not even reach 195 too!

I have already tried to increase batch size, add more neurons to the LSTM'S hidden state, increase the RNN'S sequence length and making the fully connected layer more complex - but every attempt failed as I saw enormous fluctuations in mean reward so the model hardly converged at all. May these are the sings of early overfitting?

class DQN(nn.Module):
    def __init__(self, n_input, output_size, n_hidden, n_layers, dropout=0.3):
        super(DQN, self).__init__()

        self.n_layers = n_layers
        self.n_hidden = n_hidden

        self.lstm = nn.LSTM(input_size=n_input,
            hidden_size=n_hidden,
            num_layers=n_layers,
            dropout=dropout,
            batch_first=True)

        self.dropout= nn.Dropout(dropout)

        self.fully_connected = nn.Linear(n_hidden, output_size)

    def forward(self, x, hidden_parameters):
        batch_size = x.size(0)

        output, hidden_state = self.lstm(x.float(), hidden_parameters)

        seq_length = output.shape[1]

        output1 = output.contiguous().view(-1, self.n_hidden)
        output2 = self.dropout(output1)
        output3 = self.fully_connected(output2)

        new = output3.view(batch_size, seq_length, -1)
        new = new[:, -1]

        return new.float(), hidden_state

    def init_hidden(self, batch_size, device):
        weight = next(self.parameters()).data

        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().to(device),
            weight.new(self.n_layers, batch_size, self.n_hidden).zero_().to(device))

        return hidden

Contrarily to what I expected, the simpler model gave a much better result than the other; even though RNN is supposed to be better in processing time series data.

Can anybody tell me what's the reason for this?

Also, I have to state that I applied no feature engineering and both DQN's worked with raw data. Could RNN outperform the MLP on using normalized features? (I mean feeding both models with normalized data)

Is there anything you can recommend me to improve training efficiency on RNN's to achieve the best results?

Output:
  Contrary to what I expected the simpler model gave much better result that the other; even though RNN's supposed to be better in processing time series data.

There is no time series in the cart-pole, the state contains all the information needed for optimal decision. It would be different if, for instance, you would learn from images and you would need to estimate the pole velocity from a series of images.

Also, it is not true that the more complex model should perform better. On the contrary, it is more likely to overfit. For the cart-pole you don't even need a NN, a simple linear approximator with RBFs or random Fourier features would suffice. A RNN + LSTM is for sure an overkill for such a simple problem.


Instruction: Treat a tuple/list of Tensors as a single Tensor

Input:
I'm using Pytorch for some robotics Reinforcement Learning tasks. I'd like to use both images and information about the state as observations for this task. The implementation I'm using does not directly support this so I'm making some amendments. Expected observations are either state, as a 1 dimensional Tensor, or images as a 3 dimensional Tensor (channels, width, height). In my task I would like the observation to be a tuple of Tensors.

In many places in my codebase, the observation is of course expected to be a single Tensor, not a tuple of Tensors. Is there an easy way to treat a tuple of Tensors as a single Tensor?

For example, I would like:

observation.to(device)

to work as normal when observation is a single Tensor, and call .to(device) on each Tensor when observation is a tuple of Tensors.

It should be simple enough to create a data type that can support this, but I'm wondering does such a data type already exist? I haven't found anything so far.

Output:
If your tensors are all of the same size, you can use torch.stack to concatenate them into one tensor with one more dimension.

Example:

>>> import torch
>>> a=torch.randn(2,1)
>>> b=torch.randn(2,1)
>>> c=torch.randn(2,1)
>>> a
tensor([[ 0.7691],
        [-0.0297]])
>>> b
tensor([[ 0.4844],
        [-0.9142]])
>>> c
tensor([[ 0.0210],
        [-1.1543]])
>>> torch.stack((a,b,c))
tensor([[[ 0.7691],
         [-0.0297]],

        [[ 0.4844],
         [-0.9142]],

        [[ 0.0210],
         [-1.1543]]])

You can then use torch.unbind to go the other direction.


Instruction: Pytorch RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CUDAType instead

Input:
I am trying to re-execute a GitHub project on my computer for recommendation using embedding, the goal is to first embed the user and item present in the movieLens dataset, and then use the inner product to predict a rating, when I finished the integration of all components, I got an error in the training.

Code:

from lightfm.datasets import fetch_movielens
movielens = fetch_movielens()
ratings_train, ratings_test = movielens['train'], movielens['test']
def _binarize(dataset):

    dataset = dataset.copy()

    dataset.data = (dataset.data >= 0.0).astype(np.float32)
    dataset = dataset.tocsr()
    dataset.eliminate_zeros()

    return dataset.tocoo()
train, test = _binarize(movielens['train']), _binarize(movielens['test'])
class ScaledEmbedding(nn.Embedding):
    """ Change the scale from normal to [0,1/embedding_dim] """
    def reset_parameters(self):
        self.weight.data.normal_(0, 1.0 / self.embedding_dim)
        if self.padding_idx is not None:
            self.weight.data[self.padding_idx].fill_(0)

class ZeroEmbedding(nn.Embedding):

    def reset_parameters(self):
        self.weight.data.zero_()
        if self.padding_idx is not None:
            self.weight.data[self.padding_idx].fill_(0)
class BilinearNet(nn.Module):

    def __init__(self, num_users, num_items, embedding_dim, sparse=False):
        super().__init__()

        self.embedding_dim = embedding_dim

        self.user_embeddings = ScaledEmbedding(num_users, embedding_dim,
                                               sparse=sparse)
        self.item_embeddings = ScaledEmbedding(num_items, embedding_dim,
                                               sparse=sparse)
        self.user_biases = ZeroEmbedding(num_users, 1, sparse=sparse)
        self.item_biases = ZeroEmbedding(num_items, 1, sparse=sparse)

    def forward(self, user_ids, item_ids):

        user_embedding = self.user_embeddings(user_ids)
        item_embedding = self.item_embeddings(item_ids)

        user_embedding = user_embedding.view(-1, self.embedding_dim)
        item_embedding = item_embedding.view(-1, self.embedding_dim)

        user_bias = self.user_biases(user_ids).view(-1, 1)
        item_bias = self.item_biases(item_ids).view(-1, 1)

        dot = (user_embedding * item_embedding).sum(1)

        return dot + user_bias + item_bias

def pointwise_loss(net,users, items, ratings, num_items):

    negatives = Variable(
            torch.from_numpy(np.random.randint(0,
                                               num_items,
                                                  len(users))).cuda()
    )

    positives_loss = (1.0 - torch.sigmoid(net(users, items)))
    negatives_loss = torch.sigmoid(net(users, negatives))

    return torch.cat([positives_loss, negatives_loss]).mean()

embedding_dim = 128
minibatch_size = 1024
n_iter = 10
l2=0.0
sparse = True

num_users, num_items = train.shape
net = BilinearNet(num_users,
                            num_items,
                            embedding_dim,
                            sparse=sparse).cuda()

optimizer = optim.Adagrad(net.parameters(),
                              weight_decay=l2)
for epoch_num in range(n_iter):

    users, items, ratings = shuffle(train)

    user_ids_tensor = torch.from_numpy(users).cuda()
    item_ids_tensor = torch.from_numpy(items).cuda()
    ratings_tensor = torch.from_numpy(ratings).cuda()

    epoch_loss = 0.0

    for (batch_user,
         batch_item,
         batch_ratings) in zip(_minibatch(user_ids_tensor,
                                          minibatch_size),
                               _minibatch(item_ids_tensor,
                                          minibatch_size),
                               _minibatch(ratings_tensor,
                                          minibatch_size)):

        user_var = Variable(batch_user)
        item_var = Variable(batch_item)
        ratings_var = Variable(batch_ratings)
        optimizer.zero_grad()
        loss = pointwise_loss(net,user_var, item_var, ratings_var, num_items)
        epoch_loss += loss.data[0]
        loss.backward()
        optimizer.step()
        print('Epoch {}: loss {}'.format(epoch_num, epoch_loss))

Error:

RuntimeError Traceback (most recent call last) <ipython-input-87-dcd04440363f> in <module>()
             22         ratings_var = Variable(batch_ratings)
             23         optimizer.zero_grad()
        ---> 24         loss = pointwise_loss(net,user_var, item_var, ratings_var, num_items)
             25         epoch_loss += loss.data[0]
             26         loss.backward()

        <ipython-input-86-679e10f637a5> in pointwise_loss(net, users, items, ratings, num_items)
              8
              9     positives_loss = (1.0 - torch.sigmoid(net(users, items)))
        ---> 10     negatives_loss = torch.sigmoid(net(users, negatives))
             11
             12     return torch.cat([positives_loss, negatives_loss]).mean()

        ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
        __call__(self, *input, **kwargs)
            491             result = self._slow_forward(*input, **kwargs)
            492         else:
        --> 493             result = self.forward(*input, **kwargs)
            494         for hook in self._forward_hooks.values():
            495             hook_result = hook(self, input, result)

        <ipython-input-58-3946abf81d81> in forward(self, user_ids, item_ids)
             16
             17         user_embedding = self.user_embeddings(user_ids)
        ---> 18         item_embedding = self.item_embeddings(item_ids)
             19
             20         user_embedding = user_embedding.view(-1, self.embedding_dim)

        ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
        __call__(self, *input, **kwargs)
            491             result = self._slow_forward(*input, **kwargs)
            492         else:
        --> 493             result = self.forward(*input, **kwargs)
            494         for hook in self._forward_hooks.values():
            495             hook_result = hook(self, input, result)

        ~\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
            115         return F.embedding(
            116             input, self.weight, self.padding_idx, self.max_norm,
        --> 117             self.norm_type, self.scale_grad_by_freq, self.sparse)
            118
            119     def extra_repr(self):

        ~\Anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)    1504         # remove once script supports set_grad_enabled    1505
        _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
        -> 1506     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)    1507     1508

        RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CUDAType instead (while checking arguments for embedding)

can anyone help me please ?

Output:
I would suggest you to check the input type
I had the same issue which solved by converting the input type from  int32 to int64.(running on win10)
ex:

x = torch.tensor(train).to(torch.int64)


Instruction: Pytorch method for conditional use of intermediate layer instead of final cnn layer output. ie: allow nn to learn to use more layers or less

Input:
I'm implementing a residual cnn(modified smaller version of xception) in a low latency environment. I've done a lot of manual tuning to minimize the run time speed of my network (reducing number of filters, removing layers, etc).

But now I want to try allowing my network to make its classification prediction(final fcnn layer) on the residual connection after each residual block.

basic logic-

attempt final prediction with residual connection as input

if this fcnn layer predicts a certain class with a probability > a set threshold:

return fcnn output as if it was normal final layer

else:

do next residual block like normal and try the previous conditional again unless we are already at final block

My hope is this will allow my network to learn to solve easier problems with less computation while allowing it to still do the additional layers if it is still unsure of the classification.

So my basic question is: In pytorch, whats the best way to implement this conditional in a way that allows my nn at run time to decide whether to do more processing or not

Currently Ive tested returning the intermediate x's after the blocks in the forward function, but I dont know how best to setup the conditional to chose which x to return

Also side note: I believe I may end up needing another cnn layer between the residual and fcnn to serve as a function to convert the internal representation for processing to a representation the fcnn understands for classification.

Output:
It has already been done and presented in ICLR 2018.

It appears as if in ResNets the first few bottlenecks learn representations (and therefore cannot be skipped) while the remaining bottlenecks refine the features and therefore can be skipped at a moderate loss of accuracy. (Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, Yoshua Bengio Residual Connections Encourage Iterative Inference, ICLR 2018).

This idea was taken to the extreme with sharing weights across bottlenecks in Sam Leroux, Pavlo Molchanov, Pieter Simoens, Bart Dhoedt, Thomas Breuel, Jan Kautz IamNN: Iterative and Adaptive Mobile Neural Network for efficient image classification, ICLR 2018.


Instruction: Your session crashed after using all available RAM in Google Colab

Input:
I am trying to get this to run in google colab https://github.com/oawiles/X2Face/blob/master/UnwrapMosaic/Face2Face_UnwrapMosaic.ipynb,

I was able to get it to run and display results once but since then lately I have been getting your session crashed after using all available RAM at this cell

    BASE_MODEL = '/scratch/shared/slow/ow/eccv/2018/release_models/' # Change to your path
state_dict = torch.load(BASE_MODEL + 'x2face_model.pth')

model = UnwrappedFaceWeightedAverage(output_num_channels=2, input_num_channels=3, inner_nc=128)
model.load_state_dict(state_dict['state_dict'])
model = model.cuda()

model = model.eval()

im not sure if it is a colab issue or not

here is the logs

warn("IPython.utils.traitlets has moved to a top-level traitlets package.")
/usr/local/lib/python2.7/dist-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.

Output:
Try to change settings,

Go to run-time->change run-time type->hardware accelerator->select option GPU or TPU

I think this might help.


Instruction: How to fix '_pickle.UnpicklingError: invalid load key, '<' ' error in Pytorch

Input:
The problem I encountered when I ran the official code of maskrcnn-benchmark for facebookresearch,which was wrong when loading the pre-training model.

The code runs on a remote server at the school and the graphics card is an NVIDIA P100.

checkpointer = DetectronCheckpointer(
        cfg, model, optimizer, scheduler, output_dir, save_to_disk)
extra_checkpoint_data = checkpointer.load(cfg.MODEL.WEIGHT)
arguments.update(extra_checkpoint_data)

I expect to run the code correctly and understand why this is the problem.

Output: The reason about the problem is that the previous download was not finished. So when I deleted the original file and re-downloaded it, the problem was solved.


Instruction: Pytorch second derivative returns None

Input:
I am unable to take a second derivative of the following function. When I want the second derivative with respect to u_s it works but with x_s it doesn't work.

Does anyone know what I have done wrong here?

def cost(xt, x_goal, u, Q, R):
        return (xt - x_goal).matmul(Q).matmul((xt - x_goal).transpose(0,1)) + u.matmul(R).matmul(u)

x_s = tr.tensor([ 0.0000, -1.0000,  0.0000], dtype=torch.float64,  requires_grad=True)
u_s = tr.tensor([-0.2749], dtype=torch.float64, requires_grad=True)
c = cost(x_s, x_Goal, u_s, tr.tensor(Q), tr.tensor(R))

c
   output:
   tensor([[4.0076]], dtype=torch.float64, grad_fn=<ThAddBackward>)

Cu = grad(c, u_s, create_graph=True)[0]
Cu
   output:
   tensor([-0.0550], dtype=torch.float64, grad_fn=<ThAddBackward>)

Cuu = grad(Cu, u_s, allow_unused=True)[0]
Cuu
   output:
   tensor([0.2000], dtype=torch.float64)

Cux = grad(Cu, x_s, allow_unused=True)
Cux
    output:
    (None,)

I am guessing the Cu itself is completely independent of x_s, but then the derivative should be zero at least, not None!

Output:
You haven't done anything wrong.

Suppose I have variables x, y, and z=f(y). If I compute z.backward() and then try to ask for the gradient with respect to x, I get None. For example,

import torch

x = torch.randn(1,requires_grad=True)
y = torch.randn(1,requires_grad=True)

z = y**2
z.backward()
print(y.grad) # Outputs some non-zero tensor
print(x.grad) # None

So what does this have to do with your attempt to compute the second derivative Cux? When you write create_graph=True, PyTorch keeps track of all the operations in the derivative computations which computed Cu, and since the derivatives themselves are made up of primitive operations, you can compute the gradient of the gradient, as you are doing. The problem here is that the gradient Cu never encounters the variable x_s, so effectively Cu = f(u_s). This means that when you perform Cu.backward(), the computational graph for Cu never sees the variable x_s, so it's gradient type remains None, just like in the example above.


Instruction: Why are there 3 pythons installed on my computer?

Input:
When I try to see the version of python installed on my computer, I see the followings:

(base) dhcp76:bin me$ python -V

Python 2.7.16 :: Anaconda, Inc.

(base) dhcp76:bin me$ python2 -V

Python 2.7.16

(base) dhcp76:bin me$ python3 -V

Python 3.7.3

Would this cause any issue? I have also installed anaconda3, but python3 doesn't point to that, and I don't know how to make it point to anaconda3.

So, my questions:

Would having 2 python versions both by brew and anaconda cause problems? If yes, should I remove one of them? (I prefer anaconda)
I installed pytorch using this link: http://deeplizard.com/learn/video/UWlFM0R_x6I, and now when I do import torch in all three version it works!! How is this possible if this links only installs using pip3?

Thanks!

Output:
Yes, having different version of Python can cause significant headache when you're installing python packages.

For example, if you install a package with Brew, your Anaconda installation might not be able to find it and vice-versa.

I had numerous consistency issues with maintaining all these different version of Python before I decided to completely uninstall all of them and only keep MacPorts as my general package manager.

When you use sudo pip install that could be using a different python than when you use python -m pip install which could be different from pip3 install ... etc.

There are lots of tradeoffs to each package manager.

Brew is good for people who want to get up to speed on a project quickly.
Anaconda has a great interface that allows you to minimize command line interface interactions and abstracts away some configuration stuff.
Macports has way more packages actively maintained than Brew, but requires more setup. For me, it was worth it because I've never had Python package dependency issues anymore. (Though that was also due to learning how to properly use virtualenvwrapper too.)

Most of the time, if you are fully aware of which pip/python you're calling, then you can avoid any issues and have all three.

However realistically, you may lose track of which Python versions are available with which packages in which system paths.

If things get bad, you might encounter situations where you try to pip install a package, and your system will say it already exists, but you may not be able to import <package> from python <file>.py or Terminal.


Instruction: In pytorch, how to fill a tensor with another tensor?

Input:
I'm looking for a way to expand the size of an image by adding 0 values to the right & lower edges of it. My initial plan is to use nn.padding to add the edge, until I encounter this error:

  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2796, in pad
    assert len(pad) % 2 == 0, 'Padding length must be divisible by 2'
AssertionError: Padding length must be divisible by 2

It appears that torch tries to pad the image from both side! Is there an easy way to override this and fill the tensor into the upper-left side of another image?

Output:
the only way I know is:
with torch.no_grad(): # assuming it's for init
  val = torch.distributions.MultivariateNormal(loc=zeros(2), scale=torch.eye(2))
  w.data =  val

but I doubt it's recommended.
Answering the title of the question.


Instruction: Implementing STFT with Pytorch gives a slightly different result than the STFT with Librose

Input:
I am trying to implement STFT with Pytorch. But the output from the Pytorch implementation is slightly off, when compared with the implementation from Librosa.

Librosa version

import numpy as np
from librosa.core import stft
import matplotlib.pyplot as plt

np.random.seed(3)
y = np.sin(2*np.pi*50*np.linspace(0,10,2048))+np.sin(2*np.pi*20*np.linspace(0,10,2048)) + np.random.normal(scale=1,size=2048)

S_stft = np.abs(stft(y, hop_length=512, n_fft=2048,center=False))

plt.plot(S_stft)

Pytorch version

import torch
from torch.autograd import Variable
from torch.nn.functional import conv1d

from scipy.signal.windows import hann

stride = 512

def create_filters(d,k,low=50,high=6000):
    x = np.arange(0, d, 1)
    wsin = np.empty((k,1,d), dtype=np.float32)
    wcos = np.empty((k,1,d), dtype=np.float32)
    start_freq = low
    end_freq = high
    # num_cycles = start_freq*d/44000.
    # scaling_ind = np.log(end_freq/start_freq)/k

    window_mask = hann(2048, sym=False) # same as 0.5-0.5*np.cos(2*np.pi*x/(k))
    for ind in range(k):
        wsin[ind,0,:] = window_mask*np.sin(2*np.pi*ind/k*x)
        wcos[ind,0,:] = window_mask*np.cos(2*np.pi*ind/k*x)

    return wsin,wcos

wsin, wcos = create_filters(2048,2048)

wsin_var = Variable(torch.from_numpy(wsin), requires_grad=False)
wcos_var = Variable(torch.from_numpy(wcos),requires_grad=False)

network_input = torch.from_numpy(y).float()
network_input = network_input.reshape(1,-1)

zx = np.sqrt(conv1d(network_input[:,None,:], wsin_var, stride=stride).pow(2)+conv1d(network_input[:,None,:], wcos_var, stride=stride).pow(2))
pytorch_Xs = zx.cpu().numpy()
plt.plot(pytorch_Xs[0,:1025,0])

My Question

The two graphs might look the same, but if I check the two outputs with np.allclose, we can see that they are slightly different.

np.allclose(S_stft, pytorch_Xs[0,:1025,0].reshape(1025,1))
output >>> False

Only when I tune up the tolerance to 1e-5, it gives me True result

np.allclose(S_stft, pytorch_Xs[0,:1025,0].reshape(1025,1),atol=1e-5)
output >>> True

What causes the difference in values? Is it because of the data conversion by using torch.from_numpy(y).float()?

I would like to have a difference in value less than 1e-7, 1e-8 is even better.

Output:
The difference is from the difference between their default bit.
NumPy's float is 64bit by default.
PyTorch's float is 32bit by default.


Instruction: How to run matlab .m files in google colab

Input:
I am currently trying to run this repo
https://github.com/Fanziapril/mvfnet
which requires a step:

"Run the Matlab/ModelGeneration/ModelGenerate.m to generate the shape
model "Model_Shape.mat" and copy it to the Matlab/"

Is it possible to run a .m file in colab to do this?

Also, I have looked into oct2py library
https://blink1073.github.io/oct2py/,
but was not able to successfully run the file.
I followed this How to run a MATLAB code on Python

Output:
You need to first install octave with

!apt install octave

Then you can run your m-file with

!octave -W file.m

Here's a minimal example.


Instruction: Using tensor.share_memory_() vs multiprocessing.Queue in PyTorch when training model across multiple processes

Input:
I'm using the multiprocessing package in pytorch to split the training across multiple processes. My x and y, train and test data are CUDA tensors. I'm trying to understand the difference between using the tensor.share_memory_() and the multiprocessing.Queue method to share cuda tensors. Which is preferred and why?

Here's my current code using tensor.share_memory_(). What changes should I make?

def train(model, features, target, epochs=1000):

    X_train, x_test, Y_train, y_test = train_test_split(features,
                                                    target,
                                                    test_size=0.4,
                                                    random_state=0)

    Xtrain_ = torch.from_numpy(X_train.values).float().share_memory_()
    Xtest_ = torch.from_numpy(x_test.values).float().share_memory_()

    Ytrain_ = (torch.from_numpy(Y_train.values).view(1,-1)[0]).share_memory_()
    Ytest_ = (torch.from_numpy(y_test.values).view(1,-1)[0]).share_memory_()

    optimizer = optim.Adam(model.parameters(), lr = 0.01)
    loss_fn = nn.NLLLoss()

    for epoch in range(epochs):

        #training code here

target method ends here

mp.set_start_method('spawn')

model = Net()
model.share_memory()

processes = []

for rank in range(1):
    p = mp.Process(target=train, args=(model, features, target))
    p.start()
    processes.append(p)

Env details: Python-3 and Linux

Output: They are the same.  torch.multiprocessing.Queue uses tensor.share_memory_() internally.


Instruction: SageMaker PyTorchModel passing custom variables

Input:
When deploying a model with SageMaker through the PyTorchModel class, is it possible to pass a custom environmental variable or kwargs?

I'd like to be able to switch the functionality of the serving code via a custom argument rather than needing to write multiple serve.py to handle different training model export methods.

model = PyTorchModel(name='my_model',
                     model_data=estimator.model_data,
                     role=role,
                     framework_version='1.0.0',
                     entry_point='serve.py',
                     source_dir='src',
                     sagemaker_session=sess,
                     predictor_cls=ImagePredictor,
                     <custom_argument?>
                    )

Output:
Have you tried using the env parameter in your PyTorchModel ? (cf. https://sagemaker.readthedocs.io/en/stable/model.html#sagemaker.model.Model)

model = PyTorchModel(name='my_model',
                     model_data=estimator.model_data,
                     role=role,
                     framework_version='1.0.0',
                     entry_point='serve.py',
                     source_dir='src',
                     sagemaker_session=sess,
                     predictor_cls=ImagePredictor,
                     env={'ENV_VALUE': 'val'}
                    )


Instruction: Nvcc missing when installing cudatoolkit?

Input:
I have installed cuda along pytorch with

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

However, it seems like nvcc was not installed along with it. If I want to use for example nvcc -V, I get the error that nvcc was not found, and that I should install it with sudo apt install nvidia-cuda-toolkit.
Can I do this (I dont want to just try and then find out that it is not working/messes up the whole cuda setup).
And is this a bug or expected behavior?

I am using Ubuntu 18.04 and have cuda 10.2

Output:
Met this question when installing cudatoolkit of 10.1 with PyTorch 1.4.

There is a conda-forge package https://anaconda.org/conda-forge/cudatoolkit-dev. After installing this, nvcc as well as other CUDA libraries will be then available at /home/li/anaconda3/envs/<env_name>/pkgs/cuda-toolkit in bin/ and lib/.


Instruction: PyTorch most efficient Jacobian/Hessian calculation

Input:
I am looking for the most efficient way to get the Jacobian of a function through Pytorch and have so far come up with the following solutions:
# Setup
def func(X):
    return torch.stack((X.pow(2).sum(1),
                        X.pow(3).sum(1),
                        X.pow(4).sum(1)),1)

X = Variable(torch.ones(1,int(1e5))*2.00094, requires_grad=True).cuda()

# Solution 1:
t = time()
Y = func(X)
J = torch.zeros(3, int(1e5))

for i in range(3):
    J[i] = grad(Y[0][i], X, create_graph=True, retain_graph=True, allow_unused=True)[0]

print(time()-t)
>>> Output: 0.002 s

# Solution 2:
def Jacobian(f,X):
    X_batch = Variable(X.repeat(3,1), requires_grad=True)
    f(X_batch).backward(torch.eye(3).cuda(), retain_graph=True)
    return X_batch.grad

t = time()
J2 = Jacobian(func,X)
print(time()-t)
>>> Output: 0.001 s

Since there seem to be not a big difference between using a loop in the first solution than the second one, I wanted to ask if there might still be be a faster way to calculate a Jacobian in pytorch.
My other question is then also about what might be the most efficient way to calculate the Hessian.
Finally, does anyone know if something like this can be done easier or more efficient in TensorFlow?

Output:
functorch can speed up computations even more. E.g., this code is from the functorch docs for batched Jacobian calculation (Hessian works too):
batch_size = 64
Din = 31
Dout = 33

weight = torch.randn(Dout, Din)
print(f"weight shape = {weight.shape}")
bias = torch.randn(Dout)

def predict(weight, bias, x):
    return F.linear(x, weight, bias).tanh()

x = torch.randn(batch_size, Din)
compute_batch_jacobian = vmap(jacrev(predict, argnums=2), in_dims=(None, None, 0))
batch_jacobian0 = compute_batch_jacobian(weight, bias, x)


Instruction: IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

Input:
def nms(bboxes,scores,threshold=0.5):
    '''
    bboxes(tensor) [N,4]
    scores(tensor) [N,]
    '''
    x1 = bboxes[:,0]
    y1 = bboxes[:,1]
    x2 = bboxes[:,2]
    y2 = bboxes[:,3]
    areas = (x2-x1) * (y2-y1)

    _,order = scores.sort(0,descending=True)
    keep = []
    while order.numel() > 0:
        i = order[0]
        keep.append(i)

        if order.numel() == 1:
            break

        xx1 = x1[order[1:]].clamp(min=x1[i])
        yy1 = y1[order[1:]].clamp(min=y1[i])
        xx2 = x2[order[1:]].clamp(max=x2[i])
        yy2 = y2[order[1:]].clamp(max=y2[i])

        w = (xx2-xx1).clamp(min=0)
        h = (yy2-yy1).clamp(min=0)
        inter = w*h

        ovr = inter / (areas[i] + areas[order[1:]] - inter)
        ids = (ovr<=threshold).nonzero().squeeze()
        if ids.numel() == 0:
            break
        order = order[ids+1]
    return torch.LongTensor(keep)

I tried

i=order.item()

But it does not work

Output:
I found the solution in the github issues here

Try to change

i = order[0] # works for PyTorch 0.4.1.

to

i = order # works for PyTorch>=0.5.


Instruction: Add padding in pytorch c++ API

Input:
I have a tensor with dimensions (1,3, 375, 1242). I want to reshape it to (1, 3, 384, 1248) by adding padding into it. How do i do that in Pytorch c++ API. Thank you in advance.

target = torch.zeros(1, 3, 384, 1248)
source = torch.ones(1, 3, 375, 1242)
target[: , : , :375, :1242] = source

Output:
You can use torch::constant_pad_nd

torch::Tensor source = torch::ones(torch::IntList{1, 3, 375, 1242});
// add 6 zeros to the last dimension and 9 zeros to the third dimension
torch::Tensor target = torch::constant_pad_nd(target, IntList{0, 6, 0, 9}, 0);


Instruction: how to convert series numpy array into tensors using pytorch

Input:
I am trying to convert image labels convert into tensor, but  I got some error please help me to convert to tensor:
Here My code:

features_train, features_test, targets_train, targets_test = train_test_split(X,Y,test_size=0.2,
                                                                              random_state=42)
X_train = torch.from_numpy(features_train)
X_test = torch.from_numpy(features_test)

Y_train =torch.from_numpy(targets_train).type(torch.IntTensor)
Y_test = torch.from_numpy(targets_test).type(torch.IntTensor)
train = torch.utils.data.TensorDataset(X_train,Y_train)
test = torch.utils.data.TensorDataset(X_test,Y_test)

train_loader = torch.utils.data.DataLoader(train, batch_size = train_batch_size, shuffle = False)
test_loader = torch.utils.data.DataLoader(test, batch_size = test_batch_size, shuffle = False)

Here my error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-32-f1578581ff5c> in <module>()
      5 X_test = torch.from_numpy(features_test)
      6
----> 7 Y_train =torch.from_numpy(targets_train).type(torch.IntTensor)
      8 Y_test = torch.from_numpy(targets_test).type(torch.IntTensor)
      9 train = torch.utils.data.TensorDataset(X_train,Y_train)

TypeError: expected np.ndarray (got Series)

Here my array values:

targets_train
478     1
5099    3
1203    2
5674    2
142     1
4836    2
4031    1
1553    3
4416    1
605     5
1194    3
4319    4
1498    5

Output:
Here is what I would do:

import torch
import numpy as np
n = np.arange(10)
print(n) #[0 1 2 3 4 5 6 7 8 9]
t1 = torch.Tensor(n)  # as torch.float32
print(t1) #tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
t2 = torch.from_numpy(n)  # as torch.int32
print(t2) #tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=torch.int32)


Instruction: PyTorch LSTM input dimension

Input:
I'm trying train a simple 2 layer neural network with PyTorch LSTMs and I'm having trouble interpreting the PyTorch documentation. Specifically, I'm not too sure how to go about with the shape of my training data.

What I want to do is train my network on a very large dataset through mini-batches, where each batch is say, 100 elements long. Each data element will have 5 features. The documentation states that the input to the layer should be of shape (seq_len, batch_size, input_size). How should I go about shaping the input?

I've been following this post: https://discuss.pytorch.org/t/understanding-lstm-input/31110/3
and if I'm interpreting this correctly, each minibatch should be of shape (100, 100, 5). But in this case, what's the difference between seq_len and batch_size? Also, would this mean that the first layer that the input LSTM layer should have 5 units?

Thank you!

Output:
This is an old question, but since it has been viewed 80+ times with no response, let me take a crack at it.

An LSTM network is used to predict a sequence. In NLP, that would be a sequence of words; in economics, a sequence of economic indicators; etc.

The first parameter is the length of those sequences. If you sequence data is made of sentences, then "Tom has a black and ugly cat" is a sequence of length 7 (seq_len), one for each word, and maybe an 8th to indicate the end of the sentence.

Of course, you might object "what if my sequences are of varying length?" which is a common situation.

The two most common solutions are:

Pad your sequences with empty elements. For instance, if the longest sentence you have has 15 words, then encode the sentence above as "[Tom] [has] [a] [black] [and] [ugly] [cat] [EOS] [] [] [] [] [] [] []", where EOS stands for end of sentence. Suddenly, all your sequences become of length 15, which solves your issue. As soon as the [EOS] token is found, the model will learn quickly that it is followed by an unlimited sequence of empty tokens [], and that approach will barely tax your network.
Send mini-batches of equal lengths. For instance, train the network on all sentences with 2 words, then with 3, then with 4. Of course, seq_len will be increased at each mini batch, and the size of each mini batch will vary based on how many sequences of length N you have in your data.

A best-of-both-world approach would be to divide your data into mini batches of roughly equal sizes, grouping them by approximate length, and adding only the necessary padding. For instance, if you mini-batch together sentences of length 6, 7 and 8, then sequences of length 8 will require no padding, whereas sequence of length 6 will require only 2. If you have a large dataset with sequences of widely varying length, that's the best approach.

Option 1 is the easiest (and laziest) approach, though, and will work great on small datasets.

One last thing... Always pad your data at the end, not at the beginning.

I hope that helps.