Instruction: How to enable Dict/OrderedDict/NamedTuple support in pytorch 1.1.0 JIT compiler?

Input:
From the release highlight of pytorch 1.1.0. It appears that the latest JIT compiler now supports Dict type. (Source: https://jaxenter.com/pytorch-1-1-158332.html)

    Dictionary and list support in TorchScript: Lists and dictionary types behave like Python lists and dictionaries.

Unfortunately I can't find a way to make this improvement to work properly. The following code is a simple example of exporting a Feature Pyramid Network (FPN) into tensorboard, which uses the JIT compiler:

from collections import OrderedDict

import torch
import torchvision
from torch.utils.tensorboard import SummaryWriter

torchWriter = SummaryWriter(log_dir=".tensorboard/example1")

m = torchvision.ops.FeaturePyramidNetwork([10, 20, 30], 5)
# get some dummy data
x = OrderedDict()
x['feat0'] = torch.rand(1, 10, 64, 64)
x['feat2'] = torch.rand(1, 20, 16, 16)
x['feat3'] = torch.rand(1, 30, 8, 8)
# compute the FPN on top of x
output = m.forward(x)
print([(k, v.shape) for k, v in output.items()])

torchWriter.add_graph(m, input_to_model=x)

When I run it I got the following error:

Traceback (most recent call last):
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 276, in graph
    trace, _ = torch.jit.get_trace_graph(model, args)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/jit/__init__.py", line 231, in get_trace_graph
    return LegacyTracedModule(f, _force_outplace, return_inputs)(*args, **kwargs)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/jit/__init__.py", line 284, in forward
    in_vars, in_desc = _flatten(args)
RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got collections.OrderedDict

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/peng/git-drone/gate_detection/python/gate_detection/errorcase/tb.py", line 36, in <module>
    torchWriter.add_graph(m, input_to_model=x)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 534, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose, **kwargs))
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 279, in graph
    _ = model(*args)  # don't catch, just print the error message
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 4 were given

From the error message it appears that the support is still pending. Can I trust the release highlight? Or I'm not using the API properly?

Output:
The release notes are accurate albeit a little vague. The dictionary/list/user defined classes support described in that link (and the official release notes) only apply to the TorchScript compiler (there are some code examples in the release notes), but SummaryWriter by default will run the TorchScript tracer on whatever module you pass to it, and the tracer only supports Tensors and lists/tuples of Tensors.

So the fix would be to use the TorchScript compiler rather than the tracer, but that requires:

Access to the original code
Support for the compiled output (ScriptModule) in Tensorboard

You should file an issue for (2), and there is ongoing work to fix (1), but this won't work in the short term for that model afaik.


Instruction: How to select subtensor from a tensor

Input:
I have a tensor A with the dimension of [N1, N2, N3/2, 2, N4, N5]. Another tensor B is an boolean index tensor with the dimension of [N1, N2, N3/2] = 1,0,0,0,1....
Now, I want to get a subtensor C with the dimension of [N1, N2, N3/2, N4, N5] using the index of B .
How could I implement this with pytorch?
Note that I don't incline to use for-loop because it is very slow.

I have looked up some functions in pytorch but found none of them are suitable for my task

Output:
You are making no sense: How can C have the same shape as A when B only selects part of the entries of A to copy to C??

Suppose B has k non-zero elements (that is, B selects k elements out of N1*N2*N3/2 matrices of size N4*N5 in A) then C can have shape of [k, N4, N5] only with k < N1*N2*N3/2.

You can use nonzero() to convert B from logical indices to k integer indices and then use these to select the k elements

C = A.view(-1, *A.shape[-2:])[B.nonzero(), ...]


Instruction: model.cuda() in pytorch

Input: If I call model.cuda() in pytorch where model is a subclass of nn.Module, and say if I have four GPUs, how it will utilize the four GPUs and how do I know which GPUs that are using?

Output:
If you have a custom module derived from nn.Module after model.cuda() all model parameters, (model.parameters() iterator can show you these) will end on your cuda.

To check where are your parameters just print them (cuda:0) in my case:

class M(nn.Module):
    'custom module'
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(784, 10)

m = M()
m.cuda()
for _ in m.parameters():
    print(_)

# Parameter containing:
# tensor([[-0.0201,  0.0282, -0.0258,  ...,  0.0056,  0.0146,  0.0220],
#         [ 0.0098, -0.0264,  0.0283,  ...,  0.0286, -0.0052,  0.0007],
#         [-0.0036, -0.0045, -0.0227,  ..., -0.0048, -0.0003, -0.0330],
#         ...,
#         [ 0.0217, -0.0008,  0.0029,  ..., -0.0213,  0.0005,  0.0050],
#         [-0.0050,  0.0320,  0.0013,  ..., -0.0057, -0.0213,  0.0045],
#         [-0.0302,  0.0315,  0.0356,  ...,  0.0259,  0.0166, -0.0114]],
#        device='cuda:0', requires_grad=True)
# Parameter containing:
# tensor([-0.0027, -0.0353, -0.0349, -0.0236, -0.0230,  0.0176, -0.0156,  0.0037,
#          0.0222, -0.0332], device='cuda:0', requires_grad=True)

You can also specify the device like this:

m.cuda('cuda:0')

With torch.cuda.device_count() you may check how many devices you have.


Instruction: Difference in shape of tensor torch.Size([]) and torch.Size([1]) in pytorch

Input:
I am new to pytorch. While playing around with tensors I observed 2 types of tensors-

tensor(58)
tensor([57.3895])

I printed their shape and the output was respectively -

torch.Size([])
torch.Size([1])

What is the difference between the two?

Output:
You can play with tensors having the single scalar value like this:

import torch

t = torch.tensor(1)
print(t, t.shape) # tensor(1) torch.Size([])

t = torch.tensor([1])
print(t, t.shape) # tensor([1]) torch.Size([1])

t = torch.tensor([[1]])
print(t, t.shape) # tensor([[1]]) torch.Size([1, 1])

t = torch.tensor([[[1]]])
print(t, t.shape) # tensor([[[1]]]) torch.Size([1, 1, 1])

t = torch.unsqueeze(t, 0)
print(t, t.shape) # tensor([[[[1]]]]) torch.Size([1, 1, 1, 1])

t = torch.unsqueeze(t, 0)
print(t, t.shape) # tensor([[[[[1]]]]]) torch.Size([1, 1, 1, 1, 1])

t = torch.unsqueeze(t, 0)
print(t, t.shape) # tensor([[[[[[1]]]]]]) torch.Size([1, 1, 1, 1, 1, 1])

#squize dimension with id 0
t = torch.squeeze(t,dim=0)
print(t, t.shape) # tensor([[[[[1]]]]]) torch.Size([1, 1, 1, 1, 1])

#back to beginning.
t = torch.squeeze(t)
print(t, t.shape) # tensor(1) torch.Size([])

print(type(t)) # <class 'torch.Tensor'>
print(type(t.data)) # <class 'torch.Tensor'>

Tensors, do have a size or shape. Which is the same. Which is actually a class torch.Size.
You can write help(torch.Size) to get more info.
Any time you write t.shape, or t.size() you will get that size info.

The idea of tensors is they can have different compatible size dimension for the data inside it including torch.Size([]).

Any time you unsqueeze a tensor it will add another dimension of 1.
Any time you squeeze a tensor it will remove dimensions of 1, or in general case all dimensions of one.


Instruction: Vectorized implementation of field-aware factorization

Input:
I would like to implement the field-aware factorization model (FFM) in a vectorized way. In FFM, a prediction is made by the following equation

where w are the embeddings that depend on the feature and the field of the other feature. For more info, see equation (4) in FFM.

To do so, I have defined the following parameter:

import torch

W = torch.nn.Parameter(torch.Tensor(n_features, n_fields, n_factors), requires_grad=True)

Now, given an input x of size (batch_size, n_features), I want to be able to compute the previous equation. Here is my current (non-vectorized) implementation:

total_inter = torch.zeros(x.shape[0])
for i in range(n_features):
    for j in range(i + 1, n_features):
        temp1 = torch.mm(
            x[:, i].unsqueeze(1),
            W[i, feature2field[j], :].unsqueeze(0))
        temp2 = torch.mm(
            x[:, j].unsqueeze(1),
            W[j, feature2field[i], :].unsqueeze(0))
        total_inter += torch.sum(temp1 * temp2, dim=1)

Unsurprisingly, this implementation is horribly slow since n_features can easily be as large as 1000! Note however that most of the entries of x are 0. All inputs are appreciated!

Edit:

If it can help in any ways, here are some implementations of this model in PyTorch:

pytorch-fm
ctr_model_zoo

Unfortunately, I cannot figure out exactly how they have done it.

Additional update:

I can now obtain the product of x and W in a more efficient way by doing:

temp = torch.einsum('ij, jkl -> ijkl', x, W)

Thus, my loop is now:

total_inter = torch.zeros(x.shape[0])
for i in range(n_features):
    for j in range(i + 1, n_features):
        temp1 = temp[:, i, feature2field[j], :]
        temp2 = temp[:, j, feature2field[i], :]
        total_inter += 0.5 * torch.sum(temp1 * temp2, dim=1)

It is however still too long since this loop goes over for about 500 000 iterations.

Output:
Something that could potentially help you speed up the multiplication is using pytorch sparse tensors.
Also something that might work would be the following:
Create n arrays, one for each feature i that would hold its corresponding field factors in each row. e.g. for feature i = 0

[ W[0, feature2field[0], :],
  W[0, feature2field[1], :],
  W[0, feature2field[n], :]]

Then calculate the multiplication of those arrays, lets call them F, with X

R[i] = F[i] * X

So each element in R would hold the result of the multiplication, an array, of the F[i] with X.

Next you would multiply each R[i] with its transpose

R[i] = R[i] * R[i].T

Now you can do the summation in a loop like before

for i in range(n_features):
    total_inter += torch.sum(R[i], dim=1)

Please take this with a grain of salt as i haven't tested it. In any case i think that it will point you in the right direction.

One problem that might occur is in the transpose multiplication in which each element will also be multiplied with itself and then be added in the sum. I don't think it will affect the classifier but in any case you can make the elements in the diagonal of the transpose and above 0 (including the diagonal).

Also although minor nevertheless please move the 1st unsqueeze operation outside of the nested for loop.

I hope it helps.


Instruction: Runtime error when load tensorflow and pytorch models at the same time

Input:
Here is the setup

Project A
    |-- Module A
           |--->step1 - Loads and run tensorflow pretrained model to detect
                           texts
           |
           |--->step2 - import and Instantiates Module-B from Project B
                             --> Runtime Failure here.

Project B
    |-- Module B
           |--> step 1 - Loads pretrained Pytorch model
           |--> step 2 - Runs model to classify text.

Here is the actual runtime error

self.trmodel.load_state_dict(torch.load(self.opts.saved_model, map_location='cpu'))
File/root/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 777, in load_state_dict
self.__class__.__name__,\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
Unexpected key(s) in state_dict:module.Transformation.LocalizationNetwork.conv.0.weight",module.Transformation.LocalizationNetwork.conv.1.weight",module.Transformation.LocalizationNetwork.conv.1.bias",module.Transformation.LocalizationNetwork.conv.1.running_mean",module.Transformation.LocalizationNetwork.conv.1.running_var",module.Transformation.LocalizationNetwork.conv.1.num_batches_tracked",module.Transformation.LocalizationNetwork.conv.4.weight",module.Transformation.LocalizationNetwork.conv.5.weight",module.Transformation.LocalizationNetwork.conv.5.bias",module.Transformation.LocalizationNetwork.conv.5.running_mean",module.Transformation.LocalizationNetwork.conv.5.running_var",module.Transformation.LocalizationNetwork.conv.5.num_batches_tracked",module.Transformation.LocalizationNetwork.conv.8.weight",module.Transformation.LocalizationNetwork.conv.9.weight",module.Transformation.LocalizationNetwork.conv.9.bias",module.Transformation.LocalizationNetwork.conv.9.running_mean",module.Transformation.LocalizationNetwork.conv.9.running_var",module.Transformation.LocalizationNetwork.conv.9.num_batches_tracked",module.Transformation.LocalizationNetwork.conv.12.weight",module.Transformation.LocalizationNetwork.conv.13.weight",module.Transformation.LocalizationNetwork.conv.13.bias",module.Transformation.LocalizationNetwork.conv.13.running_mean",module.Transformation.LocalizationNetwork.conv.13.running_var",module.Transformation.LocalizationNetwork.conv.13.num_batches_tracked",module.Transformation.LocalizationNetwork.localization_fc1.0.weight",module.Transformation.LocalizationNetwork.localization_fc1.0.bias",module.Transformation.LocalizationNetwork.localization_fc2.weight",module.Transformation.LocalizationNetwork.localization_fc2.bias",module.Transformation.GridGenerator.inv_delta_C",module.Transformation.GridGenerator.P_hat",module.FeatureExtraction.ConvNet.conv0_1.weight",module.FeatureExtraction.ConvNet.bn0_1.weight",module.FeatureExtraction.ConvNet.bn0_1.bias",module.FeatureExtraction.ConvNet.bn0_1.running_mean",module.FeatureExtraction.ConvNet.bn0_1.running_var",module.FeatureExtraction.ConvNet.bn0_1.num_batches_tracked",module.FeatureExtraction.ConvNet.conv0_2.weight",module.FeatureExtraction.ConvNet.bn0_2.weight",module.FeatureExtraction.ConvNet.bn0_2.bias",module.FeatureExtraction.ConvNet.bn0_2.running_mean",module.FeatureExtraction.ConvNet.bn0_2.running_var",module.FeatureExtraction.ConvNet.bn0_2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer1.0.conv1.weight",module.FeatureExtraction.ConvNet.layer1.0.bn1.weight",module.FeatureExtraction.ConvNet.layer1.0.bn1.bias",module.FeatureExtraction.ConvNet.layer1.0.bn1.running_mean",module.FeatureExtraction.ConvNet.layer1.0.bn1.running_var",module.FeatureExtraction.ConvNet.layer1.0.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer1.0.conv2.weight",module.FeatureExtraction.ConvNet.layer1.0.bn2.weight",module.FeatureExtraction.ConvNet.layer1.0.bn2.bias",module.FeatureExtraction.ConvNet.layer1.0.bn2.running_mean",module.FeatureExtraction.ConvNet.layer1.0.bn2.running_var",module.FeatureExtraction.ConvNet.layer1.0.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer1.0.downsample.0.weight",module.FeatureExtraction.ConvNet.layer1.0.downsample.1.weight",module.FeatureExtraction.ConvNet.layer1.0.downsample.1.bias",module.FeatureExtraction.ConvNet.layer1.0.downsample.1.running_mean",module.FeatureExtraction.ConvNet.layer1.0.downsample.1.running_var",module.FeatureExtraction.ConvNet.layer1.0.downsample.1.num_batches_tracked",module.FeatureExtraction.ConvNet.conv1.weight",module.FeatureExtraction.ConvNet.bn1.weight",module.FeatureExtraction.ConvNet.bn1.bias",module.FeatureExtraction.ConvNet.bn1.running_mean",module.FeatureExtraction.ConvNet.bn1.running_var",module.FeatureExtraction.ConvNet.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer2.0.conv1.weight",module.FeatureExtraction.ConvNet.layer2.0.bn1.weight",module.FeatureExtraction.ConvNet.layer2.0.bn1.bias",module.FeatureExtraction.ConvNet.layer2.0.bn1.running_mean",module.FeatureExtraction.ConvNet.layer2.0.bn1.running_var",module.FeatureExtraction.ConvNet.layer2.0.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer2.0.conv2.weight",module.FeatureExtraction.ConvNet.layer2.0.bn2.weight",module.FeatureExtraction.ConvNet.layer2.0.bn2.bias",module.FeatureExtraction.ConvNet.layer2.0.bn2.running_mean",module.FeatureExtraction.ConvNet.layer2.0.bn2.running_var",module.FeatureExtraction.ConvNet.layer2.0.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer2.0.downsample.0.weight",module.FeatureExtraction.ConvNet.layer2.0.downsample.1.weight",module.FeatureExtraction.ConvNet.layer2.0.downsample.1.bias",module.FeatureExtraction.ConvNet.layer2.0.downsample.1.running_mean",module.FeatureExtraction.ConvNet.layer2.0.downsample.1.running_var",module.FeatureExtraction.ConvNet.layer2.0.downsample.1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer2.1.conv1.weight",module.FeatureExtraction.ConvNet.layer2.1.bn1.weight",module.FeatureExtraction.ConvNet.layer2.1.bn1.bias",module.FeatureExtraction.ConvNet.layer2.1.bn1.running_mean",module.FeatureExtraction.ConvNet.layer2.1.bn1.running_var",module.FeatureExtraction.ConvNet.layer2.1.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer2.1.conv2.weight",module.FeatureExtraction.ConvNet.layer2.1.bn2.weight",module.FeatureExtraction.ConvNet.layer2.1.bn2.bias",module.FeatureExtraction.ConvNet.layer2.1.bn2.running_mean",module.FeatureExtraction.ConvNet.layer2.1.bn2.running_var",module.FeatureExtraction.ConvNet.layer2.1.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.conv2.weight",module.FeatureExtraction.ConvNet.bn2.weight",module.FeatureExtraction.ConvNet.bn2.bias",module.FeatureExtraction.ConvNet.bn2.running_mean",module.FeatureExtraction.ConvNet.bn2.running_var",module.FeatureExtraction.ConvNet.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.0.conv1.weight",module.FeatureExtraction.ConvNet.layer3.0.bn1.weight",module.FeatureExtraction.ConvNet.layer3.0.bn1.bias",module.FeatureExtraction.ConvNet.layer3.0.bn1.running_mean",module.FeatureExtraction.ConvNet.layer3.0.bn1.running_var",module.FeatureExtraction.ConvNet.layer3.0.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.0.conv2.weight",module.FeatureExtraction.ConvNet.layer3.0.bn2.weight",module.FeatureExtraction.ConvNet.layer3.0.bn2.bias",module.FeatureExtraction.ConvNet.layer3.0.bn2.running_mean",module.FeatureExtraction.ConvNet.layer3.0.bn2.running_var",module.FeatureExtraction.ConvNet.layer3.0.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.0.downsample.0.weight",module.FeatureExtraction.ConvNet.layer3.0.downsample.1.weight",module.FeatureExtraction.ConvNet.layer3.0.downsample.1.bias",module.FeatureExtraction.ConvNet.layer3.0.downsample.1.running_mean",module.FeatureExtraction.ConvNet.layer3.0.downsample.1.running_var",module.FeatureExtraction.ConvNet.layer3.0.downsample.1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.1.conv1.weight",module.FeatureExtraction.ConvNet.layer3.1.bn1.weight",module.FeatureExtraction.ConvNet.layer3.1.bn1.bias",module.FeatureExtraction.ConvNet.layer3.1.bn1.running_mean",module.FeatureExtraction.ConvNet.layer3.1.bn1.running_var",module.FeatureExtraction.ConvNet.layer3.1.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.1.conv2.weight",module.FeatureExtraction.ConvNet.layer3.1.bn2.weight",module.FeatureExtraction.ConvNet.layer3.1.bn2.bias",module.FeatureExtraction.ConvNet.layer3.1.bn2.running_mean",module.FeatureExtraction.ConvNet.layer3.1.bn2.running_var",module.FeatureExtraction.ConvNet.layer3.1.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.2.conv1.weight",module.FeatureExtraction.ConvNet.layer3.2.bn1.weight",module.FeatureExtraction.ConvNet.layer3.2.bn1.bias",module.FeatureExtraction.ConvNet.layer3.2.bn1.running_mean",module.FeatureExtraction.ConvNet.layer3.2.bn1.running_var",module.FeatureExtraction.ConvNet.layer3.2.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.2.conv2.weight",module.FeatureExtraction.ConvNet.layer3.2.bn2.weight",module.FeatureExtraction.ConvNet.layer3.2.bn2.bias",module.FeatureExtraction.ConvNet.layer3.2.bn2.running_mean",module.FeatureExtraction.ConvNet.layer3.2.bn2.running_var",module.FeatureExtraction.ConvNet.layer3.2.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.3.conv1.weight",module.FeatureExtraction.ConvNet.layer3.3.bn1.weight",module.FeatureExtraction.ConvNet.layer3.3.bn1.bias",module.FeatureExtraction.ConvNet.layer3.3.bn1.running_mean",module.FeatureExtraction.ConvNet.layer3.3.bn1.running_var",module.FeatureExtraction.ConvNet.layer3.3.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.3.conv2.weight",module.FeatureExtraction.ConvNet.layer3.3.bn2.weight",module.FeatureExtraction.ConvNet.layer3.3.bn2.bias",module.FeatureExtraction.ConvNet.layer3.3.bn2.running_mean",module.FeatureExtraction.ConvNet.layer3.3.bn2.running_var",module.FeatureExtraction.ConvNet.layer3.3.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.4.conv1.weight",module.FeatureExtraction.ConvNet.layer3.4.bn1.weight",module.FeatureExtraction.ConvNet.layer3.4.bn1.bias",module.FeatureExtraction.ConvNet.layer3.4.bn1.running_mean",module.FeatureExtraction.ConvNet.layer3.4.bn1.running_var",module.FeatureExtraction.ConvNet.layer3.4.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer3.4.conv2.weight",module.FeatureExtraction.ConvNet.layer3.4.bn2.weight",module.FeatureExtraction.ConvNet.layer3.4.bn2.bias",module.FeatureExtraction.ConvNet.layer3.4.bn2.running_mean",module.FeatureExtraction.ConvNet.layer3.4.bn2.running_var",module.FeatureExtraction.ConvNet.layer3.4.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.conv3.weight",module.FeatureExtraction.ConvNet.bn3.weight",module.FeatureExtraction.ConvNet.bn3.bias",module.FeatureExtraction.ConvNet.bn3.running_mean",module.FeatureExtraction.ConvNet.bn3.running_var",module.FeatureExtraction.ConvNet.bn3.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.0.conv1.weight",module.FeatureExtraction.ConvNet.layer4.0.bn1.weight",module.FeatureExtraction.ConvNet.layer4.0.bn1.bias",module.FeatureExtraction.ConvNet.layer4.0.bn1.running_mean",module.FeatureExtraction.ConvNet.layer4.0.bn1.running_var",module.FeatureExtraction.ConvNet.layer4.0.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.0.conv2.weight",module.FeatureExtraction.ConvNet.layer4.0.bn2.weight",module.FeatureExtraction.ConvNet.layer4.0.bn2.bias",module.FeatureExtraction.ConvNet.layer4.0.bn2.running_mean",module.FeatureExtraction.ConvNet.layer4.0.bn2.running_var",module.FeatureExtraction.ConvNet.layer4.0.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.1.conv1.weight",module.FeatureExtraction.ConvNet.layer4.1.bn1.weight",module.FeatureExtraction.ConvNet.layer4.1.bn1.bias",module.FeatureExtraction.ConvNet.layer4.1.bn1.running_mean",module.FeatureExtraction.ConvNet.layer4.1.bn1.running_var",module.FeatureExtraction.ConvNet.layer4.1.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.1.conv2.weight",module.FeatureExtraction.ConvNet.layer4.1.bn2.weight",module.FeatureExtraction.ConvNet.layer4.1.bn2.bias",module.FeatureExtraction.ConvNet.layer4.1.bn2.running_mean",module.FeatureExtraction.ConvNet.layer4.1.bn2.running_var",module.FeatureExtraction.ConvNet.layer4.1.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.2.conv1.weight",module.FeatureExtraction.ConvNet.layer4.2.bn1.weight",module.FeatureExtraction.ConvNet.layer4.2.bn1.bias",module.FeatureExtraction.ConvNet.layer4.2.bn1.running_mean",module.FeatureExtraction.ConvNet.layer4.2.bn1.running_var",module.FeatureExtraction.ConvNet.layer4.2.bn1.num_batches_tracked",module.FeatureExtraction.ConvNet.layer4.2.conv2.weight",module.FeatureExtraction.ConvNet.layer4.2.bn2.weight",module.FeatureExtraction.ConvNet.layer4.2.bn2.bias",module.FeatureExtraction.ConvNet.layer4.2.bn2.running_mean",module.FeatureExtraction.ConvNet.layer4.2.bn2.running_var",module.FeatureExtraction.ConvNet.layer4.2.bn2.num_batches_tracked",module.FeatureExtraction.ConvNet.conv4_1.weight",module.FeatureExtraction.ConvNet.bn4_1.weight",module.FeatureExtraction.ConvNet.bn4_1.bias",module.FeatureExtraction.ConvNet.bn4_1.running_mean",module.FeatureExtraction.ConvNet.bn4_1.running_var",module.FeatureExtraction.ConvNet.bn4_1.num_batches_tracked",module.FeatureExtraction.ConvNet.conv4_2.weight",module.FeatureExtraction.ConvNet.bn4_2.weight",module.FeatureExtraction.ConvNet.bn4_2.bias",module.FeatureExtraction.ConvNet.bn4_2.running_mean",module.FeatureExtraction.ConvNet.bn4_2.running_var",module.FeatureExtraction.ConvNet.bn4_2.num_batches_tracked",module.SequenceModeling.0.rnn.weight_ih_l0",module.SequenceModeling.0.rnn.weight_hh_l0",module.SequenceModeling.0.rnn.bias_ih_l0",module.SequenceModeling.0.rnn.bias_hh_l0",module.SequenceModeling.0.rnn.weight_ih_l0_reverse",module.SequenceModeling.0.rnn.weight_hh_l0_reverse",module.SequenceModeling.0.rnn.bias_ih_l0_reverse",module.SequenceModeling.0.rnn.bias_hh_l0_reverse",module.SequenceModeling.0.linear.weight",module.SequenceModeling.0.linear.bias",module.SequenceModeling.1.rnn.weight_ih_l0",module.SequenceModeling.1.rnn.weight_hh_l0",module.SequenceModeling.1.rnn.bias_ih_l0",module.SequenceModeling.1.rnn.bias_hh_l0",module.SequenceModeling.1.rnn.weight_ih_l0_reverse",module.SequenceModeling.1.rnn.weight_hh_l0_reverse",module.SequenceModeling.1.rnn.bias_ih_l0_reverse",module.SequenceModeling.1.rnn.bias_hh_l0_reverse",module.SequenceModeling.1.linear.weight",module.SequenceModeling.1.linear.bias",module.Prediction.attention_cell.i2h.weight",module.Prediction.attention_cell.h2h.weight",module.Prediction.attention_cell.h2h.bias",module.Prediction.attention_cell.score.weight",module.Prediction.attention_cell.rnn.weight_ih",module.Prediction.attention_cell.rnn.weight_hh",module.Prediction.attention_cell.rnn.bias_ih",module.Prediction.attention_cell.rnn.bias_hh",module.Prediction.generator.weight",module.Prediction.generator.bias".

Here is the catch

Independently Both Module A and Module B runs without a problem in the same Py3 env
I used Python shell and imported Module B after loading Module A and it loads just fine.

Any clue on whats going on?

For reference

Module B (code where it fails)

    def pre_load_model(self):

            print("preloading the model with opts "+str(self.opts))

            self.trmodel = Model(self.opts)

            self.trmodel = torch.nn.DataParallel(self.trmodel)
            if torch.cuda.is_available():
                self.trmodel = self.trmodel.cuda()
                self.device = torch.device('cuda:0')
            else:
                self.device = torch.device('cpu')

            # load model
            print('loading pretrained model from %s' % self.opts.saved_model)
            if torch.cuda.is_available():
                self.trmodel.load_state_dict(torch.load(self.opts.saved_model))
            else:
                self.trmodel.load_state_dict(torch.load(self.opts.saved_model, map_location='cpu'))

Output: I think you have the Python Global Interpreter Lock or GIL problem.


Instruction: Getting the accuracy from classification_report back into a list

Input:
I am using Sklean's classification_report to summarize my train and test epochs.

sklearn.metrics.classification_report

I'm getting kind of this back for each epoch:

>>> from sklearn.metrics import classification_report
>>> y_true
>>> y_pred
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
              precision    recall  f1-score   support

     class 0       0.50      1.00      0.67         1
     class 1       0.00      0.00      0.00         1
     class 2       1.00      0.67      0.80         3

    accuracy                           0.60         5
   macro avg       0.50      0.56      0.49         5
weighted avg       0.70      0.60      0.61         5

(e.g. from sklearn script)

Now I am searching for a way, to get those accuracy for each epoch in a list to calculate the mean and std of all accuracy.

This question seems to be pretty trivial but as you can see from my questions before I am pretty new to Python/Machine Learning.

Thanks for your help

Leo

Output:
Lets have a look at the documentation which contains information about the input parameter output_dict :

  output_dict : bool (default = False) If True, return output as dict

If you call classification_report(y_true, y_pred, target_names=target_names, output_dict=True) you can get the dictionary. And then you are one stackoverflow question away from your solution.


Instruction: Can you pass a variable to a data member in Python as done in Pytorch?

Input:
Here is the sample code from the nn.Module of pytorch documentation:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
       x = F.relu(self.conv1(x))
       return F.relu(self.conv2(x))

Here, we are passing x to self.conv1() in the forward function.
However, self.conv1 is a variable as can been from this line self.conv1 = nn.Conv2d(1, 20, 5) in the __init__ function.

How is this possible?

Output: Forward passes of anything in pytorch, layer or network, is done in functional way var(x). This is made in python with overwriting __call__() built-in. Try it yourself. Make a class, overwrite __call__() and use it like function.


Instruction: Initializing convolutional layers in CNN

Input:
Is there a function to initialize weights of a convolutional layer to focus more on information closer to the center of input images?

All my input images are centered, so pixels further away from the center of an image matter less than pixels closer to the center.

Output:
Please see the GIFs here for a demonstration of convolutions:

https://github.com/vdumoulin/conv_arithmetic#convolution-animations

As you can see, convolutions operate the same regardless of the position in the image, so weight initialization cannot change the focus of the image.

It is also not advisable to rush into thinking about what the net will and won't need to learn your task. There are sometimes surprising amounts of signal outside what you as a human might focus on. I would suggest training the net and seeing how it performs, and then (as others have suggested) thinking about cropping.


Instruction: How to multiply a dense matrix by a sparse matrix element-wise in pytorch

Input: I can use torch.sparse.mm() or torch.spmm() to do multiplication between sparse matrix and dense matrix directly, but which function should I choose to do element-wise multiplication?

Output:
You can implement this multiplication yourself

def sparse_dense_mul(s, d):
  i = s._indices()
  v = s._values()
  dv = d[i[0,:], i[1,:]]  # get values from relevant entries of dense matrix
  return torch.sparse.FloatTensor(i, v * dv, s.size())

Note that thanks to the linearity of the multiplication operation you do not need to worry if s is coalesced or not.


Instruction: Getting error while installing Flair (NLP Library)

Input:
I am trying to install NLP library "Flair" using pip and getting the error message:

ERROR: Could not find a version that satisfies the requirement torch>=1.0.0 (from flair) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=1.0.0 (from flair)"

What should I do

I am using python3.6 and tried to install flair using cmd and pip in virtualenv. But there is still error messages.

(env) C:\Users\HP>pip install flair

I want the installation without any issue.enter image description here

Output:
Pytorch or torch is currently cannot be installed using pypi in windows.
You will need conda to accomplish that:

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

OR

for older torch version you can use whl file, Cuda version can be 8.0,9.0,10.0

pip install torch==1.0.1 -f https://download.pytorch.org/whl/cu90/stable # CUDA 9.0 build


Instruction: Custom Loss Function becomes zero when backpropagated

Input:
I am trying to write my own custom loss function that is based on the false positive and negative rates. I made a dummy code so you can check the first 2 defenitions as well. I added the rest, so you can see how it is implemented. However, still somewhere the gradient turns out to be zero. What is now the step where the gradient turns zero, or how can I check this? Please I would like to know how I can fix this :).
I tried providing you with more information so you can play around as well, but if you miss anything please do let me know!

The gradient stays True during every step. However, still during the training of the model the loss is not updated, therefore the NN does not train.

y = Variable(torch.tensor((0, 0, 0, 1, 1,1), dtype=torch.float), requires_grad = True)
y_pred = Variable(torch.tensor((0.333, 0.2, 0.01, 0.99, 0.49, 0.51), dtype=torch.float), requires_grad = True)
x = Variable(torch.tensor((0, 0, 0, 1, 1,1), dtype=torch.float), requires_grad = True)
x_pred = Variable(torch.tensor((0.55, 0.25, 0.01, 0.99, 0.65, 0.51), dtype=torch.float), requires_grad = True)

def binary_y_pred(y_pred):
    y_pred.register_hook(lambda grad: print(grad))
    y_pred = y_pred+torch.tensor(0.5, requires_grad=True, dtype=torch.float)
    y_pred = y_pred.pow(5)  # this is my way working around using torch.where()
    y_pred = y_pred.pow(10)
    y_pred = y_pred.pow(15)
    m = nn.Sigmoid()
    y_pred = m(y_pred)
    y_pred = y_pred-torch.tensor(0.5, requires_grad=True, dtype=torch.float)
    y_pred = y_pred*2
    y_pred.register_hook(lambda grad: print(grad))
    return y_pred

def confusion_matrix(y_pred, y):
    TP = torch.sum(y*y_pred)
    TN = torch.sum((1-y)*(1-y_pred))
    FP = torch.sum((1-y)*y_pred)
    FN = torch.sum(y*(1-y_pred))

    k_eps = torch.tensor(1e-12, requires_grad=True, dtype=torch.float)
    FN_rate = FN/(TP + FN + k_eps)
    FP_rate = FP/(TN + FP + k_eps)

    return FN_rate, FP_rate

def dif_rate(FN_rate_y, FN_rate_x):
    dif = (FN_rate_y - FN_rate_x).pow(2)
    return dif

def custom_loss_function(y_pred, y, x_pred, x):
    y_pred = binary_y_pred(y_pred)
    FN_rate_y, FP_rate_y = confusion_matrix(y_pred, y)

    x_pred= binary_y_pred(x_pred)
    FN_rate_x, FP_rate_x = confusion_matrix(x_pred, x)

    FN_dif = dif_rate(FN_rate_y, FN_rate_x)
    FP_dif = dif_rate(FP_rate_y, FP_rate_x)

    cost = FN_dif+FP_dif
    return cost

# I added the rest so you can see how it is implemented, but this peace does not fully run well! If you want this part to run as well, I can add more code.
class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(FeedforwardNeuralNetModel, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)
        self.sigmoid = nn.Sigmoid()

     def forward(self, x):
        out = self.fc1(x)
        out = self.relu1(out)
        out = self.fc2(out)
        out = self.sigmoid(out)
        return out

model = FeedforwardNeuralNetModel(input_dim, hidden_dim, output_dim)

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001, betas=[0.9, 0.99], amsgrad=True)
criterion = torch.nn.BCELoss(weight=None, size_average=None, reduce=None, reduction='mean')
for epoch in range(num_epochs):

train_err = 0
for i, (samples, truths) in enumerate(train_loader):
    samples = Variable(samples)
    truths = Variable(truths)
    optimizer.zero_grad()   # Reset gradients
    outputs = model(samples)  # Do the forward pass
    loss2 = criterion(outputs, truths) # Calculate loss

    samples_y = Variable(samples_y)
    samples_x = Variable(samples_x)

    y_pred = model(samples_y)
    y = Variable(y, requires_grad=True)

    x_pred = model(samples_x)
    x= Variable(x, requires_grad=True)

    cost = custom_loss_function(y_pred, y, x_pred, x)
    loss = loss2*0+cost #checking only if cost works.
    loss.backward()
    optimizer.step()
    train_err += loss.item()
train_loss.append(train_err)

I expect the model to update during training. There is no error message.

Output:
With your definitions:TP+FN=y and TN+FP=1-y. Then you'll get FN_rate=1-y_pred and FP_rate=y_pred. Your cost is then FN_rate+FP_rate=1, the gradient of which is 0.

You can check this by hand or using a library for symbolic mathematics (e.g., SymPy):

from sympy import symbols

y, y_pred = symbols("y y_pred")

TP = y * y_pred
TN = (1-y)*(1-y_pred)
FP = (1-y)*y_pred
FN = y*(1-y_pred)

# let's ignore the eps for now
FN_rate = FN/(TP + FN)
FP_rate = FP/(TN + FP)
cost = FN_rate + FP_rate

from sympy import simplify
print(simplify(cost))
# output: 1


Instruction: How can I build a good approximation of an unknown distribution when only having samples from it in order to draw from it in torch?

Input:
Say I just have random samples from the Distribution and no other data - e.g. a list of numbers - [1,15,30,4,etc.]. What's the best way to estimate the distribution to draw more samples from it in pytorch?

I am currently assuming that all samples come from a Normal distribution and just using the mean and std of the samples to build it and draw from it. The function, however, can be of any distribution.

samples = torch.Tensor([1,2,3,4,3,2,2,1])
Normal(samples.mean(), samples.std()).sample()

Output:
If you  have enough samples (and preferably sample dimension is higher than 1), you could model the distribution using Variational Autoencoder or Generative Adversarial Networks (though I would stick with the first approach as it's simpler).

Basically, after correct implementation and training you would get deterministic decoder able to decode hidden code you would pass it (say vector of size 10 taken from normal distribution) into a value from your target distribution.

Note it might not be reliable at all though, it would be even harder if your samples are 1D only.


Instruction: Will installation from source in different conda env lead to conflict?

Input:
I want to install a nightly-version(v1.2) of pytorch from source in a new conda env, and I got v1.1 outside conda env installed from pip.

Will that lead to conflict?

Output:
If you didn't change anything else about your environment, then not. Conda shields its environment from the host environment in the sense that it comes with its own python distribution, libraries etc., hence there should be no conflicts.

Note, however, that conda also comes with pip, so you need to make sure that the pip you used to install v1.1 is indeed your host's pip and not conda's own pip.


Instruction: PyTroch, Gradient calculations

Input:
https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb

Hi I am trying to understand the NN with pytorch.
I have doubts in gradient calculations..

import torch.optim as optim

create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)

```
# in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update
```

From the about code, I understood loss.backward() calculates the gradients.
I am not sure, how these info shared with optimizer to update the gradient.

Can anyone explain this..

Thanks in advance !

Output:
When you created the optimizer in this line

optimizer = optim.SGD(net.parameters(), lr=0.01)

You provided net.parameters() with all learnable parameters that will be updated, based on gradients.

The model and the optimizer are connected only because they share the same parameters.

PyTorch parameters are tensors. They are not called variables anymore.


Instruction: LibTorch, convert deeplabv3_resnet101 to c++

Input:
I am trying to use this example code from the PyTorch website to convert a python model for use in the PyTorch c++ api (LibTorch).

Converting to Torch Script via Tracing
To convert a PyTorch model to Torch Script via tracing, you must pass an instance of your model along with an example input to the torch.jit.trace function. This will produce a torch.jit.ScriptModule object with the trace of your model evaluation embedded in the module’s forward method:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

This example works fine, and saves out the file as expected.
When i switch to this model:

model = models.segmentation.deeplabv3_resnet101(pretrained=True)

It gives me the following error:

File "convert.py", line 14, in <module>
    traced_script_module = torch.jit.trace(model, example)
  File "C:\Python37\lib\site-packages\torch\jit\__init__.py", line 636, in trace
          raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])

I assume this is because the example format is wrong, but how can I get the correct one?

Based on the comments below, my new code is:

import torch
import torchvision
from torchvision import models

model = models.segmentation.deeplabv3_resnet101(pretrained=True)
model.eval()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("model.pt")

And i now get the error:

File "convert.py", line 15, in <module>
    traced_script_module = torch.jit.trace(model, example)
  File "C:\Python37\lib\site-packages\torch\jit\__init__.py", line 636, in trace
    var_lookup_fn, _force_outplace)
RuntimeError: Only tensors and (possibly nested) tuples of tensors are supported as inputs or outputs of traced functions (toIValue at C:\a\w\1\s\windows\pytorch\torch/csrc/jit/pybind_utils.h:91)
(no backtrace available)

Output:
(from pytorch forums)

trace only supports modules that have tensor or tuple of tensor as output.
According to deeplabv3 implementation, its output is OrderedDict. That is a problem.
To solve this, make a wrapper module

class wrapper(torch.nn.Module):
    def __init__(self, model):
        super(wrapper, self).__init__()
        self.model = model

    def forward(self, input):
        results = []
        output = self.model(input)
        for k, v in output.items():
            results.append(v)
        return tuple(results)

model = wrapper(deeplap_model)
#trace...

Has my model saving out.


Instruction: Keras vs PyTorch LSTM different results

Input:
Trying to get similar results on same dataset with Keras and PyTorch.
Data
from numpy import array
from numpy import hstack

from sklearn.model_selection import train_test_split

# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
    X, y = list(), list()
    for i in range(len(sequences)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(sequences):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

def get_data():
    # define input sequence
    in_seq1 = array([x for x in range(0,500,10)])/1
    in_seq2 = array([x for x in range(5,505,10)])/1
    out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
    # convert to [rows, columns] structure
    in_seq1 = in_seq1.reshape((len(in_seq1), 1))
    in_seq2 = in_seq2.reshape((len(in_seq2), 1))
    out_seq = out_seq.reshape((len(out_seq), 1))
    # horizontally stack columns
    dataset = hstack((in_seq1, in_seq2, out_seq))

    n_features = 2 # this is number of parallel inputs
    n_timesteps = 3 # this is number of timesteps

    # convert into input/output
    X, y = split_sequences(dataset, n_timesteps)
    print(X.shape, y.shape)
    X_train,x_test,Y_train, y_test = train_test_split(X,y,test_size = 0.2,shuffle=False)

    return X_train,x_test,Y_train, y_test

Keras
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

from sklearn.metrics import mean_squared_error

import testing.TimeSeries.datacreator as dc # !!!!change this!!!!
X_train,x_test,Y_train, y_test = dc.get_data()

n_features = 2 # this is number of parallel inputs
n_timesteps = 3 # this is number of timesteps

# define model
model = Sequential()
model.add(LSTM(1024, activation='relu',
               input_shape=(n_timesteps, n_features),
               kernel_initializer='uniform',
               recurrent_initializer='uniform'))
model.add(Dense(512, activation='relu'))
model.add(Dense(1))
opt = keras.optimizers.Adam(lr=0.001,
                      beta_1=0.9,
                      beta_2=0.999,
                      epsilon=keras.optimizers.K.epsilon(),
                      decay=0.0,
                      amsgrad=False)
model.compile(optimizer=opt, loss='mse')
# fit model
model.fit(X_train, Y_train, epochs=200, verbose=1,validation_data=(x_test,y_test))

yhat = model.predict(x_test, verbose=0)

mean_squared_error(y_test, yhat)

PyTorch - module class
import numpy as np
import torch
import torch.nn.functional as F

from sklearn.metrics import mean_squared_error

import testing.TimeSeries.datacreator as dc # !!!! change this !!!!
X_train,x_test,Y_train, y_test =   dc.get_data()
n_features = 2 # this is number of parallel inputs
n_timesteps = 3 # this is number of timesteps

class MV_LSTM(torch.nn.Module):
    def __init__(self,n_features,seq_length):
        super(MV_LSTM, self).__init__()
        self.n_features = n_features # number of parallel inputs
        self.seq_len = seq_length # number of timesteps
        self.n_hidden = 1024 # number of hidden states
        self.n_layers = 1 # number of LSTM layers (stacked)

        self.l_lstm = torch.nn.LSTM(input_size = n_features,
                                 hidden_size = self.n_hidden,
                                 num_layers = self.n_layers,
                                 batch_first = True)
        # according to pytorch docs LSTM output is
        # (batch_size,seq_len, num_directions * hidden_size)
        # when considering batch_first = True
        self.l_linear = torch.nn.Linear(self.n_hidden*self.seq_len, 512)
#        self.l_linear1 = torch.nn.Linear(512, 512)
        self.l_linear2 = torch.nn.Linear(512, 1)

    def init_hidden(self, batch_size):
        # even with batch_first = True this remains same as docs
        hidden_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(next(self.parameters()).device)
        cell_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(next(self.parameters()).device)
        self.hidden = (hidden_state, cell_state)

    def forward(self, x):
        batch_size, seq_len, _ = x.size()

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)
        # lstm_out(with batch_first = True) is
        # (batch_size,seq_len,num_directions * hidden_size)
        # for following linear layer we want to keep batch_size dimension and merge rest
        # .contiguous() -> solves tensor compatibility error
        x = lstm_out.contiguous().view(batch_size,-1)
        x = F.relu(x)
        x = F.relu(self.l_linear(x))
#        x = F.relu(self.l_linear1(x))
        x = self.l_linear2(x)
        return x

PyTorch - init and train
# create NN
mv_net = MV_LSTM(n_features,n_timesteps)
criterion = torch.nn.MSELoss()
import keras # for epsilon constant
optimizer = torch.optim.Adam(mv_net.parameters(),
                             lr=1e-3,
                             betas=[0.9,0.999],
                             eps=keras.optimizers.K.epsilon(),
                             weight_decay=0,
                             amsgrad=False)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
mv_net.to(device)

train_episodes = 200
batch_size = 32
eval_batch_size = 32

for t in range(train_episodes):
    # TRAIN
    mv_net.train()
    for b in range(0,len(X_train),batch_size):
        inpt = X_train[b:b+batch_size,:,:]
        target = Y_train[b:b+batch_size]

        x_batch = torch.tensor(inpt,dtype=torch.float32).to(device)
        y_batch = torch.tensor(target,dtype=torch.float32).to(device)

        mv_net.init_hidden(x_batch.size(0))

        output = mv_net(x_batch)
        loss = criterion(output.view(-1), y_batch)

        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    # EVAL
    mv_net.eval()
    mv_net.init_hidden(eval_batch_size)
    acc = 0
    for b in range(0,len(x_test),eval_batch_size):
        inpt = x_test[b:b+eval_batch_size,:,:]
        target = y_test[b:b+eval_batch_size]

        x_batch = torch.tensor(inpt,dtype=torch.float32).to(device)
        y_batch = torch.tensor(target,dtype=torch.float32).to(device)
        mv_net.init_hidden(x_batch.size(0))

        output = mv_net(x_batch)
        acc += mean_squared_error(y_batch.cpu().detach().numpy(), output.view(-1).cpu().detach().numpy())
    print('step:' , t , 'train loss:' , round(loss.item(),3),'eval acc:',round(acc/len(x_test),3))

mv_net.init_hidden(len(x_test))
val = torch.tensor(x_test,dtype=torch.float32).to(device)
otp = mv_net(val)
print(mean_squared_error(y_test, otp.view(-1).cpu().detach().numpy()))

Results
Keras produces test MSE almost 0, but PyTorch about 6000, which is way too different
I have tried couple tweaks in PyTorch code, but none got me anywhere close to similar keras, even with identical optim params
I cant see what is wrong with (kinda tutorialic) PyTorch code

Output:
I know it is almost one year too late. But I came across the same problem and I think the problem is the following. From the keras documentation it says:

  return_sequences: Boolean. Whether to return the last output in the
  output sequence, or the full sequence.

this basically means that the input shape of your self.l_linear needs to be torch.nn.Linear(1024, 512) instead of self.n_hidden*self.seq_len, 512.

Now you also need to do the same as keras does and only use the last output in your forward pass:

    def forward(self, x):
        batch_size, seq_len, _ = x.size()

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)

        x = lstm_out[:,-1]
        x = torch.nn.functional.relu(x)
        x = torch.nn.functional.relu(self.l_linear(x))
        x = self.l_linear2(x)
        return x

when I run your example (which I needed to tweak a bit to get it run) I get very similar training losses.

Keras:

  38/38 [==============================] - 0s 6ms/step - loss: 67.6081 - val_loss: 325.9259

PyTorch:

  step: 199 train loss: 41.043 eval acc: 1142.688

I hope this helps others having a similar problem.

PS also note that keras is resetting the hidden state (stateful=False) by default.


Instruction: Use Tensorflow/PyTorch to speed up minimisation of a custom function

Input:
I do a lot of simulations, for which I often need to minimise complicated user-defined functions, for which I generally use numpy and scipy.optimize.minimize(). However, the problem with this is that I need to explicitly write down a gradient function, which can sometimes be very difficult/impossible, to find. And for large dimensional vectors, the numerical derivatives calculated by scipy are prohibitively expensive.

So, I am trying to switch to Tensorflow or PyTorch to take advantage both of their automatic differentiation capabilities, and to be able to exploit GPUs freely. Let me give an explicit example of a function who derivative is somewhat complicated to write down (would required a lot of chain rule), and thus seems ripe for Tensorflow or PyTorch -- calculating the dihedral angle between the two triangles formed by four points in 3d space:

def dihedralAngle(xyz):
## calculate dihedral angle between 4 nodes

    p1, p2, p3, p4 = 0, 1, 2, 3

    ## get unit normal vectors
    N1 = np.cross(xyz[p1]-xyz[p3] , xyz[p2]-xyz[p3])
    N2 = - np.cross(xyz[p1]-xyz[p4] , xyz[p2]-xyz[p4])
    n1, n2 = N1 / np.linalg.norm(N1), N2 / np.linalg.norm(N2)

    angle = np.arccos(np.dot(n1, n2))

    return angle

xyz1 = np.array([[0.2       , 0.        , 0.        ],
       [0.198358  , 0.02557543, 0.        ],
       [0.19345897, 0.05073092, 0.        ],
       [0.18538335, 0.0750534 , 0.        ]]) # or any (4,3) ndarray

print(dihedralAngle(xyz1)) >> 3.141

I could easily minimise this using scipy.optimize.minimize(), and I should get 0. For such a small function, I don't really the need a gradient (explicit or numerical). However, if I wish to iterate over many, many nodes, and minimise some function that depends on all the dihedral angles, then the overhead is much higher?

My questions then --

How would I implement this minimisation problem using TensorFlow or PyTorch? Both for a single dihedral angle, and for a list of such angles (i.e. we need to account for looping over lists).
Also, could I just get the gradient using automatic differentiation, to plug back into scipy.optimize.minimize() if desired? For example, scipy.optimize.minimize() allows easily for bounds and constraints, something that I haven't noticed in the Tensorflow or PyToch optimisation modules.

Output:
Here is a solution using automatic computation of the gradient by torch and then the minimizer of scipy with a lib I wrote autograd-minimize. The advantage over SGD is a better precision of the estimation (using second order methods). It is probably equivalent to using LBFGS from torch:
import numpy as np
import torch
from autograd_minimize import minimize

def dihedralAngle(xyz):
## calculate dihedral angle between 4 nodes

    p1, p2, p3, p4 = 0, 1, 2, 3

    ## get unit normal vectors
    N1 = np.cross(xyz[p1]-xyz[p3] , xyz[p2]-xyz[p3])
    N2 = - np.cross(xyz[p1]-xyz[p4] , xyz[p2]-xyz[p4])
    n1, n2 = N1 / np.linalg.norm(N1), N2 / np.linalg.norm(N2)

    angle = np.arccos(np.dot(n1, n2))

    return angle
def compute_angle(p1, p2):
    # inner_product = torch.dot(p1, p2)
    inner_product = (p1*p2).sum(-1)
    p1_norm = torch.linalg.norm(p1, axis=-1)
    p2_norm = torch.linalg.norm(p2, axis=-1)
    cos = inner_product / (p1_norm * p2_norm)
    cos = torch.clamp(cos, -0.99999, 0.99999)
    angle = torch.acos(cos)
    return angle

def compute_dihedral(v1,v2,v3,v4):
    ab = v1 - v2
    cb = v3 - v2
    db = v4 - v3
    u = torch.cross(ab, cb)
    v = torch.cross(db, cb)
    w = torch.cross(u, v)
    angle = compute_angle(u, v)
    angle = torch.where(compute_angle(cb, w) > 1, -angle, angle)

    return angle

def loss_func(v1,v2,v3,v4):
    return ((compute_dihedral(v1,v2,v3,v4)+2)**2).mean()

x0=[np.array([-17.0490,   5.9270,  21.5340]),
    np.array([-0.1608,  0.0600, -0.0371]),
    np.array([-0.2000,  0.0007, -0.0927]),
    np.array([-0.1423,  0.0197, -0.0727])]

res = minimize(loss_func, x0, backend='torch')

print(compute_dihedral(*[torch.tensor(v) for v in res.x]))


Instruction: How to convert Pytorch model parameters to long datatype?

Input:
I was able to convert pytorch model parameters to float or double but not to long.

model = model.long()

gives error, while

model = model.float()

runs.

The error I get is:

'Net' object has no attribute 'long'

Output: Most nn modules do not support long (integer) operations, e.g., convolutions, linear layer etc. Therefore, you cannot "cast" a model to torch.long.


Instruction: Is there function in pytorch similar to tf.contrib.distributions.percentile of tensorflow?

Input: Is there a function in PyTorch that does the same as tf.contrib.distributions.percentile of Tensorflow?

Output:
Interestingly, it seems PyTorch is not providing any operator on its own for this, at least not according to its search function.
Fortunately, though, PyTorch Tensors can easily be used with NumPy functions, so that you can simply call numpy.percentile, see the example below:

import torch as t
import numpy as np

x = t.Tensor([1,2,3])
print(np.percentile(x, 30)) # 30-th percentile of x
# 1.6


Instruction: Add extra dimension to an axes

Input: I have a batch of segmentation masks of shape [5,1,100,100] (batch_size x dims x ht x wd) which I have to display in tensorboardX with an RGB image batch [5,3,100,100]. I want to add two dummy dimensions to the second axes of the segmentation mask to make it [5,3,100,100] so there will not be any dimension mismatch error when I pass it to torch.utils.make_grid. I have tried unsqueeze, expand and view but I am not able to do it. Any suggestions?

Output:
You can use expand, repeat, or repeat_interleave:

import torch

x = torch.randn((5, 1, 100, 100))
x1_3channels = x.expand(-1, 3, -1, -1)
x2_3channels = x.repeat(1, 3, 1, 1)
x3_3channels = x.repeat_interleave(3, dim=1)

print(x1_3channels.shape)  # torch.Size([5, 3, 100, 100])
print(x2_3channels.shape)  # torch.Size([5, 3, 100, 100])
print(x3_3channels.shape)  # torch.Size([5, 3, 100, 100])

Note that, as stated in the docs:

torch.expand():

  Expanding a tensor does not allocate new memory, but only creates a new view on the existing tensor where a dimension of size one is expanded to a larger size by setting the stride to 0. Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory.

torch.repeat():

  Unlike expand(), this function copies the tensor’s data.


Instruction: printing image paths from the dataloader in pytorch

Input:
I am trying to learn One-shot learning with pytorch. I am experimenting with this Siamese Network in Pytorch example. Using that notebook as a guide, I simply would like to print out the image file paths for each pair of images, in addition to the dissimilarity scores.

From what I've been reading, it looks like I need to make some alterations to the dataloader in order to achieve this, as indicated here.

I do not have much experience yet in all of this. I would appreciate some guidance. I imported the altered dataloader (as in this gist) into my code.

The altered dataloader:

import torch
from torchvision import datasets

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method. this is the method that dataloader calls
    def __getitem__(self, index):
        # this is what ImageFolder normally returns
        original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
        # the image file path
        path = self.imgs[index][0]
        # make a new tuple that includes original and the path
        tuple_with_path = (original_tuple + (path,))
        return tuple_with_path

and the example usage:

data_dir = "your/data_dir/here"
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
    print(inputs, labels, paths)

my code:

from pytorch_image_folder_with_file_paths import ImageFolderWithPaths

folder_dataset_test = dset.ImageFolder(root=Config.testing_dir)
siamese_dataset = SiameseNetworkDataset(imageFolderDataset=folder_dataset_test,
                                        transform=transforms.Compose([transforms.Resize((100,100)),
                                                                      transforms.ToTensor()
                                                                      ])
                                       ,should_invert=False)

test_dataloader = DataLoader(siamese_dataset,num_workers=6,batch_size=1,shuffle=True)

dataiter = iter(test_dataloader)
x0,_,_ = next(dataiter)

for i in range(10):
    _,x1,label2 = next(dataiter)

    concatenated = torch.cat((x0,x1),0)

    output1,output2 = net(Variable(x0).cuda(),Variable(x1).cuda())
    euclidean_distance = F.pairwise_distance(output1, output2)
    imshow(torchvision.utils.make_grid(concatenated),'Dissimilarity: {:.2f}'.format(euclidean_distance.item()))

for inputs, labels, paths in test_dataloader:
    print(inputs, labels, paths)

I get the paired images with the dissimilarity score, but I don't get the paths; I get

tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]]) tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,

...etc.

Thanks

Output:
Thank you Anubhav Singh for sorting me out.

This works:

from pytorch_image_folder_with_file_paths import ImageFolderWithPaths

folder_dataset_test = ImageFolderWithPaths(root=Config.testing_dir)
siamese_dataset = SiameseNetworkDataset(imageFolderDataset=folder_dataset_test,
                                        transform=transforms.Compose([transforms.Resize((100,100)),
                                                                      transforms.ToTensor()
                                                                      ])
                                       ,should_invert=False)

test_dataloader = DataLoader(siamese_dataset,num_workers=6,batch_size=1,shuffle=True)

dataiter = iter(test_dataloader)
x0,_,_ = next(dataiter)

for i in range(10):
    _,x1,label2 = next(dataiter)

    concatenated = torch.cat((x0,x1),0)

    output1,output2 = net(Variable(x0).cuda(),Variable(x1).cuda())
    euclidean_distance = F.pairwise_distance(output1, output2)
    imshow(torchvision.utils.make_grid(concatenated),'Dissimilarity: {:.2f}'.format(euclidean_distance.item()))

for paths in folder_dataset_test:
    # use the above variables freely
    print(paths)

Incidentally, I'm working in Google Colab which doesn't allow me to edit files directly, so for the dataloader, I made a new cell and used %%writefile to get it into my notebook:

%%writefile pytorch_image_folder_with_file_paths.py

import torch
import torchvision.datasets as dset

class ImageFolderWithPaths(dset.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method. this is the method that dataloader calls
    def __getitem__(self, index):
        # this is what ImageFolder normally returns
        original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
        # the image file path
        path = self.imgs[index][0]
        # make a new tuple that includes original and the path
        tuple_with_path = (original_tuple + (path,))
        return tuple_with_path


Instruction: I am getting an error while installing PATE from SYFT

Input:
I am doing a Differential Privacy Course and have to work with syft. When running the below command, I encountered a error

I have already installed syft package required for the analysis. And I am working in Anaconda on Windows 10

from syft.frameworks.torch.differential_privacy import pate

WARNING:tf_encrypted:Falling back to insecure randomness since the required custom op could not be found for the installed version of TensorFlow (1.13.1). Fix this by compiling custom ops.

Output: It is not an error it is a warning. It comes from tf_encrypted which is used in tensorflow. The problem is that on Windows secure randomness is not supported (according to https://github.com/tf-encrypted/tf-encrypted/issues/513). And as you use PyTorch you can just ignore it.


Instruction: How to encode categorical data that have variable length so could be fetched to nn.Embedding in PyTorch

Input:
Let's say i have a data field named movie_genre for each sample movie, it is selected from the following genres:

Action
Adventure
Animation
Comedy
...

And for each movie, it might contain multiple genres:

mid    genres
1      Action | Adventure
2      Animation
3      Comedy | Adventure | Action

which means, the movie's genres is a variable list.

If i use one hot vector to encode the genre, Action can be encoded as (1, 0, 0, 0), Adventure can be encoded as(0, 1, 0, 0), and so on.

So movie with mid1 can be encoded as (1, 1, 0, 0), mid2's genre can be encoded as (0, 0, 1, 0), and so on.

However, the pytorch embedding layer nn.Embedding takes tensor containing the indices as input, but not one-hot vector. So how should i encode the data so that it can be fetched into the embedding layer?

Output:
At the moment I can think of two way to proceed :

Transform your multilabel problem to a multicategorical problem. That is to say for each combination of label create a new label (For example Action | Adventure become its own label), then embed those new labels as usual.
Embed separately each category and sums the embedding of all the category appearing in your list.

Edit : You can use pytorch nn.EmbeddingBag to perform the second operation in an efficient manner : https://pytorch.org/docs/stable/nn.html?highlight=nn%20e#torch.nn.EmbeddingBag


Instruction: How to modify Pretrained Torchvision Models in PyTorch to return two outputs for multilabel Image Classification

Input:
Input: Set of ten "Vowels", set of ten "Consonents", Image dataset where in every image both, one vowel and one consonent, are written.

Task: To identify the vowel and consonent from given image.

Approach: First apply CNN hidden layers on image, then apply two parallel fully connected/dense layers where one will classify vowel in image and other will classify consonent in image.

Problem: I am taking Pretrained Model like VGG or GoogleNet. How to modify that pretrained model to apply two parallel dense layers and return two outputs.

I have tried two different models but my query is can we modify pretrained model for this task.

Right now my model is having only one "fc" layer. I have modified number of neurons in final "fc" layer, like this

final_in_features = googlenet.fc.in_features

googlenet.fc = nn.Linear(final_in_features, 10)

But I need to add one more fc layer so that both "fc" layers are connecting with hidden layers parallel.

Right now model is returning only one output.

outputs1 = googlenet(inputs)

Task is to return two outputs from both "fc" layers, so that it should be look like this

outputs1, outputs2 = googlenet(inputs)

Output:
Here is the source for a Linear Layer in Pytorch :

class Linear(Module):
    r"""Applies a linear transformation to the incoming data: :math:`y = xA^T + b`

    Args:
        in_features: size of each input sample
        out_features: size of each output sample
        bias: If set to ``False``, the layer will not learn an additive bias.
            Default: ``True``

    Shape:
        - Input: :math:`(N, *, H_{in})` where :math:`*` means any number of
          additional dimensions and :math:`H_{in} = \text{in\_features}`
        - Output: :math:`(N, *, H_{out})` where all but the last dimension
          are the same shape as the input and :math:`H_{out} = \text{out\_features}`.

    Attributes:
        weight: the learnable weights of the module of shape
            :math:`(\text{out\_features}, \text{in\_features})`. The values are
            initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`, where
            :math:`k = \frac{1}{\text{in\_features}}`
        bias:   the learnable bias of the module of shape :math:`(\text{out\_features})`.
                If :attr:`bias` is ``True``, the values are initialized from
                :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where
                :math:`k = \frac{1}{\text{in\_features}}`

    Examples::

        >>> m = nn.Linear(20, 30)
        >>> input = torch.randn(128, 20)
        >>> output = m(input)
        >>> print(output.size())
        torch.Size([128, 30])
    """
    __constants__ = ['bias']

    def __init__(self, in_features, out_features, bias=True):
        super(Linear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        if bias:
            self.bias = Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)

    @weak_script_method
    def forward(self, input):
        return F.linear(input, self.weight, self.bias)

    def extra_repr(self):
        return 'in_features={}, out_features={}, bias={}'.format(
            self.in_features, self.out_features, self.bias is not None
        )

You can create a class DoubleLinear like this :

class DoubleLinear(Module):
    def __init__(self, Linear1, Linear2):
        self.Linear1 = Linear1
        self.Linear2 = Linear2
    @weak_script_method
    def forward(self, input):
        return self.Linear1(input), self.Linear2(input)

Then, create your two Linear layers :

Linear_vow = nn.Linear(final_in_features, 10)
Linear_con = nn.Linear(final_in_features, 10)
final_layer = DoubleLinear(Linear_vow, Linear_con)

now outputs1, outputs2 = final_layer(inputs) will work as expected.


Instruction: How to use multi-gpu during inference in pytorch framework

Input:
I am trying to make model prediction from unet3D built on pytorch framework. I am using multi-gpus

import torch
import os
import torch.nn as nn
os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES']='0,1,2'

model = unet3d()
model = nn.DataParallel(model)
model = model.to('cuda')

result = model.forward(torch.tensor(input).to('cuda').float())

But the model still uses only 1 GPU (the first one) and I get memory error.

CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 11.00 GiB total capacity; 8.43 GiB already allocated; 52.21 MiB free; 5.17 MiB cached)

How shoudl I use Multi-GPUs during inference phase? What is the mistake in my script above?

Output:
DataParallel handles sending the data to gpu.

import torch
import os
import torch.nn as nn
os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES']='0,1,2'

model = unet3d()
model = nn.DataParallel(model.cuda())

result = model.forward(torch.tensor(input).float())

if this doesn't work, please provide more details about input.

[EDIT]:

Try this:

with torch.no_grad():
    result = model(torch.tensor(input).float())


Instruction: weighted mse loss in pytorch

Input:
def weighted_mse_loss(input_tensor, target_tensor, weight = 1):
    observation_dim = input_tensor.size()[-1]
    streched_tensor = ((input_tensor - target_tensor) ** 2).view(-1, observation_dim)
    entry_num = float(streched_tensor.size())[0]
    non_zero_entry_num = torch.sum(streched_tensor[:,0] != 0).float()
    weighted_tensor = torch.mm(
        ((input_tensor - target_tensor)**2).view(-1, observation_dim),
        (torch.diag(weight.float().view(-1)))
    )
    return torch.mean(weighted_tensor) * weight.nelement() * entry_num / non_zero_entry_num

I can't understand how the code gives weighted Mean Square Error loss.
I get that observation_dim is the final output dimension, (the class number I guess), and after that line, I don't get it. Could someone help me figure out how the code calculates the loss?

Thanks a lot.

Output:
    def weighted_mse_loss(input, target, weight):
        return (weight * (input - target) ** 2).mean()

try this, hope this can help.
All arguments need tensored.


Instruction: slow Inference time for Neural Net

Input:
I have written a simple Fully connected Neural Network in Pytorch. I saved the model and loaded it in C++ using LibTorch but my inference time is pretty slow for my application field. Inference time right now is about 10 ms. Is it normal or am I doing something wrong?

I measured the inference time on python only first. then to maybe make it faster I loaded the network on C++ but it didn't help.

Here is the code for network

class network(nn.Module):
    def __init__(self):
        super(network,self).__init__()
        input_nodes = 362
        hidden_nodes1 = 50
        hidden_nodes2 = 30
        output_nodes = 1

    self.fc1 = nn.Linear(input_nodes,hidden_nodes1)
    nn.init.xavier_uniform_(self.fc1.weight)
    self.bn1 = nn.BatchNorm1d(num_features=hidden_nodes1)

    self.fc2 = nn.Linear(hidden_nodes1,hidden_nodes2)
    nn.init.xavier_uniform_(self.fc2.weight)
    self.bn2 = nn.BatchNorm1d(num_features = hidden_nodes2)

    self.fc3 = nn.Linear(hidden_nodes2,output_nodes)
    nn.init.xavier_uniform_(self.fc3.weight)

    self.out_act = nn.Sigmoid();

def forward(self,X):
    X = F.relu(self.bn1(self.fc1(X)))
    X = self.fc2(X)
    X = F.dropout2d(X,p=0.3)
    X = F.relu(X)
    X = self.fc3(X)
    out = self.out_act(X)
    return out

I want inference to somewhat take around 0.01 milliseconds.

Output:
How much data did you use for the inference? If it is only a few data points, I think there will be no much difference in execution time between python and C++. Maybe try with much more data?

Also, the architecture you are using is straightforward; it can probably run in CPU very well for inference. Don't forget to give feedback with your tests! I also want to know what is happening. :)


Instruction: Can't Backpropagate in Pytorch module with .backwards(); weights are not updated

Input:
I'm willing to create a network estimating the next sample of the signal. I've statred with a simple sin signal. But when I run the code I got noise as an output. Then checked the layer weights and figured aout that they are not updating. I can't find the mistake here.

class Model(nn.Module):
def __init__(self,in_dim,hidden_dim,num_classes):
    super(Model, self).__init__()
    self.layer1 = nn.Linear(in_dim,hidden_dim)
    self.layer2 = nn.Linear(hidden_dim,hidden_dim)
    self.layer3 = nn.Linear(hidden_dim,num_classes)
    self.relu = nn.ReLU()

def forward(self,x):
    a = self.relu(self.layer1(x))
    a = self.relu(self.layer2(a))
    return self.relu(self.layer3(a))

train:

def train(epoch,L,depth):
    criteria = nn.MSELoss()
    learning_rate = 1e-3
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    t = np.linspace(0,2,L+2)
    fs = L+2

    trn_loss = list()

   for f in range(0,epoch):
       phase = f/np.pi
       x = np.sin(2*np.pi*t*fs+phase)
       x = torch.from_numpy(x).detach().float()

       optimizer.zero_grad()

       x_hat = model(x[:-2])

       currentCost = criteria(x_hat,x[-2])
       trn_loss.append(currentCost.item())
       print(model.layer1.weight.data.clone())
       currentCost.backward()
       optimizer.step()
       print(model.layer1.weight.data.clone())
       sys.exit('DEBUG')

output:

tensor([[-0.1715, -0.1696,  0.0424,  ...,  0.0154,  0.1450, -0.0544],
    [ 0.0368,  0.1427, -0.1419,  ...,  0.0966,  0.0298, -0.0659],
    [-0.1641, -0.1551,  0.0570,  ..., -0.0227, -0.1426, -0.0648],
    ...,
    [-0.0684, -0.1707, -0.0711,  ...,  0.0788,  0.1386,  0.1546],
    [ 0.1401, -0.0922, -0.0104,  ..., -0.0490,  0.0404,  0.1038],
    [-0.0604, -0.0517,  0.0715,  ..., -0.1200,  0.0014,  0.0215]])
tensor([[-0.1715, -0.1696,  0.0424,  ...,  0.0154,  0.1450, -0.0544],
    [ 0.0368,  0.1427, -0.1419,  ...,  0.0966,  0.0298, -0.0659],
    [-0.1641, -0.1551,  0.0570,  ..., -0.0227, -0.1426, -0.0648],
    ...,
    [-0.0684, -0.1707, -0.0711,  ...,  0.0788,  0.1386,  0.1546],
    [ 0.1401, -0.0922, -0.0104,  ..., -0.0490,  0.0404,  0.1038],
    [-0.0604, -0.0517,  0.0715,  ..., -0.1200,  0.0014,  0.0215]])

Output:
Your final layer in forward call uses ReLU activation. This limits outputs of the network to [0, +inf) range.

Please notice your target is in the [-1, 1] range, so the network cannot output half (negative) of the values (and for the positive part it has to crunch +inf possible values into [0, 1] space).

You should change return self.relu(self.layer3(a)) to return self.layer3(a) in forward.

Better yet, in order to help your network accommodate to [-1, 1] range, use torch.tanh activation, so return torch.tanh(self.layer3(a)) should work best.


Instruction: How to calculate unbalanced weights for BCEWithLogitsLoss in pytorch

Input:
I am trying to solve one multilabel problem with 270 labels and i have converted target labels into one hot encoded form. I am using BCEWithLogitsLoss(). Since training data is unbalanced, I am using pos_weight argument but i am bit confused.

pos_weight (Tensor, optional) – a weight of positive examples. Must be a vector with length equal to the number of classes.

Do i need to give total count of positive values of each label as a tensor or they mean something else by weights?

Output:
The PyTorch documentation for BCEWithLogitsLoss recommends the pos_weight to be a ratio between the negative counts and the positive counts for each class.
So, if len(dataset) is 1000, element 0 of your multihot encoding has 100 positive counts, then element 0 of the pos_weights_vector should be 900/100 = 9. That means that the binary crossent loss will behave as if the dataset contains 900 positive examples instead of 100.
Here is my implementation:
(new, based on this post)
pos_weight = (y==0.).sum()/y.sum()
(original)
  def calculate_pos_weights(class_counts):
    pos_weights = np.ones_like(class_counts)
    neg_counts = [len(data)-pos_count for pos_count in class_counts]
    for cdx, pos_count, neg_count in enumerate(zip(class_counts,  neg_counts)):
      pos_weights[cdx] = neg_count / (pos_count + 1e-5)

    return torch.as_tensor(pos_weights, dtype=torch.float)

Where class_counts is just a column-wise sum of the positive samples. I posted it on the PyTorch forum and one of the PyTorch devs gave it his blessing.


Instruction: GAN measurement - FID score

Input:
I have two GANs and I want to compare their results using FID (Fréchet Inception Distance).
I have trianed the networks with the same dataset of frogs images, and by looking at the results (the generated images) one network yields better results but it's FID score is higher.
I computed the FID score between the original dataset and the generated images of each network.

I have read that lower FID values mean better image quality and diversity,
which is not consistent with the results I have seen.

Is there an explanation for that?

Output: You can have a look at this paper: https://arxiv.org/pdf/1904.06991v3.pdf. It clearly describes the problem of FID scores. However, the FID scores are also dependant on the number of samples. If you are using fewer samples to estimate it then the outcomes are not consistent.


Instruction: PyTorch Dimension out of range (expected to be in range of [-1, 0], but got 1)

Input:
I have the following PyTorch tensors:

predicted = torch.tensor([4, 4, 4, 1, 1, 1, 1, 1, 1, 4, 4, 1, 1, 1, 4, 1, 1, 4, 0, 4, 4, 1, 4, 1])

target    = torch.tensor([3, 0, 0, 1, 1, 0, 1, 1, 1, 3, 2, 4, 1, 1, 1, 0, 1, 1, 2, 1, 1, 1, 1, 1,])

I want to compute the Cross Entropy Loss (as part of an Logistic Regression implementation) between them with the following lines:

loss = nn.CrossEntropyLoss()
computed_loss = loss(predicted, target)

However, when my code runs, I get the following IndexError:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Any suggestions on what I'm doing wrong?

/ ##################################################################### /

Here is the full TraceBack:

-----------------------------------------------------------
IndexError                Traceback (most recent call last)
<ipython-input-208-3cdb253d6620> in <module>
      1 batch_size = 1000
      2 train_class = Train((training_set.shape[1]-1), number_of_target_labels, 0.01, 1000)
----> 3 train_class.train_model(training_set, batch_size)

<ipython-input-207-f3e2c7f7979a> in train_model(self, training_data, n_iters)
     42                 out = self.model(x)
     43                 _, predicted = torch.max(out.data, 1)
---> 44                 loss = self.criterion(predicted, y)
     45                 self.optimizer.zero_grad()
     46                 loss.backward()

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    940     def forward(self, input, target):
    941         return F.cross_entropy(input, target, weight=self.weight,
--> 942                                ignore_index=self.ignore_index, reduction=self.reduction)
    943
    944

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2054     if size_average is not None or reduce is not None:
   2055         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2056     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2057
   2058

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/functional.py in log_softmax(input, dim, _stacklevel, dtype)
   1348         dim = _get_softmax_dim('log_softmax', input.dim(), _stacklevel)
   1349     if dtype is None:
-> 1350         ret = input.log_softmax(dim)
   1351     else:
   1352         ret = input.log_softmax(dim, dtype=dtype)

/ ##################################################################### /

If you are interested in seeing the rest of my code, here it is:

import torch
import torch.nn as nn
from torch.autograd import Variable

class LogisticRegressionModel(nn.Module):

    def __init__(self, in_dim, num_classes):
        super().__init__()
        self.linear = nn.Linear(in_dim, num_classes)

    def forward(self, x):
        return self.linear(x)

class Train(LogisticRegressionModel):

    def __init__(self, in_dim, num_classes, lr, batch_size):
        super().__init__(in_dim, num_classes)
        self.batch_size = batch_size
        self.learning_rate = lr
        self.input_layer_dim = in_dim
        self.output_layer_dim = num_classes
        self.criterion = nn.CrossEntropyLoss()
        self.model = LogisticRegressionModel(self.input_layer_dim, self.output_layer_dim)
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.model = self.model.to(self.device)
        self.optimizer = torch.optim.SGD(self.model.parameters(), lr = self.learning_rate)

    def epochs(self, iterations, train_dataset, batch_size):
        epochs = int(iterations/(len(train_dataset)/batch_size))
        return epochs

    def train_model(self, training_data, n_iters):
        batch = self.batch_size
        epochs = self.epochs(n_iters, training_data, batch)
        training_data = torch.utils.data.DataLoader(dataset = training_data, batch_size = batch, shuffle = True)

        for epoch in range(epochs):

            for i, data in enumerate(training_data):

                X_train = data[:, :-1]
                Y_train = data[:, -1]

                if torch.cuda.is_available():
                    x = Variable(torch.Tensor(X_train).cuda())
                    y = Variable(torch.Tensor(Y_train).cuda())

                else:
                    x = Variable(torch.Tensor(X_train.float()))
                    y = Variable(torch.Tensor(Y_train.float()))

                out = self.model(x)
                _, predicted = torch.max(out.data, 1)
                loss = self.criterion(predicted, y)
                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                if i % 100 == 0:
                    print('[{}/{}] Loss: {:.6f}'.format(epoch + 1, epochs, loss))

Output:
It seems you are not quite using Cross Entropy Loss the way it is designed. CEL is primarily used for classification problems, where you have a probability distribution over some number of classes:

predicted = torch.tensor([[1,2,3,4]]).float()

(in this case, there are four classes, and the model is indicating its confidence of those four classes)

and then the target is simply an index indicating which class is correct:

target = torch.tensor([1]).long()

then, we can compute:

lossfxn = nn.CrossEntropyLoss()
loss = lossfxn(predicted, target)
print(loss) # outputs tensor(2.4402)

now, if we change the prediction to align with the target:

predicted = torch.tensor([[1,10,3,4]]).float()
target = torch.tensor([1]).long()
lossfxn = nn.CrossEntropyLoss()
loss = lossfxn(predicted, target)
print(loss) # outputs tensor(0.0035)

now the loss is much lower, because the prediction is correct!

Please consider the loss functions available and determine which is appropriate for your task: https://pytorch.org/docs/stable/nn.html#loss-functions (perhaps MSELoss?)


Instruction: How to get Docker to recognize NVIDIA drivers?

Input:
I have a container that loads a Pytorch model. Every time I try to start it up, I get this error:

Traceback (most recent call last):
  File "server/start.py", line 166, in <module>
    start()
  File "server/start.py", line 94, in start
    app.register_blueprint(create_api(), url_prefix="/api/1")
  File "/usr/local/src/skiff/app/server/server/api.py", line 30, in create_api
    atomic_demo_model = DemoModel(model_filepath, comet_dir)
  File "/usr/local/src/comet/comet/comet/interactive/atomic_demo.py", line 69, in __init__
    model = interactive.make_model(opt, n_vocab, n_ctx, state_dict)
  File "/usr/local/src/comet/comet/comet/interactive/functions.py", line 98, in make_model
    model.to(cfg.device)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 381, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 379, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/usr/local/lib/python3.7/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.7/site-packages/torch/cuda/__init__.py", line 82, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

I know that nvidia-docker2 is working.

$ docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi
Tue Jul 16 22:09:40 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:1A:00.0 Off |                  N/A |
|  0%   44C    P0    72W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:1B:00.0 Off |                  N/A |
|  0%   44C    P0    66W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:1E:00.0 Off |                  N/A |
|  0%   44C    P0    48W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:3E:00.0 Off |                  N/A |
|  0%   41C    P0    54W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 208...  Off  | 00000000:3F:00.0 Off |                  N/A |
|  0%   42C    P0    48W / 260W |      0MiB / 10989MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 208...  Off  | 00000000:41:00.0 Off |                  N/A |
|  0%   42C    P0     1W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

However, I keep getting the error above.

I've tried the following:

Setting "default-runtime": nvidia in /etc/docker/daemon.json
Using docker run --runtime=nvidia <IMAGE_ID>
Adding the variables below to my Dockerfile:

ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
LABEL com.nvidia.volumes.needed="nvidia_driver"

I expect this container to run - we have a working version in production without these issues. And I know that Docker can find the drivers, as the output above shows. Any ideas?

Output:
In order for docker to use the host GPU drivers and GPUs, some steps are necessary.

Make sure an nvidia driver is installed on the host system
Follow the steps here to setup the nvidia container toolkit
Make sure cuda, cudnn is installed in the image
Run a container with the --gpus flag (as explained in the link above)

I guess you have done the first 3 points because nvidia-docker2 is working. So since you don't have a --gpus flag in your run command this could be the issue.

I usually run my containers with the following command

docker run --name <container_name> --gpus all -it <image_name>

-it is just that the container is interactive and starts a bash environment.


Instruction: Compare the example of Pytorch and Keras on Cifar10 data

Input:
I use CIFAR10 dataset to learn how to code using Keras and PyTorch.

The environment is Python 3.6.7, Torch 1.0.0, Keras 2.2.4, Tensorflow 1.14.0.
I use the same batch size, number of epochs, learning rate and optimizer.
I use DenseNet121 as the model.

After training, Keras get 69% accuracy in test data.
PyTorch just get 54% in test data.

I know the results are different, but why is the result so bad in PyTorch?

Here is the Keras code:

import os, keras
from keras.datasets import cifar10
from keras.applications.densenet import DenseNet121
batch_size = 32
num_classes = 10
epochs = 20
# The data, split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# model
model = DenseNet121(include_top=True, weights=None, input_shape=(32,32,3), classes=10)

# initiate RMSprop optimizer
opt = keras.optimizers.SGD(lr=0.001, momentum=0.9)

model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          validation_data=(x_test, y_test),
          shuffle=True)

# Score trained model.
scores = model.evaluate(x_test, y_test, verbose=1)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])

Here is the Pytorch code:

import torch
import torchvision
import torchvision.transforms as transforms
from torch import flatten
import torch.optim as optim
from torchvision import transforms, models
from torch.nn import Linear, Softmax, Module, Sequential, CrossEntropyLoss
import numpy as np
from tqdm import tqdm
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
transform = transforms.Compose([transforms.ToTensor()])

trainset = torchvision.datasets.CIFAR10(root='./DataSet', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, num_workers=0)

testset = torchvision.datasets.CIFAR10(root='./DataSet', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=0)

import torch.nn as nn
import torch.nn.functional as F

class Net(Module):
    def __init__(self):
        super(Net, self).__init__()
        self.funFeatExtra   = Sequential(*[i for i in list(models.densenet121().children())[:-1]])
        self.funFlatten     = flatten
        self.funOutputLayer = Linear(1024, 10)
        self.funSoftmax     = Softmax(dim=1)
    def forward(self, x):
        x = self.funFeatExtra(x)
        x = self.funFlatten(x, 1)
        x = self.funOutputLayer(x)
        x = self.funSoftmax(x)
        return x

net = Net()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(20):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in tqdm(enumerate(trainloader, 0)):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net.cuda()(inputs.cuda())
        loss = criterion(outputs, labels.cuda())
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        # if i % 2000 == 1999:    # print every 2000 mini-batches
        #     print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
        #     running_loss = 0.0

print('Finished Training')

########################################################################
# The results seem pretty good.
#
# Let us look at how the network performs on the whole dataset.

correct = 0
total = 0
with torch.no_grad():
    for data in tqdm(testloader):
        images, labels = data
        outputs = net.cpu()(images.cpu())
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))

Output:
You are not supposed to softmax the model output before you pass it to CrossEntropyLoss. Per the documentation:

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

  ...

  The input is expected to contain raw, unnormalized scores for each class.

You can softmax them separately (outside of forward()) when calculating accuracy.