Instruction: How to convolute 2d image, tensor depth 3, using Linear Layer? (in PyTorch)

Input:
Given a tensor shape (3, 256, 256).
I would like to convolute or loop through it pixel by pixel to return me a tensor shape (1, 256, 256).

This may sound a bit confusing so here is my code till now so you know I mean.

class MorphNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(3, 8)
        self.fc2 = nn.Linear(8, 1)

    def forward(self, x):
        # The input here is shape (3, 256, 256)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        # Returned shape should be (1, 256, 256)
        return x

As you can see my Linear layer accept's shape 3 which matches the depth of my original tensor. What is the best way of looping through all 256x256 to return me tensor shape (1, 256, 256)

Output:
A linear layer that takes 3dim input and outputs 8dim is mathematically equivalent to a convolution with a kernel of spatial size of 1x1 (I strongly recommend that you actually "do the math" and convince yourself that this is indeed correct).

Therefore, you can use the following model, replacing the linear layers with nn.Conv2D:

class MorphNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.c1 = nn.Conv2d(3, 8, kernel_size=1, bias=True)
        self.c2 = nn.Conv2d(8, 1, kernel_size=1, bias=True)

    def forward(self, x):
        # The input here is shape (3, 256, 256)
        x = F.relu(self.c1(x))
        x = self.c2(x)
        # Returned shape should be (1, 256, 256)
        return x

If you insist on using a nn.Linear layer, you can unfold your input and then unfold it back after you apply the linear layer.


Instruction: Vectorize index identification for faster GPU processing

Input:
My input is a list of values, data_idx. In the example, the values range from [0, 5].

data_idx = [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0]

My desired output, filled_matrix is a tensor of shape max(value) by len(data_idx) where each row, r, of the tensor contains all of indices where data_idx == r and -1 for the rest of the row if the number of matched indices is fewer than len(data_idx)

For example, in the first row, r=0, data_idx==0 at indices [3, 11, 14]. The full output would look like:

filled_matrix = tensor([[ 3, 11, 14, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
    [ 5, 10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
    [ 0,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
    [ 8, 12, 13, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
    [ 4,  6, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
    [ 1,  2,  7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]],
   dtype=torch.int8)

I have working for-loop code that accomplishes my goal.

import torch

max_idx = 6
data_idx = torch.tensor([2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0]).cuda()

max_number_data_idx = data_idx.shape[0]

filled_matrix = torch.zeros([max_idx, max_number_data_idx], dtype=torch.int8, device='cuda')
filled_matrix.fill_(-1)
for i in range(max_idx):
   same_idx = (data_idx == i).nonzero().flatten()
   filled_matrix[i][:same_idx.shape[0]] = same_idx

Now, I want to speed up this code. Specifically, I want it to be faster on the GPU. In the real scenario, the input, data_idx, can be a list containing millions of values. In that case, e.g 1 M of different values, the GPU will be call 1 M of time which make it very slow. My code is sequential and GPU hate sequential code.

Is there a function which will produce the same result more efficiently? Or a way to vectorize this for-loop?

Output:
Disclaimer: I have not profiled this code too see if it is actually faster on a GPU.

One vectorized solution is to use tensor views to broadcast the comparison. Tensor views do not use additional memory. You can see more details in the documentation

First, make a matrix that contains the values that you want to compare to for each row. In this case, it's just the row indices.

comparison = torch.tensor(range(max_idx))

Now we are going to use expand and unsqueeze to make views of data_idx and comparison which are the same shape as filled_matrix.

comparison_view = comparison.unsqueeze(1).expand(max_idx, max_number_data_idx)
print(comparison_view)

# Each row is the index you want to compare to
# tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
    [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
    [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4],
    [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]])

data_idx_view = data_idx.expand(max_idx, max_number_data_idx)
print(data_idx_view)

# Each row is a copy of data_idx
# tensor([[2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0],
    [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0],
    [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0],
    [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0],
    [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0],
    [2, 5, 5, 0, 4, 1, 4, 5, 3, 2, 1, 0, 3, 3, 0]])

We can compare their equality and use nonzero to find the indices

mask = comparison_view == data_idx_view
mask_indices = mask.nonzero()

print(mask_indices)
# tensor([[ 0,  3],
    [ 0, 11],
    [ 0, 14],
    [ 1,  5],
    [ 1, 10],
    [ 2,  0],
    [ 2,  9],
    [ 3,  8],
    [ 3, 12],
    [ 3, 13],
    [ 4,  4],
    [ 4,  6],
    [ 5,  1],
    [ 5,  2],
    [ 5,  7]])

Now, you just need to manipulate these results into the format that you want for your output.

filled_matrix = torch.zeros([max_idx, max_number_data_idx], dtype=torch.int8)
filled_matrix.fill_(-1)
col_indices = [0, 1, 2, 0, 1, 0, 1, 0, 1, 2, 0, 1, 0, 1, 2]
filled_matrix[mask_indices[:, 0], col_indices] = mask_indices[:, 1].type(torch.int8)

I thought about several options for generating the col_indices list, but I couldn't come up with anything without a for loop.

col_indices = torch.zeros(mask_indices.shape[0])
for i in range(1, mask_indices.shape[0]):
    if mask_indices[i,0] == mask_indices[i-1,0]:
        col_indices[i] = col_indices[i-1]+1

You would need to do some profiling to see which code is actually faster.


Instruction: Reformer local and LSH attention in HuggingFace implementation

Input:
The recent implementation of the Reformer in HuggingFace has both what they call LSH Self Attention and Local Self Attention, but the difference is not very clear to me after reading the documentation. Both use bucketing to avoid the quadratic memory requirement of vanilla transformers, but it is not clear how they differ.

Is it the case that local self attention only allows queries to attend to keys sequentially near them (i.e., inside a given window in the sentence), as opposed to the proper LSH hashing that LSH self attention does? Or is it something else?

Output: After closely examining the source code, I found that indeed the Local Self Attention attends to the sequentially near tokens.


Instruction: How can I swap axis in a torch tensor?

Input:
I have a torch tensor of size torch.Size([1, 128, 56, 128])
1 is channel, 128 is the width, and height. 56 are the stacks of images.
How can I resize it to torch.Size([1, 56, 128, 128]) ?

Output: You could simply use permute or transpose.


Instruction: Duplicate layers when reusing pytorch model

Input:
I am trying to reuse some of the resnet layers for a custom architecture and ran into a issue I can't figure out. Here is a simplified example; when I run:

import torch
from torchvision import models
from torchsummary import summary

def convrelu(in_channels, out_channels, kernel, padding):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel, padding=padding),
        nn.ReLU(inplace=True),
    )

class ResNetUNet(nn.Module):
    def __init__(self):
        super().__init__()

        self.base_model = models.resnet18(pretrained=False)
        self.base_layers = list(self.base_model.children())

        self.layer0 = nn.Sequential(*self.base_layers[:3])

    def forward(self, x):
        print(x.shape)

        output = self.layer0(x)

        return output

base_model = ResNetUNet().cuda()
summary(base_model,(3,224,224))

Is giving me:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 112, 112]           9,408
            Conv2d-2         [-1, 64, 112, 112]           9,408
       BatchNorm2d-3         [-1, 64, 112, 112]             128
       BatchNorm2d-4         [-1, 64, 112, 112]             128
              ReLU-5         [-1, 64, 112, 112]               0
              ReLU-6         [-1, 64, 112, 112]               0
================================================================
Total params: 19,072
Trainable params: 19,072
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 36.75
Params size (MB): 0.07
Estimated Total Size (MB): 37.40
----------------------------------------------------------------

This is duplicating each layer (there are 2 convs, 2 batchnorms, 2 relu's) as opposed to giving one layer each. If I print out self.base_layers[:3] I get:

[Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False), BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True), ReLU(inplace=True)]

which shows just three layers without duplicates. Why is it duplicating my layers?

I am using pytorch version 1.4.0

Output:
Your layers aren't actually being invoked twice. This is an artifact of how summary is implemented.
The simple reason is because summary recursively iterates over all the children of your module and registers forward hooks for each of them. Since you have repeated children (in base_model and layer0) then those repeated modules get multiple hooks registered. When summary calls forward this causes both of the hooks for each module to be invoked which causes repeats of the layers to be reported.

For your toy example a solution would be to simply not assign base_model as an attribute since it's not being used during forward anyway. This avoids having base_model ever being added as a child.
class ResNetUNet(nn.Module):
    def __init__(self):
        super().__init__()
        base_model = models.resnet18(pretrained=False)
        base_layers = list(base_model.children())
        self.layer0 = nn.Sequential(*base_layers[:3])

Another solution is to create a modified version of summary which doesn't register hooks for the same module multiple times. Below is an augmented summary where I use a set named already_registered to keep track of modules which already have hooks registered to avoid registering multiple hooks.
from collections import OrderedDict
import torch
import torch.nn as nn
import numpy as np

def summary(model, input_size, batch_size=-1, device="cuda"):

    # keep track of registered modules so that we don't add multiple hooks
    already_registered = set()

    def register_hook(module):

        def hook(module, input, output):
            class_name = str(module.__class__).split(".")[-1].split("'")[0]
            module_idx = len(summary)

            m_key = "%s-%i" % (class_name, module_idx + 1)
            summary[m_key] = OrderedDict()
            summary[m_key]["input_shape"] = list(input[0].size())
            summary[m_key]["input_shape"][0] = batch_size
            if isinstance(output, (list, tuple)):
                summary[m_key]["output_shape"] = [
                    [-1] + list(o.size())[1:] for o in output
                ]
            else:
                summary[m_key]["output_shape"] = list(output.size())
                summary[m_key]["output_shape"][0] = batch_size

            params = 0
            if hasattr(module, "weight") and hasattr(module.weight, "size"):
                params += torch.prod(torch.LongTensor(list(module.weight.size())))
                summary[m_key]["trainable"] = module.weight.requires_grad
            if hasattr(module, "bias") and hasattr(module.bias, "size"):
                params += torch.prod(torch.LongTensor(list(module.bias.size())))
            summary[m_key]["nb_params"] = params

        if (
            not isinstance(module, nn.Sequential)
            and not isinstance(module, nn.ModuleList)
            and not (module == model)
            and module not in already_registered:
        ):
            already_registered.add(module)
            hooks.append(module.register_forward_hook(hook))

    device = device.lower()
    assert device in [
        "cuda",
        "cpu",
    ], "Input device is not valid, please specify 'cuda' or 'cpu'"

    if device == "cuda" and torch.cuda.is_available():
        dtype = torch.cuda.FloatTensor
    else:
        dtype = torch.FloatTensor

    # multiple inputs to the network
    if isinstance(input_size, tuple):
        input_size = [input_size]

    # batch_size of 2 for batchnorm
    x = [torch.rand(2, *in_size).type(dtype) for in_size in input_size]
    # print(type(x[0]))

    # create properties
    summary = OrderedDict()
    hooks = []

    # register hook
    model.apply(register_hook)

    # make a forward pass
    # print(x.shape)
    model(*x)

    # remove these hooks
    for h in hooks:
        h.remove()

    print("----------------------------------------------------------------")
    line_new = "{:>20}  {:>25} {:>15}".format("Layer (type)", "Output Shape", "Param #")
    print(line_new)
    print("================================================================")
    total_params = 0
    total_output = 0
    trainable_params = 0
    for layer in summary:
        # input_shape, output_shape, trainable, nb_params
        line_new = "{:>20}  {:>25} {:>15}".format(
            layer,
            str(summary[layer]["output_shape"]),
            "{0:,}".format(summary[layer]["nb_params"]),
        )
        total_params += summary[layer]["nb_params"]
        total_output += np.prod(summary[layer]["output_shape"])
        if "trainable" in summary[layer]:
            if summary[layer]["trainable"] == True:
                trainable_params += summary[layer]["nb_params"]
        print(line_new)

    # assume 4 bytes/number (float on cuda).
    total_input_size = abs(np.prod(input_size) * batch_size * 4. / (1024 ** 2.))
    total_output_size = abs(2. * total_output * 4. / (1024 ** 2.))  # x2 for gradients
    total_params_size = abs(total_params.numpy() * 4. / (1024 ** 2.))
    total_size = total_params_size + total_output_size + total_input_size

    print("================================================================")
    print("Total params: {0:,}".format(total_params))
    print("Trainable params: {0:,}".format(trainable_params))
    print("Non-trainable params: {0:,}".format(total_params - trainable_params))
    print("----------------------------------------------------------------")
    print("Input size (MB): %0.2f" % total_input_size)
    print("Forward/backward pass size (MB): %0.2f" % total_output_size)
    print("Params size (MB): %0.2f" % total_params_size)
    print("Estimated Total Size (MB): %0.2f" % total_size)
    print("----------------------------------------------------------------")
    # return summary


Instruction: Pytorch batch indexing

Input:
So the output of my network looks like this:

output = tensor([[[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.0410, -0.2234],
     [ 0.0362, -0.2111],
     [ 0.0333, -0.2018],
     [ 0.0318, -0.1951],
     [ 0.0311, -0.1904],
     [ 0.0310, -0.1873],
     [ 0.0312, -0.1851],
     [ 0.0315, -0.1837],
     [ 0.0318, -0.1828],
     [ 0.0322, -0.1822],
     [ 0.0324, -0.1819],
     [ 0.0327, -0.1817],
     [ 0.0328, -0.1815],
     [ 0.0330, -0.1815],
     [ 0.0331, -0.1814],
     [ 0.0332, -0.1814],
     [ 0.0333, -0.1814],
     [ 0.0333, -0.1814],
     [ 0.0334, -0.1814],
     [ 0.0334, -0.1814],
     [ 0.0334, -0.1814]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.0410, -0.2234],
     [ 0.0362, -0.2111],
     [ 0.0333, -0.2018],
     [ 0.0318, -0.1951],
     [ 0.0311, -0.1904],
     [ 0.0310, -0.1873],
     [ 0.0312, -0.1851],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]]])

Which is a shape of [8, 24, 2]
Now 8 is my batch size. And i would like to get a data point from every batch, at the following locations:

index = tensor([24, 10,  3,  3,  1,  1,  1,  0])

So the 24th value from the first batch, the 10th value from the second batch, and so on.

Now i have problems figuring out the syntax.
I've tried

torch.gather(output, 0, index)

But it keeps telling me, that my dimensions don't match.
And trying

output[ : ,index]

Just gets me the values at all the indexes for each batch.
What would be the correct syntax here, to get these values?

Output:
To select only one element per batch you need to enumerate the batch indices, which can be done easily with torch.arange.

output[torch.arange(output.size(0)), index]

That essentially creates tuples between the enumerated tensor and your index tensor to access the data, which results in indexing output[0, 24], output[1, 10] etc.


Instruction: Multi-dimensional tensor dot product in pytorch

Input:
I have two tensors of shapes (8, 1, 128) as follows.

q_s.shape
Out[161]: torch.Size([8, 1, 128])

p_s.shape
Out[162]: torch.Size([8, 1, 128])

Above two tensors represent a batch of eight 128 dimensional vectors. I want the dot product of batch q_s with batch p_s. How can I do this? I tried to use torch.tensordot function as follows. It works as expected as well. But it also does the extra work, which I don't want it to do. See the following example.

dt = torch.tensordot(q_s, p_s, dims=([1,2], [1,2]))

dt
Out[176]:
tensor([[0.9051, 0.9156, 0.7834, 0.8726, 0.8581, 0.7858, 0.7881, 0.8063],
        [1.0235, 1.5533, 1.2155, 1.2048, 1.3963, 1.1310, 1.1724, 1.0639],
        [0.8762, 1.3490, 1.2923, 1.0926, 1.4703, 0.9566, 0.9658, 0.8558],
        [0.8136, 1.0611, 0.9131, 1.1636, 1.0969, 0.9443, 0.9587, 0.8521],
        [0.6104, 0.9369, 0.9576, 0.8773, 1.3042, 0.7900, 0.8378, 0.6136],
        [0.8623, 0.9678, 0.8163, 0.9727, 1.1161, 1.6464, 0.9765, 0.7441],
        [0.6911, 0.8392, 0.6931, 0.7325, 0.8239, 0.7757, 1.0456, 0.6657],
        [0.8493, 0.8174, 0.8041, 0.9013, 0.8003, 0.7451, 0.7408, 1.1771]],
       grad_fn=<AsStridedBackward>)

dt.shape
Out[177]: torch.Size([8, 8])

As we can see, this produces the tensor of size (8,8) with the dot products I want lying on the diagonal. Is there any different way to obtain a smaller required tensor of shape (8,1), which just contains the elements lying on the diagonal in above result. To be more clear, the elements lying on the diagonal are the correct required dot products we want as a dot product of two batches. Element at index [0][0] is dot product of q_s[0] and p_s[0]. Element at index [1][1] is dot product of q_s[1] and p_s[1] and so on.

Is there a better way to obtain the desired dot product in pytorch?

Output:
You can do it directly:

a = torch.rand(8, 1, 128)
b = torch.rand(8, 1, 128)

torch.sum(a * b, dim=(1, 2))
# tensor([29.6896, 30.4994, 32.9577, 30.2220, 33.9913, 35.1095, 32.3631, 30.9153])

torch.diag(torch.tensordot(a, b, dim=([1,2], [1,2])))
# tensor([29.6896, 30.4994, 32.9577, 30.2220, 33.9913, 35.1095, 32.3631, 30.9153])

If you set axis=2 in the sum you will get a tensor with shape (8, 1).


Instruction: How to get padding mask from input ids?

Input:
Considering a batch of 4 pre-processed sentences (tokenization, numericalizing and padding) shown below:

batch = torch.tensor([
                      [1, 2, 0, 0],
                      [4, 0, 0, 0],
                      [3, 5, 6, 7]
])

where 0 states for [PAD] token.

Thus, what would be an efficient approach to generate a padding masking tensor of the same shape as the batch assigning zero at [PAD] positions and assigning one to other input data (sentence tokens)?

In the example above it would be something like:

padding_masking=
tensor([
                      [1, 1, 0, 0],
                      [1, 0, 0, 0],
                      [1, 1, 1, 1]
])

Output:
You can get your desired result with

padding_masking = batch > 0

If you want ints instead of booleans, use

padding_masking.type(torch.int)


Instruction: How to get the correct shape of the tensor in custom dataset

Input:
I am using custom Dataset class, but the problem is that when I get the data from the Dataloader I am left with array that has different tensor shape than I want.

shape that I get:       torch.Size([1, 56, 128, 128])
shape that I want:    torch.Size([1, 56, 1, 128, 128])

my approach was to:

1) to apply numpy.expand_dims on the array and get   torch.Size([1, 1, 56, 128, 128])

2) then to np.transpose on the array to get the shape I want torch.Size([1, 56, 1, 128, 128])

after first step I am getting the error:

raise ValueError('pic should be 2/3 dimensional. Got {} dimensions.'.format(pic.ndim))
ValueError: pic should be 2/3 dimensional. Got 4 dimensions.

if I do the transpose first, none of the combination of np.transpose(array, axes=(1,2,0)) yields the shape torch.Size([56, 1, 128, 128])

if I convert array to Tensor first and then do torch.unsqueeze I get the error:

raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>

here is my code:

class patientdataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        # np_load_old = np.load
        # np.load = lambda *a, **k: np_load_old(*a, allow_pickle=True, **k)

        image= np.asarray(np.load(img_path))

        image= np.transpose(image, axes=(1,2,0))
        image = torch.Tensor (image)

        image = torch.unsqueeze(image, dim=1)

        y_label = torch.tensor(np.asarray(self.annotations.iloc[index,1]))

        if self.transform:
            image = self.transform(image)

        return (image, y_label)

Output:
You can do using None:

x.shape
torch.Size([1, 56, 128, 128])

z = x[:,:,None,:,:]

z.shape
torch.Size([1, 56, 1, 128, 128])

I got a hint from here.


Instruction: Array indexing in pytorch

Input:
I am experiencing weird behavior while dealing with the following scenario. tne is the batch size, while N is the total number of possible locations for each example. Each example has exactly 24 scalar outputs, where their locations are stored in dof tensor (size of dof is (tne X 24)). Fint_e has a size of tneX24 (i.e., the 24 outputs for each example). I am trying to construct a large tensor, which has a size of tne X N. When I do the following, it fills in the wrong manner. Any advice?

Fint_MAT        = torch.zeros((tne,N))
Fint_MAT[:,dof]  = Fint_e

The dof tensor, which has the size of batch size X 24, has different indices for each example, but each example has in total 24 indices.

For instance,

dof[0,:] = 0, 1, 6, 9, … (24 in total)

dof[1,:] = 1,100, 151, 300,… (24 in total)

Any hint would be appreciated.

I include below a simple scenario for better elaboration:

tne = 3
N   = 48
Fint_MAT = torch.zeros((tne,N))
Fint_e   = torch.randn((tne, 24))
v1 = torch.arange(24).unsqueeze(0)
v2 = torch.arange(12, 36).unsqueeze(0)
v3 = torch.arange(24, 48).unsqueeze(0)
dof      = torch.cat((v1,v2,v3), axis=0).long()
Fint_MAT[:,dof]  = Fint_e

Output:
Okay, the key here is to use pairs of indices. Your dof tensor indexes the columns, but you also need to index the rows.

x_index = torch.arange(3).unsqueeze(1).expand(3, 24)

x_index is a 3 x 24 tensor where each row is the row index. Now you can use this together with the dof tensor to index elements in the Fint_MAT matrix

Fint_MAT[x_index, dof] = Fint_e

Basically, corresponding elements in x_index and dof form a [row, column] pair in Fint_MAT, so Fint_MAT[x_index[0], dof[0]]= Fint_e[0] etc.

I think this should give you what you want.


Instruction: Cannot Uninstall Pytorch

Input:
I've installed pytorch 0.4.1 with both pip and conda in an effort to get a specific package working. Somehow version 1.5.0 is installed somewhere on my filesystem and despite my best efforts (following all these instructions) I cannot find and uninstall this version.
Running python3.6 -c "import torch; print(torch.__version__)" results in 1.5.0 no matter how many uninstalls I attempt.
I was using some virtual environments to help in my development, using venv.
I am running macOS 10.15.4, Intel Core i9 8-core, 32 GB 2667 DDR4, AMD Radeon Pro 5500M 8 GB.

Output:
I was able to locate all installs of torch using sudo find . -name "*torch*".

My version, which I ended up deleting 'by hand', was located at /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch.

Hopefully this can help someone else.


Instruction: torch.cuda.is_available() returning false with Ubuntu 16.04

Input:
I have the following configurations

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

nvidia-smi

CUDA Toolkit (from Anaconda)

  cudatoolkit               10.2.89              hfd86e86_1    anaconda

PyTorch (from Anaconda)

  pytorch                   1.5.0          py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch

But I am getting torch.cuda.is_available() -> False

Could anyone please tell which component should I upgrade or downgrade in order to bring up the CUDA?

Output: I had the same issue on Ubuntu 18.04 with anaconda installation, recommended from the PyTorch official website. After installing cudatoolkit 10.1 rather than recommended 10.2 I got 'True'. Some people remedied this using driver version downgrade/upgrade as this is most likely a compatibility issue.


Instruction: Accuracy for every epoch in Pytorch

Input:
I'm struggling to calculate accuracy for every epoch in my training function for CNN classifier in Pytorch.

After I run this script, it always prints out 0, 0.25 or 0.75 which is obviously wrong. I'm guessing the problem are the inputs of the get_accuracy function (outputs and labels) as they are not accumulated for the entire epoch but not sure how to fix that.

Ideally, I'd like to print out both train and test accuracy for every epoch.

def get_accuracy(pred, actual):
  assert len(pred) == len(actual)

  total = len(actual)
  _, predicted = torch.max(pred.data, 1)
  correct = (predicted == actual).sum().item()
  return correct / total

def train_model(model):
    for epoch in range(epochs):
        running_loss = 0.00
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            outputs = cnn(inputs)

            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item() #calculates loss of the batch in average

        running_loss /= len(trainloader)

        training_accuracy = get_accuracy(outputs, labels)
        test_accuracy = 'todo'

        print('='*10,'iteration: {0}'.format(epoch+1),'='*10,)
        print('\n Loss: {0} \n Training accuracy:{1}% \n Test accuracy: {2}%'.format(running_loss, training_accuracy, test_accuracy))

    print('Finished Training')

Output:
I assume you want to calculate accuracy for multiclass case (so your classes are of form [0, 1, 2, 3, ..., N]).

You are using maximum while it should be argmax, for example:

def accuracy(predictions, labels):
    classes = torch.argmax(predictions, dim=1)
    return torch.mean((classes == labels).float())

This return index with maximum value, while you are returning maximum probability (or unnormalized probability). As probability is almost never equal to 1 you should have 0 accuracy always (up to float precision, it might sometimes correctly be really close to 0 or 1 so it may "hit").

For example 0.9 != 2 and you can never predict class 2, but you might, by accident, predict class 1 (0.999999999 ~= 1).

This function should be called within your inner loop, just like you calculate loss, so it would be:

for epoch in range(epochs):
    running_loss = 0.00
    running_accuracy = 0.00
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        outputs = cnn(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        running_accuracy += accuracy(outputs, labels)
    running_loss /= len(trainloader)
    running_accuracy /= len(trainloader)

Same thing would go for test and validation, just switch your model to evaluate mode (via model.eval()) and disable gradient using with torch.no_grad(): context manager).


Instruction: How to convert a tensorflow model to a pytorch model?

Input:
I'm new to pytorch. Here's an architecture of a tensorflow model and I'd like to convert it into a pytorch model.

I have done most of the codes but am confused about a few places.

1) In tensorflow, the Conv2D function takes filter as an input. However, in pytorch, the function takes the size of input channels and output channels as inputs. So how do I find the equivalent number of input channels and output channels, provided with the size of the filter.

2) In tensorflow, the dense layer has a parameter called 'nodes'. However, in pytorch, the same layer has 2 different inputs (the size of the input parameters and size of the targeted parameters), how do I determine them based on the number of the nodes.

Here's the tensorflow code.

from keras.utils import to_categorical
from keras.models import Sequential, load_model
from keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu', input_shape=X_train.shape[1:]))
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(43, activation='softmax'))

Here's my code.:

import torch.nn.functional as F
import torch

# The network should inherit from the nn.Module
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Define 2D convolution layers
        # 3: input channels, 32: output channels, 5: kernel size, 1: stride
        self.conv1 = nn.Conv2d(3, 32, 5, 1)   # The size of input channel is 3 because all images are coloured
        self.conv2 = nn.Conv2d(32, 64, 5, 1)
        self.conv3 = nn.Conv2d(64, 128, 3, 1)
        self.conv3 = nn.Conv2d(128, 256, 3, 1)
        # It will 'filter' out some of the input by the probability(assign zero)
        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        # Fully connected layer: input size, output size
        self.fc1 = nn.Linear(36864, 128)
        self.fc2 = nn.Linear(128, 10)

    # forward() link all layers together,
    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

Thanks in advance!

Output:
1) In pytorch, we take input channels and output channels as an input. In your first layer, the input channels will be the number of color channels in your image. After that it's always going to be the same as the output channels from your previous layer (output channels are specified by the filters parameter in Tensorflow).

2). Pytorch is slightly annoying in the fact that when flattening your conv outputs you'll have to calculate the shape yourself. You can either use an equation to calculate this (=(−+2)/+1), or make a shape calculating function to get the shape of a dummy image after it's been passed through the conv part of the network. This parameter will be your size of input argument; the size of your output argument will just be the number of nodes you want in your next fully connected layer.


Instruction: how to decide which decision should be taken from four given options?

Input:
For my snake AI, I'm using pytorch whose structure is:

class SnakeAI(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(24, 16, bias=True),
            nn.ReLU(),
            nn.Linear(16, 8, bias=True),
            nn.ReLU(),
            nn.Linear(8, 4, bias=True),
            nn.Sigmoid()
        )

    def forward(self, inputs):
        x = self.fc(inputs)
        return x

Based on the inputs it gives me 4 values which to my understanding should be the probability of which decision to be, whose sum should be equal to 1. which is not the case here as the output is:

[0.59388083 0.5833764  0.47855872 0.5388371 ]

based on the given output should i use argmax to decide which direction to go with? is there any better way to do that?

Output: You should change your last layer to nn.Softmax(-1). Then you would get in return a probability distribution and apply torch.max() which returns the maximum value and its index


Instruction: Expected input batch_size (1) to match target batch_size (11)

Input:
I know this seems to be a common problem but I wasn't able to find a solution. I'm running a multi-label classification model and having issues with tensor sizing.

My full code looks like this:

from transformers import DistilBertTokenizerFast, DistilBertForSequenceClassification
import torch

# Instantiating tokenizer and model
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-cased')
model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')

# Instantiating quantized model
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

# Forming data tensors
input_ids = torch.tensor(tokenizer.encode(x_train[0], add_special_tokens=True)).unsqueeze(0)
labels = torch.tensor(Y[0]).unsqueeze(0)

# Train model
outputs = quantized_model(input_ids, labels=labels)
loss, logits = outputs[:2]

Which yields the error:

ValueError: Expected input batch_size (1) to match target batch_size (11)

Input_ids looks like:

tensor([[  101,   789,   160,  1766,  1616,  1110,   170,  1205,  7727,  1113,
           170,  2463,  1128,  1336,  1309,  1138,   112,   119, 11882, 11545,
           119,   108, 15710,   108,  3645,   108,  3994,   102]])

with shape:

torch.Size([1, 28])

and labels looks like:

tensor([[0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1]])

with shape:

torch.Size([1, 11])

The size of input_ids will vary as the strings to be encoded vary in size.

I also noticed that when feeding in 5 values of Y to produce 5 labels, it yields the error:

ValueError: Expected input batch_size (1) to match target batch_size (55).

with labels shape:

torch.Size([1, 5, 11])

(Note that I didn't feed 5 input_ids, which is presumably why input size remains constant)

I've tried a few different approaches to getting these to work, but I'm currently at a loss. I'd really appreciate some guidance. Thanks!

Output:
The labels for DistilBertForSequenceClassification need to have the size torch.Size([batch_size]) as mentioned in the documentation:

  labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

In your case, your labels should have size torch.Size([1]).

That is not possible for your data, and that's because the sequence classification has one label for each sequence, but you wanted to make it a multi-label classification.

As far as I'm aware there is no multi-label model in HuggingFace's transformer library that you could use out of the box. You would need to create your own model, which is not particularly difficult, because these extra models all use the same base model and add an appropriate classifier at the end, depending on the task to be solved. HuggingFace - Multi-label Text Classification using BERT – The Mighty Transformer explains how this can be done.


Instruction: Pytorch: size mismatch error although the sizes of the matrices do match (m1: [256 x 200], m2: [256 x 200])

Input:
I trying to do transfer learning by pre training (Self supervised learning) a model on rotation (0, 90, 180, dn 270 degrees: 4 labels) on unlabelled data.
Here is the model:

class RotNet1(nn.Module):
    def __init__(self):
        keep_prob = 0.9
        super(RotNet1, self).__init__()
        self.layer1 = nn.Sequential(nn.Conv2d(in_channels = 3,
                                                  out_channels = 80,
                                                  kernel_size = 7,
                                                  stride = 1,
                                                  padding = 0),
                                        nn.ReLU(),
                                        nn.MaxPool2d(kernel_size = 2,
                      stride = 2,
                      padding = 1),
                      nn.Dropout(p=1 - keep_prob)
                      )
        self.bn1 = nn.BatchNorm2d(num_features = 80)
        self.dropout1 = nn.Dropout2d(p=0.02)
        self.layer2 = nn.Sequential(nn.Conv2d(in_channels = 80,
                                                  out_channels = 128,
                                                  kernel_size = 3,
                                                  stride = 1,
                                                  padding = 1),
                                        nn.ReLU(),
                                        nn.MaxPool2d(kernel_size = 2,
                      stride = 2,
                      padding = 1),
                      nn.Dropout(p=1 - keep_prob)
                      )
        self.bn2 = nn.BatchNorm2d(num_features = 128)
        self.layer3 = nn.Sequential(nn.Conv2d(in_channels = 128,
                                                  out_channels = 256,
                                                  kernel_size = 3,
                                                  stride = 1,
                                                  padding = 0),
                                        nn.ReLU(),
                                        nn.MaxPool2d(kernel_size = 2,
                      stride = 2,
                      padding = 1),
                      nn.Dropout(p=1 - keep_prob)
                      )
        self.bn3 = nn.BatchNorm2d(num_features = 256)
        self.layer4 = nn.Sequential(nn.Conv2d(in_channels = 256,
                                                  out_channels = 512,
                                                  kernel_size = 3,
                                                  stride = 1,
                                                  padding = 0),
                                        nn.ReLU(),
                                        nn.MaxPool2d(kernel_size = 2,
                      stride = 2,
                      padding = 1),
                      nn.Dropout(p=1 - keep_prob)
                      )
        self.bn4 = nn.BatchNorm2d(num_features = 512)
        self.layer5 = nn.Sequential(nn.Conv2d(in_channels = 512,
                                                  out_channels = 512,
                                                  kernel_size = 3,
                                                  stride = 1,
                                                  padding = 0),
                                        nn.ReLU(),
                                        nn.MaxPool2d(kernel_size = 2,
                      stride = 2,
                      padding = 1),
                      nn.Dropout(p=1 - keep_prob)
                      )
        self.bn5 = nn.BatchNorm2d(num_features = 512)
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(512* 2 * 2, 200)
        self.fc2 = nn.Linear(200, 4)
        #self.fc3 = nn.Linear(200, 100)

    def forward(self, input):
        out = self.layer1(input)
        out = self.bn1(out)
        out = self.dropout1(out)
        out = self.layer2(out)
        out = self.bn2(out)
        out = self.layer3(out)
        out = self.bn3(out)
        out = self.layer4(out)
        out = self.bn4(out)
        out = self.layer5(out)
        out = self.bn5(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        #out = self.fc3(out)
        return out

I trained this model on those 4 labels and names the model model_ssl. I then copied the model and changed the number the last fully connected layer from 4 to 200 (which is the number of labels in the labelled training and validation set where the number of example is restricted:

model_a = copy.copy(model_ssl)
#model_a.classifier
num_classes = 200
model_a.fc2 = nn.Linear(256,num_classes).cuda()

model_a.to(device)
loss_fn = torch.nn.CrossEntropyLoss()
n_epochs_a = 20
learning_rate_a = 0.01
alpha_a = 1e-5
momentum_a = 0.9
optimizer = torch.optim.SGD(model_a.parameters(),
                            momentum = momentum_a,
                            nesterov=True,
                            weight_decay = alpha_a,
                            lr=learning_rate_a)
train_losses_a, val_losses_a, train_acc_a, val_acc_a = train(model_a,
                                                             train_dataloader_sl,
                                                             val_dataloader_sl,
                                                             optimizer,
                                                             n_epochs_a,
                                                             loss_fn)

Here is the error message:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-27-f6f362ba8c53> in <module>()
     15                                                              optimizer,
     16                                                              n_epochs_a,
---> 17                                                              loss_fn)

6 frames

<ipython-input-23-df58f17c5135> in train(model, train_dataloader, val_dataloader, optimizer, n_epochs, loss_function)
     57     for epoch in range(n_epochs):
     58         model.train()
---> 59         train_loss, train_accuracy = train_epoch(model, train_dataloader, optimizer, loss_fn)
     60         model.eval()
     61         val_loss, val_accuracy = evaluate(model, val_dataloader, loss_fn)

<ipython-input-23-df58f17c5135> in train_epoch(model, train_dataloader, optimizer, loss_fn)
     10         labels = labels.to(device=device, dtype=torch.int64)
     11         # Run predictions
---> 12         output = model(images)
     13         # Set gradients to zero
     14         optimizer.zero_grad()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-11-2cd851b6d8e4> in forward(self, input)
     85         out = self.drop_out(out)
     86         out = self.fc1(out)
---> 87         out = self.fc2(out)
     88         #out = self.fc3(out)
     89         return out

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1608     if input.dim() == 2 and bias is not None:
   1609         # fused op is marginally faster
-> 1610         ret = torch.addmm(bias, input, weight.t())
   1611     else:
   1612         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [256 x 200], m2: [256 x 200] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283

The size of the matrices m1 and m2 seems to match but there is still that error message. What should I do?

Output:
The output shape of fc1 has an output size of 200, so the input size of fc2 should be 200 not 256, num_classes and 256 should be switched:

num_classes = 200
model_a.fc2 = nn.Linear(num_classes, 256).cuda()


Instruction: Does reshaping a tensor retain the characteristics of original tensor?

Input:
I have a tensor T of the shape (8, 5, 300), where 8 is the batch size, 5 is the number of documents in each batch, and 300 is the encoding of each of the document. If I reshape the Tensor as follows, does the properties of my Tensor remain the same?

T = T.reshape(5, 300, 8)
T.shape
>> Size[5, 300, 8]

So, does this new Tensor indicate the same properties as the original one? By the properties, I mean, can I say that this is also a Tensor of batch size 8, with 5 documents for each batch, and a 300 dimensional encoding for each document?

Does this affect the training of the model? If reshaping of Tensor messes up the datapoints, then there is no point in training. For example, If reshaping like above gives output as a batch of 5 samples, with 300 documents of size 8 each. If it happens so, then it's useless, since I do not have 300 documents, neither do I have batch of 5 samples.

I need to reshape it like this because my model in between produces output of the shape [8, 5, 300], and the next layer accepts input as [5, 300, 8].

Output:
NO

You need to understand the difference between reshape/view and permute.

reshape and view only changes the "shape" of the tensor, without re-ordering the elements. Therefore

orig = torch.rand((8, 5, 300))
resh = orig.reshape(5, 300, 8)

orig[0, 0, :] != resh[0, :, 0]

If you want to change the order of the elements as well, you need to permute it:

perm = orig.permute(1, 2, 0)
orig[0, 0, :] == perm[0, :, 0]


Instruction: Error: one of the variables needed for gradient computation has been modified by an inplace operation

Input:
I'm using the Soft Actor-Critic implementation available here for one of my projects. But when I try to run it, I get the following error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 1]], which is output 0 of TBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

The error arises from the gradient computation in the sac.py file. I can't see the operation that might be inplace. Any help?

The traceback:

Traceback (most recent call last)
<ipython-input-10-c124add9a61d> in <module>()
     22             for i in range(updates_per_step):
     23                 # Update parameters of all the networks
---> 24                 critic_1_loss, critic_2_loss, policy_loss, ent_loss, alpha = agent.update_parameters(memory, batch_size, updates)
     25                 updates += 1
     26

2 frames
<ipython-input-7-a2432c4c3767> in update_parameters(self, memory, batch_size, updates)
     87
     88         self.policy_optim.zero_grad()
---> 89         policy_loss.backward()
     90         self.policy_optim.step()
     91

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--> 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199
    200     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101
    102

Output:
Just downgrade the PyTorch to anything under 1.5.0 (which is latest, as of writing this).

pip uninstall torch
pip install torch==1.4.0


Instruction: Save the best model trained on Faster RCNN (COCO dataset) with Pytorch avoiding to "overfitting"

Input:
I am training a Faster RCNN neural network on COCO dataset with Pytorch.

I have followed next tutorial:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

The training results are as follows:

Epoch: [6]  [  0/119]  eta: 0:01:16  lr: 0.000050  loss: 0.3780 (0.3780)  loss_classifier: 0.1290 (0.1290)  loss_box_reg: 0.1848 (0.1848)  loss_objectness: 0.0239 (0.0239)  loss_rpn_box_reg: 0.0403 (0.0403)  time: 0.6451  data: 0.1165  max mem: 3105
Epoch: [6]  [ 10/119]  eta: 0:01:13  lr: 0.000050  loss: 0.4129 (0.4104)  loss_classifier: 0.1277 (0.1263)  loss_box_reg: 0.2164 (0.2059)  loss_objectness: 0.0244 (0.0309)  loss_rpn_box_reg: 0.0487 (0.0473)  time: 0.6770  data: 0.1253  max mem: 3105
Epoch: [6]  [ 20/119]  eta: 0:01:07  lr: 0.000050  loss: 0.4165 (0.4302)  loss_classifier: 0.1277 (0.1290)  loss_box_reg: 0.2180 (0.2136)  loss_objectness: 0.0353 (0.0385)  loss_rpn_box_reg: 0.0499 (0.0491)  time: 0.6843  data: 0.1265  max mem: 3105
Epoch: [6]  [ 30/119]  eta: 0:01:00  lr: 0.000050  loss: 0.4205 (0.4228)  loss_classifier: 0.1271 (0.1277)  loss_box_reg: 0.2125 (0.2093)  loss_objectness: 0.0334 (0.0374)  loss_rpn_box_reg: 0.0499 (0.0484)  time: 0.6819  data: 0.1274  max mem: 3105
Epoch: [6]  [ 40/119]  eta: 0:00:53  lr: 0.000050  loss: 0.4127 (0.4205)  loss_classifier: 0.1209 (0.1265)  loss_box_reg: 0.2102 (0.2085)  loss_objectness: 0.0315 (0.0376)  loss_rpn_box_reg: 0.0475 (0.0479)  time: 0.6748  data: 0.1282  max mem: 3105
Epoch: [6]  [ 50/119]  eta: 0:00:46  lr: 0.000050  loss: 0.3973 (0.4123)  loss_classifier: 0.1202 (0.1248)  loss_box_reg: 0.1947 (0.2039)  loss_objectness: 0.0315 (0.0366)  loss_rpn_box_reg: 0.0459 (0.0470)  time: 0.6730  data: 0.1297  max mem: 3105
Epoch: [6]  [ 60/119]  eta: 0:00:39  lr: 0.000050  loss: 0.3900 (0.4109)  loss_classifier: 0.1206 (0.1248)  loss_box_reg: 0.1876 (0.2030)  loss_objectness: 0.0345 (0.0365)  loss_rpn_box_reg: 0.0431 (0.0467)  time: 0.6692  data: 0.1276  max mem: 3105
Epoch: [6]  [ 70/119]  eta: 0:00:33  lr: 0.000050  loss: 0.3984 (0.4085)  loss_classifier: 0.1172 (0.1242)  loss_box_reg: 0.2069 (0.2024)  loss_objectness: 0.0328 (0.0354)  loss_rpn_box_reg: 0.0458 (0.0464)  time: 0.6707  data: 0.1252  max mem: 3105
Epoch: [6]  [ 80/119]  eta: 0:00:26  lr: 0.000050  loss: 0.4153 (0.4113)  loss_classifier: 0.1178 (0.1246)  loss_box_reg: 0.2123 (0.2036)  loss_objectness: 0.0328 (0.0364)  loss_rpn_box_reg: 0.0480 (0.0468)  time: 0.6744  data: 0.1264  max mem: 3105
Epoch: [6]  [ 90/119]  eta: 0:00:19  lr: 0.000050  loss: 0.4294 (0.4107)  loss_classifier: 0.1178 (0.1238)  loss_box_reg: 0.2098 (0.2021)  loss_objectness: 0.0418 (0.0381)  loss_rpn_box_reg: 0.0495 (0.0466)  time: 0.6856  data: 0.1302  max mem: 3105
Epoch: [6]  [100/119]  eta: 0:00:12  lr: 0.000050  loss: 0.4295 (0.4135)  loss_classifier: 0.1171 (0.1235)  loss_box_reg: 0.2124 (0.2034)  loss_objectness: 0.0460 (0.0397)  loss_rpn_box_reg: 0.0498 (0.0469)  time: 0.6955  data: 0.1345  max mem: 3105
Epoch: [6]  [110/119]  eta: 0:00:06  lr: 0.000050  loss: 0.4126 (0.4117)  loss_classifier: 0.1229 (0.1233)  loss_box_reg: 0.2119 (0.2024)  loss_objectness: 0.0430 (0.0394)  loss_rpn_box_reg: 0.0481 (0.0466)  time: 0.6822  data: 0.1306  max mem: 3105
Epoch: [6]  [118/119]  eta: 0:00:00  lr: 0.000050  loss: 0.4006 (0.4113)  loss_classifier: 0.1171 (0.1227)  loss_box_reg: 0.2028 (0.2028)  loss_objectness: 0.0366 (0.0391)  loss_rpn_box_reg: 0.0481 (0.0466)  time: 0.6583  data: 0.1230  max mem: 3105
Epoch: [6] Total time: 0:01:20 (0.6760 s / it)
creating index...
index created!
Test:  [ 0/59]  eta: 0:00:15  model_time: 0.1188 (0.1188)  evaluator_time: 0.0697 (0.0697)  time: 0.2561  data: 0.0634  max mem: 3105
Test:  [58/59]  eta: 0:00:00  model_time: 0.1086 (0.1092)  evaluator_time: 0.0439 (0.0607)  time: 0.2361  data: 0.0629  max mem: 3105
Test: Total time: 0:00:14 (0.2378 s / it)
Averaged stats: model_time: 0.1086 (0.1092)  evaluator_time: 0.0439 (0.0607)
Accumulating evaluation results...
DONE (t=0.02s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.643
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.079
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.011
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.096
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.333
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.333
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Epoch: [7]  [  0/119]  eta: 0:01:16  lr: 0.000050  loss: 0.3851 (0.3851)  loss_classifier: 0.1334 (0.1334)  loss_box_reg: 0.1845 (0.1845)  loss_objectness: 0.0287 (0.0287)  loss_rpn_box_reg: 0.0385 (0.0385)  time: 0.6433  data: 0.1150  max mem: 3105
Epoch: [7]  [ 10/119]  eta: 0:01:12  lr: 0.000050  loss: 0.3997 (0.4045)  loss_classifier: 0.1250 (0.1259)  loss_box_reg: 0.1973 (0.2023)  loss_objectness: 0.0292 (0.0303)  loss_rpn_box_reg: 0.0479 (0.0459)  time: 0.6692  data: 0.1252  max mem: 3105
Epoch: [7]  [ 20/119]  eta: 0:01:07  lr: 0.000050  loss: 0.4224 (0.4219)  loss_classifier: 0.1250 (0.1262)  loss_box_reg: 0.2143 (0.2101)  loss_objectness: 0.0333 (0.0373)  loss_rpn_box_reg: 0.0493 (0.0484)  time: 0.6809  data: 0.1286  max mem: 3105
Epoch: [7]  [ 30/119]  eta: 0:01:00  lr: 0.000050  loss: 0.4120 (0.4140)  loss_classifier: 0.1191 (0.1221)  loss_box_reg: 0.2113 (0.2070)  loss_objectness: 0.0357 (0.0374)  loss_rpn_box_reg: 0.0506 (0.0475)  time: 0.6834  data: 0.1316  max mem: 3105
Epoch: [7]  [ 40/119]  eta: 0:00:53  lr: 0.000050  loss: 0.4013 (0.4117)  loss_classifier: 0.1118 (0.1210)  loss_box_reg: 0.2079 (0.2063)  loss_objectness: 0.0357 (0.0371)  loss_rpn_box_reg: 0.0471 (0.0473)  time: 0.6780  data: 0.1304  max mem: 3105
Epoch: [7]  [ 50/119]  eta: 0:00:46  lr: 0.000050  loss: 0.3911 (0.4035)  loss_classifier: 0.1172 (0.1198)  loss_box_reg: 0.1912 (0.2017)  loss_objectness: 0.0341 (0.0356)  loss_rpn_box_reg: 0.0449 (0.0464)  time: 0.6768  data: 0.1314  max mem: 3105
Epoch: [7]  [ 60/119]  eta: 0:00:39  lr: 0.000050  loss: 0.3911 (0.4048)  loss_classifier: 0.1186 (0.1213)  loss_box_reg: 0.1859 (0.2013)  loss_objectness: 0.0334 (0.0360)  loss_rpn_box_reg: 0.0412 (0.0462)  time: 0.6729  data: 0.1306  max mem: 3105
Epoch: [7]  [ 70/119]  eta: 0:00:33  lr: 0.000050  loss: 0.4046 (0.4030)  loss_classifier: 0.1177 (0.1209)  loss_box_reg: 0.2105 (0.2008)  loss_objectness: 0.0359 (0.0354)  loss_rpn_box_reg: 0.0462 (0.0459)  time: 0.6718  data: 0.1282  max mem: 3105
Epoch: [7]  [ 80/119]  eta: 0:00:26  lr: 0.000050  loss: 0.4125 (0.4067)  loss_classifier: 0.1187 (0.1221)  loss_box_reg: 0.2105 (0.2022)  loss_objectness: 0.0362 (0.0362)  loss_rpn_box_reg: 0.0469 (0.0462)  time: 0.6725  data: 0.1285  max mem: 3105
Epoch: [7]  [ 90/119]  eta: 0:00:19  lr: 0.000050  loss: 0.4289 (0.4068)  loss_classifier: 0.1288 (0.1223)  loss_box_reg: 0.2097 (0.2009)  loss_objectness: 0.0434 (0.0375)  loss_rpn_box_reg: 0.0479 (0.0461)  time: 0.6874  data: 0.1327  max mem: 3105
Epoch: [7]  [100/119]  eta: 0:00:12  lr: 0.000050  loss: 0.4222 (0.4086)  loss_classifier: 0.1223 (0.1221)  loss_box_reg: 0.2101 (0.2021)  loss_objectness: 0.0405 (0.0381)  loss_rpn_box_reg: 0.0483 (0.0463)  time: 0.6941  data: 0.1348  max mem: 3105
Epoch: [7]  [110/119]  eta: 0:00:06  lr: 0.000050  loss: 0.4082 (0.4072)  loss_classifier: 0.1196 (0.1220)  loss_box_reg: 0.2081 (0.2013)  loss_objectness: 0.0350 (0.0379)  loss_rpn_box_reg: 0.0475 (0.0461)  time: 0.6792  data: 0.1301  max mem: 3105
Epoch: [7]  [118/119]  eta: 0:00:00  lr: 0.000050  loss: 0.4070 (0.4076)  loss_classifier: 0.1196 (0.1223)  loss_box_reg: 0.2063 (0.2016)  loss_objectness: 0.0313 (0.0375)  loss_rpn_box_reg: 0.0475 (0.0462)  time: 0.6599  data: 0.1255  max mem: 3105
Epoch: [7] Total time: 0:01:20 (0.6763 s / it)
creating index...
index created!
Test:  [ 0/59]  eta: 0:00:14  model_time: 0.1194 (0.1194)  evaluator_time: 0.0633 (0.0633)  time: 0.2511  data: 0.0642  max mem: 3105
Test:  [58/59]  eta: 0:00:00  model_time: 0.1098 (0.1102)  evaluator_time: 0.0481 (0.0590)  time: 0.2353  data: 0.0625  max mem: 3105
Test: Total time: 0:00:13 (0.2371 s / it)
Averaged stats: model_time: 0.1098 (0.1102)  evaluator_time: 0.0481 (0.0590)
Accumulating evaluation results...
DONE (t=0.02s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.649
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.079
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.011
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.095
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.334
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.334
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

I have two questions:

Overfitting: I don't know if my model is overfitting or underfitting. How I can find out looking the metrics?
Save the best model of all epochs: How I can save the best model trained during the differents epochs? Which is the best epoch according to the results?

Thank you!

Output:
You need to keep track of loss on test dataset (or some other metric like recall). Draw your attention to this part of code:

for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    evaluate(model, data_loader_test, device=device)

train_one_epoch and evaluate are defined here. Evaluate function returns object of type CocoEvaluator, but you can modify the code so that it returns test loss (you need to either extract metrics from CocoEvaluator object somehow, or write your own metric evaluation).

So, the answers are:

Keep track of test loss, it will tell you about overfitting.
Save the model state after every epoch until test loss begins to increase. Tutorial about saving models is here.


Instruction: Trouble installing Torch on Google App Engine

Input:
I've built a machine learning api that uses Torch as the ML framework. When I upload the code to Googe App Engine it runs out of memory.
After some debugging I found out that the issue it the installation of Torch.

  I'm using Torch 1.5.0 and python 3.7.4

So how do I fix this error? Maybe I can change something i app.yaml?

Error message:

Step #1 - "builder": OSError: [Errno 12] Cannot allocate memory
Step #1 - "builder":     self.pid = os.fork()
Step #1 - "builder":   File "/usr/lib/python2.7/subprocess.py", line 938, in _execute_child
Step #1 - "builder":     errread, errwrite)
Step #1 - "builder":   File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 346, in _python_version
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line     332, in GetCacheKeyRaw
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 109, in GetCacheKeyRaw
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/common/single_layer_image.py", line 60, in GetCacheKey
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/python/layer_builder.py", line 153, in BuildLayer
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 114, in Build
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
Step #1 - "builder":   File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
Step #1 - "builder":     exec code in run_globals
Step #1 - "builder":   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder":     "__main__", fname, loader, pkg_name)
Step #1 - "builder":   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": Traceback (most recent call last):

And again this error message didn't appear when i didn't include torch in my requirements.txt

to reproduce:

app.yaml

runtime: python37
resources:
    memory_gb: 16
    disk_size_gb: 10

requirements.txt

gunicorn==20.0.4
aniso8601==8.0.0
beautifulsoup4==4.9.0
boto3==1.13.3
botocore==1.16.3
bs4==0.0.1
certifi==2020.4.5.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
docutils==0.15.2
filelock==3.0.12
Flask==1.1.2
Flask-RESTful==0.3.8
googletrans==2.4.0
idna==2.9
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.9.5
joblib==0.14.1
MarkupSafe==1.1.1
numpy==1.18.4
protobuf==3.11.3
python-dateutil==2.8.1
pytz==2020.1
regex==2020.4.4
requests==2.23.0
s3transfer==0.3.3
sacremoses==0.0.43
sentencepiece==0.1.86
six==1.14.0
soupsieve==2.0
tokenizers==0.5.2
tqdm==4.46.0
transformers==2.8.0
urllib3==1.25.9
Werkzeug==1.0.1

main.py

import flask
from flask import Flask, request
from flask_restful import Api, Resource

app = Flask(__name__)
api = Api(app)

production = False

import json

# Import api code

# Create main api 'view'
class main_api(Resource):

    def get(self):
        question = request.args.get('question')

        # Run the script
        # But not necessary for the minimum working test

        return {
            'question': question,
            # 'results': results_from_script,
        }

# Adds resource
api.add_resource(main_api, '/')

# Starts the api
if __name__ == '__main__':
    host = '127.0.0.1'
    port = 8080
    app.run(host=host, port=port, debug=not production)

Output:
I fixed this error by using the flex enviroment.
The only thing I had to change was the app.yaml

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
    python_version: 3

manual_scaling:
    instances: 1
resources:
    cpu: 2
    memory_gb: 5
    disk_size_gb: 10

And then it was ready to be deployed


Instruction: How to 4 dimension PyTorch tensor multiply by 1 dimension tensor?

Input:
I'm trying to write function for mixup training. On this site i found some code and adapted to my previous code. But in original code only one random variable is generated for batch (64). But i want random value for every picture in batch.
Code with one variable for batch:

def mixup_data(x, y, alpha=1.0):
    lam = np.random.beta(alpha, alpha)
    batch_size = x.size()[0]
    index = torch.randperm(batch_size)

    mixed_x = lam * x + (1 - lam) * x[index,:]
    mixed_y = lam * y + (1 - lam) * y[index,:]

    return mixed_x, mixed_y

x and y for input come from pytorch DataLoader.
x input size: torch.Size([64, 3, 256, 256])
y input size: torch.Size([64, 3474])

This code works good. Then I changed it to this:

def mixup_data(x, y):
    batch_size = x.size()[0]
    lam = torch.rand(batch_size)
    index = torch.randperm(batch_size)

    mixed_x = lam[index] * x + (1 - lam[index]) * x[index,:]
    mixed_y = lam[index] * y + (1 - lam[index]) * y[index,:]

    return mixed_x, mixed_y

But it gives an error: RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

How i understand how the code works is it takes first image in batch and multiply by first value in lam tensor (64 values long). How can i do it?

Output:
You need to replace the following line:

lam = torch.rand(batch_size)

by

lam = torch.rand(batch_size, 1, 1, 1)

With your current code, lam[index] * x multiplication is not possible because lam[index] is of size torch.Size([64]) whereas x is of size torch.Size([64, 3, 256, 256]). So, you need to make the size of lam[index] as torch.Size([64, 1, 1, 1]) so that it becomes broadcastable.

To cope with the following statement:

mixed_y = lam[index] * y + (1 - lam[index]) * y[index, :]

We can reshape the lam tensor before the statement.

lam = lam.reshape(batch_size, 1)
mixed_y = lam[index] * y + (1 - lam[index]) * y[index, :]


Instruction: dataloader does not return correct target tensor

Input:
I am writing custom dataset but the dataloader does not return correct target tensors.

where the label is 010 I am getting 10.

I am converting the labels into int before converting to tensor.

  y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

iterating through dataloader:

Batch idx 0, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([1])
Batch idx 1, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([100])
Batch idx 2, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([10])
Batch idx 3, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([1])

csv file looks like the following:

1 p0.npy, 100
2 pl.npy, 001
3 p2.npy, 001
4 p3.npy, 001
5 p4.npy, 100
6 p5.npy, 010
7 p6.npy, 100
8 p7.npy, 100
9 p8.npy, 100
10 p9.npy, 010
11 plO.npy, 010
12 pll.npy, 010
13 p12.npy, 010
14 p13.npy, 100

code:

class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):

            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):

        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        # np_load_old = np.load
        # np.load = lambda *a, **k: np_load_old(*a, allow_pickle=True, **k)

        image= np.array(np.load(img_path))

        # y_label = torch.tensor(np.asarray(self.annotations.iloc[index,1]))

        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget

        return (image, y_label)

Output:
It seems your labels are in binary form.
Convert them into decimal and then into tensor should do the trick for you.

y_label = torch.tensor(int(self.annotations.iloc[index, 1], 2))

doing so will convert 010 into 2, 100 into 4 and so on.


Instruction: Why "ValueError: optimizer got an empty parameter list" is happened in this case?

Input:
I already studied the questions on this topic and I saw only recommendations to use ModuleList instead of usual list. But I don't understand why this error occurs in the case when I use nn.Sequential?
I tried to build AlexNet like in official implementation here: https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py
but got "ValueError: optimizer got an empty parameter list"

class AlexNet(nn.Module):

def __init__(self, input_channels, n_classes=1000):
    super(AlexNet, self).__init__()

    self.features = nn.Sequential
    (
        nn.Conv2d(input_channels, 96, kernel_size=11, stride=4),
        nn.LocalResponseNorm(size=2, alpha=2e-5),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.ReLU(inplace=True),

        nn.Conv2d(96, 256, kernel_size=5, stride=1),
        nn.LocalResponseNorm(size=2, alpha=2e-5),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.ReLU(inplace=True),

        nn.Conv2d(256, 384, kernel_size=3, stride=1),
        nn.ReLU(inplace=True),

        nn.Conv2d(384, 384, kernel_size=3, stride=1),
        nn.ReLU(inplace=True),

        nn.Conv2d(384, 256, kernel_size=3, stride=1),
        nn.ReLU(inplace=True),

    )

    self.fully_connected = nn.Sequential
    (
        nn.Dropout2d(0.5),
        nn.Linear(256, 4096),
        nn.ReLU(inplace=True),
        nn.Linear(4096, 4096),
        nn.ReLU(inplace=True),
        nn.Linear(4096, n_classes)
    )

def forward(self, x):

    x = self.features(x)
    x = nn.Flatten(x)
    x = self.fully_connected(x)

    return x

model = AlexNet(input_channels=1, n_classes=10)
optimizer = optim.Adam(model.parameters() , lr=1e-3)

Output:
You have not actually created the nn.Sequential modules, but you have assigned the nn.Sequential class to  self.features and self.fully_connected. The parentheses are on the next line and in Python that creates a tuple, because it doesn't follow an identifier and newlines usually terminate statements with some exceptions such as opening brackets.

The opening parentheses need to be on the same line:

self.features = nn.Sequential(
    nn.Conv2d(input_channels, 96, kernel_size=11, stride=4),
    nn.LocalResponseNorm(size=2, alpha=2e-5),
    nn.MaxPool2d(kernel_size=3, stride=2),
    nn.ReLU(inplace=True),
    nn.Conv2d(96, 256, kernel_size=5, stride=1),
    nn.LocalResponseNorm(size=2, alpha=2e-5),
    nn.MaxPool2d(kernel_size=3, stride=2),
    nn.ReLU(inplace=True),
    nn.Conv2d(256, 384, kernel_size=3, stride=1),
    nn.ReLU(inplace=True),
    nn.Conv2d(384, 384, kernel_size=3, stride=1),
    nn.ReLU(inplace=True),
    nn.Conv2d(384, 256, kernel_size=3, stride=1),
    nn.ReLU(inplace=True),
)

self.fully_connected = nn.Sequential(
    nn.Dropout2d(0.5),
    nn.Linear(256, 4096),
    nn.ReLU(inplace=True),
    nn.Linear(4096, 4096),
    nn.ReLU(inplace=True),
    nn.Linear(4096, n_classes),
)


Instruction: Combining feature matrixes of different shapes into a single feature

Input:
A library that I am using only supports 1 feature matrix as an input.

Therefore, I would like to merge my two features into a single feature.

Feature #1: a simple float e.g. tensor([1.9])
Feature #2: categorical that I would like to one-hot encode tensor([0., 1., 0])

tensor([
    [1.9, 0., 0.],     # row 1 for float
    [0., 1., 0.]       # row 2 for OHE
])

My plan would be to take the 1x1 feature and the 3x1 feature merge them into a 3x2.

For the float row, I would always have the 2nd and 3rd entries zeroed out. <-- is there a better approach? e.g. should i use three 1.9's?

Would this method give the affect of training on both features simultaneously?

Output:
Yes, what you propose would work, in that the model would just learn to ignore the second and third indices. But since those are never used, you can just concatenate them directly, i.e.

tensor([1.9, 0., 1., 0.])

you don't need to "indicate" in any way to the model that the first value is a scalar and the rest operate as a one-hot encoding. The model will figure out the relevant features for the task you care about.


Instruction: Sequence Labelling with BERT

Input:
I am using a model consisting of an embedding layer and an LSTM to perform sequence labelling, in pytorch + torchtext. I have already tokenised the sentences.

If I use self-trained or other pre-trained word embedding vectors, this is straightforward.

But if I use the Huggingface transformers BertTokenizer.from_pretrained and BertModel.from_pretrained there is a '[CLS]' and '[SEP]' token added to the beginning and end of the sentence, respectively. So the output of the model becomes a sequence that is two elements longer than the label/target sequence.

What I am unsure of is:

Are these two tags needed for the BertModel to embed each token of a sentence "correctly"?
If they are needed, can I take them out after the BERT embedding layer, before the input to the LSTM, so that the lengths are correct in the output?

Output:
Yes, BertModel needed them since without those special symbols added, the output representations would be different. However, my experience says, if you fine-tune BertModel on the labeling task without [CLS] and [SEP] token added, then you may not see a significant difference. If you use BertModel to extract fixed word features, then you better add those special symbols.

Yes, you can take out the embedding of those special symbols. In fact, this is a general idea for sequence labeling or tagging tasks.

I suggest taking a look at some sequence labeling or tagging examples using BERT to become confident about your modeling decisions. You can find NER tagging example using Huggingface transformers here.


Instruction: Save a Hash Value as a Tensor in pytorch

Input:
I have a dataset that contains identifiers that are saved as string.

I want to create a neural net that gets amongst other things these identifiers as labels and then checks if two identifier are exactly the same. If they are the same then I want to increase the loss if the network predicts wrong values.

As an example an identifier looks like this ec2c1cc2410a4e259aa9c12756e1d6e

It's always 32 values and uses hexadecimal characters (0-9a-f).

I want to work with this value in pytorch and save it as a tensor but I get the following problem

decimal_identifier = int(string_id, 16)
tensor_id = torch.ToTensor(decimal_identifier)

RuntimeError: Overflow when unpacking long

So I can't convert the value into a decimal because the values are too big.
Any idea how I could fix this?
I know that it's always 32 chars but I haven't found a char tensor in pytorch.

How can I feed this unique identifier in my neural net?

Output:
The problem is that int(string_id, 16) converts your 32 char long hash into a single integer. This is really a very VERY large number.
You can, instead, convert it to an array:

tensor_id = torch.tensor([int(c, 16) for c in string_id])

Resulting with (in your example):

tensor([14, 12,  2, 12,  1, 12, 12,  2,  4,  1,  0, 10,  4, 14,  2,  5,  9, 10,
        10,  9, 12,  1,  2,  7,  5,  6, 14,  1, 13,  6, 14])

You can also group the hex digits to 8 at a time (for int64 tensor):

torch.tensor([int(string_id[i:i+8], 16) for i in range(0, len(string_id), 8)], dtype=torch.int64)


Instruction: When does dataloader shuffle happen for Pytorch?

Input:
I have beening using shuffle option for pytorch dataloader for many times. But I was wondering when this shuffle happens and whether it is performed dynamically during iteration. Take the following code as an example:

namesDataset = NamesDataset()
namesTrainLoader = DataLoader(namesDataset, batch_size=16, shuffle=True)
for batch_data in namesTrainLoader:
    print(batch_data)

When we define "namesTrainLoader", does that mean the shuffling is finished and the following iteration will be based on a fixed order of data? Will there be any randomness in the for loop after namesTrainLoader was defined?

I was trying to replace half of "batch_data" with some special value:

for batch_data in namesTrainLoader:
    batch_data[：8] = special_val
    pre = model(batch_data)

Let us say there will be infinite number of epoches, will "model" eventually see all the data in "namesTrainLoader"? Or half of the data of "namesTrainLoader" is actually lost to "model"?

Output:
The shuffling happens when the iterator is created. In the case of the for loop, that happens just before the for loop starts.

You can create the iterator manually with:

# Iterator gets created, the data has been shuffled at this point.
data_iterator = iter(namesTrainLoader)

By default the data loader uses torch.utils.data.RandomSampler if you set shuffle=True (without providing your own sampler). Its implementation is very straight forward and you can see where the data is shuffled when the iterator is created by looking at the RandomSampler.__iter__ method:

def __iter__(self):
    n = len(self.data_source)
    if self.replacement:
        return iter(torch.randint(high=n, size=(self.num_samples,), dtype=torch.int64).tolist())
    return iter(torch.randperm(n).tolist())

The return statement is the important part, where the shuffling takes place. It simply creates a random permutation of the indices.

That means you will see your entire dataset every time you fully consume the iterator, just in a different order every time. Therefore there is no data lost (not including cases with drop_last=True) and your model will see all data at every epoch.


Instruction: How do i make my PyTorch DCGAN code to run on a GPU?

Input:
I am trying to train a DCGAN on a GPU but as I am starting out with PyTorch I have tried to do something from the documentation and it works but I want to confirm if it is the correct way to do it as I have done cause I looked at many other questions about running it on the GPU but they are done in different ways.

import os
import torch
import torchvision.transforms as transforms
from torch.autograd import Variable
from torch.nn import (
    BatchNorm2d,
    BCELoss,
    Conv2d,
    ConvTranspose2d,
    LeakyReLU,
    Module,
    ReLU,
    Sequential,
    Sigmoid,
    Tanh,
)
from torch.optim import Adam
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR10
from torchvision.utils import save_image
from tqdm import tqdm

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

dataset = DataLoader(
    CIFAR10(
        root="./Data",
        download=True,
        transform=transforms.Compose(
            [
                transforms.Resize(64),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
            ]
        ),
    ),
    batch_size=64,
    shuffle=True,
    num_workers=2,
)

try:
    os.mkdir("./Models")
    os.mkdir("./Results")

except FileExistsError:
    pass

class Gen(Module):
    def __init__(self):
        super(Gen, self).__init__()
        self.main = Sequential(
            ConvTranspose2d(100, 512, 4, 1, 0, bias=False),
            BatchNorm2d(512),
            ReLU(True),
            ConvTranspose2d(512, 256, 4, 2, 1, bias=False),
            BatchNorm2d(256),
            ReLU(True),
            ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
            BatchNorm2d(128),
            ReLU(True),
            ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
            BatchNorm2d(64),
            ReLU(True),
            ConvTranspose2d(64, 3, 4, 2, 1, bias=False),
            Tanh(),
        )

    def forward(self, input):
        output = self.main(input)
        return output

class Dis(Module):
    def __init__(self):
        super(Dis, self).__init__()
        self.main = Sequential(
            Conv2d(3, 64, 4, 2, 1, bias=False),
            LeakyReLU(0.2, inplace=True),
            Conv2d(64, 128, 4, 2, 1, bias=False),
            BatchNorm2d(128),
            LeakyReLU(0.2, inplace=True),
            Conv2d(128, 256, 4, 2, 1, bias=False),
            BatchNorm2d(256),
            LeakyReLU(0.2, inplace=True),
            Conv2d(256, 512, 4, 2, 1, bias=False),
            BatchNorm2d(512),
            LeakyReLU(0.2, inplace=True),
            Conv2d(512, 1, 4, 1, 0, bias=False),
            Sigmoid(),
        )

    def forward(self, input):
        output = self.main(input)
        return output.view(-1)

def weights(obj):
    classname = obj.__class__.__name__
    if classname.find("Conv") != -1:
        obj.weight.data.normal_(0.0, 0.02)

    elif classname.find("BatchNorm") != -1:
        obj.weight.data.normal_(1.0, 0.02)
        obj.bias.data.fill_(0)

gen = Gen()
gen.apply(weights)
gen.cuda().cuda().to(device)

dis = Dis()
dis.apply(weights)
dis.cuda().to(device)

criterion = BCELoss()

optimizerDis = Adam(dis.parameters(), lr=0.0002, betas=(0.5, 0.999))
optimizerGen = Adam(gen.parameters(), lr=0.0002, betas=(0.5, 0.999))

for epoch in range(25):
    for batch, data in enumerate(tqdm(dataset, total=len(dataset)), 0):
        dis.zero_grad()

        input = Variable(data[0]).cuda().to(device)
        target = Variable(torch.ones(input.size()[0])).cuda().to(device)
        output = dis(input).cuda().to(device)

        realError = criterion(output, target)

        noise = Variable(torch.randn(input.size()[0], 100, 1, 1)).cuda().to(device)
        fake = gen(noise).cuda().to(device)
        target = Variable(torch.zeros(input.size()[0])).cuda().to(device)
        output = dis(fake.detach()).cuda().to(device)

        fakeError = criterion(output, target)

        errD = realError + fakeError
        errD.backward()
        optimizerDis.step()

        gen.zero_grad()

        target = Variable(torch.ones(input.size()[0])).cuda().to(device)
        output = dis(fake).cuda().to(device)

        errG = criterion(output, target)
        errG.backward()
        optimizerGen.step()

        print(f"  {epoch+1}/25 Dis Loss: {errD.data:.4f} Gen Loss: {errG.data:.4f}")

save_image(data[0], "./Results/Real.png", normalize=True)
save_image(gen(noise).data, f"./Results/Fake{epoch+1}.png", normalize=True)
torch.save(gen, f"./Models/model{epoch+1}.pth")

Output:
A few comments about your code:

What is the value of your device variable?
Make sure is is torch.device('cuda:0') (or whatever you GPU's device id is).
Otherwise, if your device is actually torch.device('cpu') then you run in CPU.
See torch.device for more information.
You removed the "model" part of your code, but you may have skipped an important part there: Have you moved your model to GPU as well? Usually a model contains many internal parameters (aka trainable weights) and you need them also on device.
your code should also have

dis.to(device)
criterion.to(device)  # if your loss function also has trainable parameters

Note that unlike torch.tensors, calling .to on a nn.Module is an "in place" operation.

You have redundancy in your code: you do not have to call both .cuda() AND .to().
Calling .cuda() was the old way of moving things to GPU, but for a while now pytorch introduced .to() to make coding simpler.
Since both your inputs, and model are on GPU, you do not need to explicitly move the outputs to device as well. Thus, you can replace output = dis(input).cuda().to(device) with just output = dis(input).
No need to use Variable explicitly. You can replace

noise = Variable(torch.randn(input.size()[0], 100, 1, 1)).cuda().to(device)

with

noise = torch.randn(input.size()[0], 100, 1, 1), device=input.device)

You can also use torch.zeros_like and torch.ones_like for the target variable:

target = torch.zeros_like(input)

Note that zeros_like and one_like take care for the device (and data type) for you - it will be the same as input's.


Instruction: pytorch with mkl-dnn backend performance on small conv with multi thread

Input:
I'm trying pytorch model with mkl-dnn backend. But i got a problem that the multi thread performance is slower than expected, runing on small conv. Please see this table.
mkl-dnn performace table

Runs big conv, the performance is obvious faster on 8 threads compared with single thread.  But runs small conv, the speed has no big differences with 8 threads and single thread.

So my question is:
1. why 8 threads is not obviously faster than 1 thread with small conv?
2. How to improve the 8 threads performance on small conv?

My code here

import time
import torch
import torch.nn as nn
from torch.nn.utils import weight_norm

class MyConv(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.cell = nn.Conv1d(*args, **kwargs)
        self.cell.weight.data.normal_(0.0, 0.02)

    def forward(self, x):
        return self.cell(x)

def main():
    #print(*torch.__config__.show().split("\n"), sep="\n")
    torch.set_num_threads(1)
    dim = 32
    kernels = 3
    seq = 100000
    MyCell = MyConv(dim, dim, kernel_size=kernels, stride=1)
    MyCell.eval()
    inputs = []
    iter = 1000
    for i in range(iter):
        inputs.append(torch.rand(1, dim, seq))

    start = time.time() * 1000
    for i in range(iter):
        print(i)
        y = MyCell(inputs[i])
        #print(y)
    end = time.time() * 1000
    print('cost %d ms per iter\n' % ((end - start) / iter))

if __name__ == "__main__":
    main()

Mkldnn verbose info (running with "export MKLDNN_VERBOSE=1")

mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_nchw out:f32_nChw8c,num:1,1x32x1x107718,2.18091
mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_oihw out:f32_OIhw8i8o,num:1,32x32x1x1,0.00195312
mkldnn_verbose,exec,convolution,jit_1x1:avx2,forward_training,fsrc:nChw8c fwei:OIhw8i8o fbia:x fdst:nChw8c,alg:convolution_direct,mb1_ic32oc32_ih1oh1kh1sh1dh0ph0_iw107718ow107718kw1sw1dw0pw0,3.87793
mkldnn_verbose,exec,reorder,jit:uni,undef,in:f32_nChw8c out:f32_nchw,num:1,1x32x1x107718,4.69116

Thanks a lot!

Output:
More threads does not mean more speed. If you have 4 cores, you cannot go any faster than 4 times 1 core.

What you should do is tune your code for maximum performance in single-thread execution (with compiler optimization turned off), and after you have done that, turn on the compiler's optimizer and make the code multi-threaded, with no more threads than you have cores.

Source: Multithreading - How to use CPU as much as possible?


Instruction: How do I use one-hot encoding with cross-entropy loss?

Input:
I have tried using one-hot encoding for a multiclass classification of about 120 classes using a dog breed dataset. Also using resnet18. But the following error is shown when I run the code. Please help me solve the issue.

The code of my model is shown below:

model = torchvision.models.resnet18()
op = torch.optim.Adam(model.parameters(),lr=0.001)
crit = nn.NLLLoss()
model.fc = nn.Sequential(
    nn.Linear(512,120),
    nn.Dropout(inplace=True),
    nn.ReLU(),
    nn.LogSoftmax())

for i,(x,y) in enumerate(train_dl):
    # prepare one-hot vector
    y_oh=torch.zeors(y.shape[0],120)
    y_oh.scatter_(1, y.unsqueeze(1), 1)

    # do the prediction
    y_hat=model(x)
    y_=torch.max(y_hat)

    loss=crit(y,y_)
    op.zero_grad()
    loss.backward()
    op.step()

The error:

RuntimeError Traceback (most recent call last) <ipython-input-190-46a21ead759a> in <module>
       6
       7     y_hat=model(x)
 ----> 8     loss=crit(y_oh,y_hat)
       9     op.zero_grad()
      10     loss.backward()

***RuntimeError: 1D target tensor expected, multi-target not supported***

Output:
The NLLLoss you are using expects indices of the ground-truth target classes. Btw. you do not have to convert your targets into one-hot vectors and use directly the y tensor.

Note also that NLLLoss expect the distribution output distribution in the logarithmic domains, i.e., use nn.LogSoftmax instead of nn.Softmax.


Instruction: How can I send a custom dataset through ssh?

Input:
I have to train a GAN (coded in Python using pytorch) on a remote GPU that I can only access from my PC via ssh, but I have a custom dataset (that I cannot download from anywhere) which is stored in the PC without the GPU.

I've searched on Google very intensively and tried to use the scp command (which is the only solution that I've found), but it seems that the dataset is too big to be send within an acceptable time (13GB in size).

How can I transfer the dataset to the PC with the GPU within a decent amount of time, given that I cannot access the PC in any other way than an ssh connection, in order to train the network? Moreover, how can I retrieve the state_dict() and store it to my PC, once the training is complete?

Output:
It has nothing to do with the dataset itself. You can use Rsync to transfer files from you PC to the remote server using SSH and vice versa meaning you can transfer data/folders from remote server to your local PC as well.

Rsync is a utility for efficiently transferring and synchronizing files between a computer and an external hard drive and across networked computers by comparing the modification times and sizes of files. It is also well suited for transferring large files over ssh as it is able to resume from previously interrupted transfer.

From here:

  rsync is typically used for synchronizing files and directories
  between two different systems. For example, if the command rsync
  local-file user@remote-host:remote-file is run, rsync will use
  SSH to connect as user to remote-host.[7] Once connected, it will
  invoke the remote host's rsync and then the two programs will
  determine what parts of the local file need to be transferred so that
  the remote file matches the local one.

How to use:

Similar to cp, rcp and scp, rsync requires the specification of a source and of a destination, of which at least one must be local.

Generic syntax:

rsync [OPTION] … SRC … [USER@]HOST:DEST
rsync [OPTION] … [USER@]HOST:SRC [DEST]

where SRC is the file or directory (or a list of multiple files and directories) to copy from, DEST is the file or directory to copy to, and square brackets indicate optional parameters.

Simple example :

The following command will transfer all the files in the directory dataset to the home directory in the remote server:

rsync -avz dataset/ root@192.168.0.101:/home/

the -avz switch options simply mean, compress and transfer the files in archive mode and show the progress on screen:

Common options :
-v : verbose
-r : copies data recursively (but don’t preserve timestamps and permission while transferring data
-a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
-z : compress file data
-h : human-readable, output numbers in a human-readable format

You can read more here as well.


Instruction: Pytorch inconsistent size with pad_packed_sequence, seq2seq

Input:
I'm having some inconsistencies with the output of a encoder I got from this github .

The encoder looks as follows:

class Encoder(nn.Module):
    r"""Applies a multi-layer LSTM to an variable length input sequence.
    """

    def __init__(self, input_size, hidden_size, num_layers,
                 dropout=0.0, bidirectional=True, rnn_type='lstm'):
        super(Encoder, self).__init__()
        self.input_size = 40
        self.hidden_size = 512
        self.num_layers = 8
        self.bidirectional = True
        self.rnn_type = 'lstm'
        self.dropout = 0.0
        if self.rnn_type == 'lstm':
            self.rnn = nn.LSTM(input_size, hidden_size, num_layers,
                               batch_first=True,
                               dropout=dropout,
                               bidirectional=bidirectional)

    def forward(self, padded_input, input_lengths):
        """
        Args:
            padded_input: N x T x D
            input_lengths: N
        Returns: output, hidden
            - **output**: N x T x H
            - **hidden**: (num_layers * num_directions) x N x H
        """
        total_length = padded_input.size(1)  # get the max sequence length
        packed_input = pack_padded_sequence(padded_input, input_lengths,
                                            batch_first=True,enforce_sorted=False)
        packed_output, hidden = self.rnn(packed_input)
        pdb.set_trace()
        output, _ = pad_packed_sequence(packed_output, batch_first=True, total_length=total_length)
        return output, hidden

So it only consists of a rnn lstm cell, if I print the encoder this is the output:

LSTM(40, 512, num_layers=8, batch_first=True, bidirectional=True)

So it should have a 512 sized output right? But when I feed a tensor with size torch.Size([16, 1025, 40]) 16 samples of 1025 vectors with size 40 (that gets packed to fit the RNN) the output that I get from the RNN has a new encoded size of 1024 torch.Size([16, 1025, 1024]) when it should have been encoded to 512 right?

Is there something Im missing?

Output:
Setting bidirectional=True makes the LSTM bidirectional, which means there will be two LSTMs, one that goes from left to right and the other that goes from right to left.

From the nn.LSTM documentation - Outputs:

  output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.

  For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.

Your output has the size [batch, seq_len, 2 * hidden_size] (batch and seq_len are swapped in your case due to setting batch_first=True) because of using a bidirectional LSTM. The outputs of the two are concatenated in order to have the information of both, which you could easily separate if you wanted to treat them differently.