Instruction: Either too little or too many arguments for a nn.Sequential

Input:
I am new to PyTorch, so please excuse my silly question.

I define a nn.Sequential in init of my Encoder object like this:

self.list_of_blocks = [EncoderBlock(n_features, n_heads, n_hidden, dropout) for _ in range(n_blocks)]
self.blocks = nn.Sequential(*self.list_of_blocks)

The forward of EncoderBlock looks like this

def forward(self, x, mask):

In the forward() of my Encoder, I try to do:

z0 = self.blocks(z0, mask)

I expect the nn.Sequential to pass these two arguments to individual blocks.

However, I get

TypeError: forward() takes 2 positional arguments but 3 were given

When I try:

z0 = self.blocks(z0)

I get (understandably):

TypeError: forward() takes 2 positional arguments but only 1 was given

When I do not use nn.Sequential and just execute one EncoderBlock after another, it works:

for i in range(self.n_blocks):
     z0 = self.list_of_blocks[i](z0, mask)

Question: What am I doing wrong and how do I use nn.Sequential correctly in this case?

Output:
Sequential in general does not work with multiple inputs and outputs.

It is an often discussed topic, see  PyTorch forum and GitHub issues #1908 or #9979.

You can define your own version of sequential. Assuming the mask is the same for all your encoder block (e.g., like in the Transformer networks), you can do:

class MaskedSequential(nn.Sequential):
    def forward(self, x, mask):
        for module in self._modules.values():
            x = module(x, mask)
        return inputs

Or if your EncoderBlocks return tuples, you can use a more general solution suggested in one of the GitHub issues:

class MySequential(nn.Sequential):
    def forward(self, *inputs):
        for module in self._modules.values():
            if type(inputs) == tuple:
                inputs = module(*inputs)
            else:
                inputs = module(inputs)
        return inputs


Instruction: pytorch - How to Save and load model from DistributedDataParallel learning

Input:
I'm new to the Pytorch DstributedDataParallel(), but I found that most of the tutorials save the local rank 0 model during training. Which means if I get 3 machine with 4 GPU on each of them, at the final I'll get 3 model that save from each machine.

For example in pytorch ImageNet tutorial on line 252:

if not args.multiprocessing_distributed or (args.multiprocessing_distributed
                and args.rank % ngpus_per_node == 0):
            save_checkpoint({...})

They save the model if rank % ngpus_per_node == 0.

To the best of my knowledge, DistributedDataParallel() will automatic do all reduce to the loss on the backend, without doing any further job, every process can sync the loss automatically base on that.
All the model on each process will only have slightly different at the end of the process. That mean we only need to save one model is enough.

So why don't we just save model on rank == 0, but rank % ngpus_per_node == 0 ?

And which model should I used for if I get multiple model?

If this is the right way for saving model in distribute learning, should I merge them, used one of them, or inference the result base on all three models?

Please let me know if I'm wrong.

Output:
What is going on
Please correct me if I'm wrong in any place
The changes you are referring to were introduced in 2018 via this commit and described as:

in multiprocessing mode, only one process will write the checkpoint

Previously, those were saved without any if block so each node on each GPU would save a model which is indeed wasteful and would most probably overwrite saved model multiple times on each node.
Now, we are talking about multiprocessing distributed (possibly many workers each with possibly multiple GPUs).
args.rank for each process is thus modified inside the script by this line:
args.rank = args.rank * ngpus_per_node + gpu

which has the following comment:

For multiprocessing distributed training, rank needs to be the
global rank among all the processes

Hence args.rank is unique ID amongst all GPUs amongst all nodes (or so it seems).
If so, and each node has ngpus_per_node (in this training code it is assumed each has the same amount of GPUs from what I've gathered), then the model is saved only for one (last) GPU on each node. In your example with 3 machines and 4 GPUs you would get 3 saved models (hopefully I understand this code correctly as it's pretty convoluted tbh).
If you used rank==0 only one model per world (where world would be defined as n_gpus * n_nodes) would be saved.
Questions
First question

So why don't we just save model on rank == 0, but rank %
ngpus_per_node == 0 ?

I will start with your assumption, namely:

To the best of my knowledge, DistributedDataParallel() will automatic
do all reduce to the loss on the backend, without doing any further
job, every process can sync the loss automatically base on that.

Precisely, it has nothing to do with loss but rather gradient accumulation and applied corrections to weights, as per documentation (emphasis mine):

This container parallelizes the application of the given module by
splitting the input across the specified devices by chunking in the batch dimension. The module is replicated on each machine and
each device, and each such replica handles a portion of the input.
During the backwards pass, gradients from each node are averaged.

So, when the model is created with some weights it is replicated on all devices (each GPU for each node). Now each GPU gets a part of input (say, for total batch size equal to 1024, 4 nodes each with 4 GPUs, each GPU would get 64 elements), calculates forward pass, loss, performs backprop via .backward() tensor method. Now all gradients are averaged by all-gather, parameters are optimized on root machine and parameters are distributed to all nodes so module's state is always the same across all machines.
Note: I'm not sure how this averaging exactly takes place (and I don't see it explicitly said in docs), though I assume those are first averaged across GPUs and later across all nodes as it would be the most efficient I think.
Now, why would you save model for each node in such case? In principle you could only save one (as all modules will be exactly the same), but it has some downsides:

Say your node where your model was saved crashes and the file is lost. You have to redo all the stuff. Saving each model is not too costly operation (done once per epoch or less) so it can be easily done for each node/worker
You have to restart training. This means model would have to be copied to each worker (and some necessary metadata, though I don't think it's the case here)
Nodes will have to wait for every forward pass to finish anyway (so the gradients can be averaged), if the model saving takes a lot of time it would waste GPU/CPU being idle (or some other synchronization scheme would have to be applied, I don't think there is one in PyTorch). This makes it somewhat "no-cost" if you look at the overall picture.

Question 2 (and 3)

And which model should I used for if I get multiple model?

It doesn't matter as all of them will be exactly the same as the same corrections via optimizer are applied to the model with the same initial weights.
You could use something along those lines to load your saved .pth model:
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

parallel_model = torch.nn.DataParallel(MyModelGoesHere())
parallel_model.load_state_dict(
    torch.load("my_saved_model_state_dict.pth", map_location=str(device))
)

# DataParallel has model as an attribute
usable_model = parallel_model.model


Instruction: Using flatten in pytorch v1.0 Sequential module

Input:
Due to my CUDA version being 8, I am using torch 1.0.0

I need to use the Flatten layer for Sequential model. Here's my code :

import torch
import torch.nn as nn
import torch.nn.functional as F
print(torch.__version__)
# 1.0.0
from collections import OrderedDict

layers = OrderedDict()
layers['conv1'] = nn.Conv2d(1, 5, 3)
layers['relu1'] = nn.ReLU()
layers['conv2'] = nn.Conv2d(5, 1, 3)
layers['relu2'] = nn.ReLU()
layers['flatten'] = nn.Flatten()
layers['linear1'] = nn.Linear(3600, 1)
model = nn.Sequential(
layers
).cuda()

It gives me the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-38-080f7c5f5037> in <module>
      6 layers['conv2'] = nn.Conv2d(5, 1, 3)
      7 layers['relu2'] = nn.ReLU()
----> 8 layers['flatten'] = nn.Flatten()
      9 layers['linear1'] = nn.Linear(3600, 1)
     10 model = nn.Sequential(

AttributeError: module 'torch.nn' has no attribute 'Flatten'

How can I flatten my conv layer output in pytorch 1.0.0?

Output:
Just make a new Flatten layer.

from collections import OrderedDict

class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

layers = OrderedDict()
layers['conv1'] = nn.Conv2d(1, 5, 3)
layers['relu1'] = nn.ReLU()
layers['conv2'] = nn.Conv2d(5, 1, 3)
layers['relu2'] = nn.ReLU()
layers['flatten'] = Flatten()
layers['linear1'] = nn.Linear(3600, 1)
model = nn.Sequential(
layers
).cuda()


Instruction: RuntimeError: _th_normal not supported on CPUType for Long

Input:
I am trying to generate a number from the normal distribution using:

from torch.distributions import Normal
noise = Normal(th.tensor([0]), th.tensor(3.20))
noise = noise.sample()

But I am getting this error: RuntimeError: _th_normal not supported on CPUType for Long

Output:
Your first tensor th.tensor([0]) is of torch.Long type due to automatic type inference from passed value while float or FloatTensor is required by function.

You can solve it by passing 0.0 explicitly like this:

import torch

noise = torch.distributions.Normal(torch.tensor([0.0]), torch.tensor(3.20))
noise = noise.sample()

Better yet, drop torch.tensor altogether, in this case Python types will be automatically casted to float if possible, so this is valid as well:

import torch

noise = torch.distributions.Normal(0, 3.20)
noise = noise.sample()

And please, do not alias torch as th, it is not official, use fully qualified name as this only confuses everyone.


Instruction: Obtaining the shape (4, 1, 84, 84) with pytorch

Input:
Suppose I have four pytorch tensors (tensor1, tensor2, tensor3, tensor4). Each tensor is of shape (1, 1, 84, 84). The first dimension is the number of tensors, the second dimension is the number of colors (e.g. grayscale in our example) and the last two dimensions represent the height and the width of the image.

I want to stack them so that I get the shape (4, 1, 84, 84).

I tried torch.stack((tensor1, tensor2, tensor3, tensor4), dim=0), but I got a shape (4, 1, 1, 84, 84).

How can I stack those tensors so that the shape will be (4, 1, 84, 84)

Output:
You can use the concatenate function:

a = torch.ones(1,1,84,84)
b = torch.ones(1,1,84,84)
c = torch.cat((a,b), 0) # size[2,1,84,84]


Instruction: Pytorch 'Tensor' object is not callable

Input:
I am trying to apply sobel filter to my network. I am getting this error : "'Tensor' object is not callable"
Here is my code:

class SobelFilter(nn.Module):
    def __init__(self):
        super(SobelFilter, self).__init__()

        kernel1=torch.Tensor([[1, 0, -1],[2,0,-2],[1,0,-1]])
        kernela=kernel1.expand((1,1,3,3))

        kernel2=torch.Tensor([[1, 2, 1],[0,0,0],[-1,-2,-1]])
        kernelb=kernel2.expand((1,1,3,3))

        inputs = torch.randn(1,1,64,128)

        self.conv1 = F.conv2d(inputs,kernela,stride=1,padding=1)
        self.conv2 = F.conv2d(inputs,kernelb,stride=1,padding=1)

    def forward(self, x ):

        print(x.shape)
        G_x = self.conv1(x)    %GIVES ERROR AT THIS LINE
        G_y = self.conv2(x)
        out = torch.sqrt(torch.pow(G_x,2)+ torch.pow(G_y,2))

        return out

class EXP(nn.Module):
    def __init__(self, maxdisp):
        super(EXP, self).__init__()
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.SobelFilter = SobelFilter()

    def forward(self, im):
        % just think "im" to be [1,32,64,128]

        for i in range(im.shape[1]):
            out1= im[:,i,:,:].unsqueeze(1)
            im[:,i,:,:] = self.SobelFilter(out1)

what can cause the problem?
Thank you!

Output:
I think your issue is that you are using torch.nn.functional instead of just torch.
The purpose of the functional API is to perform the operations (in this case the conv2d) directly, without creating a class instance and then calling its forward method.
Therefore, the statement self.conv1 = F.conv2d(inputs,kernela,stride=1,padding=1) is already doing the convolution between the input and kernela and what you have in self.conv1is the result of such convolution.
There are two approaches here to fix the problem. Use torch.Conv2d inside __init__, where inputs is the channel of the input and not a tensor with the same shape as your real input.
And the second approach is to stick with the functional API but move it to the forward() method. What you want to achieve can be done by changing the forward to:

def forward(self, x ):
  print(x.shape)
  G_x = F.conv2d(x,self.kernela,stride=1,padding=1)
  G_y = F.conv2d(x,self.kernelb,stride=1,padding=1)
  out = torch.sqrt(torch.pow(G_x,2)+ torch.pow(G_y,2))
  return out

Note that I made kernela and kernelb class attributes. Thus you should also change the __init__() to

def __init__(self):
  super(SobelFilter, self).__init__()

  kernel1=torch.Tensor([[1, 0, -1],[2,0,-2],[1,0,-1]])
  self.kernela=kernel1.expand((1,1,3,3))

  kernel2=torch.Tensor([[1, 2, 1],[0,0,0],[-1,-2,-1]])
  self.kernelb=kernel2.expand((1,1,3,3))


Instruction: Error implementing torch.optim.lr_scheduler.LambdaLR in Pytorch

Input:
I'm working on an Image classifier and trying to implement Cyclical Learning Rates to have a better results. I'm using lr_scheduler.LambdaLR to adjust the learning rate during training, but I'm having an error that I'm not sure what is the cause of it.
this is my code:

lr_find_epochs = 2
start_lr = 1e-7
end_lr = 0.1
# Set up the model, optimizer and loss function for the experiment

optimizer = torch.optim.SGD(model.parameters(), start_lr)
criterion = nn.NLLLoss()
# LR function lambda

lr_lambda = lambda x: math.exp(x * math.log(end_lr / start_lr) / (lr_find_epochs * len( train_loader)))
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lr_lambda)

and this is the error I'm having:

The error in lr_scheduler.py.

Output:
The issue is caused by this line here

scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lr_lambda)

As the error suggests you are trying to reference value before it has been assigned,i.e. the lambda function is called with itself as the argument which is currently not assigned to anything. As a result of this, an error is raised in lr_scheduler.py

Maybe you want to pass something else to the lambda function.


Instruction: * (tuple of Tensors tensors, name dim, Tensor out)

Input:
Suppose I have the memory list list_of_tensors = [tensor1, tensor2, tensor3, tensor4]. Each element is a pytorch tensor of shape (1, 1, 84, 84).

I want to concatenate that list of tensors to get a tensor of shape (4, 1, 84, 84). torch.cat(TT, dim=0) might surely allow me to do that. TT must be a tuple of tensor, so torch.cat(*list_of_tensors, dim=0) or torch.cat((*list_of_tensors), dim=0) won't work.

How can I use list_of_tensors and torch.cat(???, dim=0) to create a new tensor of shape (4, 1, 84, 84)

Output:
You can use stack, and remove surplus dimension with squeeze

c = (torch.stack(list_of_tensors,dim=1)).squeeze(0)

now c.shape is (4, 1, 84, 84)

You can find explanation here: https://discuss.pytorch.org/t/how-to-turn-a-list-of-tensor-to-tensor/8868/6


Instruction: How can I element-wise 2 Tensors with different dimensions with Broadcasting?

Input:
I have a tensor called inputs with a size of torch.Size([20, 1, 161, 199]) and another mask with a size of torch.Size([20, 1, 199]). I want to multiply them together.

I tried:

masked_inputs = inputs * mask[..., None]

but get an error:

RuntimeError: The size of tensor a (161) must match the size of tensor b (199) at non-singleton dimension 2

I'm not quite sure what to do?

Output:
This did it:

masked_inputs = inputs * mask.unsqueeze(2)


Instruction: How do I do masking in PyTorch / Numpy with different dimensions?

Input:
I have a mask with a size of torch.Size([20, 1, 199]) and a tensor, reconstruct_output and inputs both with a size of torch.Size([20, 1, 161, 199]).

I want to set reconstruct_output to inputs where the mask is 0. I tried:

reconstruct_output[mask == 0] = inputs[mask == 0]

But I get an error:

IndexError: The shape of the mask [20, 1, 199] at index 2 does not match the shape of the indexed tensor [20, 1, 161, 199] at index 2

Output:
We can use advanced indexing here. To obtain the indexing arrays which we want to use to index both reconstruct_output and inputs, we need the indices along its axes where m==0. For that we can use np.where, and use the resulting indices to update reconstruct_output as:

m = mask == 0
i, _, l = np.where(m)
reconstruct_output[i, ..., l] = inputs[i, ..., l]

Here's a small example which I've checked with:

mask = np.random.randint(0,3, (2, 1, 4))
reconstruct_output = np.random.randint(0,10, (2, 1, 3, 4))
inputs = np.random.randint(0,10, (2, 1, 3, 4))

Giving for instance:

print(reconstruct_output)

array([[[[8, 9, 7, 2],
         [5, 4, 6, 1],
         [1, 4, 0, 3]]],

       [[[4, 3, 3, 4],
         [0, 9, 9, 7],
         [3, 4, 9, 3]]]])

print(inputs)

array([[[[7, 3, 9, 8],
         [3, 1, 0, 8],
         [0, 5, 4, 8]]],

       [[[3, 7, 5, 8],
         [2, 5, 3, 8],
         [3, 6, 7, 5]]]])

And the mask:

print(mask)

array([[[0, 1, 2, 1]],

       [[1, 0, 1, 0]]])

By using np.where to find the indices where there are zeroes in mask we get:

m = mask == 0
i, _, l = np.where(m)

i
# array([0, 1, 1])

l
# array([0, 1, 3])

Hence we'll be replacing the 0th column from the first 2D array and the 1st and 3rd from the second 2D array.

We can now use these arrays to replace along the corresponding axes indexing as:

reconstruct_output[i, ..., l] = inputs[i, ..., l]

Getting:

reconstruct_output

array([[[[7, 9, 7, 2],
         [3, 4, 6, 1],
         [0, 4, 0, 3]]],

       [[[4, 7, 3, 8],
         [0, 5, 9, 8],
         [3, 6, 9, 5]]]])


Instruction: How have the number of dimensions after LSTM been decided in Pointer Generator Model in PyTorch?

Input:
I don't understand why is the number of input and output dimensions 2 * config.hidden_dim while applying a fully connected layer in the encode class (mentioned in the last line)?

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.embedding = nn.Embedding(config.vocab_size, config.emb_dim)
        init_wt_normal(self.embedding.weight)

        self.lstm = nn.LSTM(
            config.emb_dim, config.hidden_dim, num_layers=1,
            batch_first=True, bidirectional=True)
        init_lstm_wt(self.lstm)

        self.W_h = nn.Linear(
            config.hidden_dim * 2, config.hidden_dim * 2, bias=False)

The code has been taken from https://github.com/atulkum/pointer_summarizer/blob/master/training_ptr_gen/model.py
Please Explain

Output: The reason is that the LSTM layer bidirectional, i.e., there are in fact two LSTMs each of them processing the input from each direction. They both return vectors of dimension config.hidden_dim which get concatenated into vectors of  2 * config.hidden_dim.


Instruction: size mismatch, m1: [288 x 9], m2: [2592 x 256] at /tmp/pip-req-build-4baxydiv/aten/src/TH/generic/THTensorMath.cpp:197

Input:
Here is the error I got

*** RuntimeError: size mismatch, m1: [288 x 9], m2: [2592 x 256] at /tmp/pip-req-build-4baxydiv/aten/src/TH/generic/THTensorMath.cpp:197

Here my model :

class DQN(nn.Module):
    def __init__(self, num_actions, lr):
        super(DQN, self).__init__()

        self.conv1 = nn.Conv2d(
            in_channels=4, out_channels=16, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(
            in_channels=16, out_channels=32, kernel_size=4, stride=2)

        # You have to respect the formula ((W-K+2P/S)+1)
        self.fc = nn.Linear(in_features=32*9*9, out_features=256)
        self.out = nn.Linear(in_features=256, out_features=num_actions)

    def forward(self, state):
        import ipdb; ipdb.set_trace()
        # (1) Hidden Conv. Layer
        self.layer1 = F.relu(self.conv1(state))

        # (2) Hidden Conv. Layer
        self.layer2 = F.relu(self.conv2(self.layer1))

        # (3) Hidden Linear Layer
        self.layer3 = self.fc(self.layer2)

        # (4) Output
        actions = self.out(self.layer3)

        return actions

The error is triggered at line self.layer3 = self.fc(self.layer2). state is a pytorch tensor of shape (1, 4, 84, 84).

The full traceback is

Traceback (most recent call last):
  File "/home/infinity/Projects/Exercice_Project/AI_Exercices/Atari_2600_Breakout.py", line 228, in <module>
    action = agent.choose_action(state, policy_network)
  File "/home/infinity/Projects/Exercice_Project/AI_Exercices/Atari_2600_Breakout.py", line 166, in choose_action
    return policy_net(state).argmax(dim=1).to(self.device)
  File "/home/infinity/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/infinity/Projects/Exercice_Project/AI_Exercices/Atari_2600_Breakout.py", line 101, in forward
    self.layer3 = self.fc(self.layer2)
  File "/home/infinity/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/infinity/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/infinity/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: size mismatch, m1: [288 x 9], m2: [2592 x 256] at /tmp/pip-req-build-4baxydiv/aten/src/TH/generic/THTensorMath.cpp:197

Output: Just before self.layer3 = self.fc(self.layer2), I had to add the line input_layer3 = self.layer2.reshape(-1, 32*9*9).


Instruction: How to generate accurate masks for an image from Mask R-CNN prediction in PyTorch?

Input:
I have trained a Mask RCNN network for instance segmentation of apples.  I am able to load the weights and generate predictions for my test images. The masks being generated seem to be in the correct location, but the mask itself has no real form.. it just looks like a bunch of pixels

Training is done based on the dataset from this paper, and here is the github link to code being used to train and generate weights

code for prediction is as follows. (i have omitted the parts where i create path variables and assign the paths)

import os
import glob
import numpy as np
import pandas as pd
import cv2 as cv
import fileinput

import torch
import torch.utils.data
import torchvision

from data.apple_dataset import AppleDataset
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

import utility.utils as utils
import utility.transforms as T

from PIL import Image
from matplotlib import pyplot as plt
%matplotlib inline

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def get_maskrcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model

num_classes = 2
device = torch.device('cpu')

model = get_maskrcnn_model_instance(num_classes)
checkpoint = torch.load('model_49.pth', map_location=device)
model.load_state_dict(checkpoint['model'], strict=False)

dataset_test = AppleDataset(test_image_files_path, get_transform(train=False))
img, _ = dataset_test[1]
model.eval()

with torch.no_grad():
    prediction = model([img.to(device)])

prediction

Image.fromarray(img.mul(255).permute(1, 2, 0).byte().numpy())

(unable to load image here since its over 2MB.

Image.fromarray(prediction[0]['masks'][0, 0].mul(255).byte().cpu().numpy())

Here is an Imgur link to the original image.. below is the predicted mask for one of the instances

Also, could you please help me understand the data structure of the generated prediction matrix shown below.. How do i access the masks so as to generate a single image with all masks displayed???

[{'boxes': tensor([[ 966.8143, 1633.7491, 1106.7389, 1787.6367],
          [1418.7872, 1467.0619, 1732.0828, 1796.1527],
          [1608.0396, 2064.6482, 1710.7534, 2206.5535],
          [2326.3750, 1690.3418, 2542.2112, 1883.2626],
          [2213.2024, 1864.3657, 2299.8933, 1963.0178],
          [1112.9083, 1732.5953, 1236.7600, 1823.0170],
          [1150.8256,  614.0334, 1218.8584,  711.4094],
          [ 942.7086,  794.6043, 1138.2318, 1008.0430],
          [1065.4371,  723.0493, 1192.7570,  870.3763],
          [1002.3103,  883.4616, 1146.9994, 1006.6841],
          [1315.2816, 1680.8625, 1531.3210, 1989.3317],
          [1244.5769, 1925.0903, 1459.5417, 2175.3252],
          [1725.2191, 2082.6187, 1934.0227, 2274.2952],
          [ 936.3065, 1554.3765, 1014.2722, 1659.4229],
          [ 934.8851, 1541.3331, 1090.4736, 1657.3751],
          [2486.0120,  776.4577, 2547.2329,  847.9725],
          [2336.1675,  698.6327, 2508.6492,  921.4550],
          [2368.4077, 1954.1102, 2448.4004, 2049.5796],
          [1899.1403, 1775.2371, 2035.7561, 1962.6923],
          [2176.0664, 1075.1553, 2398.6084, 1267.2555],
          [2274.8899,  641.6769, 2395.9634,  791.3353],
          [2535.1580,  874.4780, 2642.8213,  966.4614],
          [2183.4236,  619.9688, 2288.5676,  758.6825],
          [2183.9832, 1122.9382, 2334.9583, 1263.3226],
          [1135.7822,  779.0529, 1225.9871,  890.0135],
          [ 317.3954, 1328.6995,  397.3900, 1467.7740],
          [ 945.4811, 1833.3708,  997.2318, 1878.8607],
          [1992.4447,  679.4969, 2134.6667,  835.8701],
          [1098.5416, 1452.7799, 1429.1808, 1771.4460],
          [1657.3193, 1405.5405, 1781.6273, 1574.6780],
          [1443.8911, 1747.1544, 1739.0361, 2076.9724],
          [1092.6003, 1165.3340, 1206.0881, 1383.8314],
          [2466.4170, 1945.5931, 2555.1931, 2039.8368],
          [2561.8508, 1616.2659, 2672.1033, 1742.2332],
          [1894.4806,  907.9214, 2097.1875, 1182.6473],
          [2321.5005, 1701.3344, 2368.3699, 1865.3914],
          [2180.0781,  567.5969, 2344.6357,  763.4360],
          [1845.7612,  668.6808, 2045.2688,  899.8501],
          [1858.9216, 2145.7097, 1961.8870, 2273.5088],
          [ 261.4607, 1314.0154,  396.9288, 1486.9498],
          [2488.1682, 1585.2357, 2669.0178, 1794.9926],
          [2696.9548,  936.0087, 2802.7961, 1025.2294],
          [1593.6837, 1489.8641, 1720.3124, 1627.8135],
          [2517.9468,  857.1713, 2567.1125,  929.4335],
          [1943.2167,  636.3422, 2151.4419,  853.8924],
          [2143.5664, 1100.0521, 2308.1570, 1290.7125],
          [2140.9231, 1947.9692, 2238.6956, 2000.6249],
          [1461.6316, 2105.2593, 1559.7675, 2189.0264],
          [2114.0781,  374.8153, 2222.8838,  559.9851],
          [2350.5320,  726.5779, 2466.8140,  878.2617]]),
  'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1]),
  'scores': tensor([0.9916, 0.9841, 0.9669, 0.9337, 0.9118, 0.7729, 0.7202, 0.7193, 0.6928,
          0.6872, 0.6690, 0.5913, 0.4877, 0.4683, 0.3781, 0.3327, 0.3164, 0.2364,
          0.1696, 0.1692, 0.1502, 0.1365, 0.1316, 0.1171, 0.1119, 0.1094, 0.1041,
          0.0865, 0.0853, 0.0835, 0.0822, 0.0816, 0.0797, 0.0796, 0.0788, 0.0780,
          0.0757, 0.0736, 0.0736, 0.0689, 0.0681, 0.0644, 0.0642, 0.0630, 0.0612,
          0.0598, 0.0563, 0.0531, 0.0525, 0.0522]),
  'masks': tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          ...,

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]]])}]

Output:
The prediction from the Mask R-CNN has the following structure:

  During inference, the model requires only the input tensors, and returns the post-processed predictions as a List[Dict[Tensor]], one for each input image. The fields of the Dict are as follows:

boxes (FloatTensor[N, 4]): the predicted boxes in [x1, y1, x2, y2] format, with values between 0 and H and 0 and W
labels (Int64Tensor[N]): the predicted labels for each image
scores (Tensor[N]): the scores or each prediction
masks (UInt8Tensor[N, 1, H, W]): the predicted masks for each instance, in 0-1 range.

You can use OpenCV's findContours and drawContours functions to draw masks as follows:

img_cv = cv2.imread('input.jpg', cv2.COLOR_BGR2RGB)

for i in range(len(prediction[0]['masks'])):
    # iterate over masks
    mask = prediction[0]['masks'][i, 0]
    mask = mask.mul(255).byte().cpu().numpy()
    contours, _ = cv2.findContours(
            mask.copy(), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)
    cv2.drawContours(img_cv, contours, -1, (255, 0, 0), 2, cv2.LINE_AA)

cv2.imshow('img output', img_cv)

Sample output:


Instruction: Torch sum each row excluding an index

Input:
Given a Tensor A of shape (N,C) and an indices Tensor Idx of shape (N,), i'd like to sum all the elements of each row in A excluding the corresponding column index in I. For example:

A = torch.tensor([[1,2,3],
                  [4,5,6]])

Idx = torch.tensor([0,2])

#result:
torch.tensor([[5],
              [9]])

A solution using loops is known.

Output:
You can set excluded elements to zero:

A[range(A.shape[0]),Idx] = 0

and sum tensor along rows:

b = A.sum(dim = 1,keepdim = True ) # b = torch.tensor([[5],  [9]])


Instruction: "No such file" when loading csv data stored in G drive to torchtext format using torchtext.data.TabularDataset,

Input:
I have stored a csv file in G drive and try to load it to torchtext data.TabularDataset. The error message is "FileNotFoundError: [Errno 2] No such file or directory: 'https://.....'"

Is it impossible to load csv file from g drive directly to torchtext TabularDataset?

Here is the code. I have also made a public colab notebook with data publicly available.

import torch
from torchtext import data, datasets

!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

TEXT = data.Field(tokenize = 'spacy', batch_first = True, lower=False)
LABEL = data.LabelField(sequential=False, dtype = torch.float)

train = data.TabularDataset(path = 'https://drive.google.com/open?id=1eWMjusU3H34m0uml5SdJvYX6gQuB8zta',
                            format = 'csv',
                            fields = [('Insult', LABEL), (None, None), ('Comment', TEXT)],
                            skip_header=False)

Output:
Let's assume you can afford to download this CSV file. I would suggest you to use a functionally built-in on torchtext: download_from_url.

import os
import torch
from torchtext import data, datasets
from torchtext.utils import download_from_url

# download the file
CSV_FILENAME = 'data.csv'
CSV_GDRIVE_URL = 'https://drive.google.com/uc?export=download&id=1eWMjusU3H34m0uml5SdJvYX6gQuB8zta'
download_from_url(CSV_GDRIVE_URL, CSV_FILENAME)

TEXT = data.Field(tokenize = 'spacy', batch_first = True, lower=False)  #from torchtext import data
LABEL = data.LabelField(sequential=False, dtype = torch.float)

# if you're on Colab, you'll need this /content
train = data.TabularDataset(path=os.path.join('/content', CSV_FILENAME),
                            format='csv',
                            fields = [('Insult', LABEL), (None, None), ('Comment', TEXT)],
                            skip_header=False )

Notice that the Google Drive link should not be the one with open?id, but change it to uc?export=download&id.


Instruction: Anaconda, update Pytorch to the latest version 1.5

Input:
How could I update PyTorch from 1.4 -> 1.5 using Anaconda either through terminal or navigator?

Updating Anaconda with conda update --all updated some of the packages, not all, PyTorch included.

Initially, I installed PyTorch by running conda install -c pytorch pytorch

From PyTorch Github page there's the command

conda install -c pytorch magma-cuda90 # or [magma-cuda92 | magma-cuda100 | magma-cuda101 ] depending on your cuda version

but I wonder if there's going to be any kind of conflict between the already installed version and this one.

Thanks

Output:
PyTorch latest stable release (as of April 06, 2020) is still 1.4, as you can see here.

Therefore, if you want to install the nightly build (which is on track to the 1.5) using conda, you can follow the official instructions:

Cuda 10.1:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch-nightly -c defaults -c conda-forge

Cuda 9.2:

conda install pytorch torchvision cudatoolkit=9.2 -c pytorch-nightly -c defaults -c conda-forge -c numba/label/dev

or the CPU-only version:

conda install pytorch torchvision cpuonly -c pytorch-nightly -c defaults -c conda-forge

Or, you can just wait 1.5 to be an stable release (currently, we are in the release candidate 2) and update the pytorch package as you'd have done otherwise.

Be aware that:

  PyTorch 1.4 is the last release that supports Python 2

So, if you're moving to PyTorch 1.5, say goodbye to Python 2 (Yay!!).


Instruction: Reduce the number of hidden units in hugging face transformers (BERT)

Input:
I have been given a large csv each line of which is a set of BERT tokens made with hugging face BertTokenizer (https://huggingface.co/transformers/main_classes/tokenizer.html).
1 line of this file looks as follows:

101, 108, 31278, 90939, 70325, 196, 199, 71436, 10107, 29190, 10107, 106, 16680, 68314, 10153, 17015, 15934, 10104, 108, 10233, 12396, 14945, 10107, 10858, 11405, 13600, 13597, 169, 57343, 64482, 119, 119, 119, 100, 11741, 16381, 10109, 68830, 10110, 20886, 108, 10233, 11127, 21768, 100, 14120, 131, 120, 120, 188, 119, 11170, 120, 12132, 10884, 10157, 11490, 12022, 10113, 10731, 10729, 11565, 14120, 131, 120, 120, 188, 119, 11170, 120, 162, 11211, 11703, 12022, 11211, 10240, 44466, 100886, 102

and there are 9 million lines like this

Now, I am trying to get embeddings from these tokens like this:

def embedding:
    tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)
    model = BertModel.from_pretrained('bert-base-multilingual-cased')
    input_ids = torch.tensor([101, 108, 31278, 90939, 70325, 196, 199, 71436, 10107, 29190, 10107, 106, 16680, 68314, 10153, 17015, 15934, 10104, 108, 10233, 12396, 14945, 10107, 10858, 11405, 13600, 13597, 169, 57343, 64482, 119, 119, 119, 100, 11741, 16381, 10109, 68830, 10110, 20886, 108, 10233, 11127, 21768, 100, 14120, 131, 120, 120, 188, 119, 11170, 120, 12132, 10884, 10157, 11490, 12022, 10113, 10731, 10729, 11565, 14120, 131, 120, 120, 188, 119, 11170, 120, 162, 11211, 11703, 12022, 11211, 10240, 44466, 100886, 102]).unsqueeze(0)  # Batch size 1
    outputs = model(input_ids)
    last_hidden_states = outputs[0][0][0]  # The last hidden-state is the first element of the output tuple

Output of this is embedding correspending to the line. The size is 768*1 tensor. Semantically, everything is ok. But, when I do this for the full file the output is 768 * 9,0000,000 torch tensors. So I get a memory error even with large machine with a 768 GB of RAM.
Here is how I call this function:
tokens['embeddings'] = tokens['text_tokens'].apply(lambda x: embedding(x))

tokens is the pandas data frame with 9 million lines each of which contains BERT tokens.

Is it possible to reduce the default size of the hidden units, which is 768 here: https://huggingface.co/transformers/main_classes/model.html

Thank you for your help.

Output:
Changing the dimensionality would mean changing all the model parameters, i.e., retraining the model. This could be achievable by knowledge distillation, but it will be probably still quite computationally demanding.

You can also use some dimensionality reduction techniques on the BERT outputs, like PCA (available e.g., in scikit-learn).  In that case, I would suggest taking several thousand BERT vectors, fit the PCA and then apply the PCA on all the remaining vectors.


Instruction: Understanding when to call zero_grad() in pytorch, when training with multiple losses

Input:
I am going through an open-source implementation of a domain-adversarial model (GAN-like). The implementation uses pytorch and I am not sure they use zero_grad() correctly. They call zero_grad() for the encoder optimizer (aka the generator) before updating the discriminator loss. However zero_grad() is hardly documented, and I couldn't find information about it.

Here is a psuedo code comparing a standard GAN training (option 1), with their implementation (option 2). I think the second option is wrong, because it may accumulate the D_loss gradients with the E_opt. Can someone tell if these two pieces of code are equivalent?

Option 1 (a standard GAN implementation):

X, y = get_D_batch()
D_opt.zero_grad()
pred = model(X)
D_loss = loss(pred, y)
D_opt.step()

X, y = get_E_batch()
E_opt.zero_grad()
pred = model(X)
E_loss = loss(pred, y)
E_opt.step()

Option 2 (calling zero_grad() for both optimizers at the beginning):

E_opt.zero_grad()
D_opt.zero_grad()

X, y = get_D_batch()
pred = model(X)
D_loss = loss(pred, y)
D_opt.step()

X, y = get_E_batch()
pred = model(X)
E_loss = loss(pred, y)
E_opt.step()

Output:
It depends on params argument of torch.optim.Optimizer subclasses (e.g. torch.optim.SGD) and exact structure of the model.
Assuming E_opt and D_opt have different set of parameters (model.encoder and model.decoder do not share weights), something like this:
E_opt = torch.optim.Adam(model.encoder.parameters())
D_opt = torch.optim.Adam(model.decoder.parameters())

both options MIGHT indeed be equivalent (see commentary for your source code, additionally I have added backward() which is really important here and also changed model to discriminator and generator appropriately as I assume that's the case):
# Starting with zero gradient
E_opt.zero_grad()
D_opt.zero_grad()

# See comment below for possible cases
X, y = get_D_batch()
pred = discriminator(x)
D_loss = loss(pred, y)
# This will accumulate gradients in discriminator only
# OR in discriminator and generator, depends on other parts of code
# See below for commentary
D_loss.backward()
# Correct weights of discriminator
D_opt.step()

# This only relies on random noise input so discriminator
# Is not part of this equation
X, y = get_E_batch()
pred = generator(x)
E_loss = loss(pred, y)
E_loss.backward()
# So only parameters of generator are updated always
E_opt.step()

Now it's all about get_D_Batch feeding data to discriminator.
Case 1 - real samples
This is not a problem as it does not involve generator, you pass real samples and only discriminator takes part in this operation.
Case 2 - generated samples
Naive case
Here indeed gradient accumulation may occur. It would occur if get_D_batch would simply call X = generator(noise) and passed this data to discriminator.
In such case both discriminator and generator have their gradients accumulated during backward() as both are used.
Correct case
We should take generator out of the equation. Taken from PyTorch DCGan example there is a little line like this:
# Generate fake image batch with G
fake = generator(noise)
label.fill_(fake_label)
# DETACH HERE
output = discriminator(fake.detach()).view(-1)

What detach does is it "stops" the gradient by detaching it from the computational graph. So gradients will not be backpropagated along this variable. This effectively does not impact gradients of generator so it has no more gradients so no accumulation happens.
Another way (IMO better) would be to use with.torch.no_grad(): block like this:
# Generate fake image batch with G
with torch.no_grad():
    fake = generator(noise)
label.fill_(fake_label)
# NO DETACH NEEDED
output = discriminator(fake).view(-1)

This way generator operations will not build part of the graph so we get better performance (it would in the first case but would be detached afterwards).
Finally
Yeah, all in all first option is better for standard GANs as one doesn't have to think about such stuff (people implementing it should, but readers should not). Though there are also other approaches like single optimizer for both generator and discriminator (one cannot zero_grad() only for subset of parameters (e.g. encoder) in this case), weight sharing and others which further clutter the picture.
with torch.no_grad() should alleviate the problem in all/most cases as far as I can tell and imagine ATM.


Instruction: AttributeError: 'Example' object has no attribute 'Insult' when build vocab using build_vocab() from torchtext

Input:
I tried to build vocabulary using .build_vocab() from torchtext in colab. It returns the error message: AttributeError: 'Example' object has no attribute 'Insult'

My question is similar to @szymix12 s question. His answer is to make sure the order of the fields passed is the same as the csv headers. I confirm that the order I assign is correct. The csv data got two columns:"Insult" (LABEL) and "Comment"(TEXT). "Insult" is a binary label indicator (0 or 1).

The code is as below, I have also made a colab notebook. Feel free to run.

import os
import torch
from torchtext import data, datasets
from torchtext.utils import download_from_url

CSV_FILENAME = 'data.csv'
CSV_GDRIVE_URL = 'https://drive.google.com/open?id=1ctPKO-_sJbmc8RodpBZ5-3EyYBWlUy5Pbtjio5pyq00'
download_from_url(CSV_GDRIVE_URL, CSV_FILENAME)

TEXT = data.Field(tokenize = 'spacy', batch_first = True, lower=True)  #from torchtext import data
LABEL = data.LabelField(sequential=False, dtype = torch.float)

train = data.TabularDataset(path=os.path.join('/content', CSV_FILENAME),
                            format='csv',
                            fields = [('Insult', LABEL), ('Comment', TEXT)],
                            skip_header=True)

print(vars(train[0]),vars(train[1]),vars(train[2]))

TEXT.build_vocab(train)

Output:
You don't need to manually download&upload the file as suggested by @knoop. As I provided in my previous answer to your question, you should not use the link with open. You should change to uc?export=download instead. In this way, the correct CSV_GDRIVE_URL should be:

CSV_GDRIVE_URL = 'https://drive.google.com/uc?export=download&id=1eWMjusU3H34m0uml5SdJvYX6gQuB8zta'

Then, if you cat it, you'll get:

,Insult,Date,Comment
11,0,20120530044519Z,"""Be careful,Jimbo.OG has a fork with your name on it."""
1478,0,,"""@dale0987654321 @\\'Micah Carter So you\\'re saying those claims are all false...?  I don\\'t know... just from my understanding these are in his book.... but I can be wrong... I\\'m just saying if they are true....should I just assume he\\'s racist?\\n\\nFrom \\'Dreams of My Father\\',""I CEASED TO ADVERISE MY MOTHER\\'S RACE AT THE AGE OF12 OR 13, when I began to suspect that by doing so I was ingratiating myself to whites""From Dreams of My Father, "" I FOUND A SOLACE IN NURSING A PERVASIVE SENSE OF GRIEVANCE AND ANIMOSITY AGAINST MY MOTHER\\'S RACE"".From \\'Dreams of my Father\\', ""The emotion between the races could never be pure..... the THE OTHER RACE (WHITE) WOULD ALWAYS REMAIN JUST THAT: MENACING, ALIEN AND APART""From Dreams Of My Father, ""never emulate white men and brown men whose fates didn\\'t speak to my own. IT WAS INTO MY FATHER\\'S IMAGE , THE BLACK MAN, THE SON OF AFRICA, THAT I\\'D PACKED ALL THE ATTRIBUTES I SOUGHT IN MYSELF.From Dreams Of My Father:""THAT HATE HADN\\'T GONE AWAY,"" he wrote, BLAMING WHITE PEOPLE,- SOME CRUEL, SOME IGNORANT, sometimes a single face, sometimes just a faceless image of a system claiming power over our lives."""""
1983,0,20120610152024Z,"""Only fucking with number 2... that's prolly the only pair i can pull off \xa0out these."""
3201,1,,"""S T F U and pay your dues"""
3549,0,20120530024115Z,"""Worry about your current opponent...."""
3402,1,20120618222736Z,"""And while you're at it, please prove how your ""point"" related to past draconian laws that banned inter-racial marriage?\xa0 That intent was to ""ban"" mixed races.\xa0 Just stop it please! You are insulting intelligence."""
2233,0,20120620034735Z,"""I'd rather have an old, straight white guy than the stupid fcking zebra we have now....everything he touches turns toshit............"""
3219,1,20120619020631Z,"""Only a black racists idiot would claim that the NAACP is not racist in todays world, especially since you're a member.\n\nAnd I don't give a 'bo' whether you take that personally or not."""
3148,0,,"""Fight for money !!! go ahead ,kill yourselfs, all for greed,common, take decision that is not in accordance with what the supreme court says, find some connection by the INEC ,and say to yourself ,HO IS MY TURN TO LOOT NOT YOURS,OR I KILL U.bounch of shameless idiots."""
386,0,,"""The cleverest comment that I have seen for a long time, I think that it was in the Mail\\n\\n \\n\\n\\'Arshavin is the worst Russian Sub since the Kursk"""""

And TEXT.build_vocab(train) will run successfully.


Instruction: Pytorch sending inputs/targets to device

Input:
Currently using the pycocotools 2.0 library.

My train_loader is:

train_loader, test_loader = get_train_test_loader('dataset', batch_size=16, num_workers=4)

However the training lines of code:

        for i, data in enumerate(train_loader, 0):
            images, targets = data
            images = images.to(device)
            targets = targets.to(device)

Results in error. Variables data, images, and targets are all of class tuple

Traceback (most recent call last):
  File "train.py", line 40, in <module>
    images = images.to(device)
AttributeError: 'tuple' object has no attribute 'to'

How can I properly send these to cuda device?'

Edit:

I can send images[0].to(device) no problem. How can I send the rest?

Output:
You should open the for loop with as many items as your dataset returns at each iteration.
Here is an example to illustrate my point:

Consider the following dataset:

class CustomDataset:
    def __getitem__(self, index):
        ...
        return a, b, c

Notice that it returns 3 items at each iteration.

Now let us make a dataloader out of this:

from torch.utils.data import DataLoader
train_dataset = CustomDataset()
train_loader = DataLoader(train_dataset, batch_size=50, shuffle=True)

Now when we use train_loader we should open the for loop with 3 items:

for i, (a_tensor, b_tensor, c_tensor) in enumerate(train_loader):
   ...

Inside the for loop's context, a_tensor, b_tensor, c_tensor will be tensors with 1st dimension 50 (batch_size).

So, based on the example that you have given, it seems that whatever dataset class your get_train_test_loader function is using has some problem. It is always better to seprately instantiate the dataset and then create the dataloader rather than use a blanket function like the one you have.


Instruction: How can I parallelize a for loop for use in PyTorch?

Input:
I realize that for loops are slow with Python in general. I have some code that messes around with some tensors:

            for batch_index, mask_batch in enumerate(mask):
                mask_len = torch.sum(mask_batch).int()

                if mask_len == 0:
                    side_input = torch.zeros((max_inp_len, side_input.shape[1])).to(mask.device)
                else:

                    m_nonzero = mask_batch.nonzero().flatten()
                    first_nonzero = m_nonzero[0]
                    last_nonzero = m_nonzero[-1]

                    if side == 'left':
                        end_index = first_nonzero - 1
                        start_index = 0
                    elif side == 'right':
                        start_index = last_nonzero + 1
                        end_index = inputs[batch_index].size(1)

                    side_input = inputs[batch_index][start_index:end_index]

                    if end_index - start_index < max_inp_len:
                        pad_zeros = torch.zeros(
                            (max_inp_len - side_input.shape[0], side_input.shape[1])).to(mask.device)
                        if side == 'left':
                            side_input = torch.cat((pad_zeros, side_input), 0)
                        elif side == 'right':
                            side_input = torch.cat((side_input, pad_zeros), 0)

                side_inputs.append(side_input)

        return torch.stack(side_inputs)

I feel like this loop is REALLY slowing things down. Is there some way for me to do it without the loop?

Output:
Python does not have true parallelism within any given process. You would have to spawn a ProcessPool and make the inside of your loop a function taking batch_index, mask_batch, then map that function over the mask object in your current for loop. Thing is, I don't know if PyTorch will play nicely with this.

Like so

def f(batch_index, mask_batch):
    mask_len = torch.sum(mask_batch).int()

    if mask_len == 0:
        side_input = torch.zeros((max_inp_len, side_input.shape[1])).to(mask.device)
    else:
        m_nonzero = mask_batch.nonzero().flatten()
        first_nonzero = m_nonzero[0]
        last_nonzero = m_nonzero[-1]

        if side == 'left':
            end_index = first_nonzero - 1
            start_index = 0
        elif side == 'right':
            start_index = last_nonzero + 1
            end_index = inputs[batch_index].size(1)

            side_input = inputs[batch_index][start_index:end_index]

            if end_index - start_index < max_inp_len:
                pad_zeros = torch.zeros((max_inp_len - side_input.shape[0], side_input.shape[1])).to(mask.device)
                if side == 'left':
                    side_input = torch.cat((pad_zeros, side_input), 0)
                elif side == 'right':
                    side_input = torch.cat((side_input, pad_zeros), 0)
    return side_input

The other things you can look at are further vectorizing the code. Most things in PyTorch and Numpy can be vectorized away by using builtin functions and adding another dimension onto your tensors that represents the "loop" dimension. This will allow PyTorch to handle the parallelism for you.

PyTorch may have a concept of devices that you can put different iterations of the loop on, again this will require you to make a function for this loop and maybe take the device it goes on as an input.

Lastly you can look into just in time compliation like Numba or torch.jit to perform auto-vectorization for you.

None of this will work (most likely) if the mask is of an unknown length. If it is of a known length, I think vectorization, as hard as it is, is likely your best choice.


Instruction: What is running loss in PyTorch and how is it calculated

Input:
I had a look at this tutorial in the PyTorch docs for understanding Transfer Learning. There was one line that I failed to understand.

After the loss is calculated using loss = criterion(outputs, labels),  the running loss is calculated using running_loss += loss.item() * inputs.size(0) and finally, the epoch loss is calculated using running_loss / dataset_sizes[phase].

Isn't loss.item() supposed to be for an entire mini-batch (please correct me if I am wrong). i.e, if the batch_size is 4, loss.item() would give the loss for the entire set of 4 images. If this is true, why is loss.item() being multiplied  with inputs.size(0) while calculating running_loss? Isn't this step like an extra multiplication in this case?

Any help would be appreciated. Thanks!

Output:
It's because the loss given by CrossEntropy or other loss functions is divided by the number of elements i.e. the reduction parameter is mean by default.
torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean')

Hence, loss.item() contains the loss of entire mini-batch, but divided by the batch size. That's why loss.item() is multiplied with batch size, given by inputs.size(0), while calculating running_loss.


Instruction: NCCL Connection Failed Using PyTorch Distributed

Input:
I am trying to send a PyTorch tensor from one machine to another with torch.distributed. The dist.init_process_group function works properly. However, there is a connection failure in the dist.broadcast function. Here is my code on node 0:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 0
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.from_numpy(np.random.rand(3, 3)).cuda()
dist.broadcast(tensor=a, src=0)

Here is my code on node 1:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 1
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.zeros((3,3)).cuda()
dist.broadcast(tensor=a, src=0)

I set NCCL_DEBUG=INFO before running the code. Here is the information I got on Node 1:

iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying

iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143<59811> failed : Connection timed out
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:100 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:326 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:695 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:951 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO misc/group.cc:69 -> 2 [Async thread]
Traceback (most recent call last):
  File "test_dist_1.py", line 25, in <module>
    dist.broadcast(tensor=a, src=0)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 806, in broadcast
    work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-req-build-58y_cjjl/torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error

And Node 0 seems to stuck in function dist.broadcast:

iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff

Can anyone help me with this? How can I send the tensor from node 0 to node 1? I would really appreciate any help!

Output: I found that the two servers that I used were not under the same VPC. Therefore, they can never communicate. So I used other servers under the same VPC and it worked.


Instruction: PyTorch tensor advanced indexing

Input:
Say i have one matrix and one vector as follows:

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

y = torch.tensor([0, 2, 1])

is there a way to slice it x[y] so the result is:

res = [1, 6, 8]

so basically i take the first element of y and take the element in x that corresponds to the first row and the elements' column.

Cheers

Output:
You can specify the corresponding row index as:

import torch
x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

y = torch.tensor([0, 2, 1])

x[range(x.shape[0]), y]
tensor([1, 6, 8])


Instruction: How can I add an element to a PyTorch tensor along a certain dimension?

Input: I have a tensor inps, which has a size of [64, 161, 1] and I have some new data d which has a size of [64, 161]. How can I add d to inps such that the new size is [64, 161, 2]?

Output:
There is a cleaner way by using .unsqueeze() and torch.cat(), which makes direct use of the PyTorch interface:

import torch

# create two sample vectors
inps = torch.randn([64, 161, 1])
d = torch.randn([64, 161])

# bring d into the same format, and then concatenate tensors
new_inps = torch.cat((inps, d.unsqueeze(2)), dim=-1)
print(new_inps.shape)  # [64, 161, 2]

Essentially, unsqueezing the second dimension already brings the two tensors into the same shape; you just have to be careful to unsqueeze along the right dimension.
Similarly, the concatenation is unfortunately named differently from the otherwise similarly named NumPy function, but behave the same. Note that instead of letting torch.cat figure out the dimension by providing dim=-1, you can also explicitly provide the dimension to concatenate along, in this case by replacing it with dim=2.

Keep in mind the difference between concatenation and stacking, which is helpful for similar problems with tensor dimensions.


Instruction: What is the difference between tensor[:] and tensor in pytorch?

Input:
import numpy as np
import time
features, labels = d2l.get_data_ch7()

def init_adam_states():
    v_w, v_b = torch.zeros((features.shape[1], 1),dtype=torch.float32), torch.zeros(1, dtype=torch.float32)
    s_w, s_b = torch.zeros((features.shape[1], 1),dtype=torch.float32), torch.zeros(1, dtype=torch.float32)
    return ((v_w, s_w), (v_b, s_b))

def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    for p, (v, s) in zip(params, states):
        v[:] = beta1 * v + (1 - beta1) * p.grad.data
        s = beta2 * s + (1 - beta2) * p.grad.data**2
        v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
        s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
        p.data -= hyperparams['lr'] * v_bias_corr / (torch.sqrt(s_bias_corr) + eps)
    hyperparams['t'] += 1

def train_ch7(optimizer_fn, states, hyperparams, features, labels, batch_size=10, num_epochs=2):
    # 初始化模型
    net, loss = d2l.linreg, d2l.squared_loss

    w = torch.nn.Parameter(torch.tensor(np.random.normal(0, 0.01, size=(features.shape[1], 1)), dtype=torch.float32),
                           requires_grad=True)
    b = torch.nn.Parameter(torch.zeros(1, dtype=torch.float32), requires_grad=True)

    def eval_loss():
        return loss(net(features, w, b), labels).mean().item()

    ls = [eval_loss()]
    data_iter = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(features, labels), batch_size, shuffle=True)

    for _ in range(num_epochs):
        start = time.time()
        print(w)
        print(b)
        for batch_i, (X, y) in enumerate(data_iter):
            l = loss(net(X, w, b), y).mean()  # 使⽤平均损失

            # 梯度清零
            if w.grad is not None:
                w.grad.data.zero_()
                b.grad.data.zero_()

            l.backward()
            optimizer_fn([w, b], states, hyperparams)  # 迭代模型参数
            if (batch_i + 1) * batch_size % 100 == 0:
                ls.append(eval_loss())  # 每100个样本记录下当前训练误差

    # 打印结果和作图
    print('loss: %f, %f sec per epoch' % (ls[-1], time.time() - start))
    d2l.set_figsize()
    d2l.plt.plot(np.linspace(0, num_epochs, len(ls)), ls)
    d2l.plt.xlabel('epoch')
    d2l.plt.ylabel('loss')
train_ch7(adam, init_adam_states(), {'lr': 0.01, 't': 1}, features, labels)

I want to implement the Adam algorithm in the follow code and I feel confused in the function named adam.

v = beta1 * v + (1 - beta1) * p.grad.data
s = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 1.
figure 1

v[:] = beta1 * v + (1 - beta1) * p.grad.data
s = beta2 * s + (1 - beta2) * p.grad.data**2
or
v = beta1 * v + (1 - beta1) * p.grad.data
s[:] = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 2.
figure 2

v[:] = beta1 * v + (1 - beta1) * p.grad.data
s[:] = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 3.
figure 3

The loss function curve in case 3 has always been smoother than that in case 1.
The loss function curve in case 2 sometimes can't converge.

Why is different?

Output:
To answer the first question,

v = beta1 * v + (1 - beta1) * p.grad.data

is an out-of-place operation. Remember that python variables are references to objects. By assigning a new value to variable v, the underlying object which v referred to before this assignment will not be changed. Instead the expression beta1 * v + (1 - beta1) * p.grad.data results in a new tensor which is then referred to by v.

On the other hand

v[:] = beta1 * v + (1 - beta1) * p.grad.data

is an in-place operation. After this operation v still refers to the same underlying object, and the elements of that tensor are modified and replaced with the values of the new tensor beta1 * v + (1 - beta1) * p.grad.data.

Take a look at the following 3 lines to see why this matters

for p, (v, s) in zip(params, states):
    v[:] = beta1 * v + (1 - beta1) * p.grad.data
    s[:] = beta2 * s + (1 - beta2) * p.grad.data**2

v and s are actually referring to tensors which are stored in states. If we do in-place operations then the values in states are changed to reflect the value assigned to v[:] and s[:].

If out-of-place operations are used then the values in states remain unchanged.


Instruction: How I can swap 3 dimensions with each other in Pytorch?

Input:
I have a a= torch.randn(28, 28, 8) and I want to swap the dimensions of the tensor and move the third dimension to the first place, first one to the second place and the second one to the third place. I used  b = a.transpose(2, 0, 1) , but I received this error:

TypeError: transpose() received an invalid combination of arguments - got (int, int, int), but expected one of:
 * (name dim0, name dim1)
 * (int dim0, int dim1)

Should I use transpose several times, each time only to swap two dimensions? Is there any way that I can swap all at once?

Thanks.

Output:
You can use Pytorch's permute() function to swap all at once,

>>>a = torch.randn(28, 28, 8)
>>>b = a.permute(2, 0, 1)
>>>b.shape
torch.Size([8, 28, 28])


Instruction: How do I remove elements from a 3D tensor with PyTorch?

Input: I have a tensor with shape torch.Size([4, 161, 325]). How do I remove the first element along dim=2 so that the resulting tensor has a shape of torch.Size([4, 161, 324])?

Output:
You can use simple slicing,

>>>a = torch.randn(4, 161, 325)
>>>b = a[:, :, 1:]
>>>b.shape
torch.Size([4, 161, 324])


Instruction: Transformer Pipeline for NER returns partial words with ##s

Input:
How should I interpret the partial words with '##'s in them returned by the Transformer NER pipelines? Other tools like Flair and SpaCy return the word and their tag. I have worked with the CONLL dataset before and never noticed anything like this. Moreover, why are words being divided like this?

Example from the HuggingFace:

from transformers import pipeline

nlp = pipeline("ner")

sequence = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
           "close to the Manhattan Bridge which is visible from the window."

print(nlp(sequence))

Output:

[
    {'word': 'Hu', 'score': 0.9995632767677307, 'entity': 'I-ORG'},
    {'word': '##gging', 'score': 0.9915938973426819, 'entity': 'I-ORG'},
    {'word': 'Face', 'score': 0.9982671737670898, 'entity': 'I-ORG'},
    {'word': 'Inc', 'score': 0.9994403719902039, 'entity': 'I-ORG'},
    {'word': 'New', 'score': 0.9994346499443054, 'entity': 'I-LOC'},
    {'word': 'York', 'score': 0.9993270635604858, 'entity': 'I-LOC'},
    {'word': 'City', 'score': 0.9993864893913269, 'entity': 'I-LOC'},
    {'word': 'D', 'score': 0.9825621843338013, 'entity': 'I-LOC'},
    {'word': '##UM', 'score': 0.936983048915863, 'entity': 'I-LOC'},
    {'word': '##BO', 'score': 0.8987102508544922, 'entity': 'I-LOC'},
    {'word': 'Manhattan', 'score': 0.9758241176605225, 'entity': 'I-LOC'},
    {'word': 'Bridge', 'score': 0.990249514579773, 'entity': 'I-LOC'}
]

Output:
Pytorch transformers and BERT make 2 tokens, the regular words as tokens and words + sub-words as tokens; which divide words by their base meaning + their complement, addin "##" at the start.

Let's say you have the phrease: I like hugging animals

The first set of tokens would be:

["I", "like", "hugging", "animals"]

And the second list with the sub-words would be:

["I", "like", "hug", "##gging", "animal", "##s"]

You can learn more here:
https://www.kaggle.com/funtowiczmo/hugging-face-tutorials-training-tokenizer


Instruction: PyTorch broadcast multiplication of 4D and 2D matrix?

Input:
How do I broadcast to multiply these two matrices together?

x: torch.Size([10, 120, 180, 30]) # (N, H, W, C)
W: torch.Size([64, 30]) # (Y, C)

The output should be:

(10, 120, 180, 64) == (N, H, W, Y)

Output:
I assume x is some kind of example with batches and w matrix is the corresponding weight. In this case you could simply do:

out = x @ w.T

which is a tensor multiplication, not an element-wise one. You can't do element-wise multiplication to get such shape and this operation would not make sense. All you could do is to unsqueeze both of the matrics in some way to have their shape broadcastable and apply some operation over dimension you don't want for some reason like this:

x : torch.Size([10, 120, 180, 30, 1])
W: torch.Size([1, 1, 1, 30, 64]) # transposition would be needed as well

After such unsqueezing you could do x*w and sum or mean along the third dim to get desired shape.

For clarity, both ways are not equivalent.


Instruction: PyTorch DataLoader shuffle

Input:
I did an experiment and I did not get the result I was expecting.

For the first part, I am using

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
                                          shuffle=False, num_workers=0)

I save trainloader.dataset.targets to the variable a, and trainloader.dataset.data to the variable b before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable c, and trainloader.dataset.data to the variable d. Finally, I check a == c and b == d and they both give True, which was expected because the shuffle parameter of the DataLoader is False.

For the second part, I am using

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
                                          shuffle=True, num_workers=0)

I save trainloader.dataset.targets to the variable e, and trainloader.dataset.data to the variable f before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable g, and trainloader.dataset.data to the variable h. I expect e == g and f == h to be both False since shuffle=True, but they give True again. What am I missing from the definition of DataLoader class?

Output:
I believe that the data that is stored directly in the trainloader.dataset.data or .target will not be shuffled, the data is only shuffled when the DataLoader is called as a generator or as iterator

You can check it by doing next(iter(trainloader)) a few times without shuffling and with shuffling and they should give different results

import torch
import torchvision

transform = torchvision.transforms.Compose([
        torchvision.transforms.ToTensor(),
        ])
MNIST_dataset = torchvision.datasets.MNIST('~/Desktop/intern/',download = True, train = False,
                                           transform = transform)
dataLoader = torch.utils.data.DataLoader(MNIST_dataset,
                                         batch_size = 128,
                                         shuffle = False,
                                         num_workers = 10)
target = dataLoader.dataset.targets

MNIST_dataset = torchvision.datasets.MNIST('~/Desktop/intern/',download = True, train = False,
                                           transform = transform)

dataLoader_shuffled= torch.utils.data.DataLoader(MNIST_dataset,
                                         batch_size = 128,
                                         shuffle = True,
                                         num_workers = 10)

target_shuffled = dataLoader_shuffled.dataset.targets

print(target == target_shuffled)

_, target = next(iter(dataLoader));
_, target_shuffled = next(iter(dataLoader_shuffled))

print(target == target_shuffled)

This will give :

tensor([True, True, True,  ..., True, True, True])
tensor([False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False,  True,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False,  True, False, False, False, False, False,
        False,  True, False, False, False, False, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False,  True,  True, False, False, False, False,
        False, False, False, False, False, False, False, False, False, False,
        False, False, False, False, False,  True, False, False,  True, False,
        False, False, False, False, False, False, False, False])

However the data and label stored in data and target is a fixed list and since you are trying to access it directly, they will not be shuffled.


Instruction: Append feature maps to intermediate layers of network in PyTorch

Input:
I have the following PyTorch network architecture shown below. It takes in the input of size 16x8x2048, and gives an output of size 256x128x3. I have a feature map that I want to append along the channel dimension after every upsampling layer in the below architecture. I would be scaling the feature maps to the appropriate resolution before appending. How can I append these feature maps in the intermediate layers of the network?

model = []
model += [Conv2dBlock(2048, 256, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(256, 128, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(128, 64, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(64, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(32, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model += [Conv2dBlock(32, 3, 3, 1, 1, norm='none', activation=activ, pad_type=pad_type)]
model = nn.Sequential(*model)

Output:
So the solution was just changing the way in which the network is written as pointed out by https://stackoverflow.com/users/10229754/mercury. The following change shows a way in which the network can be written so that is possible to append additional input. In the forward() function, the input can be directly appended using torch.cat()-

conv1 = Conv2dBlock(2063, 256, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)
up1 = nn.Upsample(scale_factor=2, mode='bilinear')
conv2 = Conv2dBlock(256, 128, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)
up2 = nn.Upsample(scale_factor=2, mode='bilinear')
conv3 = Conv2dBlock(128, 64, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)
up3 = nn.Upsample(scale_factor=2, mode='bilinear')
conv4 = Conv2dBlock(64, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)
up4 = nn.Upsample(scale_factor=2, mode='bilinear')
conv5 = Conv2dBlock(32, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)
conv6 = Conv2dBlock(32, 3, 3, 1, 1, norm='none', activation=activ, pad_type=pad_type)


Instruction: Interpretation of in_channels and out_channels in Conv2D in Pytorch Convolution Neural Networks (CNN)

Input:
Let us suppose I have a CNN with starting 2 layers as:

inp_conv = Conv2D(in_channels=1,out_channels=6,kernel_size=(3,3))

Please correct me if I am wrong but I think what this line if code can be thought as that

There is a single grayscale image coming as input where we have to use 6 different kernals of same size (3,3) to make 6 different feature maps from a single image.

And if I have a second Conv2D layer just after first one as

second_conv_connected_to_inp_conv = Conv2D(in_channels=6,out_channels=12,kernel_size=(3,3))

What does this mean in terms of out_channels? Is there going to be 12 new feature maps for each of the 6 feature maps coming as output from first layer OR are there going to be a total of 12 feature maps from 6 incoming features?

Output:
For each out_channel, you have a set of kernels for each in_channel.
Equivalently, each out_channel has an in_channel x height x width kernel:
for i in nn.Conv2d(in_channels=2, out_channels=3, kernel_size=(4, 5)).parameters():
    print(i)

Output:
Parameter containing:
tensor([[[[-0.0012,  0.0848, -0.1301, -0.1164, -0.0609],
      [ 0.0424, -0.0031,  0.1254, -0.0140,  0.0418],
      [-0.0478, -0.0311, -0.1511, -0.1047, -0.0652],
      [ 0.0059,  0.0625,  0.0949, -0.1072, -0.0689]],

     [[ 0.0574,  0.1313, -0.0325,  0.1183, -0.0255],
      [ 0.0167,  0.1432, -0.1467, -0.0995, -0.0400],
      [-0.0616,  0.1366, -0.1025, -0.0728, -0.1105],
      [-0.1481, -0.0923,  0.1359,  0.0706,  0.0766]]],

    [[[ 0.0083, -0.0811,  0.0268, -0.1476, -0.1142],
      [-0.0815,  0.0998,  0.0927, -0.0701, -0.0057],
      [ 0.1011,  0.1572,  0.0628,  0.0214,  0.1060],
      [-0.0931,  0.0295, -0.1226, -0.1096, -0.0817]],

     [[ 0.0715,  0.0636, -0.0937,  0.0478,  0.0868],
      [-0.0200,  0.0060,  0.0366,  0.0981,  0.1518],
      [-0.1218, -0.0579,  0.0621,  0.1310,  0.1376],
      [ 0.1395,  0.0315, -0.1375,  0.0145, -0.0989]]],

    [[[-0.1474,  0.1405,  0.1202, -0.1577,  0.0296],
      [-0.0266, -0.0260, -0.0724,  0.0608, -0.0937],
      [ 0.0580,  0.0800,  0.1132,  0.0591, -0.1565],
      [-0.1026,  0.0789,  0.0331, -0.1233, -0.0910]],

     [[ 0.1487,  0.1065, -0.0689, -0.0398, -0.1506],
      [-0.0028, -0.1191, -0.1220, -0.0087,  0.0237],
      [-0.0648,  0.0938, -0.0962,  0.1435,  0.1084],
      [-0.1333, -0.0394,  0.0071,  0.0231,  0.0375]]]], requires_grad=True)
Parameter containing:
tensor([ 0.0620,  0.0095, -0.0771], requires_grad=True)

A more detailed example going from 1 channel input, through 2 and 4 channel convolutions:
import torch

torch.manual_seed(0)

input0 = torch.randint(-1, 1, (1, 1, 8, 8)).type(torch.FloatTensor)
print('input0:', input0.size())
print(input0.data)

layer0 = nn.Conv2d(in_channels=1, out_channels=2, kernel_size=2, stride=2, padding=0, bias=False)
print('\nlayer1:')
for i in layer0.parameters():
    print(i.size())
    i.data = torch.randint(-1, 1, i.size()).type(torch.FloatTensor)
    print(i.data)

output0 = layer0(input0)
print('\noutput0:', output0.size())
print(output0.data)

print('\nlayer1:')
layer1 = nn.Conv2d(in_channels=2, out_channels=4, kernel_size=2, stride=2, padding=0, bias=False)
for i in layer1.parameters():
    print(i.size())
    i.data = torch.randint(-1, 1, i.size()).type(torch.FloatTensor)
    print(i.data)
output1 = layer1(output0)
print('\noutput1:', output1.size())
print(output1.data)

output:
input0: torch.Size([1, 1, 8, 8])
tensor([[[[-1.,  0.,  0., -1.,  0.,  0.,  0.,  0.],
  [ 0.,  0.,  0., -1., -1.,  0., -1., -1.],
  [-1., -1., -1.,  0., -1.,  0.,  0., -1.],
  [-1.,  0.,  0.,  0.,  0., -1.,  0., -1.],
  [ 0., -1.,  0.,  0., -1.,  0.,  0., -1.],
  [-1.,  0., -1.,  0.,  0.,  0.,  0.,  0.],
  [-1.,  0., -1.,  0.,  0.,  0.,  0., -1.],
  [ 0., -1., -1.,  0.,  0., -1.,  0., -1.]]]])

layer1:
torch.Size([2, 1, 2, 2])
tensor([[[[-1., -1.],
          [-1.,  0.]]],

        [[[ 0., -1.],
          [ 0., -1.]]]])

output0: torch.Size([1, 2, 4, 4])
tensor([[[[1., 1., 1., 1.],
          [3., 1., 1., 1.],
          [2., 1., 1., 1.],
          [1., 2., 0., 1.]],

         [[0., 2., 0., 1.],
          [1., 0., 1., 2.],
          [1., 0., 0., 1.],
          [1., 0., 1., 2.]]]])

layer1:
torch.Size([4, 2, 2, 2])
tensor([[[[-1., -1.],
          [-1., -1.]],

         [[ 0., -1.],
          [ 0., -1.]]],

        [[[ 0.,  0.],
          [ 0.,  0.]],

         [[ 0., -1.],
          [ 0.,  0.]]],

        [[[ 0.,  0.],
          [-1.,  0.]],

         [[ 0., -1.],
          [-1.,  0.]]],

        [[[-1., -1.],
          [-1., -1.]],

         [[ 0.,  0.],
          [-1., -1.]]]])

output1: torch.Size([1, 4, 2, 2])
tensor([[[[-8., -7.],
          [-6., -6.]],

         [[-2., -1.],
          [ 0., -1.]],

         [[-6., -3.],
          [-2., -2.]],

         [[-7., -7.],
          [-7., -6.]]]])

Breaking down the linear algebra:
np.sum(
    # kernel for layer1, in_channel 0, out_channel 0
    # multiplied by output0, channel 0, top left corner
    (np.array([[-1., -1.],
              [-1., -1.]]) * \
    np.array([[1., 1.],
              [3., 1.]])) + \

    # kernel for layer1, in_channel 1, out_channel 0
    # multiplied by output0, channel 1, top left corner
    (np.array([[ 0., -1.],
              [ 0., -1.]]) * \
    np.array([[0., 2.],
              [1., 0.]]))
)

This will be equal to output1, channel 0, top left corner:
-8.0


Instruction: How to use custom torch.autograd.Function in nn.Sequential model

Input:
Is there any way that I can use custom torch.autograd.Function in a nn.Sequential object or should I use explicitly an nn.Module object with forward function. Specifically I am trying to implement a sparse autoencoder and I need to add L1 distance of the code(hidden representation) to the loss.
I have defined custom torch.autograd.Function L1Penalty below then tried to use it inside a nn.Sequential object as below. However when I run I got the error TypeError: __main__.L1Penalty is not a Module subclass How can I solve this issue?

class L1Penalty(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, l1weight = 0.1):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input, None

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(ctx.l1weight)
        grad_input+=grad_output
        return grad_input

model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 6),
    nn.ReLU(),
    # sparsity
    L1Penalty(),
    nn.Linear(6, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU()
).to(device)

Output:
The right way to do that would be this
import torch, torch.nn as nn

class L1Penalty(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, l1weight = 0.1):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(ctx.l1weight)
        grad_input+=grad_output
        return grad_input

Creating a Lambda class that acts as a wrapper
class Lambda(nn.Module):
    """
    Input: A Function
    Returns : A Module that can be used
        inside nn.Sequential
    """
    def __init__(self, func):
        super().__init__()
        self.func = func

    def forward(self, x): return self.func(x)

TA-DA!
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 6),
    nn.ReLU(),
    # sparsity
    Lambda(L1Penalty.apply),
    nn.Linear(6, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU())

a = torch.rand(50,10)
b = model(a)
print(b.shape)


Instruction: How to change the values of a 2d tensor in certain rows and columns

Input:
Suppose I have an all-zero mask tensor like this:

mask = torch.zeros(5,3, dtype=torch.bool)

Now I want to set the value of mask at the intersection of the following rows and cols indices to True:

rows = torch.tensor([0,2,4])
cols = torch.tensor([1,2])

I would like to produce the following result:

tensor([[False, True,  True ],
        [False, False, False],
        [False, True,  True ],
        [False, False, False],
        [False, True,  True ]])

When I try the following code, I receive an error:

mask[rows, cols] = True

IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [3], [2]

How can I do that efficiently in PyTorch?

Output:
You need proper shape for that you can use torch.unsqueeze

mask = torch.zeros(5,3, dtype=torch.bool)
mask[rows, cols.unsqueeze(1)] = True
mask
tensor([[False,  True,  True],
        [False, False, False],
        [False,  True,  True],
        [False, False, False],
        [False,  True,  True]])

or torch.reshape

mask[rows, cols.reshape(-1,1)] = True
mask
tensor([[False,  True,  True],
        [False, False, False],
        [False,  True,  True],
        [False, False, False],
        [False,  True,  True]])


Instruction: Pytorch Error: Optimizer got an Empty Parameter List on Linux server

Input:
I am running FasterRCNN code and was experimenting with various backbones (resnet18...101). I have created FasterRCNN class with init function. While training on my linux server I am facing the "Empty Parameter" error (can be seen in the attached image 1)

While the same code is running without any error on my local machine.
I am using Anaconda distribution 4.0.1 and Python 3.7 with pytorch 1.2
Below you can refer my code (model and train) -

class Faster_RCNN(nn.Module):
    def __init__(self, num_classes=2, backbone = 'mobilenet_v2', test= False):
        super(Faster_RCNN, self).__init__()
        self.classes = num_classes # replace the classifier with a new one, that has num_classes 2 for our use.
        # 1 class (bar) + background
        self.test = test
        self.backbone = backbone
        print('....Initializing Model....\n')
        print(f'....This model uses {self.backbone} as backbone....\n')
        if self.backbone == 'mobilenet_v2':
            backbone = torchvision.models.mobilenet_v2(pretrained=True).features
            backbone.out_channels = 1280
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

        if self.backbone == 'resnet18':
            bb = torchvision.models.resnet18(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 512
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

        if self.backbone == 'resnet34':
            bb = torchvision.models.resnet34(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 512
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)
        if self.backbone == 'resnet50':
            bb = torchvision.models.resnet50(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 2048
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)
        if self.backbone == 'resnet101':
            bb = torchvision.models.resnet50(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 2048
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

    def forward(self, x,y):
        if self.test:
            pred = self.model(x)
        else:
            pred = self.model(x,y)
        return pred

Here is the train code:

import dataloader
import model as md
import numpy as np
#import matplotlib.pyplot as plt
import torch
from torch.optim import Adam
from torch.utils.tensorboard import SummaryWriter
import argparse
writer = SummaryWriter()

bkbone = ['resnet18','resenet34','resenet50','resnet101']

for bbone in bkbone:

    ap = argparse.ArgumentParser()
    ap.add_argument("-b", "--backbone", required = False, default = bbone, help = "resenet18, resnet34, resnet50, resenet101 and mobilenet_v2 can be given")
    args = vars(ap.parse_args())
    epochs = 1000
    batch_size = 10
    lr = 0.005
    path = 'D:\\beantech_Data\\objtect_detection'
    # path = '/media/TBDataStudent/pankaj/beantech/object_detection'
    data= dataloader.Bar(root=path, batch_size=batch_size)

    model = md.Faster_RCNN(backbone=args['backbone']).cuda()
    model.train()

    optimizer = Adam(model.parameters(), lr = lr, weight_decay=0.0005)

    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=3,gamma=0.1)

    ### Train ###
    for epoch in range(epochs):
        loss_classifier=[]
        loss_box_reg =[]
        loss_objectness = []
        loss_rpn_box_reg = []
        Tloss = []

        for _, (img, label) in enumerate(data.train_loader):
            images = [image.cuda() for image in img ]
            for i in range(len(label)):
                label[i]['boxes']=label[i]['boxes'].cuda()
                label[i]['labels']=label[i]['labels'].cuda()
            optimizer.zero_grad()
            output = model(images, label)

            l1, l2, l3, l4 = output
            loss_classifier.append(output[l1].item())
            loss_box_reg.append(output[l2].item())
            loss_objectness.append(output[l3].item())
            loss_rpn_box_reg.append(output[l4].item())

            loss = sum(list(output.values()))
            loss.backward()
            optimizer.step()
            Tloss.append(loss.item())

        writer.add_scalar(l1, np.mean(loss_classifier), epoch)
        writer.add_scalar(l2, np.mean(loss_box_reg), epoch)
        writer.add_scalar(l3, np.mean(loss_objectness), epoch)
        writer.add_scalar(l4, np.mean(loss_rpn_box_reg), epoch)
        writer.add_scalar('Total Loss', np.mean(Tloss), epoch)

        print(f'\n\n --{l1}: {np.mean(loss_classifier)}\n --{l2}: {np.mean(loss_box_reg)}\n --{l3}: {np.mean(loss_objectness)}\n --{l4}: {np.mean(loss_rpn_box_reg)}')
        print(f'Total loss of epoch {epoch}is: {np.mean(Tloss)}')
        writer.close()

    torch.save(model.state_dict(), f'FasterRCNN_{args["backbone"]}'+'.pt')

Output:
There was pytorch version mismatch. I updated the server with latest pytorch and everything is running smooth.
thanks everyone for the help.


Instruction: Asking gpt-2 to finish sentence with huggingface transformers

Input:
I am currently generating text from left context using the example script run_generation.py of the huggingface transformers library with gpt-2:

$ python transformers/examples/run_generation.py \
  --model_type gpt2 \
  --model_name_or_path gpt2 \
  --prompt "Hi, " --length 5

=== GENERATED SEQUENCE 1 ===
Hi,  could anyone please inform me

I would like to generate short complete sentences. Is there any way to tell the model to finish a sentence before length words?

Note: I don't mind changing model, but would prefer an auto-regressive one.

Output:
Unfortunately there is no way to do so. You can set the length parameter to a greater value and then just discard the incomplete part at the end.
Even GPT3 doesn't support completing a sentence before a specific length. GPT3 support "sequences" though. Sequences force the model to stop when certain condition is fulfilled. You can find more information about in thi article


Instruction: training by batches leads to more over-fitting

Input:
I'm training a sequence to sequence  (seq2seq) model and I have different values to train on for the input_sequence_length.

For values 10 and 15, I get acceptable results but when I try to train with 20, I get memory errors so I switched the training to train by batches but the model over-fit and the validation loss explodes, and even with the accumulated gradient I get the same behavior, so I'm looking for hints and leads to more accurate ways to do the update.

Here is my training function (only with batch section) :

    if batch_size is not None:
        k=len(list(np.arange(0,(X_train_tensor_1.size()[0]//batch_size-1), batch_size )))
        for epoch in range(num_epochs):
            optimizer.zero_grad()
            epoch_loss=0
            for i in list(np.arange(0,(X_train_tensor_1.size()[0]//batch_size-1), batch_size )): # by using equidistant batch till the last one it becomes much faster than using the X.size()[0] directly
                sequence = X_train_tensor[i:i+batch_size,:,:].reshape(-1, sequence_length, input_size).to(device)
                labels = y_train_tensor[i:i+batch_size,:,:].reshape(-1, sequence_length, output_size).to(device)
                # Forward pass
                outputs = model(sequence)
                loss = criterion(outputs, labels)
                epoch_loss+=loss.item()
                # Backward and optimize
                loss.backward()

            optimizer.step()
            epoch_loss=epoch_loss/k
            model.eval
            validation_loss,_= evaluate(model,X_test_hard_tensor_1,y_test_hard_tensor_1)
            model.train()
            training_loss_log.append(epoch_loss)
            print ('Epoch [{}/{}], Train MSELoss: {}, Validation : {} {}'.format(epoch+1, num_epochs,epoch_loss,validation_loss))

EDIT:
here are the parameters that I'm training with :

batch_size = 1024
num_epochs = 25000
learning_rate = 10e-04

optimizer=torch.optim.Adam(model.parameters(), lr=learning_rate)
criterion = nn.MSELoss(reduction='mean')

Output:
Batch size affects regularization. Training on a single example at a time is quite noisy, which makes it harder to overfit. Training on batches smoothes everything out, which makes it easier to overfit. Translating back to regularization:

Smaller batches add regularization.
Larger batches reduce regularization.

I am also curious about your learning rate. Every call to loss.backward() will accumulate the gradient. If you have set your learning rate to expect a single example at a time, and not reduced it to account for batch accumulation, then one of two things will happen.

The learning rate will be too high for the now-accumulated gradient, training will diverge, and both training and validation errors will explode.
The learning rate won't be too high, and nothing will diverge. The model will just train more quickly and effectively. If the model is too large for the data being fit, then training error will go to 0 but validation error will explode due to overfitting.

Update

Here is a bit more detail regarding the gradient accumulation.

Every call to loss.backward() will accumulate gradient, until you reset it with optimizer.zero_grad(). It will be acted on when you call optimizer.step(), based on whatever it has accumulated.

The way your code is written, you call loss.backward() for every pass through the inner loop, then you call optimizer.step() in the outer loop before resetting. So the gradient has been accumulated, that is summed, over all examples in the batch and not just one example at a time.

Under most assumptions, that will make the batch-accumulated gradient larger than the gradient for a single example. If the gradients are all aligned, for B batches, it will be larger by B times. If the gradients are i.i.d. then it will be more like sqrt(B) times larger.

If you do not account for this, then you have effectively increased your learning rate by that factor. Some of that will be mitigated by the smoothing effect of larger batches, which can then tolerate a higher learning rate. Larger batches reduce regularization, larger learning rates add it back. But that will not be a perfect match to compensate, so you will still want to adjust accordingly.

In general, whenever you change your batch size you will also want to re-tune your learning rate to compensate.

Leslie N. Smith has written some excellent papers on a methodical approach to hyperparameter tuning. A great place to start is A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. He recommends you start by reading the diagrams, which are very well done.


Instruction: How can I trim / remove part of a Tensor to match the shape of another Tensor with PyTorch?

Input:
I have 2 tensors:

outputs: torch.Size([4, 27, 161])       pred: torch.Size([4, 30, 161])

I want to cut pred (from the end) so that it'll have the same dimensions as outputs.

What's the best way to do it with PyTorch?

Output:
You can use Narrow

e.g:

a = torch.randn(4,30,161)
a.size() # torch.Size([4, 30, 161])
a.narrow(1,0,27).size() # torch.Size([4, 27, 161])