Instruction: PyTorch paste values into tensor by row index with increasing column index

Input:
I have a tensor output into which I want to put some values. I know the row that each value should go in, but I don't have an index tensor describing the columns. Instead, if there are k values that belong to one row, they should go in columns 0, 1, ..., k-1. This is perhaps better explained with an example:

import torch

n = 4
max_cols = 5
output = torch.zeros(n, max_cols)
row_idx = torch.tensor([0, 0, 0, 0, 1, 1, 2, 2, 2, 3])
values = torch.arange(len(row_idx)).float() # the values could be anything, not just arange
# output[??] = values

Here the first 4 values should be in output at [0, 0], ... [0, 3], the next 2 values at [1, 0], [1, 1], and so on.

Here's how I'm doing this now

_, counts = torch.unique(row_idx, return_counts=True)
range_ = torch.arange(max_cols)
col_idx = torch.cat([range_[:c] for c in counts])
output[row_idx, col_idx] = values
output

tensor([[0., 1., 2., 3., 0.],
        [4., 5., 0., 0., 0.],
        [6., 7., 8., 0., 0.],
        [9., 0., 0., 0., 0.]])

Is there any more efficient way to paste these values into the appropriate positions?

(feel free to suggest a better title for this if you can think of one)

Output:
I think your solution has linear time complexity. So, I am not sure if it can be further improved. However, I think the solution you provided is not correct. Let me give an example.

For the following input:

row_idx = torch.tensor([0, 0, 1, 0, 0, 1, 2, 2, 2, 3])

Your solution outputs the following.

tensor([[4., 1., 3., 3., 0.],
        [2., 5., 2., 0., 0.],
        [6., 7., 8., 0., 0.],
        [9., 0., 0., 0., 0.]])

However, I think your expected output is:

tensor([[0., 1., 3., 4., 0.],
        [2., 5., 0., 0., 0.],
        [6., 7., 8., 0., 0.],
        [9., 0., 0., 0., 0.]])

So, I suggest the following solution that I believe is correct.

def helper(a):
    idx = a.cumsum(-1)
    id_arr = torch.ones(idx[-1], dtype=int)
    id_arr[0] = 0
    id_arr[idx[:-1]] = -a[:-1]+1
    return id_arr.cumsum(-1)

n = 4
max_cols = 5
output = torch.zeros(n, max_cols)
row_idx = torch.tensor([0, 0, 1, 0, 0, 1, 2, 2, 2, 3])
values = torch.arange(len(row_idx)).float()

count = torch.unique(row_idx, return_counts=True)[1]
col_idx = helper(count)[row_idx.argsort().argsort()]
output[row_idx, col_idx] = values
print(output)

Update

You can simply add one line to your code as follows to make it work correctly.

_, counts = torch.unique(row_idx, return_counts=True)
range_ = torch.arange(max_cols)
col_idx = torch.cat([range_[:c] for c in counts])
col_idx = col_idx[row_idx.argsort().argsort()] # <== UPDATE
output[row_idx, col_idx] = values
print(output)


Instruction: Calculate the standard deviation of a moving windows using 2d convolution

Input:
I am doing a image processing project.
I want to calculate the standard deviation of a silding window using 2d convolution. I can now calculate the mean, but I cannot find a way to calculate the standard deviation. Here is my code:

import torch
import numpy as np
import cv2
import matplotlib.pyplot as plt
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
torch.set_default_tensor_type('torch.cuda.FloatTensor')
testvideo=cv2.VideoCapture ("test.mp4")
if testvideo.isOpened():
    ret, frame = testvideo.read()
    if ret == True:
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        imgarr=np.asarray(gray)
        #plt.imshow(imgarr,cmap="gray")
        imgtensor=torch.from_numpy(imgarr).float().to(device).unsqueeze(0).unsqueeze(0)
        print(imgtensor.device)
        #print(intensity.device)
        MeanConv=torch.nn.Conv2d(1, 1, kernel_size=100)
        kernel=(torch.ones(100,100)/10000).unsqueeze(0).unsqueeze(0)
        MeanConv.weight = torch.nn.Parameter(kernel)
        print(kernel)
        intensity=MeanConv(imgtensor)
        plt.imshow(intensity.squeeze(0).squeeze(0).detach().cpu().numpy(),cmap="gray")
        #for i in range(714):
            #for j in range(714):
                #intensity[i,j]=torch.mean(imgtensor[i:i+7,j:j+7])
                #sd[i,j]=torch.std(imgtensor[i:i+7,j:j+7])
        #x=(sd/intensity).cpu()
        #plt.imshow(x.numpy(),cmap="gray")
testvideo.release()

Output:
I think it can be somehow like this:

mean (img) using convolution.
Subtract from the original image of p. 1
Calculate the square of each element.
By convolution we find the average, p. 3
Calculates the square root of the elements of p. 4


Instruction: Multiple threads accessing same model on GPU for inference

Input: I have a cnn model that is loaded onto the GPU and for every image, a new thread has to be created and detached to run the model on this image. Is this possible and if so, Is it safe?

Output:
Yes, you definitely can. There are two aspects to it. If you want to run each model in parallel, then you have to load the same model in multiple GPUs. If you don't need that (just want the threading part), then you can load the model and use concurrent.futures.ThreadPoolExecutor(). In each call, you can pass an image.
I demonstrated one example with the darknet framework.
I loaded the model in two separate GPUs (for parallel operation, you can avoid that too) and each time I get a request, I use ThreadPoolExecutor to pass the images to the processing function.
from darknet import *
import concurrent.futures
import time

# you can avoid this part if you don't need multiple GPUs
set_gpu(0) # running on GPU 0
net1 = load_net(b"cfg/yolov3-lp_vehicles.cfg", b"backup/yolov3-lp_vehicles.backup", 0)
meta1 = load_meta(b"data/lp_vehicles.data")

set_gpu(1) # running on GPU 1
net2 = load_net(b"cfg/yolov3-lp_vehicles.cfg", b"backup/yolov3-lp_vehicles.backup", 0)
meta2 = load_meta(b"data/lp_vehicles.data")

def f(x):
    if x[0] == 0: # gpu 0
        return detect_np_lp(net1, meta1, x[1])
    else:
        return detect_np_lp(net2, meta2, x[1])

def func2(): # with threading
    a1 = cv2.imread("lp_tester/bug1.jpg")
    a2 = cv2.imread("lp_tester/bug2.jpg")
    nums = [(0, a1), (1, a2)] # the first element in tuple denotes GPU ID
    with concurrent.futures.ThreadPoolExecutor() as executor:
        r_m = [val for val in executor.map(f, nums)]
    print('out f2')
    #return r_m

t1 = time.time()
func2()
t2 = time.time()
print(t2-t1)


Instruction: fp16 inference on cpu Pytorch

Input:
I have a pretrained pytorch model I want to inference on fp16 instead of fp32, I have already tried this while using the gpu but when I try it on cpu I get:
"sum_cpu" not implemented for 'Half' torch.
any fixes?

Output: As I know, a lot of CPU-based operations in Pytorch are not implemented to support FP16; instead, it's NVIDIA GPUs that have hardware support for FP16(e.g. tensor cores in Turing arch GPU) and PyTorch followed up since CUDA 7.0(ish). To accelerate inference on CPU by quantization to FP16, you may wanna try torch.bfloat16 dtype(https://github.com/pytorch/pytorch/issues/23509).


Instruction: pytorch got None after backward()

Input:
I am learning pytorch and write a simple code as below.

import torch
x = torch.randn(3,requires_grad=True).cuda()
print(x)
y = x * x
print(y)
y.backward(torch.tensor([1,1.0,1]).cuda())
print(x.grad)

tensor([ 0.5934, -1.8813, -0.7817], device='cuda:0', grad_fn=<CopyBackwards>)
tensor([0.3521, 3.5392, 0.6111], device='cuda:0', grad_fn=<MulBackward0>)
None

if I change the code as

from torch.autograd import Variable
import torch
# x = torch.randn(3,requires_grad=True).cuda()
x = Variable(torch.randn(3).cuda(),requires_grad=True)
print(x)
y = x * x
print(y)
y.backward(torch.tensor([1,1.0,1]).cuda())
print(x.grad)

tensor([0.9800, 0.3597, 1.6315], device='cuda:0', requires_grad=True)
tensor([0.9605, 0.1294, 2.6617], device='cuda:0', grad_fn=<MulBackward0>)
tensor([1.9601, 0.7194, 3.2630], device='cuda:0')

The grad is ok. But why? I hate the Variable class.

env

python:3.8

pytorch:1.5

cuda :10.2

Output:
I got it.

x = torch.randn(3,requires_grad=True).cuda()

x is create by cuda(). So x is not a leaf tensor.

Change the code as below will be ok.

x = torch.randn(3,requires_grad=True,device=0)


Instruction: Training 1D CNN in Pytorch

Input:
I want to train the model given below. I am developing 1D CNN model in PyTorch. Usually we use dataloaders in PyTorch. But I am not using dataloaders for my implementation. I need guidance on how i can train my model in pytorch.

import torch
import torch.nn as nn
import torch.nn.functional as F

class  CharCNN(nn.Module):
def __init__(self,num_labels=11):
    super(CharCNN, self).__init__()

    self.conv1 = nn.Sequential(
        nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),
        nn.Dropout(0.1),
    )

    self.conv2 = nn.Sequential(
        nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_2, stride=stride_size),
        nn.Dropout(0.25)
    )

    self.fc1 = nn.Sequential(
        nn.Linear(depth_2*kernel_size_2, num_hidden),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

    self.fc2 = nn.Sequential(
        nn.Linear(num_hidden, num_labels),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

def forward(self, x):
    out = self.conv1(x)
    out = self.conv2(out)

    # collapse
    out = x.view(x.size(0), -1)
    # linear layer
    out = self.fc1(out)
    # output layer
    out = self.fc2(out)
    #out = self.log_softmax(x,dim=1)

    return out

I am training my network like this:

criterion = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(),lr=learning_rate)

for e in range(training_epochs):
  if(train_on_gpu):
     net.cuda()
    train_losses = []

   for batch in iterate_minibatches(train_x, train_y, batch_size):
      x, y = batch
      inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
      if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()
      opt.zero_grad()
      output = model(inputs, batch_size)

    loss = criterion(output, targets.long())
    train_losses.append(loss.item())
    loss.backward()
    opt.step()
val_losses = []
accuracy=0
f1score=0
print("Epoch: {}/{}...".format(e+1, training_epochs),
              "Train Loss: {:.4f}...".format(np.mean(train_losses)))

But i am getting the following error

          TypeError                                 Traceback (most recent call last)
          <ipython-input-60-3a3df06ef2f8> in <module>
   14             inputs, targets = inputs.cuda(), targets.cuda()
   15         opt.zero_grad()
   ---> 16         output = model(inputs, batch_size)
   17
   18         loss = criterion(output, targets.long())

   ~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
   * input, **kwargs)
   530             result = self._slow_forward(*input, **kwargs)
   531         else:
   --> 532             result = self.forward(*input, **kwargs)
   533         for hook in self._forward_hooks.values():
   534             hook_result = hook(self, input, result)

   TypeError: forward() takes 2 positional arguments but 3 were given

Please guide me how i can resolve this issue.

Output:
The forward method of your model only takes one argument, but you are calling it with two arguments:

output = model(inputs, batch_size)

It should be:

output = model(inputs)


Instruction: is there any similar function with clamp_ in tensorflow > 2.0

Input:
I'm converting torch code to tensorflow 2.0

prior_boxes = torch.FloatTensor(prior_boxes).to(device)  # (8732, 4)
prior_boxes.clamp_(0, 1)  # (8732, 4)

is there any replacement of clamp_(0,1) in tensorflow > 2.0?

Output:
Try tf.clip_by_value, though unlike clamp_, it is not in-place:

t = tf.constant([[-10., -1., 0.], [0., 2., 10.]])
t2 = tf.clip_by_value(t, clip_value_min=-1, clip_value_max=1)
t2.numpy()
# gives [[-1., -1., 0.], [0., 1., 1.]]


Instruction: No performance improvement using quantization model in pytorch

Input:
I have trained a model in pytorch with float data type. I want to improve my inference time by converting this model to quantized model. I have used torch.quantization.convert api to convert my model's weight to uint8 data type. However, when I use this model for inference, I do not get any performance improvement. Am I doing something wrong here ?

The Unet Model code:

def gen_initialization(m):
    if type(m) == nn.Conv2d:
        sh = m.weight.shape
        nn.init.normal_(m.weight, std=math.sqrt(2.0 / (sh[0]*sh[2]*sh[3])))
        nn.init.constant_(m.bias, 0)
    elif type(m) == nn.BatchNorm2d:
        nn.init.constant_(m.weight, 1)
        nn.init.constant_(m.bias, 0)

class TripleConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(TripleConv, self).__init__()
        mid_ch = (in_ch + out_ch) // 2
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, mid_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.BatchNorm2d(num_features=mid_ch),
            nn.LeakyReLU(negative_slope=0.1),
            nn.Conv2d(mid_ch, mid_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.BatchNorm2d(num_features=mid_ch),
            nn.LeakyReLU(negative_slope=0.1),
            nn.Conv2d(mid_ch, out_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.BatchNorm2d(num_features=out_ch),
            nn.LeakyReLU(negative_slope=0.1)
        )
        self.conv.apply(gen_initialization)

    def forward(self, x):
        return self.conv(x)

class Down(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Down, self).__init__()
        self.triple_conv = TripleConv(in_ch, out_ch)
        self.avg_pool_conv = nn.AvgPool2d(2, 2)
        self.in_ch = in_ch
        self.out_ch = out_ch

    def forward(self, x):
        self.cache = self.triple_conv(x)
        pad = torch.zeros(x.shape[0], self.out_ch - self.in_ch, x.shape[2], x.shape[3], device=x.device)
        x = torch.cat((x, pad), dim=1)
        self.cache += x
        return self.avg_pool_conv(self.cache)

class Center(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Center, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.BatchNorm2d(num_features=out_ch),
            nn.LeakyReLU(negative_slope=0.1, inplace=True)
        )
        self.conv.apply(gen_initialization)

    def forward(self, x):
        return self.conv(x)

class Up(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Up, self).__init__()
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear',
                                    align_corners=True)
        self.triple_conv = TripleConv(in_ch, out_ch)

    def forward(self, x, cache):
        x = self.upsample(x)
        x = torch.cat((x, cache), dim=1)
        x = self.triple_conv(x)
        return x

class UNet(nn.Module):
    def __init__(self, in_ch, first_ch=None):
        super(UNet, self).__init__()

        if not first_ch:
            first_ch = 32

        self.down1 = Down(in_ch, first_ch)
        self.down2 = Down(first_ch, first_ch*2)
        self.down3 = Down(first_ch*2, first_ch*4)
        self.down4 = Down(first_ch*4, first_ch*8)
        self.center = Center(first_ch*8, first_ch*8)
        self.up4 = Up(first_ch*8*2, first_ch*4)
        self.up3 = Up(first_ch*4*2, first_ch*2)
        self.up2 = Up(first_ch*2*2, first_ch)
        self.up1 = Up(first_ch*2, first_ch)
        self.output = nn.Conv2d(first_ch, in_ch, kernel_size=3, stride=1,
                                padding=1, bias=True)
        self.output.apply(gen_initialization)

    def forward(self, x):
        x = self.down1(x)
        x = self.down2(x)
        x = self.down3(x)
        x = self.down4(x)
        x = self.center(x)
        x = self.up4(x, self.down4.cache)
        x = self.up3(x, self.down3.cache)
        x = self.up2(x, self.down2.cache)
        x = self.up1(x, self.down1.cache)
        return self.output(x)

The inference code:

from tqdm import tqdm
import os
import numpy as np
import torch
import gan_network
import torch.nn.parallel
from torch.utils.data import DataLoader
import torch.utils.data as data
import random
import glob
import scipy.io
import time
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"

class DataFolder(data.Dataset):
    def __init__(self, file):
        super(DataFolder, self).__init__()
        self.image_names = []
        fid = file
        for line in fid:
            # line = line[:-1]
            if line == '':
                continue
            # print(line)
            self.image_names.append(line)
        random.shuffle(self.image_names)
        self.image_names = self.image_names[0:]

    def __len__(self):
        return len(self.image_names)

    def __getitem__(self, index):
        path = self.image_names[index]
        img = np.load(path)
        img = np.rollaxis(img, 2, 0)
        img = torch.from_numpy(img[:, :, :])
        return img, path

if __name__ == '__main__':
    batch_size = 1
    image_size = 2048
    channels = 6
    model_path = 'D:/WorkProjects/Network_Training_Aqusens/FullFovReconst/network/network_epoch9.pth'
    test_data = glob.glob('D:/save/temp/*.npy')
    dest_dir = 'D:/save/temp/results/'

    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    net = gan_network.UNet(6, 32)
    if torch.cuda.device_count() > 1:
        net = torch.nn.DataParallel(net)
    net.to(device)
    net.load_state_dict(torch.load(model_path))
    quantized_model = torch.quantization.convert(net, {torch.nn.Conv2d, torch.nn.BatchNorm2d}, inplace=False)

    dataset = DataFolder(file=test_data)
    print(f'{len(dataset)}')
    data_loader = DataLoader(dataset=dataset, num_workers=4,
                             batch_size=batch_size, shuffle=False,
                             drop_last=False, pin_memory=True)
    input = torch.Tensor(batch_size, channels, image_size, image_size).to(device)

    t0 = time.time()
    with torch.no_grad():
        for i, batch in enumerate(tqdm(data_loader)):
            input.copy_(batch[0])
            output = net(input).cpu().clone().numpy()
            np.array(output)
            output = np.rollaxis(output, 1, 4)
            for num in range(batch_size):
                arr = output[num, :, :, :]
                file_name = os.path.basename(batch[1][num])
                save_name = os.path.join(dest_dir, file_name)
                save_name = save_name.replace(".npy", "")
                scipy.io.savemat(save_name+'.mat', {'output': arr})
    t1 = time.time()
    print(f'Elapsed time = {t1-t0}')

For models net and quantized model, i get the elapsed time around 30 seconds for 12 images passed through them.

Output: PyTorch documentation suggests three ways to perform quantization. You are doing post-training dynamic quantization (the simplest quantization method available) which only supports torch.nn.Linear and torch.nn.LSTM layers as listed here. To quantize CNN layers, you would want to check out the other two techniques (these are the ones that support CNN layers): post-training static quantization and quantization aware training. This tutorial shows both these techniques applied on CNNs.


Instruction: Difference between src_mask and src_key_padding_mask

Input:
I am having a difficult time in understanding transformers. Everything is getting clear bit by bit but one thing that makes my head scratch is
what is the difference between src_mask and src_key_padding_mask which is passed as an argument in forward function in both encoder layer and decoder layer.

https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer

Output:
Difference between src_mask and src_key_padding_mask
The general thing is to notice the difference between the use of the tensors _mask vs _key_padding_mask.
Inside the transformer when attention is done we usually get an squared intermediate tensor with all the comparisons
of size [Tx, Tx] (for the input to the encoder), [Ty, Ty] (for the shifted output - one of the inputs to the decoder)
and [Ty, Tx] (for the memory mask - the attention between output of encoder/memory and input to decoder/shifted output).
So we get that this are the uses for each of the masks in the transformer
(note the notation from the pytorch docs is as follows where Tx=S is the source sequence length
(e.g. max of input batches),
Ty=T is the target sequence length (e.g. max of target length),
B=N is the batch size,
D=E is the feature number):

src_mask [Tx, Tx] = [S, S] – the additive mask for the src sequence (optional).
This is applied when doing atten_src + src_mask. I'm not sure of an example input - see tgt_mask for an example
but the typical use is to add -inf so one could mask the src_attention that way if desired.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

tgt_mask [Ty, Ty] = [T, T] – the additive mask for the tgt sequence (optional).
This is applied when doing atten_tgt + tgt_mask. An example use is the diagonal to avoid the decoder from cheating.
So the tgt is right shifted, the first tokens are start of sequence token embedding SOS/BOS and thus the first
entry is zero while the remaining. See concrete example at the appendix.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

memory_mask [Ty, Tx] = [T, S]– the additive mask for the encoder output (optional).
This is applied when doing atten_memory + memory_mask.
Not sure of an example use but as previously, adding -inf sets some of the attention weight to zero.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

src_key_padding_mask [B, Tx] = [N, S] – the ByteTensor mask for src keys per batch (optional).
Since your src usually has different lengths sequences it's common to remove the padding vectors
you appended at the end.
For this you specify the length of each sequence per example in your batch.
See concrete example in appendix.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

tgt_key_padding_mask [B, Ty] = [N, t] – the ByteTensor mask for tgt keys per batch (optional).
Same as previous.
See concrete example in appendix.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

memory_key_padding_mask [B, Tx] = [N, S] – the ByteTensor mask for memory keys per batch (optional).
Same as previous.
See concrete example in appendix.
If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged.
If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.
If a FloatTensor is provided, it will be added to the attention weight.

Appendix
Examples from pytorch tutorial (https://pytorch.org/tutorials/beginner/translation_transformer.html):
1 src_mask example
    src_mask = torch.zeros((src_seq_len, src_seq_len), device=DEVICE).type(torch.bool)

returns a tensor of booleans of size [Tx, Tx]:
tensor([[False, False, False,  ..., False, False, False],
         ...,
        [False, False, False,  ..., False, False, False]])

2 tgt_mask example
    mask = (torch.triu(torch.ones((sz, sz), device=DEVICE)) == 1)
    mask = mask.transpose(0, 1).float()
    mask = mask.masked_fill(mask == 0, float('-inf'))
    mask = mask.masked_fill(mask == 1, float(0.0))

generates the diagonal for the right shifted output which the input to the decoder.
tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf,
         -inf, -inf, -inf],
        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf,
         -inf, -inf, -inf],
        [0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf,
         -inf, -inf, -inf],
         ...,
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., -inf],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0.]])

usually the right shifted output has the BOS/SOS at the beginning and it's the tutorial gets the right shift simply
by appending that BOS/SOS at the front and then triming the last element with tgt_input = tgt[:-1, :].
3 _padding
The padding is just to mask the padding at the end.
The src padding is usually the same as the memory padding.
The tgt has it's own sequences and thus it's own padding.
Example:
    src_padding_mask = (src == PAD_IDX).transpose(0, 1)
    tgt_padding_mask = (tgt == PAD_IDX).transpose(0, 1)
    memory_padding_mask = src_padding_mask

Output:
tensor([[False, False, False,  ...,  True,  True,  True],
        ...,
        [False, False, False,  ...,  True,  True,  True]])

note that a False means there is no padding token there (so yes use that value in the transformer forward pass) and a True means that there is a padding token (so masked it out so the transformer pass forward does not get affected).

The answers are sort of spread around but I found only these 3 references being useful
(the separate layers docs/stuff wasn't very useful honesty):

long tutorial: https://pytorch.org/tutorials/beginner/translation_transformer.html
MHA docs: https://pytorch.org/docs/master/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention
transformer docs: https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html


Instruction: Convolution in PyTorch with non-trainable pre-defined kernel

Input:
I would like to introduce a custom layer to my neural network. The mathematical operation should be a discrete 2D cross correlation (or convolution) with a non-trainable kernel. The values in the kernel depend on three things: kernel shape, strides and padding. I intend to multiply the output element-wise with a weight matrix.

PyTorch already has an implementation of a discrete 2D cross correlation class called 'Conv2d', however it generates a random kernel and trains using the entries of said kernel. If possible I would like a class similar to 'Conv2d' that does what I need, to make sure to use my GPU most effectively. I tried implementing this on my own, but couldn't figure out how to obtain the correct shapes for the input array. 'Conv2d' only uses 'in_channels', if I understood correctly.

Output:
If I understand correctly,you want a Conv2d layer with defined kernel and you don't want it to be learnable.

In that case,you can use the conv2d function like this:

import torch.nn.functional as F
output_tensor = F.conv2d(input_tensor, your_kernel, ...)

the parameter your_kernel is your weight matrix,also you need to declare other parameters of the function like padding and stride

Then you need to set the requires_grad attribute to 'False', and exclude it from the optimizer if you don't want it to be learnable.

And about the shape issue ,maybe you wanna check this out


Instruction: Pytorch: accessing a subtensor using lists of indices

Input:
I have a pair of tensors S and T of dimensions (s1,...,sm) and (t1,...,tn) with si < ti. I want to specify a list of indices in each dimensions of T to "embed" S in T. If I1 is a list of s1 indices in (0,1,...,t1) and likewise for I2 up to In, I would like to do something like
T.select(I1,...,In)=S
that will have the effect that now T has entries equal to the entries of S over the indices (I1,...,In).
for example

`S=
[[1,1],
[1,1]]

T=
[[0,0,0],
[0,0,0],
[0,0,0]]

T.select([0,2],[0,2])=S

T=
[[1,0,1],
[0,0,0],
[1,0,1]]`

Output:
If you're flexible with using NumPy only for the indices part, then here's one approach by constructing an open mesh using numpy.ix_() and using this mesh to fill-in the values from the tensor S. If this is not acceptable, then you can use torch.meshgrid()

Below is an illustration of both approaches with descriptions interspersed in comments.

# input tensors to work with
In [174]: T
Out[174]:
tensor([[0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]])

# I'm using unique tensor just for clarity; But any tensor should work.
In [175]: S
Out[175]:
tensor([[10, 11],
        [12, 13]])

# indices where we want the values from `S` to be filled in, along both dimensions
In [176]: idxs = [[0,2], [0,2]]

Now we will leverage np.ix_() or torch.meshgrid() to generate an open mesh by passing in the indices:

# mesh using `np.ix_`
In [177]: mesh = np.ix_(*idxs)

# as an alternative, we can use `torch.meshgrid()`
In [191]: mesh = torch.meshgrid([torch.tensor(lst) for lst in idxs])

# replace the values from tensor `S` using basic indexing
In [178]: T[mesh] = S

# sanity check!
In [179]: T
Out[179]:
tensor([[10,  0, 11],
        [ 0,  0,  0],
        [12,  0, 13]])


Instruction: How to solve this question "RuntimeError: CUDA out of memory."?

Input:
I'm going to extract a feature from pictures.I first define a tensor data_feature_map, and then use torch.cat to stack the features of one picture.

My code is :

data_feature_map = torch.ones(1,2048)
for i, data in enumerate(train_loader, 0):
    img, _ = data
    img.requires_grad_=False
    if torch.cuda.is_available():
        img = img.cuda()
    out = model(img)
    # out.shape = [1,2048]
    out = out.view(1,-1).cpu()
    data_feature_map = torch.cat((data_feature_map, out), 0)

but when i run it, it will show the error "RuntimeError: CUDA out of memory."

please tell me why this error occurs.Thank you very much.

Output:
Since your GPU is running out of memory, you can try few things:

1.) Reduce your batch size

2.) Reduce your network size


Instruction: Differential Privacy decreases the model performance significantly

Input:
Background Information

I trained a classifier to predict three labels: COVID/Pneumonia/Healthy based on chest X-Ray images. It's a PyTorch implementation of COVID-Net. I use a training set to train on, validation set to save the best performing model, and then a test set to measure the "real" performance of the model. However, I noticed that my model "learned" to classify normal/pneumonia really good, but it just ignored the underpopulated COVID set. Therefore I choose to undersample (reduce the number of training instances of the other classes (normal and pneumonia) in order to get equal populations). This worked well, but my sample set has been reduced to ~1500 samples (low!). The results are somewhat worse than COVID-Net, I achieve an accuracy of ~80% and lower sensitivity on underpopulated classes (COVID) then they report. I suppose that they report better performance because they do not use a validation-set and use the test-set each epoch. I figured that they might indirectly overfit on the test-set because of that. I have chosen to explain this so that the reader gets a context.

Question

I tried adding privacy to the training procedure by using Differential Privacy. Specifically, I used Facebook's PyTorch-DP module. Training works just as well if I choose to add almost no-privacy (this can be achieved by choosing a really low noise multiplier value (sigma), i.e. 1e-7) and a really high delta. So it's not that the module itself is not working/faulty, but, if I use a lower sigma (so I add more noise) then I get more privacy (epsilon decreases) but the model fails to fit the data at all.
The question is: how do I manage to add privacy to a somewhat meaningful degree while making sure that my model somewhat fits the data still?

Performance differences

Confusion Matrix of Model without Differential Privacy added. It's not "good" but it's at least somewhat meaningful and the model reaches an accuracy of ~80%.

Confusion Matrix of Model with Differential Privacy (epsilon: 2.3) after 100 epochs. It looks as if the model does not know what to do, at all.

Possible explanations

I read a paper that stated that adding Differential Privacy can cause bad performance in the sense that the accuracy decreases for underpopulated classes. But, I used undersampling and I think this should've solved that, but the accuracy stays bad (for all classes!).

Maybe because my sample set is so small, differential privacy is much harder to achieve, and therefore the performance is bad? However, even if add a really tiny bit of privacy, with an epsilon value >20000, the model still struggles in learning how to classify. So I'm not sure.

Output:
It seems the PyTorch Differential Privacy library from Facebook Research is built on the concept of Renyi differential privacy guarantee that is well-suited for expressing guarantees of privacy-preserving algorithms and for composition of heterogeneous mechanisms. We need to have a good estimation of the heterogenity in this COVID-Net dataset.

In particular, Rényi divergence satisfies the data processing inequality. It seems the current library is more suitable for machine learning problems with more heterogeneity in datasets. This library is using an implementation of the differentially private stochastic gradient descent (SGD) algorithm. It follows the sequence of random initialisation, computation of gradient, clipping gradient, adding noise and doing the descent. The clipping and noise parameters may vary with the number of training steps and epochs.

The success of differential privacy on deep learning problems are driven by the  extent of pre-processing the gradient to protect privacy and privacy_accounting which keeps track of the privacy spending over the course of training. It is highlighted that in the differentially private deep learning the model accuracy is more sensitive to training parameters such as batch size and noise level than to the structure of a neural network.

In the PyTorch library, we can see the examples on ImageNet, MNIST, DCGAN etc. In all these examples we can see how each of the mentioned parameters such as clipping, batch size etc. can be varied for getting the required accuracy levels. Kindly refer to the following example scripts in the PyTorch DP library.

PyTorch DP Example Scripts for Various Models


Instruction: How to get the imagenet dataset on which pytorch models are trained on

Input:
Can anyone please tell me how to download the complete imagenet dataset on which the pytorch torchvision models are trained on and their Top-1 error is reported on?

I have downloaded Tiny-Imagenet from Imagenet website and used pretrained resnet-101 model which provides only 18% Top-1 accuracy.

Output:
Download the ImageNet dataset from http://www.image-net.org/  (you have to sign in)

Then, you should move validation images to labeled subfolders, which could be done automatically using the following shell script:
https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh


Instruction: Passing variable to nn.Conv2d arguments within a class init definition python

Input:
So I want to pass some new variables such as kernel_size when I initiate a new object. Let's say net=Net10(5,2,4,3,1,1). so that I get an object of this class with the parameters I want not something always constant, cos otherwise I will have to define lots of classes. Now, I want to pass kernel_size within the self.Conv2d, and I get a syntax error: positional argument follows keyword argument
Does anyone know how to fix this? Should I change it all to functions instead of classes?

    class Net10(nn.Module):
    def __init__ (self, kernel_size, stride, pooling, num_classes, neurons, ActFunn, *args):
        super(Net10, self).__init__()
        self.kernel = kernel_size
        self.stride = stride
        self.pooling = pooling
        self.num_classes = num_classes
        self.neurons = neurons
        self.Actfun = ActFunn
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel, padding=2, stride=1)
        self.pool = nn.MaxPool2d(pooling, pooling)
        self.fcinput= round (28 / pooling)

Output:
Try:

self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=self.kernel, padding=2, stride=1)

In more details: python functions (e.g., __init__ of the conv layer) can have input arguments is two "flavors": positional arguments: that is associating an input argument to a function variable according to its position in the argument list.
The other "flavor" is keyword argument: argument that is given with its keyword, e.g., in_channels=1 etc.

As a rule python does not allow wild mixing of positional and keyword arguments.
You can have positional arguments followed by keyword argument, but you cannot have a positional argument once you started declaring keyword arguments.

self.conv1 = nn.Conv2d(in_channels=1,   # keyword argument
   out_channels=32,   # keyword argument
   kernel,   # positional argument (no "keyword" defining this argument)
   padding=2,  # keyword
   stride=1)   # keyword


Instruction: How to load dataset from pickle files into PyTorch?

Input: I have X_train(inputs) and Y_train(labels) in separate pickle files in form of integer matrices. Now, I need to load them and train using PyTorch. I tried torch.utils.data.DataLoader and torchvision.datasets.DatasetFolder but nothing worked or I might be getting wrong somewhere. Please suggest a proper way for the same.

Output:
You should really give a clear description of your problem with some examples. Anyway, as far as I understand you are looking for something like this.

import pickle
from torch.utils.data import Dataset
from torchvision import transforms
from torch.utils.data import DataLoader

class YourDataset(Dataset):

    def __init__(self, X_Train, Y_Train, transform=None):
        self.X_Train = X_Train
        self.Y_Train = Y_Train
        self.transform = transform

    def __len__(self):
        return len(self.X_Train)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        x = self.X_Train[idx]
        y = self.Y_Train[idx]

        if self.transform:
            x = self.transform(x)
            y = self.transform(y)

        return x, y

file = open('FILENAME_X_train', 'rb')
X_train = pickle.load(file)
file.close()

file = open('FILENAME_Y_train', 'rb')
Y_train = pickle.load(file)
file.close()

your_dataset = YourDataset(X_train, Y_train, transform=transforms.Compose([transforms.ToTensor()]))

your_data_loader = DataLoader(your_dataset, batch_size=8, shuffle=True, num_workers=0)

Note that I have not tested the code, but I think that it gives the general idea. Hope it helps.


Instruction: Downloading transformers models to use offline

Input:
I have a trained transformers NER model that I want to use on a machine not connected to the internet. When loading such a model, currently it downloads cache files to the .cache folder.

To load and run the model offline, you need to copy the files in the .cache folder to the offline machine. However, these files have long, non-descriptive names, which makes it really hard to identify the correct files if you have multiple models you want to use. Any thoughts on this?

Example of model files

Output:
One relatively easy way to deal with this issue is to simply "rename" the pretrained models, as is detailed in this thread.
Essentially, all you have to do is something like this for whatever model you're trying to work with:
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased")
model.save_pretrained("./my_named_bert")

The thread also details how the local model folders are named, see LysandreJik's post:

Hi, they are named as such because that's a clean way to make sure the model on the S3 is the same as the model in the cache. The name is created from the etag of the file hosted on the S3. [...]


Instruction: what is torch's unsqueeze equivalence with tensorflow?

Input:
what is torch's unsqueeze  equivalence with tensorflow?

#tensorflow auto-broadcasts singleton dimensions
lower_bounds = tf.argmax(set_1[:, :2].unsqueeze(1), set_2[:, :2].unsqueeze(0))  # (n1, n2, 2)
upper_bounds = tf.argmin(set_1[:, 2:].unsqueeze(1), set_2[:, 2:].unsqueeze(0))  # (n1, n2, 2)

Output:
Maybe you wanna try this:
tf.expand_dims(x, axis)


Instruction: PyTorch LSTM dropout vs Keras LSTM dropout

Input:
I'm trying to port my sequential Keras network to PyTorch. But I'm having trouble with the LSTM units:

LSTM(512,
     stateful = False,
     return_sequences = True,
     dropout = 0.5),
LSTM(512,
     stateful = False,
     return_sequences = True,
     dropout = 0.5),

How should I formulate this in PyTorch? Especially dropout appears to work very differently in PyTorch than it does in Keras.

Output:
The following should work for you.

lstm = nn.LSTM(
    input_size = ?,
    hidden_size = 512,
    num_layers = 1,
    batch_first = True,
    dropout = 0.5
)

You need to set the input_size. Check out the documentation on LSTM.

Update

In a 1-layer LSTM, there is no point in assigning dropout since dropout is applied to the outputs of intermediate layers in a multi-layer LSTM module. So, PyTorch may complain about dropout if num_layers is set to 1. If we want to apply dropout at the final layer's output from the LSTM module, we can do something like below.

lstm = nn.Sequential(
    nn.LSTM(
        input_size = ?,
        hidden_size = 512,
        num_layers = 1,
        batch_first = True
    ),
    nn.Dropout(0.5)
)

According to the above definition, the output of the LSTM would pass through a Dropout layer.


Instruction: Where is the numpy data stored?

Input:
In python, If only import torch (but not import numpy), "torch.numpy()" can still work. Is that means the numpy data can be stored and displayed without numpy package? Where is the numpy data stored and how does it display (without numpy package)?

example codes:

import torch
a = torch.tensor([[1,2,3],[4,5,6]])
a = a.numpy()
print(a)
array([[1, 2, 3],
       [4, 5, 6]])

Output: PyTorch uses NumPy internally. You don't need to manually import everything a package uses, that is one of the core principles of modules. It's still an object of the same NumPy class and you need to have NumPy installed for it to work, otherwise you would get an import error, just that the import happens in one of PyTorch's files, rather than your own.


Instruction: what is the torch's torch.cat equivalence with tensorflow?

Input:
def cxcy_to_xy(cxcy):
    """
    Convert bounding boxes from center-size coordinates (c_x, c_y, w, h) to boundary coordinates (x_min, y_min, x_max, y_max).

    :param cxcy: bounding boxes in center-size coordinates, a tensor of size (n_boxes, 4)
    :return: bounding boxes in boundary coordinates, a tensor of size (n_boxes, 4)
    """
    return torch.cat([cxcy[:, :2] - (cxcy[:, 2:] / 2),  # x_min, y_min
                      cxcy[:, :2] + (cxcy[:, 2:] / 2)], 1)  # x_max, y_max

I want to change
this torch.cat with tensorflow 2.0

Output:
Few options depending on the API in TF you're using:

tf.concat - most similar to torch.cat:

tf.concat(values, axis, name='concat')

tf.keras.layers.concatenate - if you're using Keras sequential API:

tf.keras.layers.concatenate(values, axis=-1, **kwargs)

tf.keras.layers.Concatenate - if you're using Keras functional API:

x = tf.keras.layers.Concatenate(axis=-1, **kwargs)(values)

If you're using the Keras API, this answer is informative for understanding the differences between all the Keras concatenation functions.


Instruction: what does clamp_ does in pytorch and how to change it to the tensorflow 2.0?

Input:
   prior_boxes = torch.FloatTensor(prior_boxes).to(device)  # (8732, 4)
   prior_boxes.clamp_(0, 1)  # (8732, 4)

what dooes clamp_ do in pytorch and how to change it to the tensorflow 2.0?
I'm not sure what clamp_ do exactly?

Output:
clamp_(0, 1) Clamp all elements in prior_boxes into the range [ 0, 1].

Tensorflow:

tf.clip_by_value
https://www.tensorflow.org/api_docs/python/tf/clip_by_value


Instruction: Array Slicing with step 2

Input:
Have array like

arr = [1,2,3,4,5,6,7,8,9,10].

How I can get array like this:

[1,2,5,6,9,10]

take 2 elements with step 2(::2)

I try something like arr[:2::2].it's not work

Output:
[:2::2] is not valid Python syntax.  A slice only takes 3 values - start, stop, step.  You are trying to provide 4.

Here's what you need to do:

In [233]: arr = np.arange(1,11)
In [234]: arr
Out[234]: array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

first reshape to form groups of 2:

In [235]: arr.reshape(5,2)
Out[235]:
array([[ 1,  2],
       [ 3,  4],
       [ 5,  6],
       [ 7,  8],
       [ 9, 10]])

now slice to get every other group:

In [236]: arr.reshape(5,2)[::2 ,:]
Out[236]:
array([[ 1,  2],
       [ 5,  6],
       [ 9, 10]])

and then back to 1d:

In [237]: arr.reshape(5,2)[::2,:].ravel()
Out[237]: array([ 1,  2,  5,  6,  9, 10])

You have to step back a bit, and imagine the array as a whole, and ask how to make it fit the desire pattern.


Instruction: Pytorch loss does't change in vgg 19 model

Input:
In pytorch I made a model vgg19 for classification tiny imagenet:

model = nn.Sequential(
    nn.BatchNorm2d(3),
    nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=128, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Flatten(),
    nn.Linear(25088, 4096),
    nn.Linear(4096, 1000),
    nn.Linear(1000, 200),
    nn.Softmax(),
    nn.Dropout2d(),
)

In the learning process, the loss remains at approximately the same value (5.3 +- 0.01).

###############
iter = 4000 / 80000
loss = 5.295811176300049
###############
iter = 4800 / 80000
loss = 5.298299789428711
###############
iter = 5600 / 80000
loss = 5.309792995452881
###############
iter = 6400 / 80000
loss = 5.3179707527160645
###############
iter = 7200 / 80000
loss = 5.3179707527160645

I have already tried to increase and decrease lr, batch_size. But still no idea how to fix it.

Training code
(epochs = 10, loss = cross_entropy, optim = Adam(lr = 0.01), X_batch.shape = (batch_size, 3, 224, 224):

for epoch in range(num_epochs):
    i = 0
    for (X_batch, y_batch) in train_batch_gen:
        X_batch = Variable(torch.FloatTensor(X_batch)).cuda()
        y_batch = Variable(torch.LongTensor(y_batch)).cuda()
        logits = model.cuda().forward(X_batch)
        opt.zero_grad()
        loss = lossFunc(logits, y_batch)
        loss.backward()
        opt.step()

        train_loss.append(loss.data.cpu().numpy())
        if i % (batch_size*100) == 0:
            print("###############")
            print(f"iter = {i} / {80000}")
            print(f"loss = {np.mean(train_loss[-len(train_dataset) // batch_size :])}")
        i += batch_size

Output:
nn.CrossEntropyLoss applies log-softmax, but you also apply softmax in the model:

nn.Linear(1000, 200),
nn.Softmax(),
nn.Dropout2d(),

The output of your model must be the raw logits, without the nn.Softmax().

Additionally, dropout should not be used just before the output of the model, since that effectively wipes out some of the classes, making the loss punish something that would have been correct otherwise. Dropout should only be used between layers as a regularisation.


Instruction: Why torch.dot(a,b) makes requires_grad=False

Input:
I have some losses in a loop storing them in a tensor loss. Now I want to multiply a weight tensor to the loss tensor to have final loss, but after torch.dot(), the result scalar, ll_new, has requires_grad=False. The following is my code.

loss_vector = torch.FloatTensor(total_loss_q)
w_norm = F.softmax(loss_vector, dim=0)
ll_new = torch.dot(loss_vector,w_norm)

How can I have requires_grad=False for the ll_new after doing the above?

Output:
I think the issue is in the line: loss_vector = torch.FloatTensor(total_loss_q) as requires_grad for loss_vector is False (default value). So, you should do:

loss_vector = torch.FloatTensor(total_loss_q, requires_grad=True)


Instruction: Whenever I try to install torch, it displays killed

Input:
I just want to install pytorch, I ran this in the terminal:

pip install torch

And it displays:

Collecting torch
Killed

What is the problem?

Output:
It says your your free ram is not enough to install the package, but there is a method that you can still use it.

pip install torch --no-cache-dir


Instruction: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Input:
****I set my model and data to the same device, but always raise the error like this:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same**
The following is training code**

total_epoch = 1
best_epoch = 0
training_losses = []
val_losses = []

for epoch in range(total_epoch):

    epoch_train_loss = 0

    for X, y in train_loader:

    X, y = X.cuda(), y.cuda()

    optimizer.zero_grad()
    result = model(X)
    loss = criterion(result, y).
    epoch_train_loss += loss.item()
    loss.backward()
    optimizer.step()

training_losses.append(epoch_train_loss)

epoch_val_loss = 0
correct = 0
total = 0

with torch.no_grad():
    for X, y in val_loader:

        X, y = X.cuda(), y.cuda()

        result = model(X)
        loss = criterion(result, y)
        epoch_val_loss += loss.item()
        _, maximum = torch.max(result.data, 1)
        total += y.size(0)
        correct += (maximum == y).sum().item()

val_losses.append(epoch_val_loss)
accuracy = correct/total
print("EPOCH:", epoch, ", Training Loss:", epoch_train_loss, ", Validation Loss:", epoch_val_loss, ", Accuracy: ", accuracy)

if min(val_losses) == val_losses[-1]:
    best_epoch = epoch
    checkpoint = {'model': model,
                        'state_dict': model.state_dict(),
                        'optimizer' : optimizer.state_dict()}

    torch.save(checkpoint, models_dir + '{}.pth'.format(epoch))
    print("Model saved")

when i Run the following code for detection using cv2.capture(0) .

import cvlib as cv
from PIL import Image

cap = cv2.VideoCapture(0)

font_scale=1
thickness = 2
red = (0,0,255)
green = (0,255,0)
blue = (255,0,0)
font=cv2.FONT_HERSHEY_SIMPLEX

face_cascade = cv2.CascadeClassifier( cv2.data.haarcascades +'haarcascade_frontalface_default.xml')

while(cap.isOpened()):
    ret, frame = cap.read()
    if ret == True:

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.4, 4)

    for (x, y, w, h) in faces:

        cv2.rectangle(frame, (x, y), (x+w, y+h), blue, 2)

        croped_img = frame[y:y+h, x:x+w]
        pil_image = Image.fromarray(croped_img, mode = "RGB")
        pil_image = train_transforms(pil_image)
        image = pil_image.unsqueeze(0)

        result = loaded_model(image)
        _, maximum = torch.max(result.data, 1)
        prediction = maximum.item()

        if prediction == 0:
            cv2.putText(frame, "Masked", (x,y - 10), font, font_scale, green, thickness)
            cv2.rectangle(frame, (x, y), (x+w, y+h), green, 2)
        elif prediction == 1:
            cv2.putText(frame, "No Mask", (x,y - 10), font, font_scale, red, thickness)
            cv2.rectangle(frame, (x, y), (x+w, y+h), red, 2)

    cv2.imshow('frame',frame)

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        break
else:
    break

cap.release()
cv2.destroyAllWindows()

about the function loaded_model declared as below

def load_checkpoint(filepath):
checkpoint = torch.load(filepath)
model = checkpoint['model']
model.load_state_dict(checkpoint['state_dict'])
for parameter in model.parameters():
parameter.requires_grad = False

return model.eval()

filepath = models_dir + str(best_epoch) + ".pth"
loaded_model = load_checkpoint(filepath)

ERROR:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-29-b3a630684f44> in <module>()
     43
     44
---> 45             result = loaded_model(image)
     46             _, maximum = torch.max(result.data, 1)
     47             prediction = maximum.item()

5 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

**I hope you can answer it.Thanks!**

Output:
The model's weights are on the GPU, while the image is on the CPU. You need to put it onto the GPU as well.

image = pil_image.unsqueeze(0)
image = image.cuda()

result = loaded_model(image)

It looks like you didn't manually put the model onto the GPU, but rather that you saved the model's weights, which were originally on the GPU, and PyTorch keeps the device information when saving the state dict. If you want to run the model on the CPU, you should make sure that the weights are on the CPU. torch.load accepts a map_location argument, which forces the loaded data to be on the specified device, rather than using the saved device.

# Load weights onto the CPU regardless of saved device.
checkpoint = torch.load(filepath, map_location="cpu")


Instruction: Installing PyTorch3D fails with anaconda and pip on Windows 10

Input:
I saw that more people seem to have the same issue, but it was not resolved. I am trying to install Pytorch3D with Anaconda and got the following PackageNotFound error.
Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
  - pytorch3d
Current channels:
  - https://conda.anaconda.org/pytorch3d/win-64
  - https://conda.anaconda.org/pytorch3d/noarch
  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're looking for, navigate to
    https://anaconda.org
and use the search bar at the top of the page.

I have also tried using pip install 'git+https://github.com/facebookresearch/pytorch3d.git'
and get the following:
C:\Users\Alexandra>pip install 'git+https://github.com/facebookresearch/pytorch3d.git'
ERROR: Invalid requirement: "'git+https://github.com/facebookresearch/pytorch3d.git'"

C:\Users\Alexandra>pip install git+https://github.com/facebookresearch/pytorch3d.git
Collecting git+https://github.com/facebookresearch/pytorch3d.git
  Cloning https://github.com/facebookresearch/pytorch3d.git to c:\users\alexan~1\appdata\local\temp\pip-req-build-uspo7an4
  Running command git clone -q https://github.com/facebookresearch/pytorch3d.git 'C:\Users\ALEXAN~1\AppData\Local\Temp\pip-req-build-uspo7an4'
  ERROR: Error [WinError 2] The system cannot find the file specified while executing command git clone -q https://github.com/facebookresearch/pytorch3d.git 'C:\Users\ALEXAN~1\AppData\Local\Temp\pip-req-build-uspo7an4'
ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

I am on Windows 10, using python 3.8, PyTorch 1.5 and CUDA 10.2.
I am very new to python, so I have no idea how to fix this (you can tell, that I've never installed from git before..) (please be lenient!)
Thank you!
EDIT: Thank you for your answers. I did install Git and it got me a bit further, but still not completeing the build..

Also, interesting enough, when I run the commands
!pip install torch torchvision
!pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'

in Google Collab it seems to work, but I cannot run it , let's say, in jupyter.
Any more ideas?

Output:
Edit 10-17-2022
With CUDA 11.6 downloading CUB and setting CUB_HOME is no longer necessary. Trying to use CUB_HOME will give nvcc.exe compile error. Any previous CUB_HOME environment variable should be deleted and command line restarted before running setup.

Original Answer
I have also tried to install pytorch3d on windows 10. As of writing this there is no windows package in, https://anaconda.org/pytorch3d/pytorch3d.
Pytorch3d install doc has detailed instructions but some information is missing and only found inside various issues. Following various issues I was able get pytorch3d installed by compiling from source on pytorch 1.8.1 and 1.10.0(This version is not supported yet in official docs for pytorch3d 0.6.0).
I have tested on pytorch 1.8.1 with CUDA 10.2 and pytorch 1.10.0 with CUDA 11.3. I had CUDA Toolkit 11.0, CuDNN installed separately with environment variables set to be used by tensorflow gpu. For both environment a new python 3.9 was used. Visual studio 16.11.5 was used with Desktop Development with C++ enabled, CMake 3.21.3.
It is probably better to have same CUDA Toolkit version as the Pytorch GPU version. There was warning regarding version but in my case it was installed successfully.
The pytorch3d source code must be downloaded and extracted in order to compile. When running python setup.py install from pytorch3d folder it looked for CUDA_HOME. It was able to find correct CUDA path probably based on other flags.
I initially faced these errors,

RuntimeError: Error compiling objects for extension.
xutility(...): error: expected a "("

Install on windows

Create conda enviroment, install torch and dependencies.

conda create -n pytorch3d python=3.9
conda activate pytorch3d
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
conda install -c fvcore -c iopath -c conda-forge fvcore iopath

Install appropriate CUDA Toolkit, CuDNN and set enviroment variables.

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\libnvvp
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include

Set correct path to cl.exe Visual C++ compiler. This will allow to run compilation from conda prompt with correct environment selected. Host x64 was used and x86 for the folder inside as x64 one gave error for me. In my case it was,

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x86

Following this issue, PYTORCH3D_NO_NINJA system environment variable was set with value of 1.

FORCE_CUDA with value of 1 (though not required if CUDA is available in pytorch) and CUB_HOME system environment variable was set by downloading it from, https://github.com/NVIDIA/cub/releases.

PYTORCH3D_NO_NINJA 1
CUB_HOME C:\portable\cub-1.9.9

The following env variable were probably set by CUDA Toolkit install.

CUDA_PATH_V11_0 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0
CUDA_PATH C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0

Following this issue instead of removing -std=c++14 from extra_compile_args in setup.py, commenting out "-std=c++14" in nvcc_args worked for me. I did check if the suggested method in the issue works.

Now running python setup.py install from pytorch3d source folder should start compile and install it.

Install following requirements in conda env to run demos and examples.

conda install jupyter
pip install scikit-image matplotlib imageio plotly opencv-python

Some of the steps mentioned above is probably not needed. There is no need to modify any header files. This is the source that worked for me, https://github.com/facebookresearch/pytorch3d/tree/bfeb82efa38f29ed5b9cf8d8986fab744fe559ea.


Instruction: Pytorch installation on windows

Input:
I have tried many suggestions online to install in my virtual env "torch" but to no avail. It won't let me import torch. I am able to install torchvision through conda though following this link: https://pytorch.org/get-started/locally/. Any suggestions are welcome!

Here is the error message (I downgrade to python 3.5 in the virtualenv)

(env_peem) PS E:\Users\Maggie\TS_MatSeg_Share> pip install torch
Collecting torch\Users\Maggie\TS_MatSeg_Share>
  Using cached torch-0.1.2.post2.tar.gz (128 kB)
Requirement already satisfied: pyyaml in e:\users\maggie\ts_matseg_share\env_peem\lib\site-packages (from torch) (5.3.1)
Building wheels for collected packages: torch
  Building wheel for torch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'E:\Users\Maggie\AppData\Local\Temp\7\pip-wheel-xn9ld0bu'
       cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\
  Complete output (30 lines):
  running bdist_wheel
  running build
  running build_deps
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 265, in <module>
      description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\setuptools\__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 51, in run
      from tools.nnwrap import generate_wrappers as generate_nn_wrappers
  ImportError: No module named 'tools.nnwrap'
  ----------------------------------------
  ERROR: Failed building wheel for torch
  Running setup.py clean for torch
  ERROR: Command errored out with exit status 1:
   command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch
  Complete output (2 lines):
  running clean
  error: [Errno 2] No such file or directory: '.gitignore'
  ----------------------------------------
  ERROR: Failed cleaning build dir for torch
Failed to build torch
Installing collected packages: torch
    Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'E:\Users\Maggie\AppData\Local\Temp\7\pip-record-z6gt4ig9\install-record.txt' --single-version-externally-managed --compile --install-headers 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\include\site\python3.5\torch'
         cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\setuptools\__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ImportError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'E:\Users\Maggie\AppData\Local\Temp\7\pip-record-z6gt4ig9\install-record.txt' --single-version-externally-managed --compile --install-headers 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\include\site\python3.5\torch' Check the logs for full command output.

Output:
It tries to install torch-0.1.2.post2.tar.gz, which is an extremely outdated version, and it probably was more involved to install it back then. You don't want that version, but rather the most recent one, which currently is 1.5.0.

You didn't select the appropriate configuration from PyTorch - Getting Started Locally, because if you select Windows, which is the operating system you are using, the installation command is:

pip install torch===1.5.0 torchvision===0.6.0 -f https://download.pytorch.org/whl/torch_stable.html

That is because PyTorch does not publish the Windows versions to PyPI anymore, and it needs to be installed from their custom registry.


Instruction: How did Pytorch process images in ImageNet when training resnet pretrained models in torchvision.models.resnet34?

Input:
I downloaded the pretrained parameters of resnet34 in torchvision.models and put them to a tensorflow1.X network, but just get 58% accurary testing on the ImageNet2015 Validation set (50,000 picture).

I guess it may be caused by the different precessing method to the data set. The Validation I am using is in TFRecord format processed by my friend.

So I am wondering how Pytorch process images of ImageNet when training resnet34 pretrained models? Turn RGB to BGR? Scale pictures values to 0--1?

Output:
From the documentation:

  All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. You can use the following transform to normalize:

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])```

The documentation links this example for preprocessing ImageNet data.


Instruction: Prunning model doesn't improve inference speed or reduce model size

Input:
I'm trying to prune my model in PyTorch with torch.nn.utils.prune, which provides 2 tensors,

one is the original weight and
the other is a mask contain 0s and 1s that help us close certain connections in the network.

I have tried both of the solutions, but none improve the inference speed:

Use the network after pruning to infer which will first close some connections with the mask and then run inference.
Zeros out the original weights with the mask and then remove the mask from the state_dict to infer.

Is there a way to improve the speed with the model tensor and the mask? Doesn't multiply with a non-zero float number with 0 will faster than multiply 2 floats with each other?
Here is my prune function and the pruning speed calculating procedure:

def prune_net(net):
    """Prune 20% net's weights that have abs(value) approx. 0
    Function that will be use when an iteration is reach
    Args:

    Return:
        newnet (nn.Module): a newnet contain mask that help prune network's weight
    """
    if not isinstance(net,nn.Module):
        print('Invalid input. Must be nn.Module')
        return
    newnet = copy.copy(net)
    modules_list = []

    for name, module in newnet.named_modules():
        if isinstance(module, torch.nn.Conv2d):
            modules_list += [(module,'weight'),(module,'bias')]
        if isinstance(module, torch.nn.Linear):
            modules_list += [(module,'weight'),(module,'bias')]

    prune.global_unstructured(
        modules_list,
        pruning_method=prune.L1Unstructured,
        amount=0.2,)
    return newnet

Test inference speed 1st case:

import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
import time
from torch.autograd import Variable

torch.set_default_tensor_type('torch.cuda.FloatTensor')
old_net = init_your_net()

new_net = prune_net(old_net)
new_net = prune_net(new_net)

old_net.eval()
new_net.eval()

old_net = old_net.cuda()
new_net = new_net.cuda()
dataset = load_your_dataset()

for i in range(100):
    x = dataset[i]
    x = x.cuda()
    y = x.cuda()

    #new infer
    start_time = time.perf_counter()
    detections = new_net(x).data
    time_new += time.perf_counter() - start_time

    #old infer
    start_time = time.perf_counter()
    detections = old_net(y).data
    time_old += time.perf_counter() - start_time
print('old ',time_old)
print('new ', time_new)

Test inference speed 2nd case:

import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
import time
from torch.autograd import Variable

torch.set_default_tensor_type('torch.cuda.FloatTensor')
old_net = init_your_net()

new_net = prune_net(old_net)
new_net = prune_net(new_net)
# Apply mask to model tensor and remove mask from state_dict
for name, module in new_net.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        prune.remove(module,'weight')
        prune.remove(module,'bias')
    if isinstance(module, torch.nn.Linear):
        prune.remove(module,'weight')
        prune.remove(module,'bias')

old_net.eval()
new_net.eval()

old_net = old_net.cuda()
new_net = new_net.cuda()
dataset = load_your_dataset()

for i in range(100):
    x = dataset[i]
    x = x.cuda()
    y = x.cuda()

    #new infer
    start_time = time.perf_counter()
    detections = new_net(x).data
    time_new += time.perf_counter() - start_time

    #old infer
    start_time = time.perf_counter()
    detections = old_net(y).data
    time_old += time.perf_counter() - start_time
print('old ',time_old)
print('new ', time_new)

UPDATE
I found torch have a sparse module that can reduce memory usage if we prune enough parameter but it hasn't support nn.Module yet, only Tensor object. Here are some useful link:
https://github.com/pytorch/pytorch/issues/36214#issuecomment-619586452
https://pytorch.org/docs/stable/sparse.html

Output:
It is important to understand the difference between unstructured pruning and structured pruning.

Structured pruning: the dimensions of the weight tensors are reduced by removing entire rows/columns of the tensors. This translates into removing neurons with all their incoming and outgoing connections (in dense layers) or entire convolutional filters (in convolutional layers).

Unstructured pruning: individual weights can be "removed" (zeroed-out) without constraints of the shape of the final tensor. This translates into removing individual connections between neurons (in dense layers) or removing individual weights of the convolutional filters (in convolutional layers). Notice that the resulting weight tensors can be sparse but maintain their original shape.

Currently, torch.nn.utils.prune only supports unstructured pruning, which hardly helps to reduce the inference cost because GPUs are not optimized for sparse matrix multiplications. While you might want to reduce the dimensions of your weight tensors to reduce the number of floating-point operations, unstructured pruning produces weight tensors with many zeros but does not automatically reduce the size of such tensors.
Unstructured pruning can help improve the performance only when a lot of weights are removed. In this case, you can either rely on PyTorch sparse operations or try to find rows/columns that contain all zeros and thus can be removed.
Instead, if you want to look into structured pruning, you can take a look at TorchPruner, a library that I have developed myself for research purposes and that provides utilities to find the least important neurons and slice the weight tensors accordingly.


Instruction: How to rename Pytorch object name?

Input:
My Pytorch Model:

EfficientDet(
  (backbone): EfficientNetFeatures(
    (conv_stem): Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
...
...

Is there a way to rename backbone object to some other name?

Output:
We can rename an attribute of an instance using the following function.

def rename_attribute(obj, old_name, new_name):
    obj._modules[new_name] = obj._modules.pop(old_name)

Example

class EfficientNetFeatures(nn.Module):
    def __init__(self):
        super(EfficientNetFeatures, self).__init__()
        self.conv_stem = nn.Conv2d(4, 48, kernel_size=(3, 3),
                                   stride=(2, 2), padding=(1, 1), bias=False)
        self.bn1 = nn.BatchNorm2d(48, eps=0.001, momentum=0.1,
                                  affine=True, track_running_stats=True)

class EfficientDet(nn.Module):
    def __init__(self):
        super(EfficientDet, self).__init__()
        self.backbone = EfficientNetFeatures()

model = EfficientDet()
print(model)

rename_attribute(model, 'backbone', 'newname')
print(model)

Outputs:

EfficientDet(
  (backbone): EfficientNetFeatures(
    (conv_stem): Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
)

EfficientDet(
  (newname): EfficientNetFeatures(
    (conv_stem): Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
)


Instruction: TypeError: __call__() takes 2 positional arguments but 3 were given. To train Raccoon prediction model using FastRCNN through Transfer Learning

Input:
 from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
 from engine import train_one_epoch, evaluate
 import utils
 import torchvision.transforms as T

 num_epochs = 10
 for epoch in range(num_epochs):
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
    lr_scheduler.step()
    evaluate(model, data_loader_test, device=device)

I am using the same code as provided in this link Building Raccoon Model but mine is not working.

This is the error message I am getting
TypeError                                 Traceback (most recent call last)
 in ()

  2 for epoch in range(num_epochs):

  3    # train for one epoch, printing every 10 iterations

  4   ----> train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)

  5     # update the learning rate

  6   lr_scheduler.step()

7 frames

 in getitem(self, idx)

 29         target["iscrowd"] = iscrowd

 30         if self.transforms is not None:

 31        ---> img, target = self.transforms(img, target)

 32         return img, target
 33

TypeError: call() takes 2 positional arguments but 3 were given

Output:
The above answer is incorrect, I accidentally upvoted before noticing. You are using the wrong Compose, note that it says
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#putting-everything-together
"In references/detection/, we have a number of helper functions to simplify training and evaluating detection models. Here, we will use references/detection/engine.py, references/detection/utils.py and references/detection/transforms.py. Just copy them to your folder and use them here."
there are helper scripts. They subclass the compose and flip methods
https://github.com/pytorch/vision/blob/6315358dd06e3a2bcbe9c1e8cdaa10898ac2b308/references/detection/transforms.py#L17
I did the same thing before noticing this. Do not use the compose method from torchvision.transforms, or else you will get the error above. Download their module and load it.


Instruction: How to use permute on this Input and Target?

Input:
I am having errors on my semantic segmentation masks with 5 classes + 1 (background).

How do I use permute to avoid this?

Target size (torch.Size([4, 1, 320, 480, 6])) must be the same as input size (torch.Size([4, 6, 320, 480]))

Output:
You can combine permute and unsqueeze:

import torch

x = torch.rand((4, 6, 320, 480))
new_x = x.permute((0,2,3,1)).unsqueeze(1)
# new_x.shape = torch.Size([4, 1, 320, 480, 6])


Instruction: Python Neural Network: 'numpy.ndarray' object has no attribute 'dim'

Input:
I am using this database for modeling
http://archive.ics.uci.edu/ml/datasets/Car+Evaluation

after preprocessing

    X_train = df.drop('class', axis=1).to_numpy()
    y_train = df['class'].to_numpy()

    X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size=0.2)

class

class network(nn.Module):

def __init__(self, input_size, hidden1_size, hidden2_size, num_classes):
    super(network, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden1_size)
    self.relu1 = nn.ReLU()
    self.fc2 = nn.Linear(hidden1_size, hidden2_size)
    self.relu2 = nn.ReLU()
    self.fc3 = nn.Linear(hidden2_size, num_classes)

  def forward(self, x):
    out = self.fc1(x)
    out = self.relu1(out)
    out = self.fc2(out)
    out = self.relu2(out)
    out = self.fc3(out)
    return out

net = network(input_size=6, hidden1_size=5, hidden2_size=4, num_classes=4)
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

Error is in this block

plt.ion()

for t in range(200):
    prediction = net(X_train)     # input x and predict based on x

    loss = loss_func(prediction, y_train)     # must be (1. nn output, 2. target)

    optimizer.zero_grad()   # clear gradients for next train
    loss.backward()         # backpropagation, compute gradients
    optimizer.step()        # apply gradients

    if t % 5 == 0:
        # plot and show learning process
        plt.cla()
        plt.scatter(x.data.numpy(), y.data.numpy())
        plt.plot(x.data.numpy(), prediction.data.numpy(), 'r-', lw=5)
        plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 20, 'color':  'red'})
        plt.pause(0.1)

plt.ioff()
plt.show()

Error message

AttributeError                            Traceback (most recent call last)
 in ()
      2
      3 for t in range(200):
----> 4     prediction = net(X_train)     # input x and predict based on x
      5
      6     loss = loss_func(prediction, y_train)     # must be (1. nn output, 2. target)

> 4 frames /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py
> in linear(input, weight, bias)    1606         if any([type(t) is not
> Tensor for t in tens_ops]) and has_torch_function(tens_ops):    1607
> return handle_torch_function(linear, tens_ops, input, weight,
> bias=bias)
> -> 1608     if input.dim() == 2 and bias is not None:    1609         # fused op is marginally faster    1610         ret = torch.addmm(bias, input, weight.t())
>
> AttributeError: 'numpy.ndarray' object has no attribute 'dim'

Output:
in prediction = net(X_train), X_train is a numpy array, but torch expects a tensor.

You need to convert to torch tensor, and move to gpu if you want

the 1st line should be

X_train = torch.from_numpy(df.drop('class', axis=1).to_numpy())


Instruction: PyTorch ToTensor Changes C x H x W (5 x 600 x 900) to H x C x W (900 x 5 x 600)

Input:
Here is my DataLoader. When I use ToTensor, it changes the dimensions of the image to H x C x W. Is permute okay to fix this or this might change some orientation?

class DPWHDataset(Dataset):
  def __init__(self, mean=None, std=None, phase=None, dataset=None):
    self.data = dataset
    self.mean = mean
    self.std = std
    self.phase = phase
    self.transforms = get_transforms(phase, mean, std)

  def __len__(self):
    return len(self.data)

  def __getitem__(self, idx):
    image_name = self.data[idx]

    image_path = image_prefix + image_name + ".jpg"
    mask_path = binary_mask_prefix + image_name + "_mask.png"

    mask = cv2.imread(mask_path, 0)
    print(image_path)

    # image = np.array(Image.open(image_path))
    # mask = np.array(Image.open(mask_path))
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    mask = create_channel_mask(mask)
    # augmented = self.transforms(image=image, mask=mask)
    # image = augmented['image']
    # mask = augmented['mask']

    image = torchvision.transforms.ToTensor()(image)
    image = torchvision.transforms.Normalize(mean=self.mean, std=self.std)(image)
    mask = torchvision.transforms.ToTensor()(mask)
    return image, mask

Output:
According to the documentation, torchvision.transforms.ToTensor converts a PIL Image or numpy.ndarray (H x W x C) to a torch.FloatTensor of shape (C x H x W).
So, in the following line:

image = torchvision.transforms.ToTensor()(image)

The resultant image tensor is of shape (C x H x W) and the input tensor is of shape (H x W x C). You can verify this by printing the tensor shapes.

And yes, you can adjust the shape using torch.permute, it won't cause any issue.


Instruction: Is A PyTorch Dataset Accessed by Multiple DataLoader Workers?

Input:
When using more than 1 DataLoader workers in PyTorch, does every worker access the same Dataset instance? Or does each DataLoader worker have their own instance of Dataset?

from torch.utils.data import DataLoader, Dataset

class NumbersDataset(Dataset):
    def __init__(self):
        self.samples = list(range(1, 1001))

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        return self.samples[idx]

dataset = NumbersDataset()
train_loader = DataLoader(dataset, num_workers=4)

Output:
It seams like they are accessing to the same instance. I have tried adding a static variable inside the dataset class and incrementing it every time a new instance is created. Code can be found below.

from torch.utils.data import DataLoader, Dataset

class NumbersDataset(Dataset):
    i = 0

    def __init__(self):
        NumbersDataset.i += 1
        self.samples = list(range(1, 1001))

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        return self.samples[idx]

dataset_1 = NumbersDataset()
train_loader = DataLoader(dataset_1, num_workers=4)

for i, data in enumerate(train_loader):
    pass

dataset_2 = NumbersDataset()
train_loader = DataLoader(dataset_2, num_workers=4)

for i, data in enumerate(train_loader):
    pass

print(NumbersDataset.i)

The output is 2. Hope it helps :D


Instruction: Installing torchaudio in Windows

Input:
I am trying to install torchaudio in Windows from source. I installed sox and added it in the path env variable. Then I run

python setup.py install

cloned from the GitHub. When I import torchaudio, I get warning

  No audio backend is available.

I think this means that the sox is not configured correctly or I have done something very wrong. How to install torchaudio in Windows? My Linux machine is not good and I can't use Linux which has CUDA.

Output:
You need to install the backend.
On Windows it's PySoundFile
pip install PySoundFile

This should do the trick
You can follow torch documentation here
https://pytorch.org/audio/backend.html


Instruction: PyTorch: how to apply another transform to an existing Dataset?

Input:
This is a code example:

dataset = datasets.MNIST(root=root, train=istrain, transform=None)  #preserve raw img

print(type(dataset[0][0]))
# <class 'PIL.Image.Image'>

dataset = torch.utils.data.Subset(dataset, indices=SAMPLED_INDEX) # for resample

for ind in range(len(dataset)):
    img, label = dataset[ind] # <class 'PIL.Image.Image'> <class 'int'>/<class 'numpy.int64'>
    img.save(fp=os.path.join(saverawdir, f'{ind:02d}-{int(label):02d}.png'))

dataset.transform = transforms.Compose([
                transforms.RandomResizedCrop(image_size),
                transforms.ToTensor(),
                transforms.Normalize((0.1307,), (0.3081,))
            ])
#transform for net forwarding

print(type(dataset[0][0]))
# expected <class 'torch.Tensor'>, however it's still <class 'PIL.Image.Image'>

Since dataset is randomly resampled, I don't want to reload a new dataset with transform, but just apply transform to the already existing dataset.

Thanks for your help :D

Output:
You can create a small wrapper Dataset that will take care  of applying the given transform to the underlying dataset on the fly:
Here's an example that was posted over on the pytorch forums: https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/4
class MyDataset(Dataset):
    def __init__(self, subset, transform=None):
        self.subset = subset
        self.transform = transform

    def __getitem__(self, index):
        x, y = self.subset[index]
        if self.transform:
            x = self.transform(x)
        return x, y

    def __len__(self):
        return len(self.subset)

With your code it could look something like:
dataset = datasets.MNIST(root=root, train=istrain, transform=None)  #preserve raw img

print(type(dataset[0][0]))
# <class 'PIL.Image.Image'>

dataset = torch.utils.data.Subset(dataset, indices=SAMPLED_INDEX) # for resample

transformed_dataset = TransformDataset(dataset, transform=transforms.Compose([
                transforms.RandomResizedCrop(image_size),
                transforms.ToTensor(),
                transforms.Normalize((0.1307,), (0.3081,))
            ]))


Instruction: (pytorch / mse) How can I change the shape of tensor?

Input:
Problem definition:

I have to use MSELoss function to define the loss to classification problem. Therefore it keeps saying the error message regarding the shape of tensor.

Entire error message:

  torch.Size([32, 10]) torch.Size([32])
  --------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call
  last)  in
       53     output = model.forward(images)
       54     print(output.shape, labels.shape)
  ---> 55     loss = criterion(output, labels)
       56     loss.backward()
       57     optimizer.step()

  /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in
  call(self, *input, **kwargs)
      530             result = self._slow_forward(*input, **kwargs)
      531         else:
  --> 532             result = self.forward(*input, **kwargs)
      533         for hook in self._forward_hooks.values():
      534             hook_result = hook(self, input, result)

  /opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in
  forward(self, input, target)
      429
      430     def forward(self, input, target):
  --> 431         return F.mse_loss(input, target, reduction=self.reduction)
      432
      433

  /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in
  mse_loss(input, target, size_average, reduce, reduction)    2213
  ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
  2214     else:
  -> 2215         expanded_input, expanded_target = torch.broadcast_tensors(input, target)    2216         ret =
  torch._C._nn.mse_loss(expanded_input, expanded_target,
  _Reduction.get_enum(reduction))    2217     return ret

  /opt/conda/lib/python3.7/site-packages/torch/functional.py in
  broadcast_tensors(*tensors)
       50                 [0, 1, 2]])
       51     """
  ---> 52     return torch._C._VariableFunctions.broadcast_tensors(tensors)
       53
       54

  > RuntimeError: The size of tensor a (10) must match the size of tensor
  b (32) at non-singleton dimension 1

How can I reshape the tensor, and which tensor (output or labels) should I change to calculate the loss?

Entire code is attached below.

import numpy as np
import torch

# Loading the Fashion-MNIST dataset
from torchvision import datasets, transforms

# Get GPU Device
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (0.5,))])
# Download and load the training data
trainset = datasets.FashionMNIST('MNIST_data/', download = True, train = True, transform = transform)
testset = datasets.FashionMNIST('MNIST_data/', download = True, train = False, transform = transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size = 32, shuffle = True, num_workers=4)
testloader = torch.utils.data.DataLoader(testset, batch_size = 32, shuffle = True, num_workers=4)

# Examine a sample
dataiter = iter(trainloader)
images, labels = dataiter.next()

# Define the network architecture
from torch import nn, optim
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim = 1))
model.to(device)

# Define the loss
criterion = nn.MSELoss()

# Define the optimizer
optimizer = optim.Adam(model.parameters(), lr = 0.001)

# Define the epochs
epochs = 5

train_losses, test_losses = [], []

for e in range(epochs):
  running_loss = 0
  for images, labels in trainloader:
    # Flatten Fashion-MNIST images into a 784 long vector
    images = images.to(device)
    labels = labels.to(device)
    images = images.view(images.shape[0], -1)

    # Training pass
    optimizer.zero_grad()
    output = model.forward(images)
    print(output.shape, labels.shape)
    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()

    running_loss += loss.item()
  else:
    test_loss = 0
    accuracy = 0

    # Turn off gradients for validation, saves memory and computation
    with torch.no_grad():
      # Set the model to evaluation mode
      model.eval()

      # Validation pass
      for images, labels in testloader:
        images = images.to(device)
        labels = labels.to(device)
        images = images.view(images.shape[0], -1)
        ps = model(images)
        test_loss += criterion(ps, labels)
        top_p, top_class = ps.topk(1, dim = 1)
        equals = top_class == labels.view(*top_class.shape)
        accuracy += torch.mean(equals.type(torch.FloatTensor))

    model.train()

    print("Epoch: {}/{}..".format(e+1, epochs),
          "Training loss: {:.3f}..".format(running_loss/len(trainloader)),
          "Test loss: {:.3f}..".format(test_loss/len(testloader)),
          "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

Output:
From the output you print before it error, torch.Size([32, 10]) torch.Size([32]).

The left one is what the model gives you and the right one is from trainloader, normally you use this for something like nn.CrossEntropyLoss.

And from the full error log, the error is from this line

loss = criterion(output, labels)

The way to make this work is called One-hot Encoding, if it's me for sake of my laziness I'll write it like this.

ones = torch.sparse.torch.eye(10).to(device)  # number of class class
labels = ones.index_select(0, labels)


Instruction: Horovod Elasticity Adjust number of Workers in RunTime

Input:
I have been using Horovod using both TensorFlow and PyTorch in docker, every thing works fine under a fixed number of containers as explained in Horovod docker

I have checked the Horovod Elastic Demos Horovod examples but they doesn't show how to change the number of workers in runtime.

What I need is to know how to change the number of workers up or down by runtime?

Output:
What you need for that is a Docker-specific host discovery that tells Elastic Horovod about all available containers. A generic way to do this is by using horovodrun and providing a host discovery script via --host-discovery-script. When invoked, the script returns a list of available hosts. See the Running with horovodrun section of the Elastic Horovod documentation.

In the near future there will be service provider specific host discoveries built into Horovod so users do not need to implement scripts for common providers.


Instruction: Pytorch Error, RuntimeError: expected scalar type Long but found Double

Input:
I have run into the following error while training a BERT classifier. The

type(b_input_mask) = type(b_labels) = torch.Tensor

type(b_labels[i]) = tensor(1., dtype=torch.float64)

type(b_input_masks[i]) = class'torch.Tensor'

What could be the possible data type error here since I have not typecasted any variable to either long or double?

Thanks in advance!

Output:
In a classification task, the data type for input labels should be Long but you assigned them as float64

type(b_labels[i]) = tensor(1., dtype=torch.float64)

=>

type(b_labels[i]) = tensor(1., dtype=torch.long)