Instruction: How to differentiate a gradient in Pytorch

Input:
I'm trying to differentiate a gradient in PyTorch. I found this link but can't get it to work.
My code looks as follows:
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(2, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 4)

        def forward(self, x):
            x=self.fc1(x)
            x=self.fc2(x)
            x=self.out(x)
            return x

nx = net_x()
r = torch.tensor([1.0,2.0])
nx(r)
>>>tensor([-0.2356, -0.7315, -0.2100, -0.6741], grad_fn=<AddBackward0>)

But when I try to differentiate the function with respect to the first parameter
grad(nx, r[0])

I get the error
TypeError: 'net_x' object is not iterable

Update
Trying to extend this to tensors:
For some reason the gradient is the same for all inputs.
a = torch.rand((8,2), requires_grad=True)
s = []
s_t = []
for input_tensor in a:
    output_tensor = nx(input_tensor)
    s.append(output_tensor[0])
    s_t_value = grad(output_tensor[0], input_tensor)[0][0]
    s_t.append(s_t_value)
print(s_t)

But the output is:
[tensor(-0.1326), tensor(-0.1326), tensor(-0.1326), tensor(-0.1326), tensor(-0.1326), tensor(-0.1326), tensor(-0.1326), tensor(-0.1326)]

Output:
First thing to change if you want to have the gradients with respect to r is to set the requires_grad flag to True for this tensor :
nx = net_x()
r = torch.tensor([1.0,2.0], requires_grad=True)

Then, as explained in autograd documentation, grad computes the gradients of oputputs with respect to the inputs, so you need to save the output of the model :
y = nx(r)

Now you can compute the gradients with respect to r. But there is one last issue : grad only knows how to propagate gradients from a scalar tensor, which y is not. So you need to compute the gradients with respect to each coordinate :
for x in y:
    print(grad(x, r, retain_graph=True))

or equivalently:
for i in range(y.shape[0]):
    # prints the vector (dy_i/dr_0, dy_i/dr_1, ... dy_i/dr_n)
    print(grad(y[i], r, retain_graph=True))

You need to retain_graph because without this flag, the computational graph is cleared after the first gradient propagation. And there you have it, the derivative of each coordinate of nx(r) with respect to r !
To answer your question in the comments :
Not an error, it's normal. So you have a batched input of size (B, 2), with B = 8. You get a batched output of shape (B, 4). Now, for each vector of the batched output, for each coordinate of this vector, you can compute the derivative with respect to the batched input, which will yield a gradient of size (B,2), like that :
for b in y: # There a B vectors b of shape (4)
    for x in b: # There are 4 coordinates
        # This prints a tensor of shape (B, 2)
        print(grad(x, r, retain_graph=True))

Now remember the way batches work : all batches are computed together to harvest the power of GPU, but they are actually completely independant. So al b vectors are actually results of the network from different inputs. Which means, the gradient of the i-th vector b with respect to the j-th vector of the input must be 0 if i!=j. Does that make sense ? It's like computing f(x,y) = (x^2, y^2). The derivative of y^2 with respect to x is obviously 0 ! Well consider x and y to be two samples from one batch, and you have you explaination for why there are a lot of 0 in your results.
A last sample of code to make it even clearer :
inputs = [torch.randn(1, 2, requires_grad=True) for i in range(8)]
r = torch.cat(inputs) # shape : (8, 2)
y = nx(r) # shape : (8, 4)
for i in range(len(y)):
    print(f"Gradients of y[{i}] wrt r[{i}]")
    for x in y[i]:
        # prints a tensor of size (2)
        print(grad(x, inputs[i], retain_graph=True))

On to why all the gradients are the same. This is because your neural network is completely linear. You have 3 nn.Linear layers, and no non-linear activation function (as a consequence, this is literally equivalent to a network with only one layer). One property of linear layers is that their gradient is constant : d(alpha*x)/dx = alpha (independant of x). Therefore the gradients will be identical along all dimensions. Just add non-linear activation layers like sigmoids and this behavior will not happen again.


Instruction: PyTorch ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([15]))

Input:
I'm currently using this repo to perform NLP and learn more about CNN's using my own dataset, and I keep running into an error regarding a shape mismatch:
ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([15]))

     10 }
     11 for epoch in tqdm(range(params['epochs'])):
---> 12     train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
     13     valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
     14     epoch_mins, epoch_secs = epoch_time(start_time, end_time)

     57         print("PredictionShapeAfter:")
     58         print(predictions.shape)
---> 59         loss = criterion(predictions, batch.l)
     60
     61         acc = binary_accuracy(predictions, batch.l)

Doing some digging, I found that my CNN's prediction is a different size compared to the training data truth it's being compared to:
  Input Shape:
    torch.Size([15, 64])
    Truth Shape:
    torch.Size([64])
    embedded unsqueezed: torch.Size([15, 1, 64, 100])
    cat shape: torch.Size([15, 300])
    Prediction Shape Before Squeeze:
    torch.Size([15, 1])
    PredictionShapeAfter:
    torch.Size([15])

The model is making the prediction shape (the last value in this list) as the first dimension of the inputs. Is this a common problem and is there a way to rectify this issue?
My Model:
class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim,
                 dropout, pad_idx):
        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)

        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1,
                                              out_channels = n_filters,
                                              kernel_size = (fs, embedding_dim))
                                    for fs in filter_sizes
                                    ])

        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        embedded = self.embedding(text)
        embedded = embedded.unsqueeze(1)
        print(f"embedded unsqueezed: {embedded.shape}")
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        cat = self.dropout(torch.cat(pooled, dim = 1))
        print(f"cat shape: {cat.shape}")
        return self.fc(cat)

My Training function:
def train(model, iterator, optimizer, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        print("InputShape:")
        print(batch.t.shape)
        print("Truth Shape:")
        print(batch.l.shape)

        predictions = model(batch.t)
        print("Prediction Shape Before Squeeze:")
        print(predictions.shape)

        predictions = predictions.squeeze(1)
        print("PredictionShapeAfter:")
        print(predictions.shape)
        loss = criterion(predictions, batch.l)

        acc = binary_accuracy(predictions, batch.l)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

My full code can be found at this link.

Output:
Your issue is here:
self.convs = nn.ModuleList([
                            nn.Conv2d(in_channels = 1,
                                      out_channels = n_filters,
                                      kernel_size = (fs, embedding_dim))
                            for fs in filter_sizes
                            ])

You are inputting data of shape [15, 1, 64, 100], which the convolutions are interpreting as batches of size 15, of 1-channel images of HxW 64x100.
What it appears you want is a batch of size 64, so swap those dimensions first:
...
embedded = embedded.swapdims(0,2)
conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
...


Instruction: The best way to implement stateful LSTM/ConvLSTM in Pytorch?

Input:
I am trying to boost the performance of a object detection task with sequential information, using ConvLSTM.
A typical ConvLSTM model takes a 5D tensor with shape (samples, time_steps, channels, rows, cols) as input.
as stated in this post, a long sequence of 500 images need to be split into smaller fragments in the Pytorch ConvLSTM layer. For example, it could be split into 10 fragements with each having 50 time steps.

I have two goals:

I want the network to remember the state across the 10 fragment sequences. I.e. how to pass the hidden state between the fragements?

I want to feed in the images (of the video) one by one. I.e. the long sequence of 500 images is split into 500 fragments with each one having only one image. The input should be like (all_samples, channels, rows, cols). This only make sense if the 1.goal could be achieved.

I found some good answers for Tensorflow, but I am using Pytorch.
TensorFlow: Remember LSTM state for next batch (stateful LSTM)
The best way to pass the LSTM state between batches
What is the best way to implement stateful LSTM/ConvLSTM in Pytorch?

Output:
I found this post has a good example
model = nn.LSTM(input_size = 20, hidden_size = h_size)
out1, (h1,c1) = model(x1)
out2, (h2,c2) = model(x2, (h1,c1))


Instruction: filter class/subfolder with pytorch ImageFolder

Input:
Here's my folder structure
image-folders/
   ├── class_0/
   |   ├── 001.jpg
   |   ├── 002.jpg
   └── class_1/
   |   ├── 001.jpg
   |   └── 002.jpg
   └── class_2/
       ├── 001.jpg
       └── 002.jpg

By using ImageFolder from torchvision, I can create dataset with this syntax :
dataset = ImageFolder("image-folders",...)
But this will read the entire subfolder and create 3 target classes. I don't want to include the class_2 folder, I want my dataset to only contains class_0 and class_1 only, is there any way to achieve this besides delete/move the class_2 folder?

Output:
You can do this by using torch.utils.data.Subset of the original full ImageFolder dataset:
from torchvision.datasets import ImageFolder
from torch.utils.data import Subset

# construct the full dataset
dataset = ImageFolder("image-folders",...)
# select the indices of all other folders
idx = [i for i in range(len(dataset)) if dataset.imgs[i][1] != dataset.class_to_idx['class_s']]
# build the appropriate subset
subset = Subset(dataset, idx)


Instruction: Target value with torch.nn.MultiLabelSoftMarginLoss should be 0 or -1?

Input:
I have a multi-label classification problem (A single sample can be classified as several classes at the same time).
I want to use torch.nn.MultiLabelSoftMarginLoss but I got confused with the documentation where the ground truth are written like this :
Target: (N, C)(N,C) , label targets padded by -1 ensuring same shape as the input.

Does that mean the target is in one hot form, but the zero replace with -1?
Let's say I want to classified several attributes for object detection such as : Man, Tall, Long hair.
My first image is a tall woman with long hair, does my Target become 0 1 1 or -1 1 1 ? I can't fathom why use -1 instead of 0
It's quite hard to find example in internet since a lot of people mistook multi-label task as multiple class classification and keep using BCELoss.

Output:
Look closer at the doc:

The targets are expected to be {0, 1} and not -1.
I'm not sure what this -1 is doing, it might be for "ignore", but you are correct that the doc there is not very clear.
There is an open issue on pytorch's github about this. Feel free to contribute.


Instruction: How to add two separate layers on the top of one layer using pytorch?

Input: I want to add two separate layers on the top of one layer (or a pre-trained model) Is that possible for me to do using Pytorch?

Output:
Yes, when defining your model's forward function, you can specify how the inputs should be passed through the layers.
For example:
def forward(self, X):
    X = self.common_layer(X)
    X = self.activation_fn(X)
    Xa = self.layer_a(X)
    Xb = self.layer_b(X)
    # now combine outputs of the parallel layers however you please
    return self.combining_layer(torch.cat([Xa, Xb]))

Where forward is a member of MyNet:
class MyNet(nn.Module):
   def __init__(self):
       # define common_layer, activation_fn, layer_a, layer_b, and combining_layer

Training
The model should be trained, just like any other pytorch model.
Because the combined output goes through both layer_a and layer_b, computing gradient of the loss will optimize paramaters for both layers. The layers will be optimized seperately as the parameters of their object are independent.
For example.
model = MyNet()
...
optimizer.zero_grad()
predictions = model(input_batch)
loss = my_loss_fn(predictions, ground_truth)
loss.backward()
optimizer.step()


Instruction: Align feature map with ego motion (problem of zooming ratio )

Input:
I want to align the feature map using ego motion, as mentioned in the paper An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
I use VoxelNet as backbone, which will shrink the image for 8 times. The size of my voxel is 0.1m x 0.1m x 0.2m(height)
So given an input bird-eye-view image size of 1408 x 1024,
the extracted feature map size would be 176 x 128, shrunk for 8 times.

The ego translation of the car between the "images"(point clouds actually) is 1 meter in both x and y direction. Am I right to adjust the feature map for 1.25 pixels?
1m/0.1m = 10  # meter to pixel
10/8 = 1.25   # shrink ratio of the network

However, though experiments, I found the feature maps align better if I adjust the feature map with only 1/32 pixel for the 1 meter translation in real world.

Ps. I am using the function torch.nn.functional.affine_grid to perform the translation, which takes a 2x3 affine matrix as input.

Output:
It's caused by the function torch.nn.functional.affine_grid I used.
I didn't fully understand this function before I use it.
These vivid images would be very helpful on showing what this function actually do(with comparison to the affine transformations in Numpy.


Instruction: CUDA Illegal Memory Access error when using torch.cat

Input:
I was playing around with pytorch concatenate and wanted to see if I could use an output tensor that had a different device to the input tensors, here is the code:
import torch
a = torch.ones(4)
b = torch.ones(4)
c = torch.zeros(8).cuda()
print(c)
ab = torch.cat([a,b], out=c)
print(c)

I am running this inside a jupyter notebook. pytorch version: 1.7.1
I get the following error:
...
\Anaconda3\envs\...\lib\site-packages\torch\_tensor_str.py in __init__(self, tensor)
     87
     88         else:
---> 89             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
     90
     91             if nonzero_finite_vals.numel() == 0:

RuntimeError: CUDA error: an illegal memory access was encountered

It happens if you try to access the tensors c (in this case with a print).
I couldnt find anything in the  documentation that said I couldn't do this, other than perhaps this line:
" ... any python sequence of tensors of the same type ... "
The error is kind of curious though... any ideas?

Output:
It appears that the behaviors changes according to the version of pytorch. With the version 1.3.0 I get the error expected object of backend CUDA but got CPU, but the version 1.5.0 I do indeed get the same error as you do. This would probably be worth mentioning on their github, because I believe the former error is more useful than the latter.
Anyway, both errors come from the fact that you concatenate cpu tensors into a GPU one. You can solve it very easily :
# Move the tensors to the GPU prior to concatenating
ab = torch.cat([a.cuda(),b.cuda()], out=c)

or
# Move the tensor after concatenating
c.copy_(torch.cat([a,b]).cuda())

I don't have a notebook but I believe you will have to restart your kernel, the error you get seems to break it down really bad. My python shell just cannot compute anything anymore after getting the illegal memory access.


Instruction: image translation in Pytorch, using affine_grid & grid_sample functions

Input:
I am going to move the image for 1 or 2 pixels, as I specified a small number (1.25 , 1.9) in the affine matrix.
BUT, the image is moved far far away, like hundreds of pixels:

( my input image is fully filled with yellow pineapples)
Below is a working example.
import torch
import numpy as np
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
import torch.nn.functional as F

rotation_simple = np.array([[1,0, 1.25],
                           [ 0,1, 1.9]])

#load image
transform = transforms.Compose([transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor()])
dataloader = torch.utils.data.DataLoader(datasets.ImageFolder('/home/Pictures',transform=transform,), shuffle=True)
dtype =  torch.FloatTensor

i = 0
while i<3:
    img, labels = next(iter(dataloader))
    img = img#.double() # 有时候要转为double有时候不用转

    rotation_simple = torch.as_tensor(rotation_simple)[None]

    grid = F.affine_grid(rotation_simple, img.size()).type(dtype)
    x = F.grid_sample(img, grid)

    plt.imshow(x[0].permute(1, 2, 0))
    plt.show()
    i+=1

I wonder why does the function move the the image so far away instead of moving it for just 1 pixel in x and y direction.
Ps. Setting "align_corners=True" didn't help for this case.
Pps. My pytorch version is 1.4.0+cu100

Output:
The "unit of measures" for the grid and the affine transformation are not pixels, but rather normalized coordinates:

grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. For example, values x = -1, y = -1 is the left-top pixel of input, and values x = 1, y = 1 is the right-bottom pixel of input.

Therefore, translating by [1.25, 1.9] is actually translating by almost the entire image size. You need to divide the translation values by 2*img.shape to get pixel-wise translations.
See the doc for grid_sample for more information.


Instruction: How does one use Pytorch (+ cuda) with an A100 GPU?

Input:
I was trying to use my current code with an A100 gpu but I get this error:
---> backend='nccl'
/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

which is reather confusing because it points to the usual pytorch installation but doesn't tell me which combination of pytorch version + cuda version to use for my specific hardware (A100). What is the right way to install pytorch for an A100?

These are some versions I've tried:
# conda install -y pytorch==1.8.0 torchvision cudatoolkit=10.2 -c pytorch
# conda install -y pytorch torchvision cudatoolkit=10.2 -c pytorch
#conda install -y pytorch==1.7.1 torchvision torchaudio cudatoolkit=10.2 -c pytorch -c conda-forge
# conda install -y pytorch==1.6.0 torchvision cudatoolkit=10.2 -c pytorch
#conda install -y pytorch==1.7.1 torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge

# conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
# conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
# conda install -y pytorch torchvision cudatoolkit=9.2 -c pytorch # For Nano, CC
# conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge

note that this can be subtle because I've had this error with this machine + pytorch version in the past:
How to solve the famous `unhandled cuda error, NCCL version 2.7.8` error?

Bonus 1:
I still have errors:
ncclSystemError: System call (socket, malloc, munmap, etc) failed.
Traceback (most recent call last):
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1423, in <module>
    main()
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1365, in main
    train(args=args)
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1385, in train
    args.opt = move_opt_to_cherry_opt_and_sync_params(args) if is_running_parallel(args.rank) else args.opt
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/distributed.py", line 456, in move_opt_to_cherry_opt_and_sync_params
    args.opt = cherry.optim.Distributed(args.model.parameters(), opt=args.opt, sync=syn)
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py", line 62, in __init__
    self.sync_parameters()
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py", line 78, in sync_parameters
    dist.broadcast(p.data, src=root)
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1090, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8

one of the answers suggested to have nvcca & pytorch.version.cuda to match but they do not:
(meta_learning_a100) [miranda9@hal-dgx ~]$ python -c "import torch;print(torch.version.cuda)"

11.1
(meta_learning_a100) [miranda9@hal-dgx ~]$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

How do I match them? I this the error? Can someone display their pip, conda and nvcca version to see what set up works?
More error messages:
hal-dgx:21797:21797 [0] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83<0> [1]virbr0:192.168.122.1<0>
hal-dgx:21797:21797 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
hal-dgx:21797:21797 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83<0>
hal-dgx:21797:21797 [0] NCCL INFO Using network IB
NCCL version 2.7.8+cuda11.1
hal-dgx:21805:21805 [2] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83<0> [1]virbr0:192.168.122.1<0>
hal-dgx:21799:21799 [1] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83<0> [1]virbr0:192.168.122.1<0>
hal-dgx:21805:21805 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
hal-dgx:21799:21799 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
hal-dgx:21811:21811 [3] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83<0> [1]virbr0:192.168.122.1<0>
hal-dgx:21811:21811 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
hal-dgx:21811:21811 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83<0>
hal-dgx:21811:21811 [3] NCCL INFO Using network IB
hal-dgx:21799:21799 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83<0>
hal-dgx:21805:21805 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83<0>
hal-dgx:21799:21799 [1] NCCL INFO Using network IB
hal-dgx:21805:21805 [2] NCCL INFO Using network IB

hal-dgx:21797:27906 [0] misc/ibvwrap.cc:280 NCCL WARN Call to ibv_create_qp failed
hal-dgx:21797:27906 [0] NCCL INFO transport/net_ib.cc:360 -> 2
hal-dgx:21797:27906 [0] NCCL INFO transport/net_ib.cc:437 -> 2
hal-dgx:21797:27906 [0] NCCL INFO include/net.h:21 -> 2
hal-dgx:21797:27906 [0] NCCL INFO include/net.h:51 -> 2
hal-dgx:21797:27906 [0] NCCL INFO init.cc:300 -> 2
hal-dgx:21797:27906 [0] NCCL INFO init.cc:566 -> 2
hal-dgx:21797:27906 [0] NCCL INFO init.cc:840 -> 2
hal-dgx:21797:27906 [0] NCCL INFO group.cc:73 -> 2 [Async thread]

hal-dgx:21811:27929 [3] misc/ibvwrap.cc:280 NCCL WARN Call to ibv_create_qp failed
hal-dgx:21811:27929 [3] NCCL INFO transport/net_ib.cc:360 -> 2
hal-dgx:21811:27929 [3] NCCL INFO transport/net_ib.cc:437 -> 2
hal-dgx:21811:27929 [3] NCCL INFO include/net.h:21 -> 2
hal-dgx:21811:27929 [3] NCCL INFO include/net.h:51 -> 2
hal-dgx:21811:27929 [3] NCCL INFO init.cc:300 -> 2
hal-dgx:21811:27929 [3] NCCL INFO init.cc:566 -> 2
hal-dgx:21811:27929 [3] NCCL INFO init.cc:840 -> 2
hal-dgx:21811:27929 [3] NCCL INFO group.cc:73 -> 2 [Async thread]

after putting
import os
os.environ["NCCL_DEBUG"] = "INFO"

Output:
From the link pytorch site from @SimonB 's answer, I did:
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This solved the problem for me.


Instruction: Concatination on a 3 dimensional tensor (Tensor Re-Shaping)

Input:
I have 2 tensors,
their format currently is [13, 2] respectively. I am trying to combine both into a 3 dimensional tensor with dimensions [2, 13, 2] such that they stack on top of each other, however are separated as batches.
here is an example of one of the tensors in format [13, 2]:
tensor([[[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]])

I would like to maintain the shape, but have them in 2 groups in the same tensor. below is an example of the format I am after:
tensor([[[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]],

         [[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]]])

Anyone have any ideas on how to do this using concatination? I have tried using .unsqueeze when using torch.cat((a, b.unsqueeze(0)), dim=-1) however it changed the format to [13, 4, 1] which is not the format I am after.
solution below works, however, my idea was that I would keep stacking to y via a loop without being restricted by the shape. Sorry for not projecting my idea clearly enough.
They will all be of size [13,2] so it will go up in the form of [1,13,2], [2,13,2], [3,13,2], [4,13,2] etc...

Output:
In this case, you need torch.stack rather than torch.cat, at least it's more convenient :
x1 = torch.randn(13,2)
x2 = torch.randn(13,2)
y = torch.stack([x1,x2], 0) # creates a new dimension 0
print(y.shape)
>>> (2, 13, 2)

You can indeed use unsqueeze and cat though, but you need to unsqueeze both input tensors :
x1 = torch.randn(13,2).unsqueeze(0) # shape: (1,13,2)
x2 = torch.randn(13,2).unsqueeze(0) # same
y = torch.cat([x1, x2], 0)
print(y.shape)
>>> (2,13,2)

Here is a useful thread to understand the difference : difference between cat and stack
If you need to stack more tensors, it's not really much harder, stack works on an arbitrary number of tensors:
# This list of tensors is what you will build in your loop
tensors = [torch.randn(13, 2) for i in range(10)]
# Then at the end of the loop, you stack them all together
y = torch.stack(tensors, 0)
print(y.shape)
>>> (10, 13, 2)

Or, if you don't want to use the list :
# first, build the y tensor to which the other ones will be appended
y = torch.empty(0, 13, 2)
# Then the loop, and don't forget to unsqueeze
for i in range(10):
    x = torch.randn(13, 2).unsqueeze(0)
    y = torch.cat([y, x], 0)

print(y.shape)
>>> (10, 13, 2)


Instruction: Why can't I use my graphic card with PyTorch?

Input:
I am trying to use PyTorch with GPU in my Ubuntu 18.04. The GPU is a GeForce GTX 1070.
nvidia-smi:
| NVIDIA-SMI 460.67       Driver Version: 460.67       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:0B:00.0 Off |                  N/A |
| 21%   49C    P2    60W / 180W |   4598MiB /  8119MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 00000000:42:00.0 Off |                  N/A |
|  0%   48C    P8     7W / 180W |     20MiB /  8117MiB |      0%      Default |
|                               |                      |                  N/A |

nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

.bashrc file:
export PATH="/usr/local/cuda-10.2/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH"

I installed pytorch using the command below (from here):
pip install torch torchvision torchaudio

Torch version:
PyTorch Version: 1.8.0

Python version:
Python 3.8.7

gcc/g++ versions:
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

If I try to get the GPU, I get the following:
>>> import torch
>>> print(torch.cuda.is_available())
False

Can anyone advice me please

Output:
If you look at pytorch page, they advise to use special command to install torch with cuda, so probably, you would like to use this one:
pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Large datasets and Cuda memory Issue

Input:
I was processing a large dataset and ran into this error: "RuntimeError: CUDA out of memory. Tried to allocate 1.35 GiB (GPU 0; 8.00 GiB total capacity; 3.45 GiB already allocated; 1.20 GiB free; 4.79 GiB reserved in total by PyTorch).
Any thought on how to solve this?

Output:
I met the same problem before. It's not a bug, you just ran out of memory on your GPU.

One way to solve it is to reduce the batch size until your code will run without this error.
if it not works, better to understand your model. A single 8GiB GPU may not handle a large and deep model. You should consider changing a GPU with larger memory and find a lab to help you (Google Colab can help)
if you are just doing evaluate, force a tensor to be run CPU would be fine
Try model compression algorithm


Instruction: How to use pytorch lightning Accuracy with ignore class?

Input:
I have some training pipeline which uses CrossEntropyLoss with an ignore class.
The model outputs log_probs of shape (150, 3) - meaning 3 possible classes in batches of 150.
The label_batch is of shape 150, and torch.max(label_batch) == tensor(3, device='cuda:0'), meaning there is an extra class labeled 3, which is the ignore class.
The loss handles it fine:
self._criterion = nn.CrossEntropyLoss(
    reduction='mean',
    ignore_index=3
)

But the accuracy metric thinks class 3 is valid and gives very wrong results:
self.train_acc = pl.metrics.Accuracy()
wrong result with self.train_acc.update(log_probs, label_batch) because of the 3 label should be ignored.

How to correctly use pl.metrics.Accuracy() with an ignore class?

Output:
Copying response from the discussion thread in the github forum https://github.com/PyTorchLightning/pytorch-lightning/discussions/6890

It is currently not supported in the accuracy metric, but we have an open PR for implementing that exact feature PyTorchLightning/metrics#155
Currently what you can is instead calculate the confusion matrix and then ignore some classes based on that (remember that the true positive/correctly classified are found on the diagonal of the confusion matrix):
ignore_index = 3
metric = ConfusionMatrix(num_classes=3)
confmat = metric(preds, target)
confmat = confmat[:2,:2] # remove last column and row corresponding to class 3
acc = confmat.trace() / confmat.sum()


Instruction: Give correct datatype to pytorch, but it does not accept

Input:
class Model(nn.Module):
    def __init__(self,
                input_size=12175,
                hidden_size=6,
                num_layers=1,
                batch_size=1,
                sequence_length=1,
                num_classes=6):
        """RNN and FC. hidden_size and num_classes MUST equal."""
        super().__init__()
        self.rnn = nn.RNN(input_size=input_size,
                         hidden_size=hidden_size,
                         batch_first=True)
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.batch_size = batch_size
        self.sequence_length = sequence_length
        self.num_classes = num_classes

        # Fully-Connected layer
        self.fc = nn.Linear(num_classes, num_classes)

    def forward(self, x, hidden):
        import ipdb; ipdb.set_trace()
        # Reshape input in (batch_size, sequence_length, input_size)
        x = x.view(self.batch_size, self.sequence_length, self.input_size)
        x = x.double()
        hidden = hidden.double()
        out, hidden = self.rnn(x, hidden)
        out = self.fc(out) # Add here
        return hidden, out

    def init_hidden(self):
        return torch.zeros(self.num_layers, self.batch_size, self.hidden_size)

And training
import itertools
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

CLASS_LENGTH = 6

def train(model, device, train_loader, optimizer, epoch, criterion):
    """
    This function has one line different from the ordinary `train()` function
    It has `make_variables()` to convert tuple of names to be a tensor
    """
    model.train()
    hidden = model.init_hidden()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
#         hidden = hidden.view(model.batch_size, model.sequence_length, CLASS_LENGTH)
        output, hidden = model(data, hidden)

        tmp = output.view(model.batch_size, CLASS_LENGTH)
        loss = criterion(tmp, target)
        loss.backward(retain_graph=True)
        optimizer.step()
        if batch_idx % 1000 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test(model, device, test_loader, criterion):
    model.eval()
    test_loss = 0
    correct = 0
    y_test = []
    y_pred = []
    with torch.no_grad():
        for data, target in tqdm(test_loader):
            data, target = data.to(device), target.to(device)
            output, hidden = model(data, hidden)
            tmp = output.view(-1, COUNTRY_LENGTH)
            test_loss += criterion(tmp, target).item() # sum up batch loss
            pred = tmp.max(1, keepdim=True)[1] # get the index of the max log-probability

            pred_tmp = pred.view(-1)
            pred_list = pred_tmp.tolist()
            target_list = target.tolist()

            y_test += target_list
            y_pred += pred_list
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

    # Confusion matrix
    confusion_mtx = confusion_matrix(y_test, y_pred)
    plot_confusion_matrix(confusion_mtx, classes=test_loader.dataset.countries, normalize=True,
                          title='Confusion matrix')

Problem:
I do not understand why it raise error since I am sure that I put correct datatype into it
ipdb> n
> /tmp/ipykernel_16/2018411812.py(30)forward()
     29         hidden = hidden.double()
---> 30         out, hidden = self.rnn(x, hidden)
     31         out = self.fc(out) # Add here

ipdb> n
RuntimeError: expected scalar type Double but found Float
> /tmp/ipykernel_16/2018411812.py(30)forward()
     29         hidden = hidden.double()
---> 30         out, hidden = self.rnn(x, hidden)
     31         out = self.fc(out) # Add here

ipdb> x
tensor([[[-5.6964e-01, -5.1070e-01, -5.9109e-01,  ...,  1.5597e-15,
           1.5597e-15,  1.5597e-15]]], dtype=torch.float64)
ipdb> hidden
tensor([[[0., 0., 0., 0., 0., 0.]]], dtype=torch.float64)

pytorch version: 1.8.1+cu102
OSX 10.15.7
i7
Question:
How to solve this problem?

Output:
You are ensure type for RNNs input and hidden state, but it also contain some tensor parameters, weights, biases etc. To make them double, try torch.set_default_tensor_type(t) before RNN object construction, but I personaly would use Float.
...
super().__init__()
torch.set_default_tensor_type(torch.DoubleTensor)
self.rnn = nn.RNN(input_size=input_size,
                     hidden_size=hidden_size,
                     batch_first=True)
# may be turn defaults back to floats:
# torch.set_default_tensor_type(torch.FloatTensor)


Instruction: Multi-output regression using skorch & sklearn pipeline gives runtime error due to dtype

Input:
I want to use skorch to do multi-output regression. I've created a small toy example as can be seen below. In the example, the NN should predict 5 outputs. I also want to use a preprocessing step that is incorporated using sklearn pipelines (in this example PCA is used, but it could be any other preprocessor). When  executing this example I get the following error in the Variable._execution_engine.run_backward step of torch:
RuntimeError: Found dtype Double but expected Float

Am I forgetting something? I suspect, somewhere something has to be cast, but as skorch handles a lot of the pytorch stuff, I don't see what and where.
Example:
import torch
import skorch
from sklearn.datasets import make_classification, make_regression
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.decomposition import PCA

X, y = make_regression(n_samples=1000, n_features=40, n_targets=5)
X = X.astype('float32')

class RegressionModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 5)

    def forward(self, X):
        y = self.l0(X)
        y = self.l1(y)
        return y

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

net = skorch.NeuralNetRegressor(
    RegressionModule,
    callbacks=[InputShapeSetter()],
)

pipe = make_pipeline(PCA(n_components=10), net)

pipe.fit(X, y)
print(pipe.predict(X))

Edit 1:
Casting X to float32 at the start won't work for every preprocessor as can be seen from this example:
import torch
import skorch
from sklearn.datasets import make_classification, make_regression
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from category_encoders import OneHotEncoder

X, y = make_regression(n_samples=1000, n_features=40, n_targets=5)
X = pd.DataFrame(X,columns=[f'feature_{i}' for i in range(X.shape[1])])
X['feature_1'] = pd.qcut(X['feature_1'], 3, labels=["good", "medium", "bad"])

y = y.astype('float32')

class RegressionModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 5)

    def forward(self, X):
        y = self.l0(X)
        y = self.l1(y)
        return y

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

net = skorch.NeuralNetRegressor(
    RegressionModule,
    callbacks=[InputShapeSetter()],
)

pipe = make_pipeline(OneHotEncoder(cols=['feature_1'], return_df=False), net)

pipe.fit(X, y)
print(pipe.predict(X))

Output:
By default OneHotEncoder returns numpy array of dtype=float64. So one could simply cast the input-data X when being fed into forward() of the model:
class RegressionModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 5)

    def forward(self, X):
        X = X.to(torch.float32)
        y = self.l0(X)
        y = self.l1(y)
        return y


Instruction: Pytorch "Unfold" equivalent in Tensorflow

Input:
Say I have grayscale images of size (50*50), with a batch size of 2 in this case, and i use the Pytorch Unfold function as follows:
import numpy as np
from torch import nn
from torch import tensor

image1 = np.random.rand(1,50,50)
image2 = np.random.rand(1,50,50)
image = np.stack((image1,image2))
image = tensor(image)

ds = nn.Unfold(kernel_size=(2,2),stride=2)
x = ds(image).numpy()
x.shape

## OUTPUT: (2, 4, 625)

What would be the equivalent tensorflow implementation so that the output of the tensorflow implementation would exactly match 'x'? I've tried using the tf.image.extract_patches function but it seems to not be giving me what I quite want.
The question is then: What is the tensorflow implementation of Unfold?

Output:
tf.image.extract_patches() is analogous to torch.nn.Unfold, but you need to rejig the parameters slightly:
tf.image.extract_patches(image, sizes=[1,2,2,1], strides=[1,2,2,1], rates=[1,1,1,1], padding='SAME')


Instruction: Working of nn.Linear with multiple dimensions

Input:
PyTorch's nn.Linear(in_features, out_features) accepts a tensor of size (N_batch, N_1, N_2, ..., N_end), where N_end = in_features. The output is  a tensor of size (N_batch, N_1, N_2, ..., out_features).
It isn't very clear to me how it behaves in the following situations:

If v is a row, the output will be A^Tv+b
If M is a matrix, it is treated as a batch of rows, and for every row v, A^Tv+b is performed, and then everything is put back into matrix form
What if the input tensor is of a higher rank? Say the input tensor has dimensions (N_batch, 4, 5, 6, 7). Is it true that the layer will output a batch of size N_batch of (1, 1, 1, N_out)-shaped vectors, everything shaped into a (N_batch, 4, 5, 6, N_out) tensor?

Output:
for 1 dimension, the input is vector with dim in_features, output is out_features. calculated as what you said
for 2 dimensions, the input is N_batch vectors with dim in_features, output is N_batch vectors with dim out_features. calculated as what you said
for 3 dimensions, the input is (N_batch, C, in_features), which is N_batch matrices, each with C rows of vectors with dim in_features, output is N_batch matrices, each with C rows of vectors with dim out_features.
If you feel it's hard to think of 3 dimensional case. One simple way is to flatten the shape to (N_batch * C, in_features), so the input becomes N_batch * C rows of vectors with dim in_features which is the same case as the two dimensional case. This flatten part involves no computation, just rearrange the input.
So in your case 3, its output is N_batch of (3, 4, 5, 6, N_out) vectors, or after rearrange its N_batch * 3 * 4 * 5 * 6 vectors with dim N_out. your shape with all 1 dims are not correct, since there are only N_batch * N_out elements in total.
If you dig into the internal C implementations of pytorch, you can find the matmul implementation actually flatten the dimensions as I have described native matmul which is the exact function used by nn.Linear


Instruction: How to call method without method name in python

Input:
class LinearRegression(nn.Module):
def __init__(self,input_size,output_size):
    # super function. It inherits from nn.Module and we can access everythink in nn.Module
    super(LinearRegression,self).__init__()
    # Linear function.
    self.linear = nn.Linear(input_dim,output_dim)

def forward(self,x):
    return self.linear(x)

input_dim = 1
output_dim = 1
model = LinearRegression(input_dim,output_dim)

In this code block when I want to call forward method in model object I can call 2 different ways

The first way
results = model.forward(car_price_tensor)

Second way
results = model(car_price_tensor)

And I try the second way on a different class and object it doesnt work. How it possible?

Output:
This behavior that lets you call a Python object like a function is enabled through a special method __call__, some explanation here.
A torch.nn.Module is a class that implements this behavior. You class LinearRegression is a subclass of it so it inherits this behavior. By default the __call__ method is derived from your forward implementation but it not the same.
The difference in behavior is explained here.


Instruction: How to achieve removing/pruning the near-zero parameters in neural network?

Input:
I need to remove the near-zero weights of the Neural network so that the distribution of parameters is far away from the zero point.
The distribution of weights after removing nearzero weights and weight-scaling
I met the problem from this paper: https://ieeexplore.ieee.org/document/7544366
I wonder how can I achieve this in my PyTorch/TensorFlow program, such as use a customized activation layer? Or Define a loss function that punishes the near-zero weight?
Thank you if you can provide any help.

Output:
You're looking for L1 regularization, read the docs.
import tensorflow as tf

tf.keras.layers.Dense(units=128,
                      kernel_regularizer=tf.keras.regularizers.L1(.1))

Smaller coefficients will be turned to zero.


Instruction: Python TypeError: 'Tensor' object is not callable when sorting dictionary

Input:
Here is my code. The packages imported are not shown. I am trying to feed the CIFAR-10 test data into alexnet. The dictionary at the end needs to be sorted so I can find the most common classification. Please help, I have tried everything!
............................................................................................................................................................................................................................................................................................................
alexnet = models.alexnet(pretrained=True)

transform = transforms.Compose([            #[1]
 transforms.Resize(256),                    #[2]
 transforms.CenterCrop(224),                #[3]
 transforms.ToTensor(),                     #[4]
 transforms.Normalize(                      #[5]
 mean=[0.485, 0.456, 0.406],                #[6]
 std=[0.229, 0.224, 0.225]                  #[7]
 )])

# Getting the CIFAR-10 dataset
dataset = CIFAR10(root='data/', download=True, transform=transform)
test_dataset = CIFAR10(root='data/', train=False, transform=transform)

classes = dataset.classes
#print(classes)

torch.manual_seed(43)
val_size = 10000
train_size = len(dataset) - val_size

train_ds, val_ds = random_split(dataset, [train_size, val_size])

#print(len(train_ds), len(val_ds))

batch_size=100

train_loader = DataLoader(train_ds, batch_size, shuffle=True, num_workers=8, pin_memory=True)
val_loader = DataLoader(val_ds, batch_size, num_workers=8, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size, num_workers=8, pin_memory=True)

with open("/home/shaan/Computer Science/CS4442/Ass4/imagenet_classes.txt") as f:
    classes = eval(f.read())

holder = []
dic = {}

current = ''
#data_iter = iter(test_loader)
#images,labels = data_iter.next()

#alexnet.eval()
with torch.no_grad():
    for data in test_loader:
        images, labels = data
        out = alexnet(images)
        #print(out.shape)

        for j in range(0,batch_size):
            sorted, indices = torch.sort(out,descending=True)
            percentage = F.softmax(out,dim=1)[j]*100
            results = [(classes[i.item()],percentage[i].item()) for i in indices[j][:5]]
            holder.append(results[0][0])

holder.sort()
for z in holder:
    if current != z:
        count = 1
        dic[z] = count
        current = z

    else:
        count = count + 1
        dic[z] = count
        current = z

This is where im getting the error:
for w in sorted(dic, key=dic.get, reverse=True):
    print(w, dic[w])

Output:
This line is the problem
sorted, indices = torch.sort(out,descending=True)

You created a variable named sorted, which is exactly the same name as sorted function you call when it error.
Just change this to something else like
sorted_out, indices = torch.sort(out,descending=True)


Instruction: Load a saved NN model in different Python file

Input:
I am trying to implement the code from a Pytorch beginner's tutorial. But I have written the code for loading the saved model in another Python file.
The FashionClassify file contains the code exactly as its in the tutorial.
Below is the code:
from FashionClassify import NeuralNetwork
from FashionClassify import test_data
import torch

model = NeuralNetwork()
model.load_state_dict(torch.load("model.pth"))
classes = [
    "T-shirt/top", "Trouser","Pullover","Dress","Coat","Sandal","Shirt","Sneaker","Bag","Ankle boot",
]

model.eval()
x, y = test_data[0][0], test_data[0][1]
with torch.no_grad():
    pred = model(x)
    predicted, actual = classes[pred[0].argmax(0)],classes[y]
    print(f'Predicted: "{predicted}", Actual: "{actual}"')

However, when I run this, the entire training process starts again. Why is that so ?
OR
Is it an expected behavior ?
(I have gone through a couple of webpages and StackOverflow answers but couldn't find my problem)
FashionClassify file code:
import torch
from torch import nn
from torch.utils.data import DataLoader  # wraps an iterable around dataset
from torchvision import datasets  # stores samples and their label
from torchvision.transforms import ToTensor, Lambda, Compose
import matplotlib as plt

training_data = datasets.FashionMNIST(root='data', train=True, download=True, transform=ToTensor(), )
test_data = datasets.FashionMNIST(root='data', train=False, download=True, transform=ToTensor(), )

batch_size = 64
train_dataLoader = DataLoader(training_data, batch_size=batch_size)
test_dataLoader = DataLoader(test_data, batch_size=batch_size)

for X, y in test_dataLoader:
    print('Shape of X [N,C,H,W]:', X.size())
    print('Shape of y:', y.shape, y.dtype)
    break

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Using {} device'.format(device))

# to define a NN, we inherit a class from nn.Module
class NeuralNetwork(nn.Module):
    def __init__(self):
        # will specify how data will proceed in the forward pass
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28 * 28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork().to(device)
print(model)

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X,y) in enumerate(dataloader):
        X,y = X.to(device), y.to(device)

        #compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        #backprop
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch%100 ==0:
            loss,current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}]")

def test(dataloader, model):
    size = len(dataloader.dataset)
    model.eval()
    test_loss, correct = 0,0
    with torch.no_grad():
        for X, y in dataloader:
            X,y = X.to(device), y.to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct +=  (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= size
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

epochs = 5
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataLoader, model, loss_fn, optimizer)
    test(test_dataLoader, model)
print("Done!")

torch.save(model.state_dict(), "model.pth")
print("Saved PyTorch Model State to model.pth")

Output:
That's what happens when you import another file. All the code gets rerun.
Instead, in your training file:
class FancyNetwork(nn.Module):
    [...]

def train():
    [train code]

if __name__ == "__main__":
    train()

Now when you run this file train() will get called, but when you import this file in another one, train won't get called automatically.


Instruction: Apply vectorised function to Cartesian product of two ranges in PyTorch

Input:
I am kind of new to pytorch, and have a very simple question. Let's say we have a scalar function f():
def f(x,y):
    return np.cos(x)+y

What I want to do is use the GPU to generate all pairs of data-points from two ranges x and y. For simple case, take x=y=[0,1,2].
Can I do that without changing the function? If not, how would you change the function?

Output:
You can take the Cartesian product of the values before applying your function to their first and second elements:
x = y = torch.tensor([0,1,2])

pairs = torch.cartesian_prod(x,y)
# tensor([[0, 0], [0, 1], [0, 2], [1, 0], [1, 1], [1, 2], [2, 0], [2, 1], [2, 2]])

x_, y_ = pairs[:,0], pairs[:,1]
f(x_,y_)


Instruction: How to split duplicate samples to train test with no overlapping?

Input:
I have a nlp datasets (about 300K samples) where there exits duplicate data. I want to split it to train test （70%-30%）, and they should have no overlapping.
For instance:
|dataset:      |   train   |     test   |
|   a          |     a     |       c    |
|   a          |     a     |       c    |
|   b          |     b     |       c    |
|   b          |     b     |            |
|   b          |     b     |            |
|   c          |     d     |            |
|   c          |     d     |            |
|   c          |           |            |
|   d          |           |            |
|   d          |           |            |

I have tired exhaustively random sample, but it too time consuming.

Output:
It is doable, but requires a few steps to be accomplished.
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# original dataset with duplicates
dataset = pd.DataFrame(["a", "a", "b", "b", "b", "c", "c", "c", "d", "d"])

# get unique values, remove duplicates, but keep original counts
data_no_dup, counts = np.unique(dataset, return_counts=True)

# split using the standard Scikit-Learn way
train_no_dup, test_no_dup = train_test_split(data_no_dup, test_size=0.2, random_state=0)

# retrieve original counts
train, test = [], []
for sample in train_no_dup:
    train.extend([sample] * counts[list(data_no_dup).index(sample)])
for sample in test_no_dup:
    test.extend([sample] * counts[list(data_no_dup).index(sample)])

print("Train: {}".format(train))
print("Test: {}".format(test))

Output
Train: ['d', 'd', 'b', 'b', 'b', 'a', 'a']
Test: ['c', 'c', 'c']


Instruction: Numpy: get only elements on odd or even diagonal offsets in a matrix, change the rest to zeros

Input:
I have a 2D (square) matrix, for example it can be like this:
1 2 3
4 5 6
7 8 9

I want to get only the elements on the odd or even diagonal offsets of it, and let the rest be zeros. For example, with even diagonal offsets (%2 = 0), the resulting matrix is:
1 0 3
0 5 0
7 0 9

Explanation: main diagonal has offset 0, which is 1 5 9. The next diagonal offsets are 2, 6 and 4, 8 thus they are changed to zeros. Repeat the process until we reach the last diagonal.
And with odd diagonal index, the resulting matrix is:
0 2 0
4 0 6
0 8 0

I look at the np.diag(np.diag(x)) but it only returns the main diagonal and the rest are zeros. How can I extend it to odd/even offsets?
I can also use PyTorch.

Output:
I would do it following way using numpy
import numpy as np
arr = np.array([[1,2,3],[4,5,6],[7,8,9]])
masktile = np.array([[True,False],[False,True]])
mask = np.tile(masktile, (2,2))  # this must be at least as big as arr
arr0 = np.where(mask[:arr.shape[0],:arr.shape[1]], arr, 0)
arr1 = np.where(mask[:arr.shape[0],:arr.shape[1]], 0, arr)
print(arr0)
print(arr1)

output:
[[1 0 3]
 [0 5 0]
 [7 0 9]]
[[0 2 0]
 [4 0 6]
 [0 8 0]]

Explanation: I am creating mask which is array of Trues and Falses to use to decide if given element is to remain or should be replaced by 0. I create single tile which then I feed into np.tile to get "chessboard" of sufficient size, then I use part of apprioate size of it together with np.where to replace selected elements with 0.


Instruction: PyTorch Lightning: includes some Tensor objects in checkpoint file

Input:
As Pytorch Lightning provides automatic saving for model checkpoints, I use it to save top-k best models. Specifically in Trainer setting,
    checkpoint_callback = ModelCheckpoint(
                            monitor='val_acc',
                            dirpath='checkpoints/',
                            filename='{epoch:02d}-{val_acc:.2f}',
                            save_top_k=5,
                            mode='max',
                            )

This is working well but it does not save some attribute of the model object. My model stores some Tensor at every training epoch end such that
class SampleNet(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.save_hyperparameters()
        self.layer = torch.nn.Linear(100, 1)
        self.loss = torch.nn.CrossEntropy()

        self.some_data = None # Initialize as None

    def training_step(self, batch):
        x, t = batch
        out = self.layer(x)
        loss = self.loss(out, t)
        results = {'loss': loss}

        return results

    def training_epoch_end(self, outputs):

        self.some_data = some_tensor_object

This is a simplified example but I want the checkpoint file made by above checkpoint_callback to remember the attribute self.some_data but when I load the model from checkpoint, it always reset to None. I confirmed that it is successfully updated during the training.
I tried not to initialize it as None in init but then the attribute will disappear when loading model.
Saving the attribute as a distinct pt file is something I want to avoid as it is associated with model configuration so I manually need to match the file with corresponding checkpoint file later.
Would it be possible to include such tensor attribute in checkpoint file?

Output:
Simply use the model class hooks on_save_checkpoint() and on_load_checkpoint() for all sorts of objects that you want to save alongside the default attributes.
def on_save_checkpoint(self, checkpoint) -> None:
    "Objects to include in checkpoint file"
    checkpoint["some_data"] = self.some_data

def on_load_checkpoint(self, checkpoint) -> None:
    "Objects to retrieve from checkpoint file"
    self.some_data= checkpoint["some_data"]

See module docs


Instruction: how to exclude particular dependency package version from setup file in python

Input:
I've a small package that does not work on torch v1.8.0 but it works fine on the new v1.8.1 version and other older versions v1.7.1 so want to exclude the v1.8.0 version.

I could just set
install_requires=[
    "torch>=1.8.1",
      ...

but the torch package size is huge and also want mypackage to work on the older versions of the torch.
I've tried
install_requires=[
    "torch>=1.8.1,!=1.8.0,<=1.7.1",
       ...

but when installing the package with pip install mypackage getting the following error:
ERROR: Could not find a version that satisfies the requirement torch!=1.8.0,<=1.7.1,>=1.8.1 (from mypackage) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1)
ERROR: No matching distribution found for torch!=1.8.0,<=1.7.1,>=1.8.1 (from mypackage)

how can exclude the v1.8.0 version? thank you.

Output:
One can specify a minimum version and also exclude certain versions. Below, the minimum version is 1.0.0. This should be set to a reasonable value depending on the project.
torch>=1.0.0,!=1.8.0

The issue with torch>=1.8.1,!=1.8.0,<=1.7.1 is that it requests torch greater than or equal to 1.8.1 and less than or equal to 1.7.1. That is not possible, so pip cannot fulfill the request.
PEP 508 and PEP 440 are relevant here.


Instruction: I don't find a way to use my wav file as dataset in PyTorch

Input:
hello I am new to PyTorch and I want to make a simple speech recognition but I don't want to use pytorch.datasets I have some voices for dataset but I don't find anywhere to help me.
I want to use .wav files. I saw a tutorial but he used pytorch dataset.
import torch
from torch import nn, optim
import torch.nn.functional as F
import torchaudio

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

from torchaudio.datasets import SPEECHCOMMANDS
import os

class SpeechSubset(SPEECHCOMMANDS):
    def __init__(self, subset, str=None):
        super().__init__("./", download=True)

        def load_list(filename):
            filepath = os.path.join(self._path, file.name)
            with open(filepath) as fileob:
                return [os.path.join(self._path, line.strip())]

        if subset == "validation":
            self._walker = load_list("validation_list.txt")
        elif subset == "testing":
            self._walker = load_list("testing_list.txt")
        elif subset == "training":
            excludes = load_list("validation_list.txt") + load_list("testing_list.txt")
            excludes = set(excludes)
            self._walker = [w for w in self._walker if w not in excludes]

train_set = SpeechSubset("training")
test_set = SpeechSubset("testing")

waveform, sample_rate, label, speaker_id, utterance_number = train_set[0]

sorry my english isn't too good.
EDIT
Im using the SPEECHCOMMANDS dataset but I want to use my own
thank you for reading.

Output:
Since you are talking about the speech recognition and pytorch, I would recommend you to use a well-developed set of tools instead of doing speech-related training tasks from scratch.
A good repo on github is Espnet. It contains some quite recent work on text-to-speech and speech-to-text models as well as ready-to-use scripts to train on popular open-source dataset in different languages. It also includes trained models for you to use directly.
Back to your question, if you want to use pytorch to train your own speech recognition model on your own dataset, I would recommend you to go to this Espnet Librispeech ASR recipe. Although it uses .flac files, some little modifications on data preparation script and change some parameters in the major entry script asr.sh may feed your demand.
Note that, in addition to knowledge on python and torch, espnet needs you to be familiar with shell scripts as well. Their asr.sh script is quite long. This may not be an easy task for people who are more comfort with minimal pytorch codes for one specific model. Espnet is designed to accomodate many models and many datasets.  It contains many preprocessing stages, e.g. speech feature extracting, length filtering, token preparation, language model training and so on, which are necessary for good speech recognition models.
If you insist on the repo that you found. You need to write a custom Dataset and Dataloader classes. You can refer to pytorch dataloading tutorial, but this link uses images as an example, if you want an audio example, maybe from some github repos like deepspeech pytorch dataloader


Instruction: Tensorboard - ValueError: too many values to unpack (expected 2)

Input:
I have tried to use tensorboard to visualize a model. I was following the pytorch.org tutorial. Here is the code for dataloader.
writer_train = SummaryWriter('runs/training')

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=config.train_batch_size, shuffle=True,
                                           num_workers=config.num_workers, pin_memory=True)

images, labels = next(iter(train_loader))
writer_train.graph_model(light_net, images)

and I got this error in the iter line.

images, labels = next(iter(train_loader))
ValueError: too many values to unpack (expected 2)

I have debugged the code and found this.

Output:
The error is likely caused by the use of a built-in function instead of the .next() method of the train_loader object.
next() and iter() are builtin methods in Python. See from the docs iter and next.
In the tutorial is shows the following
# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

Where it uses the next() method to unpack values into the 2 variables. This is not the same as your usage of next(iter(train_loader)). Do it in the way as shown and it should solve your problem.


Instruction: How does a Pytorch neural network load dataset into GPU

Input:
When loading a dataset into the GPU for training, would a Pytorch NN load the entire dataset or just the batch?
I have a 33GB dataset that fits comfortably on my normal RAM (64GB) but i only have a 16GB of GPU RAM (T4). As long as Pytorch only loads one batch at a time into the GPU, that should work fine without any memory problems?

Output:
You can load one batch of data at a time into GPU. You should use data loader to fetch a batch of data and also initialize a torch device instance to use GPU.
You can check the following tutorial. It uses data loader to get data as batches and load them to GPU by using torch device.
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html


Instruction: Combining multiple models trained in different parts of the dataset in PyTorch

Input:
In PyTorch, is it theoretically possible to 'merge' multiple models together into one model - effectively combining all the data learnt so far? The models are exactly identical, however, are trained with different parts of the training data.
If so, would it be possible to split a dataset into equal parts and distribute training between many computers in a similar manner to folding@home? Would the new model be as good as if it was not distributed?

Output:
I believe what you are asking for is a distributed implementation of a tensorflow/pytorch model. Similar to distributed databases, chunks of data can be used on separate clusters to train a single model in parallel on each cluster. The resultant model will be trained on all the separate chunks of data on different clusters, as though it has been trained on the complete data together.
While there are multiple tools that do the same, the one I can recommend is Analytics Zoo.

A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray. You can easily apply AI models (e.g., TensorFlow, Keras, PyTorch, BigDL, OpenVINO, etc.) to distributed big data.

More details here.


Instruction: Difference between torch.Size([64]) and (64,)?

Input:
I created a Pytorch dataset class to store 64 lines of text. The file only has text, no label so I artificially generated an index list y (just to follow along with a tutorial https://medium.com/swlh/how-to-use-pytorch-dataloaders-to-work-with-enormously-large-text-files-bbd672e955a0#4fe0). After I created the dataset object and wrapped it around a dataloader, y.shape returned torch.Size([64]) while the tutorial said it would return (64,). (torch version is 1.8.1. torchvision version is 0.9.1. Python is 3.7.10.)
Is there a difference between torch.Size([64]) and (64,)? Thank you.
##### IMPORT PACKAGES #####
import nltk
import string
from nltk import word_tokenize
from torch.utils.data import IterableDataset, DataLoader, Dataset

##### DEFINE CLASS #####
class CustomDataset(Dataset):
    # A Pytorch Dataset class to store text
    def __init__(self, filename):
        '''
        Input: filename (Each line is a string.)
        Output: member variable X (list of unprocessed strings)
                member variable y (index list of X)
        '''
        # Open file and store contents in list
        with open(filename) as f:
            lines = f.read().split('\n')
        X, y = [], []
        i = 0
        for line in lines:
          X.append(line)
          y.append(i)
          i +=1
        # Store in member variables
        self.X = X
        self.y = y

     def preprocess(self, text):
        '''
        Input: a string from X
        Output: a preprocessed string
        '''
        text_pp = text.lower() # lower case
        return text_pp

    def __len__(self):
        return len(self.y)

    def __getitem__(self, index):
       '''
       Input: a number (within range of X's indices)
       Output: string at specified index
       '''
       return self.preprocess(self.X[index]), self.y[index]

##### CREATE OBJECT #####
dataset = CustomDataset('micro.txt')
dataloader = DataLoader(dataset, batch_size = 64, num_workers = 2)
for X, y in dataloader:
  print(y.shape) # torch.Size([64]) [Is it same as (64,)?])

Output:
In a way they are same thing. You are printing shape of a one dimensional tensor. Shape written in tutorial is shape format of one dimensional numpy array.
If shape of y is printed after converting it into numpy array, mentioned format will appear. You can see both format with following code.
print((torch.rand(64)).shape)  # torch.Size([64])
print((torch.rand(64)).numpy().shape)  # (64,)


Instruction: Dropout Layer with zero dropping rate

Input:
I'm having trouble understanding a certain aspect of dropout layers in PyTorch.
As stated in the Pytorch Documentation the method's signature is torch.nn.Dropout(p=0.5, inplace=False) where p is the dropout rate.
What does this layer do when choosing p=0? Does it change its input in any way?

Output:
Dropout with p=0 is equivalent to the identity operation.
In fact, this is the exact behaviour of Dropout modules when set in eval mode:

During evaluation the module simply computes an identity function.


Instruction: TypeError: forward() takes 2 positional arguments but 3 were given in pytorch

Input:
I have the following error in my training loop and I don't really understand what the issue is. I am currently in the process of writing this code so stuff isn't final but I cannot figure out what this problem is.
I have tried googling the error and read some of the answers but still couldn't seem to understand the crux of the issue.
Dataset and Dataloader
(X and Y are already given to me, they are both [2000, 40, 1] tensors)
class TrainingDataset(data.Dataset):
  def __init__(self, X, y):
    self.X = X
    self.y = y

  def __len__(self):
    return Nf

  # returns corresponding input/output pairs
  def __getitem__(self, t):
    X = self.X[t]
    y = self.y[t]

    #print(X.shape, y.shape)

    return X, y

# prints torch.Size([2000, 40, 1]) torch.Size([2000, 40, 1])
print(x.size(), y.size())

dataset = TrainingDataset(x,y)
batchSize = 20
dataIter = data.DataLoader(dataset, batchSize)

Model:
class Encoder(nn.Module):
  def __init__(self, num_inputs = 40, num_outputs = 40):
    super(Encoder, self).__init__()

    self.num_inputs = num_inputs
    self.num_hidden = num_hidden
    self.num_outputs = num_outputs

    self.layers = nn.Sequential(
        nn.Linear(num_inputs, num_outputs),
        nn.ReLU(),
        nn.Linear(num_outputs, num_outputs),
        nn.ReLU(),
        nn.Linear(num_outputs, num_outputs)
    )

  def forward(self, x_c, y_c):
    return self.layers(x_c, y_c)

Training Loop:
for epoch in range(epochs):
  for batch in dataIter:
    optimiser.zero_grad()
    l = loss(encoder(x_c=batch[0], y_c=batch[1]), batch[1])
    l.backward()
    optimiser.step()

Error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-aa1c60616d82> in <module>()
      6   for batch in dataIter:
      7     optimiser.zero_grad()
----> 8     l = loss(encoder(x_c=batch[0], y_c=batch[1]), batch[1])
      9     l.backward()
     10     optimiser.step()

2 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

TypeError: forward() takes 2 positional arguments but 3 were given

Can anyone point me in the right direction? I have just started to learn and do pytorch so I am not good at any of this yet.

Output:
  def forward(self, x_c, y_c):
    return self.layers(x_c, y_c)

Your error lies here, this function should have only 1 argument apart from self.


Instruction: How to speed up the process of loading data while training my neural network?

Input:
I am now training my LSTM neural network with my GPU.
The problem is:
I have 23,000 .csv files in my training set, which has a shape of (40,76). Each time I got a batch(64) to load my data, I found that it took about 1s to load the data(read 64 .csv files), and it took about 0.08s to compute the loss and update the parameters. When I checked the power and utilization of my GPU, I found it was of low efficiency. Therefore, how can I improve the organization of my training data?
Here is my own dataset class.enter image description here

Output:
Combine the CSV files into a single file. Or, load the data from the CSVs and save the data in some other form. This way, you only have to read from one file instead of 23,000. Reading files is relatively very slow because it requires a system call (your program has to ask the operating system to read the file).
Easiest thing to do is just combine the csvs and then save them as a new csv. Then just use that csv to load the data. I would bet most of the run time of your code is from opening/closing the files


Instruction: Dataset size is smaller than memory, What's wrong with my code?

Input:
The following is part of the code, epoch=300, each npz file is 2.73M, but the batch size of my dataloader gives 64, a total of 8 gpuss, so a mini batch should be 64×8×2.73M≈1.1G, my actual memory is 128G. Even if it becomes larger after decompression, it will not reach the size of 128G. The following figure link shows that all 128G of memory is occupied. How should I change my code?
class VimeoDataset(Dataset):
def __init__(self, dataset_name, batch_size=64):
    def __init__(self, dataset_name, batch_size=32):
    self.batch_size = batch_size
    self.path = '/data/train_sample/dataset/'
    self.dataset_name = dataset_name
    #self.load_data()
    self.h = 256
    self.w = 448
    xx = np.arange(0, self.w).reshape(1,-1).repeat(self.h,0) #xx shape is(256,448)
    yy = np.arange(0, self.h).reshape(-1,1).repeat(self.w,1) #yy shape is(448,256)
    self.grid = np.stack((xx,yy),2).copy()

def __len__(self):
    return len(self.meta_data)

def getimg(self, index):
    f = np.load('/data/train_sample/dataset/'+ str(index) + '.npz')
    if index < 8000:
        train_data = f['i0i1gt']
        flow_data = f['ft0ft1']
    elif 8000 <= index < 10000:
        val_data = f['i0i1gt']
    else:
        pass

    if self.dataset_name == 'train':
        meta_data = train_data
    else:
        meta_data = val_data

    data = meta_data
    img0 = data[0:3].transpose(1, 2, 0)
    img1 = data[3:6].transpose(1, 2, 0)
    gt = data[6:9].transpose(1, 2, 0)
    flow_gt = flow_data.transpose(1, 2, 0)
    return img0, gt, img1, flow_gt

    dataset = VimeoDataset('train')
def __getitem__(self, index):
    img0, gt, img1, flow_gt = self.getimg(index)
    ...

sampler = DistributedSampler(dataset)
train_data = DataLoader(dataset, batch_size=args.batch_size, num_workers=8, pin_memory=True,
drop_last=True, sampler=sampler)

system usage figure

Output:
I have given fixing your dataset given our comments above a go. Essentially you need to pass more variables into your class so that it can easily differentiate between your train and validation data. This is without loading all of your data into memory, although sometimes this is necessary (sequentially, not at once) to calculate some data statistics and such.
Disclaimer: I took a guess at using glob to find your npz files and that you use flow_data in your validation set (missing in your code for validation data).
from glob import glob
class VimeoDataset(Dataset):
    def __init__(self, npzs, batch_size=64,train_set=False):
        self.batch_size = batch_size
        self.train_set = train_set
        self.h = 256
        self.w = 448
        xx = np.arange(0, self.w).reshape(1,-1).repeat(self.h,0) #xx shape is(256,448)
        yy = np.arange(0, self.h).reshape(-1,1).repeat(self.w,1) #yy shape is(448,256)
        self.grid = np.stack((xx,yy),2).copy()
        self.npzs = npzs

    def __len__(self):
        return len(self.npzs)

    def getimg(self, index):
        f = np.load(self.npzs[index])
        data = f['i0i1gt']
        if self.train_set:
            flow_data = f['ft0ft1']
        else:
            flow_data = np.zeros([self.h,self.w,4])
        img0 = data[0:3].transpose(1, 2, 0)
        img1 = data[3:6].transpose(1, 2, 0)
        gt = data[6:9].transpose(1, 2, 0)
        flow_gt = flow_data.transpose(1, 2, 0)
        return img0, gt, img1, flow_gt

    def __getitem__(self, index):
        img0, gt, img1, flow_gt = self.getimg(index)

npzs = glob('/data/train_sample/dataset/*.npz')
train_val_split = 8000
train_dataset = VimeoDataset(npzs[:train_val_split],train_set = True)
val_dataset = VimeoDataset(npzs[train_val_split:])


Instruction: Python ValueError: Inserting a list of values using df.loc

Input:
I have a dataframe df with paths of images. I am performing object detection and want to save the names of the detected objects in a new column (named objects) of the dataframe.
However the code works fine when only 1 object is detected, when there is a list of multiple objects the code gives an error.
ValueError: cannot set using a multi-index selection indexer with a different length than the value
I think there is some problem with my way of inserting values in the dataframe. I have actually created an empty column first and then using loop inserting vaues in the objects column.
Code
train_image_paths = 'data/train_images/' + df['image']    # train_images is the folder containing images and
                                                          #  df['image'] contains all image paths

df['objects'] = ''     # Creating an empty column for storing detected objects

for idx, img in enumerate(train_image_paths):
    ..
    #object detection code    # Performs object detection and
                              # stores the detected objects in a list named detected_objects
    ..

    df.loc[idx, 'objects'] = detected_objects   # Adding the detected objects to the dataframe

return df

Output of df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 34250 entries, 0 to 34249
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   image   34250 non-null  object
dtypes: object(1)
memory usage: 267.7+ KB

Output:
I think you should create new list or Pandas Series then append it to dataframe.
train_image_paths = 'data/train_images/' + df['image']    # train_images is the folder containing images and
                                                          #  df['image'] contains all image paths

objects = []     # Creating an empty column for storing detected objects

for idx, img in enumerate(train_image_paths):
    ..
    #object detection code    # Performs object detection and
                              # stores the detected objects in a list named detected_objects
    ..

    objects.append(detected_objects)   # Adding the detected objects to the dataframe

df['objects'] = objects
return df

Here is the references: https://www.geeksforgeeks.org/adding-new-column-to-existing-dataframe-in-pandas/


Instruction: Computing gradients only for the front-end network in Pytorch

Input:
I have a very simple question.
Let's say that I have two networks to train (i.e., net1, net2).
The output of net1 will be fed into net2 while training.
In my case, I would like to only update net1:
optimizer=Optimizer(net1.parameters(), **kwargs)
loss=net2(net1(x))
loss.backward()
optimizer.step()

Although this will achieve what I'm aiming for, it takes up too much redundant memory since this will compute the gradients for net2 (causes OOM error).
Therefore I have tried out several attempts to solve this issue:

torch.no_grad:

z=net1(x)
with torch.no_grad():
    loss=net2(z)

Didn't raise OOM but removed all the gradients including the ones from net1.

requires_grad=False:

net2.requires_grad=False
loss=net2(net1(x))

Raised OOM.

detach():

z=net1(x)
loss=net2(z).detach()

Didn't raise OOM but removed all the gradients including the ones from net1.

eval():

net2.eval()
loss=net2(net1(x))

Raised OOM.
Is there any way to compute the gradients only for the front-end network (net1) for memory efficiency?
Any suggestions would be appreciated.

Output:
First let's try to understand why your methods don't work.

This context manager disables all gradient computation.
Since net1 requires a gradient, the subsequent requires_grad=False are ignored.
If you detach at that state, this means the gradient computation already stops right there
Eval just sets net2 to eval mode, that doesn't influence the gradient computation at all.

Depending on your architecture, the OOM error could already come from saving all the intermediate values in your computation graph (frequently an issue in CNNs), or it could come from having to store gradients (more common in fully connected networks).
What you probably are looking for is called "checkpointing" which you don't even have to implement on your own, you can use the checkpointing API of pytorch, check out the documentation.
This basically lets you compute and process the gradient for net1 and net2 separately. Note that you do need all the gradient informatino to go through net2, otherwise you cannot compute the gradients wrt. net1!


Instruction: CUDA out of memory error when calling PyTorch train script in a loop from another python script

Input:
I'm trying to run the same pytorch training script with different arguments (argparse) from another python script. I'm using os.system() for the same.
Here's what I'm trying to do -
train.py = > the script which contains the train-loop.
runner.py => the file which runs the train script in a loop.
# runner.py
for hp in hyperparams:
    os.system(f"CUDA_VISIBLE_DEVICES=1 python train.py --arg1 hp")

A few models get trained but I eventually end up getting a CUDA out of memory error. For instance, if there were 10 models, it will successfully train 8 and then give a CUDA error for 9 and 10.
My guess is that the GPU memory is not being cleared after every loop. What can I do to mitigate this?

Output: torch.cuda.empty_cache() along with deleting the models helped.


Instruction: mat1 dim 1 must match mat2 dim 0 -Pytorch

Input:
I have attempted to solve this error but it has been to no avail. My CNN model is below:
The shape of X_train and X_test are:
X_train shape: torch.Size([12271, 3, 100, 100]) | X_test shape: torch.Size([3068, 3, 100, 100])
class Net(nn.Module):
    def __init__(self, num_classes=7):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size= 5,stride=1,padding=2)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5,stride=1,padding=2)
        self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5,stride=1,padding=2)

        self.pool = nn.MaxPool2d(2, 2)
        self.drop = nn.Dropout2d(p=0.2)

        self.fc1 = nn.Linear(9216, 1000)
        self.fc2 = nn.Linear(1000, 500)
        self.fc3 = nn.Linear(500, out_features=num_classes)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))

        x = x.reshape(x.size(0), -1)

        x = F.dropout(self.drop(x), training=self.training)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

print(net)

The training on the model on data code as required is show below:
The required traceback is below:
count = 0
total_step = len(trainloader)
loss_list = []
acc_list = []
iteration_list = []
for epoch in range(20):
    for i, data  in enumerate(trainloader,0):
      images, labels = data[0].to(device), data[1].to(device)
        # Run the forward pass
      outputs = net(images.float())
      loss = criterion(outputs, labels)
      loss_list.append(loss.item())
      iteration_list.append(i)
      # Backprop and perform Adam optimisation
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      #count += 1

      #if count % 100 == 0:
        # Track the accuracy
      total = labels.size(0)
      _, predicted = torch.max(outputs.data, 1)
      correct = (predicted == labels).sum().item()
      acc_list.append(correct / total)

      if (i + 1) % 1000 == 0:
          print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                        (correct / total) * 100))

Output:
Your final convolution's (conv3) output dimensions don't match the input dimension of first Linear layer. self.conv3's output shape will be BatchSize x 128 x 12 x 12 when resized:
x = x.reshape(x.size(0), -1)
print(x.shape)  # torch.Size([BS, 18432])

So 18432 doesn't match 9216. You can do one of these changes and it will match the dimension and work:
# Reduce output channels here
self.conv3 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=5,stride=1,padding=2)
# OR change input dimension of Linear layer
self.fc1 = nn.Linear(18432, 1000)

Just a side note, you don't need to use a functional dropout in forward pass, you have already created a callable dropout in init. You can just call that:
x = self.drop(x)    # Instead of x = F.dropout(self.drop(x), training=self.training)

Training/testing mode for the dropout is handled internally when you switch between modes  net.eval() or net.train(). So you don't need to worry about that.


Instruction: Inference Discrepancy between PyTorch and DJL implementation via Kotlin

Input:
I have a PyTorch model trained on the 17flowers dataset, and converted via PyTorch's tracing to a JIT model. I have tested the inference output for the PyTorch model and the JIT converted model, and the results are equivalent there. This leads me to believe there is an issue with my implementation of the DJL framework.
There is an issue when I attempt to utilized DJL for inference utilizing the converted JIT model, which is necessary for DJL. I am not getting 100% match, which I expected.
The Kotlin implementation for djl.ai is straightforward and essentially follows the instructions here.
I have a sanitized version of the Kotlin code below:
@Throws(IOException::class, ModelException::class, TranslateException::class)
internal fun main(args: Array<String>) {
    val artifactId = "ai.djl.localmodelzoo:torchscript_17flowers"
    val pipeline = Pipeline()
    pipeline.add(CenterCrop(224, 224))
        .add(Resize(224, 224))
        .add(ToTensor())
        .add(Normalize(floatArrayOf(0.485f, 0.456f, 0.406f), floatArrayOf(0.229f, 0.224f, 0.225f)))
    val translator = ImageClassificationTranslator.builder()
        .setPipeline(pipeline)
        .optSynsetArtifactName("synset.txt")
        .optApplySoftmax(true)
        .build();

    System.setProperty("ai.djl.repository.zoo.location","build/pytorch_models/torchscript_17flowers")

    val criteria = Criteria.builder()
        .setTypes(Image::class.java, Classifications::class.java) // defines input and output data type
        .optTranslator(translator)
        .optArtifactId(artifactId) // defines which model to load
        .optProgress(ProgressBar())
        .build()

    val model = ModelZoo.loadModel(criteria)

    // single image test
    var img = ImageFactory.getInstance().fromUrl("https://image.jpg");
    img.getWrappedImage()

    val predictor: Predictor<Image, Classifications> = model.newPredictor()
    val classifications: Classifications = predictor.predict(img)
    val best = classifications.best<Classifications.Classification>()
}

My issue isn't getting things to run so much as it is getting the inference results to match. It is my understanding that they should match, and that Kotlin should work fine as DJL is meant to work for Java. I'm curious if there are any thoughts surrounding this encountered issue.

Output:
The discrepancy most likely comes from image pre-processing:
pipeline.add(CenterCrop(224, 224))
        .add(Resize(224, 224))
        .add(ToTensor())
        .add(Normalize(floatArrayOf(0.485f, 0.456f, 0.406f), floatArrayOf(0.229f, 0.224f, 0.225f)))

Many of PyTorch CV models, they don't do a center crop.
In order to get the same result as python, you must to make sure you process them the same way as python code does.
DJL image operation align with openCV operators, if you use PILLOW in python, you might see some difference in the result.


Instruction: Is this a right way to train and test the model using Pytorch?

Input:
I am trying to implement a binary classification with Neural Networks with Pytorch to train a model as shown below:

model = nn.Sequential(
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=196, out_features=300),
    nn.ReLU(),
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=300, out_features=196),
)
cross_entropy_loss = nn.CrossEntropyLoss()
klloss = bnn.BKLLoss(reduction='mean', last_layer_only=False)
klweight = 0.01
optimizer = optim.Adam(model.parameters(), lr=0.01)

Traing part:
 #training
    for step in range(200):
        models = model(data_tensor)
        cross_entropy = cross_entropy_loss(models, target_tensor)
        #cross_entropy = 0
        kl = klloss(model)
        total_cost = cross_entropy + klweight*kl

        optimizer.zero_grad()
        total_cost.backward()
        optimizer.step()

    _, predicted = torch.max(models.data, 1)
    final = target_tensor.size(0)
    correct = (predicted == target_tensor).sum()
    print('- Accuracy: %f %%' % (100 * float(correct) / final))
    print('- CE : %2.2f, KL : %2.2f' % (cross_entropy.item(), kl.item()))

Question 1: Is it a right way to train a model? In many articles I found that there is a section to iterate over the DataLoader for training data, such as :
for i, data in enumerate(trainloader, 0):
 # Get inputs
 inputs, targets = data

Question 2: What is the use of this loop if I can directly give the data features(inputs) as data_tensor and data labels(target) as target_tensor? Because when iterating through data loader, it takes more time.
I am not sure how to test my model. I did as given below:
correct = 0
total = 0
with torch.no_grad():
    for step in range(data_tensor_test.size(0)):
        models = model(data_tensor_test)
        _, predicted = torch.max(models.data, 1)
        total += target_tensor_test.size(0)
        correct += (predicted == target_tensor_test).sum().item()

print('Accuracy of the test data: %d %%' % (
    100 * correct / total))

Question 3: Is it right way to test the model that I trained above?

Output:
I'll attempt to answer your questions:
Question 1 & 2:
Is it a right way to train a model? In many articles I found that there is a section to iterate over the DataLoader for training data.
You should use a dataset with a dataloader when training in pytorch for several reasons:

It allows us to sample our data randomly
It doesn't preload data into memory, which is particularly useful for huge datasets
It operates in the background of code so fetches data parallel to training, thus saving time
It's very efficient at batching your data

What you're doing here is seemingly running your model on every element in your data at once. If you only have 32 points in your data this might be fine (although really not optimal because you have such limited data) but there is a balance to strike between running your optimizer and exposing your model to learning opportunities.
I would guess this takes longer because your model is very small and it probably takes longer to run the data fetching than it does when it's already preloaded in memory. It is hard to answer this without knowing the size of your dataset and the batch size of your processing.
Question 3: Is it right way to test the model that I trained above?
You need to set your model to its evaluation stage using model.eval() before you run any inference code. I also don't understand the point of your for loop since you just pass through the same data every time. I would generally run something like this:
correct = 0
total = 0
with torch.no_grad():
    model.eval()
    for step,(dat,lab) in enumerate(dataloader_test):
        models = model(dat)
        _, predicted = torch.max(models.data, 1)
        total += dat.size(0)
        correct += (predicted == lab).sum().item()

print('Accuracy of the test data: %d %%' % (
    100 * correct / total))


Instruction: PyTorch - How to use Avg 2d Pooling as a dataset transform?

Input:
In Pytorch, I have a dataset of 2D images (or alternatively, 1 channel images) and I'd like to apply average 2D pooling as a transform. How do I do this? The following does not work:
    omniglot_dataset = torchvision.datasets.Omniglot(
        root=data_dir,
        download=True,
        transform=torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.CenterCrop((80, 80)),
            # torchvision.transforms.Resize((10, 10))
            torch.nn.functional.avg_pool2d(kernel_size=3, strides=1),
        ])
    )

Output:
Transforms have to be a callable object. But torch.nn.functional.avg_pool2d doesn't return a callable object, but rather it is just a function you can call to process, that is why they are packaged under torch.nn.functional where all functionals receives the input and parameters. You need to use the other version:
torch.nn.AvgPool2d(kernel_size=3, stride=1)

Which returns a callable object, that can be called to process a given input, for example:
pooler = torch.nn.AvgPool2d(kernel_size=3, stride=1)
output = pooler(input)

With this change here you can see different versions how you can use callable version:
import torchvision
import torch
import matplotlib.pyplot as plt

omniglotv1 = torchvision.datasets.Omniglot(
        root='./dataset/',
        download=True,
        transform=torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.CenterCrop((80, 80))
        ])
    )

x1, y = omniglotv1[0]
print(x1.size())      # torch.Size([1, 80, 80])

omniglotv2 = torchvision.datasets.Omniglot(
        root='./dataset/',
        download=True,
        transform=torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.CenterCrop((80, 80)),
            torch.nn.AvgPool2d(kernel_size=3, stride=1)
        ])
    )

x2, y = omniglotv2[0]
print(x2.size())      # torch.Size([1, 78, 78])

pooler = torch.nn.AvgPool2d(kernel_size=3, stride=1)
omniglotv3 = torchvision.datasets.Omniglot(
        root='./dataset/',
        download=True,
        transform=torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.CenterCrop((80, 80)),
            pooler
        ])
    )

x3, y = omniglotv3[0]
print(x3.size())      # torch.Size([1, 78, 78])

Here, I just added a short code for image printing to see how the transform looks:
x_img   = x1.squeeze().cpu().numpy()
ave_img = x2.squeeze().cpu().numpy()
combined = np.zeros((158,80))
combined[0:80,0:80] = x_img
combined[80:,0:78] = ave_img
plt.imshow(combined)
plt.show()


Instruction: In pytorch, how to train a model with two or more outputs?

Input:
output_1, output_2 = model(x)
loss = cross_entropy_loss(output_1, target_1)
loss.backward()
optimizer.step()

loss = cross_entropy_loss(output_2, target_2)
loss.backward()
optimizer.step()

However, when I run this piece of code, I got this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Then, I really wanna know what I am supposed to do to train a model with 2 or more outputs

Output:
The entire premise on which pytorch (and other DL frameworks) is founded on is the backporpagation of the gradients of a scalar loss function.
In your case, you have a vector (of dim=2) loss function:
[cross_entropy_loss(output_1, target_1), cross_entropy_loss(output_2, target_2)]

You need to decide how to combine these two losses into a single scalar loss.
For instance:
weight = 0.5  # relative weight
loss = weight * cross_entropy_loss(output_1, target_1) + (1. - weight) * cross_entropy_loss(output_2, target_2)
# now loss is a scalar
loss.backward()
optimizer.step()