Instruction: Is there a way to train a neural network with untrainable params in torch?

Input:
So, I am trying to learn the transfer learning and trying to implement it in an autoencoder while I have frozen some of my initial layers and then i added new layers in it as well now i am trying to train it but for some reasons i cannot train them. Any concept on how to training really works with the untrainable params?
This is the code for my training the error occurs in backward propagation and it thinks due to some modification in neural network it cannot work.
nb_epoch = 10
for epoch in range(1, nb_epoch+1):
    train_loss = 0
    s = 0. #Compute RMSE
    for id_user in range(nb_users):
        input = Variable(training_set_torch[id_user]).unsqueeze(0)#Batch of 1 input vector or else wont work
        target = input.clone()
        if torch.sum(target.data > 0) > 0: #Target.data is rating
            output = net(input)
            #Now we create optimize the memory
            target.require_grad = False
            output[target == 0] = 0
            #calculating loss
            loss = criterion(output, target)
            #Average of movie considered like non zero ratings
            mean_corrector = nb_users/float(torch.sum(target.data > 0)+ 1e-10)# 1e-10 is to avoid 0/0
            #Now calling backward method
            loss.backward()*
            train_loss += np.sqrt(loss.data*mean_corrector)
            s += 1.
            #Using the Optimzer
            optimizer.step()

    #Printing what happens at every epoch
    print('Epoch: ' +str(epoch)+ 'Loss: '+str(train_loss/s))

*Error occured
ERROR: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1000]], which is output 0 of SigmoidBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
and when I set that to True it shows another error
ERROR: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1000]], which is output 0 of SigmoidBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Output:
I suspect this line is causing you the issue:
output[target == 0] = 0

Try removing it and see if you get an error.
Update:
You can remove these entries, and still avoid the "in-place" error by replacing the assignment with a multiplication by a mask:
output = output * (target != 0)


Instruction: PyTorch loss function that depends on gradient of network with respect to input

Input:
I'm trying to implement a loss function that depends on the gradient of the network with respect to its inputs. That is, the loss function has a term like
sum(u - grad_x(network(x)))
where u is computed by forward propagating x through the network.
I'm able to compute the gradient by calling
funcApprox = funcNetwork.forward(X)
funcGrad = grad(funcApprox, X, grad_outputs=torch.ones_like(funcApprox))

Here, funcNetwork is my NN and X is the input. These computations are done in the loss function.
However, now if I attempt to do the following
opt.zero_grad()
loss = self.loss(X) # My custom loss function that calculates funcGrad, etc., from above

opt.zero_grad()
loss.backward()
opt.step()

I see the following error:
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

on the loss.backward() line from above.
I've tried playing around with create_graph, retain_graph, etc. but to no avail.
Any help is appreciated!

Output: As per comment by @aretor, setting retain_graph=True, create_graph=False in the grad call in the loss function, and retain_graph=True in backward solves the issue.


Instruction: Training loss is increasing in CNN?

Input:
I am in the process of training my first CNN to solve a multi-class classification problem. I am feeding in images of animals corresponding to one of 182 classes, however I have ran into some issues. Firstly my code appears to get stuck on optimiser.step(), it has been calculating this for roughly 30 minutes. Secondly my training loss is increasing:
EPOCH: 0 BATCH: 1999 LOSS: 1.5790680234357715
EPOCH: 0 BATCH: 3999 LOSS: 2.9340945997834207

If any one would be able to provide some guidance that would be greatly appreciated. Below is my code
#loading data
train_data = dataset.get_subset(
    "train",
    transform=transforms.Compose(
        [transforms.Resize((448, 448)), transforms.ToTensor()]
    ),
)

train_loader = get_train_loader("standard", train_data, batch_size=16)

#definind model
class ConvNet(nn.Module):

  def __init__(self):
    super(ConvNet, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 3, 1)
    self.conv2 = nn.Conv2d(6, 16, 3, 3)
    self.fc1 = nn.Linear(37*37*16, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 182)

  def forward(self, X):
    X = F.relu(self.conv1(X))
    X = F.max_pool2d(X, 2, 2)
    X = F.relu(self.conv2(X))
    X = F.max_pool2d(X, 2, 2)
    X = torch.flatten(X, 1)
    X = F.relu(self.fc1((X)))
    X = F.relu(self.fc2((X)))
    X = self.fc3(X)
    return F.log_softmax(X, dim=1)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(modell.parameters(), lr=0.001)

import time

start_time = time.time()

#VARIABLES  (TRACKER)
epochs = 2
train_losses = []
test_losses = []
train_correct = []
test_correct = []

# FOR LOOP EPOCH
for i in range(epochs):
  trn_corr = 0
  tst_corr = 0

  running_loss = 0.0
  #TRAIN
  for b, (X_train, Y_train, meta) in enumerate(train_loader):

    b+=1 #batch starts at 1

    #zero parameter gradients
    optimizer.zero_grad()

    # pass training to model as float (later compute loss)
    output = modell(X_train.float())

    #Calculate the loss of outputs with respect to ground truth values
    loss = criterion(output, Y_train)

    #Backpropagate the loss through the network
    loss.backward()

    #perform parameter update based on the current gradient
    optimizer.step()

    predicted = torch.max(output.data, 1)[1]

    batch_corr = (predicted == Y_train).sum() # True (1) or False (0)
    trn_corr += batch_corr

    running_loss += loss.item()

    if b%2000 == 1999:
      print(f"EPOCH: {i} BATCH: {b} LOSS: {running_loss/2000}")
      running_loss = 0.0

train_losses.append(loss)
train_correct.append(trn_corr)

Output:
As for the loss, it may be due to the model. The model has some rooms for improvement. Only 2 convolution layers is not sufficient for your data, as well as only expanding to 16 channels. Use more convolution layers with more channels. For example, 5 conv layers with channels of 16, 32, 32, 64, 64. Experiment with different numbers of layers and channels to see which one is best. Also, a good learning rate for Adam is 3e-4.To more easily track the models progress, I’d recommend decreasing the interval at which it prints the loss so you can more easily track progress.
About the data, are there enough instances of each class? Is it normalized between 0 and 1?


Instruction: Pytorch error: TypeError: adaptive_avg_pool3d(): argument 'output_size' (position 2) must be tuple of ints, not list

Input:
I got this error when trying to use the AdaptiveAvgPool3D in Pytorch. Below is the error trace
Traceback (most recent call last):

File "/scratch/a.bip5/BraTS 2021/./sisa.py", line 395, in
outputs = model(inputs)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/scratch/a.bip5/BraTS 2021/./sisa.py", line 96, in forward
x1 = self.pool1(x)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/scratch/a.bip5/BraTS 2021/./sisa.py", line 135, in forward
x1=aa(x)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/modules/pooling.py", line 1166, in forward
return F.adaptive_avg_pool3d(input, self.output_size)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/nn/functional.py", line 1148, in adaptive_avg_pool3d
return torch._C._nn.adaptive_avg_pool3d(input, _output_size)
TypeError: adaptive_avg_pool3d(): argument 'output_size' (position 2) must be tuple of ints, not list
When looking into the error stack I found this piece of code in ../functional.py:
if has_torch_function_unary(input):
    return handle_torch_function(
        adaptive_max_pool3d_with_indices, (input,), input, output_size, return_indices=return_indices
    )
_output_size = _list_with_default(output_size, input.size())
return torch._C._nn.adaptive_max_pool3d(input, _output_size)

Printing out types for output_size and _output_size shows one's a tuple(as it is supposed to be) and the other is turned into a list before being passed through in the lib function itself. What I don't understand is why this conversion to list is taking place if the function using this list doesn't like it? If the purpose of this is to give out an error, what are the conditions that satisfy the has_torch_function_unary condition?
Any help would be appreciated.
Edit:
I tried to side-step the problem by using output_size in the final return statement instead of _output_size. That led to an even more cryptic error-
TypeError: adaptive_avg_pool3d(): argument 'output_size' (position 2) must be tuple of ints, not tuple
How does pytorch differentiate between tuple of ints and a tuple?

Output: The reason behind the error turned out to be a type error. When passing the size to AdaptiveAvgPool3d I was getting the values for the size through division. Although the remainder was 0, division by default saves values as float in python. This meant the tuple being passed as the output_size was in fact a tuple of float which the _list_with_default turns into a list but likely doesn't if the tuple was made of int. Simply using int(..) for each dimension passed to AdaptiveAvgPool3d was the solution.


Instruction: TypeError: 'module' object is not callable error?

Input:
I studying on a model training. when i called the training function i got this error "TypeError: 'module' object is not callable" and i can't see where i missed it.
here is my calling function:
train(
            model,
            optimizer,
            loss,
            train_loader,
            hyperparams["epoch"],
            scheduler=hyperparams["scheduler"],
            device=hyperparams["device"],
            val_loader=val_loader,

)
the error i got

Output:
You are calling tqdm module, instead of tqdm method from tqdm module.
Replace:
import tqdm

with:
from tqdm import tqdm


Instruction: output from bert into cnn model

Input:
i am trying to concatenate bert model with Cnn 1d using pytorch . I used this code but I do not understand what is meaning of in_channels and out_channels in function conv1d
if input shape into cnn model is torch(256,64,768)
class MixModel(nn.Module):
    def __init__(self,pre_trained='distilbert-base-uncased'):
        super().__init__()
        self.bert =  AutoModel.from_pretrained('distilbert-base-uncased')
        self.hidden_size = self.bert.config.hidden_size
        self.conv = nn.Conv1d(in_channels=1, out_channels=256, kernel_size=5, padding='valid', stride=1)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool1d(kernel_size= 256- 5 + 1)
        self.dropout = nn.Dropout(0.3)
        self.clf = nn.Linear(self.hidden_size*2,6)

    def forward(self,inputs, mask , labels):

        cls_hs = self.bert(input_ids=inputs,attention_mask=mask, return_dict= False)
        x=cls_hs
       # x = torch.cat(cls_hs[0]) # x= [416, 64, 768]
        x = self.conv(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.dropout(x)
        x = self.clf(x)

        return x

Edit
I use recommended answer and change the parameters but i got error
class MixModel(nn.Module):
    def __init__(self,pre_trained='bert-base-uncased'):
        super().__init__()
        self.bert =  AutoModel.from_pretrained('distilbert-base-uncased')
        self.hidden_size = self.bert.config.hidden_size
        self.conv = nn.Conv1d(in_channels=768, out_channels=256, kernel_size=5, padding='valid', stride=1)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool1d(kernel_size= 64- 5 + 1)
        print(11)
        self.dropout = nn.Dropout(0.3)
        print(12)
        self.clf = nn.Linear(self.hidden_size*2,6)
        print(13)

    def forward(self,inputs, mask , labels):

        cls_hs = self.bert(input_ids=inputs,attention_mask=mask, return_dict= False)
        x=cls_hs[0]
        print(cls_hs[0])
        print(len(cls_hs[0]))
        print(cls_hs[0].size())
        #x = torch.cat(cls_hs,0) # x= [416, 64, 768]
        x = x.permute(0, 2, 1)
        x = self.conv(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.dropout(x)
        x = self.clf(x)
return x

the error is
5 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
1846     if has_torch_function_variadic(input, weight, bias):
1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
1849
1850
RuntimeError: mat1 and mat2 shapes cannot be multiplied (65536x1 and 1536x6)

Output:
The dimension of the output prediction of BERT (and many other transformer-based models) is of shape batchxseq-lenxfeature-dim: That is, your input is a batch of 256 sequences of length (probably with padding) of 64 tokens, each token is represented by a feature vector of dimension 768.
In order to apply 1-d convolution along the sequence-len dimension, you will need first to permute x to be of shape batchxdimxlen:
x = x.permute(0, 2, 1)

Now you can apply nn.Conv1d, where the in_channels is the dimension of x = 768. the out_channels is up to you - what is going to be the hidden dimension of your model.


Instruction: Numpy to pyTorch: are there different data types?

Input:
Question: Can somebody help me to align this two approaches of data generation so that both of them can be used by the nn-model below ? When using appraoch (2) with numpy and torch.from_numpy(x) a run time error occurs ("expected scalar type Float but found Double")
For data generation I have these two approaches:
import torch
import torch.nn as nn
import numpy as np

def get_training_data_1():
    x = torch.randn(batch_size, n_in)
    y = torch.tensor([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]])
    return x,y

def get_training_data_2():
    x = np.random.rand(batch_size, n_in)
    y = np.array([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]])

    x = torch.from_numpy(x)
    y = torch.from_numpy(y)
    return x,y

n_in, n_h, n_out, batch_size = 2, 5, 1, 10
x, y = get_training_data_2()

With this model I run into problems when using appraoch (2) with numpy and torch.from_numpy(x), while it is OK when using approach (1)
#---- Create a NN-model
model = nn.Sequential( nn.Linear(n_in, n_h),     # hidden layer
                       nn.ReLU(),                # activation layer
                       nn.Linear(n_h, n_out),    # output layer
                       nn.Sigmoid() )            # final 0, 1 rounding

#---- Construct the loss function
criterion = torch.nn.MSELoss()

#---- Construct the optimizer (Stochastic Gradient Descent in this case)
optimizer = torch.optim.SGD(model.parameters(), lr = 0.1)

#---- Gradient Descent
for epoch in range(1501):
    y_pred = model(x)                       # Forward pass: Compute predicted y by passing x to the model
    loss = criterion(y_pred, y)             # Compute and print loss
    if epoch%50 == 0:
        print(epoch, loss.item())
    optimizer.zero_grad()                   # Zero gradients, perform a backward pass, and update the weights.
    loss.backward()                         # perform a backward pass (backpropagation)
    optimizer.step()                        # Update the parameters

Output:
The default floating point type in torch is float32 (i.e. single precision).  In NumPy the default is float64 (double precision).  Try changing get_training_data_2 so that it explicitly sets the data type of the numpy arrays numpy.float32 before converting them to torch tensors:
def get_training_data_2():
    x = np.random.rand(batch_size, n_in).astype(np.float32)
    y = np.array([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]],
                 dtype=np.float32)

    x = torch.from_numpy(x)
    y = torch.from_numpy(y)
    return x,y

Note. With the newer NumPy random API, you can generate float32 samples directly instead of casting float64 values to float32.
def get_training_data_2(rng):
    x = rng.random(size=(batch_size, n_in), dtype=np.float32)
    y = np.array([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]],
                 dtype=np.float32)

    x = torch.from_numpy(x)
    y = torch.from_numpy(y)
    return x,y

rng = np.random.default_rng()
x, y = get_training_data_2(rng)


Instruction: conda env broken after installing pytorch on M1 - Intel MKL FATAL ERROR

Input:
I installed pytorch on my M1 mac book, following some instructions on-line (via conda command). Then my whole environment got corrupted. Whenever I try to import some library (pandas, numpy, whatever) I get this:
Intel MKL FATAL ERROR: This system does not meet the minimum requirements for use of the Intel(R) Math Kernel Library.
The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions.
The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions.
The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.

How can I fix it?
How can I install pytorch successully?

Output:
Here is what worked for me a few weeks ago:

Create a new conda environment and activate it.
conda install ipykernel jupyter numpy pandas matplotlib nomkl (the key part being to include nomkl and don't include PyTorch).
pip install torch torchvision

I could not get step 3 to work using conda (possibly related issue).
This doesn't feel like a robust solution (mixing conda and pip), but the environment has worked for me the past few weeks.


Instruction: How to get input size for a operator in pytorch script model?

Input:
I use this code to transfer the model to script model:
scripted_model = torch.jit.trace(detector.model, images).eval()
Then I print the scripted_model. A part of the output is as follows:
 (base): DLA(
    original_name=DLA
    (base_layer): Sequential(
      original_name=Sequential
      (0): Conv2d(original_name=Conv2d)
      (1): BatchNorm2d(original_name=BatchNorm2d)
      (2): ReLU(original_name=ReLU)
    )
    (level0): Sequential(
      original_name=Sequential
      (0): Conv2d(original_name=Conv2d)
      (1): BatchNorm2d(original_name=BatchNorm2d)
      (2): ReLU(original_name=ReLU)
    )
    (level1): Sequential(
      original_name=Sequential
      (0): Conv2d(original_name=Conv2d)
      (1): BatchNorm2d(original_name=BatchNorm2d)
      (2): ReLU(original_name=ReLU)
    )
    (level2): Tree(
      original_name=Tree
      (tree1): BasicBlock(
        original_name=BasicBlock
        (conv1): Conv2d(original_name=Conv2d)
        (bn1): BatchNorm2d(original_name=BatchNorm2d)
        (relu): ReLU(original_name=ReLU)
        (conv2): Conv2d(original_name=Conv2d)
        (bn2): BatchNorm2d(original_name=BatchNorm2d)
      )
      (tree2): BasicBlock(
        original_name=BasicBlock
        (conv1): Conv2d(original_name=Conv2d)
        (bn1): BatchNorm2d(original_name=BatchNorm2d)
        (relu): ReLU(original_name=ReLU)
        (conv2): Conv2d(original_name=Conv2d)
        (bn2): BatchNorm2d(original_name=BatchNorm2d
      )
      (root): Root(
        original_name=Root
        (conv): Conv2d(original_name=Conv2d)
        (bn): BatchNorm2d(original_name=BatchNorm2d)
        (relu): ReLU(original_name=ReLU)
      )
      (downsample): MaxPool2d(original_name=MaxPool2d)
      (project): Sequential(
        original_name=Sequential
        (0): Conv2d(original_name=Conv2d)
        (1): BatchNorm2d(original_name=BatchNorm2d)
      )
    )
...

I just want to get the input size for the operator, such as how many inputs for the operator (0): Conv2d(original_name=Conv2d). I print the graph of this script model, the output is as follows:
  %4770 : __torch__.torch.nn.modules.module.___torch_mangle_11.Module = prim::GetAttr[name="wh"](%self.1)
  %4762 : __torch__.torch.nn.modules.module.___torch_mangle_15.Module = prim::GetAttr[name="tracking"](%self.1)
  %4754 : __torch__.torch.nn.modules.module.___torch_mangle_23.Module = prim::GetAttr[name="rot"](%self.1)
  %4746 : __torch__.torch.nn.modules.module.___torch_mangle_7.Module = prim::GetAttr[name="reg"](%self.1)
  %4738 : __torch__.torch.nn.modules.module.___torch_mangle_3.Module = prim::GetAttr[name="hm"](%self.1)
  %4730 : __torch__.torch.nn.modules.module.___torch_mangle_27.Module = prim::GetAttr[name="dim"](%self.1)
  %4722 : __torch__.torch.nn.modules.module.___torch_mangle_19.Module = prim::GetAttr[name="dep"](%self.1)
  %4714 : __torch__.torch.nn.modules.module.___torch_mangle_31.Module = prim::GetAttr[name="amodel_offset"](%self.1)
  %4706 : __torch__.torch.nn.modules.module.___torch_mangle_289.Module = prim::GetAttr[name="ida_up"](%self.1)
  %4645 : __torch__.torch.nn.modules.module.___torch_mangle_262.Module = prim::GetAttr[name="dla_up"](%self.1)
  %4461 : __torch__.torch.nn.modules.module.___torch_mangle_180.Module = prim::GetAttr[name="base"](%self.1)
  %5100 : (Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor) = prim::CallMethod[name="forward"](%4461, %input.1)
  %5082 : Tensor, %5083 : Tensor, %5084 : Tensor, %5085 : Tensor, %5086 : Tensor, %5087 : Tensor, %5088 : Tensor, %5089 : Tensor = prim::TupleUnpack(%5100)
  %5101 : (Tensor, Tensor, Tensor) = prim::CallMethod[name="forward"](%4645, %5082, %5083, %5084, %5085, %5086, %5087, %5088, %5089)
  %5097 : Tensor, %5098 : Tensor, %5099 : Tensor = prim::TupleUnpack(%5101)
  %3158 : None = prim::Constant()

I even can find the operator name. How can I get input size for a specific operator in the script model?

Output:
One solution is to try summary from torchinfo and the output shape of the first layer is the input shape for the next one and so on:
!pip install torchinfo

from torchinfo import summary
summary(model, input_size=(batch_size, 3, 224, 224)) # input size to your NN

#output

===============================================================================================
    Layer (type:depth-idx)                        Output Shape              Param #
    ===============================================================================================
    ResNet50                                      --                        --
    ├─ResNet: 1-1                                 [64, 10]                  --
    │    └─Conv2d: 2-1                            [64, 64, 112, 112]        9,408
    │    └─BatchNorm2d: 2-2                       [64, 64, 112, 112]        128
    │    └─ReLU: 2-3                              [64, 64, 112, 112]        --
    │    └─MaxPool2d: 2-4                         [64, 64, 56, 56]          --
    │    └─Sequential: 2-5                        [64, 64, 56, 56]          --
    │    │    └─BasicBlock: 3-1                   [64, 64, 56, 56]          73,984
    │    │    └─BasicBlock: 3-2                   [64, 64, 56, 56]          73,984
    │    └─Sequential: 2-6                        [64, 128, 28, 28]         --
    │    │    └─BasicBlock: 3-3                   [64, 128, 28, 28]         230,144
    │    │    └─BasicBlock: 3-4                   [64, 128, 28, 28]         295,424
    │    └─Sequential: 2-7                        [64, 256, 14, 14]         --
    │    │    └─BasicBlock: 3-5                   [64, 256, 14, 14]         919,040
    │    │    └─BasicBlock: 3-6                   [64, 256, 14, 14]         1,180,672
    │    └─Sequential: 2-8                        [64, 512, 7, 7]           --
    │    │    └─BasicBlock: 3-7                   [64, 512, 7, 7]           3,673,088
    │    │    └─BasicBlock: 3-8                   [64, 512, 7, 7]           4,720,640
    │    └─AdaptiveAvgPool2d: 2-9                 [64, 512, 1, 1]           --
    │    └─Linear: 2-10                           [64, 10]                  5,130
    ===============================================================================================
    Total params: 11,181,642
    Trainable params: 11,181,642
    Non-trainable params: 0
    Total mult-adds (G): 116.07
    ===============================================================================================
    Input size (MB): 38.54
    Forward/backward pass size (MB): 2543.33
    Params size (MB): 44.73
    Estimated Total Size (MB): 2626.59
    ===============================================================================================


Instruction: Loaded PyTorch model has a different result compared to saved model

Input:
I have a python script that trains and then tests a CNN model. The model weights/parameters are saved after testing through the use of:
checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, path + filename)

After saving I immediately load the model through the use of a function:
model_load = create_model(cnn_type="vgg", numberofclasses=len(cases))

And then, I load the model weights/parameters through:
model_load.load_state_dict(torch.load(filePath+filename), strict = False)
model_load.eval()

Finally, I feed this model the same testing data I used before the model was saved.
The problem is that the testing results are not the same when I compare the testing results of the model before saving and after loading. My hunch is that due to strict = False, some of the parameters are not being passed through to the model. However, when I make strict = True. I receive errors. Is there a work around this?
The error message is:
RuntimeError: Error(s) in loading state_dict for CNN:
        Missing key(s) in state_dict: "linear.weight", "linear.bias", "linear 2.weight", "linea r2.bias", "linear 3.weight", "linear3.bias". Unexpected key(s) in state_dict: "state_dict", "optimizer".

Output:
You are loading a dictionary containing the state of your model as well as the optimizer's state. According to your error stack trace, the following should solve the issue:
>>> model_state = torch.load(filePath+filename)['state_dict']
>>> model_load.load_state_dict(model_state, strict=True)


Instruction: Plotting contour of pytorch functions

Input:
I have a function like the one below:
x0 = torch.tensor([1.5, .1])
Q = torch.tensor([[10.0, 6.0],
                  [9.0, 8.0]])

def f1(x):
    z = x - x0
    Qz = z @ Q
    return 0.5 * Qz@z

How do I get a contour plot for this?
x is a 2D tensor.
I am messing up somewhere while using meshgrid.

Output:
First, let's make your code work with a batch of 2d points, that is, x of shape nx2:
def f1(x):
  z = x - x0  # z of shape n-2
  Qz = z @ Q  # Qz of shape n-2
  return 0.5 * (Qz * z).sum(dim=-1)  # we want output of size n and not n-n

Now we can create a grid over which we want to plot f1(x):
grid = torch.stack(torch.meshgrid(torch.linspace(-20., 20., 100), torch.linspace(-20., 20., 100), indexing='xy'))
# convert the grid to a batch of 2d points:
grid = grid.reshape(2, -1).T
# get the output on all points in the grid
out = f1(grid)
# plot
plt.matshow(out.detach().numpy().reshape(100,100))

You'll get:


Instruction: torchmetric calculate accuracy with threshold

Input:
How does torchmetrics.Accuracy threshold keyword work? I have the following setup:
import torch, torchmetrics

preds = torch.tensor([[0.3600, 0.3200, 0.3200]])
target = torch.tensor([0])
torchmetrics.functional.accuracy(preds, target, threshold=0.5, num_classes=3)

output:
tensor(1.)

None of the values in preds have a probability higher than 0.5, why is it giving 100% accuracy?

Output:
threshold=0.5 sets each probability under 0.5 to 0. It is used only in case you are dealing with binary (which is not your case, since num_classes=3) or multilabel classification (which seems not the case because multiclass is not set). Therefore threshold is not actually involved.
In your case, preds represents a prediction related to one observation. Its largest probability (0.36) is set at position 0, thus argmax(preds) and target are equal to 0, therefore accuracy is set to 1, because the prediction of the (only) observation is correct.


Instruction: BERT Classifier ValueError: Target size (torch.Size([4, 1])) must be the same as input size (torch.Size([4, 2]))

Input:
I'm training a Classifier Model but it's a few days that I cannot overcame a problem!
I have the ValueError: Target size (torch.Size([4, 1])) must be the same as input size (torch.Size([4, 2])) error but actually it seems correct to me ! Indeed I used unsqueeze(1) to put them of the same size. WHat else I can try? Thank you!
class SequenceClassifier(nn.Module):

  def __init__(self, n_classes):
    super(SequenceClassifier, self).__init__()
    self.bert = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME,return_dict=False)
    self.drop = nn.Dropout(p=0.3)
    self.out = nn.Linear(self.bert.config.hidden_size, n_classes)

  def forward(self, input_ids, attention_mask):
    _, pooled_output = self.bert(
      input_ids=input_ids,
      attention_mask=attention_mask
    )
    output = self.drop(pooled_output)
    return self.out(output)

model = SequenceClassifier(len(class_names))
model = model.to(device)

EPOCHS = 10

optimizer = AdamW(model.parameters(), lr=2e-5, correct_bias=False)
total_steps = len(train_data_loader) * EPOCHS

scheduler = get_linear_schedule_with_warmup(
  optimizer,
  num_warmup_steps=0,
  num_training_steps=total_steps
)
weights=[0.5,1]
pos_weight=torch.FloatTensor(weights).to(device)
loss_fn=nn.BCEWithLogitsLoss(pos_weight=pos_weight)

def train_epoch(
  model,
  data_loader,
  loss_fn,
  optimizer,
  device,
  scheduler,
  n_examples
):
  model = model.train()

  losses = []
  correct_predictions = 0

  for d in data_loader:
    input_ids = d["input_ids"].to(device)
    attention_mask = d["attention_mask"].to(device)
    targets = d["targets"].to(device)

    outputs = model(
      input_ids=input_ids,
      attention_mask=attention_mask
    )

    _, preds = torch.max(outputs, dim=1)

    targets = targets.unsqueeze(1)
    loss = loss_fn(outputs, targets)

    correct_predictions += torch.sum(preds == targets)
    losses.append(loss.item())

    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()

  return correct_predictions.double() / n_examples, np.mean(losses)

%%time

history = defaultdict(list)
best_accuracy = 0

for epoch in range(EPOCHS):

  print(f'Epoch {epoch + 1}/{EPOCHS}')
  print('-' * 10)

  train_acc, train_loss = train_epoch(
    model,
    train_data_loader,
    loss_fn,
    optimizer,
    device,
    scheduler,
    len(df_train)
  )

  print(f'Train loss {train_loss} accuracy {train_acc}')

  val_acc, val_loss = eval_model(
    model,
    val_data_loader,
    loss_fn,
    device,
    len(df_val)
  )

  print(f'Val   loss {val_loss} accuracy {val_acc}')
  print()

  history['train_acc'].append(train_acc)
  history['train_loss'].append(train_loss)
  history['val_acc'].append(val_acc)
  history['val_loss'].append(val_loss)

  if val_acc > best_accuracy:
    torch.save(model.state_dict(), 'best_model_state.bin')
    best_accuracy = val_acc

ValueError: Target size (torch.Size([4, 1])) must be the same as input size (torch.Size([4, 2]))

EDIT
I have a binary classifier problem, indeed i have 2 classes encoded 0 ("bad") and 1 ("good").

Output:
In case anyone stumbles on this like I did, I'll write out an answer since there aren't a lot of google hits for this target size/input size error and the previous answer has some factual inaccuracies.
Unlike the previous answer would suggest, the real problem isn't with the loss function but with the output of the model.nn.BCEWithLogitsLoss is completely fine for multi-label and multi-class applications. Chiara updated her post saying that in fact she has a binary classification problem, but even that should not be a problem for this loss function. So why the error?
The original code has:
outputs = model(
  input_ids=input_ids,
  attention_mask=attention_mask
)
_, preds = torch.max(outputs, dim=1)

This means "Run the model, then create preds with the row indeces of the highest output of the model". Obviously, there is only a "index of highest" if there are multiple  predicted values. Multiple output values usually means multiple input classes, so I can see why Shai though this was multi-class. But why would we get multiple outputs from a binary classifier?
As it turns out, BERT (or Huggingface anyway) for binary problems expects that n_classes is set to 2 -- setting classes to 1 puts the model in regression mode. This means that under the hood, binary problems are treated like a two-class problem, outputting predictions with the size [2, batch size] -- one column predicting the chance of it being a 1 and one for the chance of it being 0. The loss fucntion throws an error because it is supplied with only one row of one-hot encoded labels: targets = d["targets"].to(device) so the  labels have dimensions [batch size] or after the unsqueeze, [1, batch size]. Either way, the dimensions don't match up.
Some loss functions can deal with this fine, but others require the exact same dimensions. To make things more frustrating, for version 1.10, nn.BCEWithLogitsLoss requires matching dimensions but later versions do not.
One solution may therefore be to update your pytorch (version 1.11 would work for example).
For me, this was not an option, so I ended up going with a different loss function. nn.CrossEntropyLoss, as suggested by Shai, indeed does the trick as it accepts any input with the same length. In other words, they had a working solution for the wrong reasons.


Instruction: Pytorch Custom dataset's __getitem__ calls itself indefinitely when handling exception

Input:
I'm writing a script for my customdatset class but I get Index out of range error whenever I access data using for loop like so:
cd = CustomDataset(df)
for img, target in cd:
   pass

I realized I might have a problem reading a few images (if they are corrupt) so I implemented a random_on_error feature which chooses a random image if something is wrong with the current image. And I'm sure that's where the problem is. As I've noticed that all the 2160 images in the dataset are read without any hiccups(i print the index number for every iteration) but the loop would not stop and reads the 2161st image which results in an Index out of range exception that gets handled by reading a random image. This continues forever.
Here is my class:
class CustomDataset(Dataset):
    def __init__(self, data: pd.DataFrame, augmentations=None, exit_on_error=False, random_on_error: bool = True):
        """
        :param data: Pandas dataframe with paths as first column and target as second column
        :param augmentations: Image transformations
        :param exit_on_error: Stop execution once an exception rises. Cannot be used in conjunction with random_on_error
        :param random_on_error: Upon an exception while reading an image, pick a random image and process it instead.
        Cannot be used in conjuntion with exit_on_error.
        """

        if exit_on_error and random_on_error:
            raise ValueError("Only one of 'exit_on_error' and 'random_on_error' can be true")

        self.image_paths = data.iloc[:, 0].to_numpy()
        self.targets = data.iloc[:, 1].to_numpy()
        self.augmentations = augmentations
        self.exit_on_error = exit_on_error
        self.random_on_error = random_on_error

    def __len__(self):
        return self.image_paths.shape[0]

    def __getitem__(self, index):
        image, target = None, None
        try:
            image, target = self.read_image_data(index)
        except:
            print(f"Exception occurred while reading image, {index}")
            if self.exit_on_error:
                print(self.image_paths[index])
                raise
            if self.random_on_error:
                random_index = np.random.randint(0, self.__len__())
                print(f"Replacing with random image, {random_index}")
                image, target = self.read_image_data(random_index)

            else:  # todo implement return logic when self.random_on_error is false
                return

        if self.augmentations is not None:
            aug_image = self.augmentations(image=image)
            image = aug_image["image"]

        image = np.transpose(image, (2, 0, 1))

        return (
            torch.tensor(image, dtype=torch.float),
            torch.tensor(target, dtype=torch.long)
        )

    def read_image_data(self, index: int) -> ImagePlusTarget:
        # reads image, converts to 3 channel ndarray if image is grey scale and converts rgba to rgb (if applicable)
        target = self.targets[index]
        image = io.imread(self.image_paths[index])
        if image.ndim == 2:
            image = np.expand_dims(image, 2)
        if image.shape[2] > 3:
            image = color.rgba2rgb(image)

        return image, target

I believe the problem is with the except block (line 27) in __getitem__(), as when I remove it the code works fine. But I cannot see what the problem here is.
Any help is appreciated, thanks

Output:
How do you expect python to know when to stop reading from your CustomDataset?
Defining a method __getitem__ in CustomDataset makes it an iterable object in python. That is, python can iterate over CustomDataset's items one by one. However, the iterable object must raise either StopIteration or IndexError for python to know it reached the end of the iterations.
You can either change the loop to expicitly use the __len__ of your dataset:
for i in range(len(cd)):
  img, target = cd[i]

Alternatively, you should make sure you raise IndexError from your dataset if index is outside the range. This can be done using multiple except clauses.
Something like:
try:
  image, target = self.read_image_data(index)
except IndexError:
  raise  # do not handle this error
except:
  # treat all other exceptions (corrupt images) here
  ...


Instruction: How to model this function in PyTorch

Input:
I want to train a feed-forward Neural Network with a single hidden layer that models the below equation.
h = g(W1.input1 + V1.input2 + b)
output1 = f(W2.h + b_w)
output2 = f(V2.h + b_v)

f and g are activation functions, h is the hidden representation, W1, W2, V1, V2 are Weight matrices, b, b_w, b_v are respective biases.
I can't concatenate 2 inputs because that will result in a single Weight matrix. I can't train two separate NNs because the latent representation will miss the interaction between 2 inputs. Any help is much appreciated. I have also attached the NN diagram below

Output:
I decided to write my own Linear layer which calculates h = g(W1.input1 + V1.input2 + b) I do this by creating 2 parameters W1 and V1 multiply input1 and input2 with the 2 parameters and then add everything. The code is given below:
import torch
import torch.nn as nn
import math

class MyLinearLayer(nn.Module):
    def __init__(self, size_in1, size_out1):
        super().__init__()
        self.size_in1, self.size_out1 = size_in1, size_out1
        W_1 = torch.Tensor(size_out1, size_in1)
        V_1 = torch.Tensor(size_out1, size_in1)
        self.W1 = nn.Parameter(W_1)
        self.V1 = nn.Parameter(V_1)
        bias = torch.Tensor(size_out1)
        self.bias = nn.Parameter(bias)

    def forward(self, x):
        w_times_x= torch.mm(x[0], self.W1.t())
        v_times_x= torch.mm(x[1], self.V1.t())
        weight_times_x = torch.add(w_times_x, v_times_x)
        return torch.add(weight_times_x, self.bias)  # w times x + w times v + b

class NN(nn.Module):
    def __init__(self, in_ch, h_ch, out_ch):
        super().__init__()
        self.input = MyLinearLayer(in_ch, h_ch)
        self.W2 = nn.Linear(h_ch, out_ch)
        self.V2= nn.Linear(h_ch, out_ch)
        self.act = nn.ReLU()

    def forward(self, i1, i2):
        # I pass in stacked input
        inp = torch.stack([i1,i2])
        h = self.act(self.input(inp))
        o1 = self.act(self.W2(h))
        o2 = self.act(self.V2(h))
        return o1, o2

model = NN(5, 10, 5)
o1,o2 = model(torch.rand(2, 5), torch.rand(2, 5))
for name, param in model.named_parameters():
    if param.requires_grad:
        print(name, '->',param.data.shape)

output 7 parameters to be trained:
input.W1 -> torch.Size([10, 5])
input.V1 -> torch.Size([10, 5])
input.bias -> torch.Size([10])
W2.weight -> torch.Size([5, 10])
W2.bias -> torch.Size([5])
V2.weight -> torch.Size([5, 10])
V2.bias -> torch.Size([5])

Thanks for all the inputs @aretor, @Ivan, and @DerekG


Instruction: How can I reduce dimension of a tensor after using Softmax?

Input:
I got a tensor of scores (lets' call it logits_tensor) that has shape: (1910, 164, 33).
Taking a look at it, logits_tensor[0][0]:
tensor([-2.5916, -1.5290, -0.8218, -0.8882, -2.0961, -2.1064, -0.7842, -1.5200,
        -2.1324, -1.5561, -2.4731, -2.1933, -2.8489, -1.8257, -1.8033, -1.8771,
        -2.8365,  0.6690, -0.6895, -1.7054, -2.4862, -0.8104, -1.5395, -1.1351,
        -2.7154, -1.7646, -2.6595, -2.0591, -2.7554, -1.8661, -2.7512, -2.0655,
         5.7374])

Now, by applying a softmax
probs_tensor = torch.nn.functional.softmax(logits_tensor, dim=-1)

I obtain another tensor with the same dimensions that contain probabilities, probs_tensor[0][0]:
tensor([2.3554e-04, 6.8166e-04, 1.3825e-03, 1.2937e-03, 3.8660e-04, 3.8263e-04,
        1.4356e-03, 6.8778e-04, 3.7283e-04, 6.6341e-04, 2.6517e-04, 3.5078e-04,
        1.8211e-04, 5.0665e-04, 5.1810e-04, 4.8127e-04, 1.8438e-04, 6.1396e-03,
        1.5782e-03, 5.7138e-04, 2.6173e-04, 1.3984e-03, 6.7454e-04, 1.0107e-03,
        2.0812e-04, 5.3857e-04, 2.2009e-04, 4.0118e-04, 1.9996e-04, 4.8660e-04,
        2.0079e-04, 3.9860e-04, 9.7570e-01])

What I'd like to obtain is a tensor of shape 1910, 164) that contains the indices of the max probabilities (for each of 164 elements) shown above, like this:
precitions[0]
> tensor([32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,  1, 17, 17, 17,
       17, 17, 17, 17, 17, 17, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
       32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]

Note that "32" is the index of the higher probability element in probs_tensor[0][0]. The same task can be achieved by using torch.argmax but I need the softmax step.

Output:
Indeed you can apply torch.argmax on the tensor:
>>> logits_tensor = torch.rand(1910, 164, 33)
>>> probs_tensor = logits_tensor.softmax(-1)

>>> probs_tensor.argmax(-1).shape
torch.Size([1910, 164])

Do note, applying argmax on probs_tensor is identical to applying it on logits_tensor. The logit with the highest value will remain the logit with the highest probability mass.


Instruction: didn't match because some of the keywords were incorrect: dtype

Input:
The following code is an instance method built in a class
def get_samples_from_component(self,batchSize):
    SMALL = torch.tensor(1e-10, dtype=torch.float64, device=local_device)
    a_inv = torch.pow(self.kumar_a,-1)
    b_inv = torch.pow(self.kumar_b,-1)
    r1    = torch.tensor(SMALL, dtype=torch.float64,device=self.device)
    r2    = torch.tensor(1-SMALL, dtype=torch.float64, device=self.device)
    v_means = torch.mul(self.kumar_b, beta_fn(1.+a_inv, self.kumar_b)).to(device=self.device)
    u       = torch.distributions.uniform.Uniform(low=r1, high=r2).sample([1]).squeeze()
    v_samples  = torch.pow(1 - torch.pow(u, b_inv), a_inv).to(device=self.device)
    if v_samples.ndim > 2:
        v_samples = v_samples.squeeze()
    v0 = v_samples[:, -1].pow(0).reshape(v_samples.shape[0], 1)
    v1 = torch.cat([v_samples[:, :self.z_dim - 1], v0], dim=1)
    n_samples = v1.size()[0]
    n_dims = v1.size()[1]
    components = torch.zeros((n_samples, n_dims)).to(device=self.device)

    for k in range(n_dims):
        if k == 0:
            components[:, k] = v1[:, k]
        else:
            components[:, k] = v1[:, k] * torch.stack([(1 - v1[:, j]) for j in range(n_dims) if j < k]).prod(axis=0)
    # ensure stick segments sum to 1
    assert_almost_equal(torch.ones(n_samples,device=self.device).cpu().numpy(), components.sum(axis=1).detach().cpu().numpy(),
                        decimal=4, err_msg='stick segments do not sum to 1')
    print(f'size of sticks: {components}')

    components = torch.IntTensor(torch.argmax(torch.cat( self.compose_stick_segments(v_means),1) ,1), dtype=torch.long, device=self.device)
    components = torch.cat( [torch.range(0, batchSize).unsqueeze(1), components.unsqueeze(1)], 1)
    print(f'size of sticks: {components}')
    all_z = []
    for d in range(self.z_dim):
        temp_z = torch.cat(1, [self.z_sample_list[k][:, d].unsqueeze(1) for k in range(self.K)])
        all_z.append(gather_nd(temp_z, components).unsqueeze(1))
    out       = torch.cat( all_z,1)
    return out

By running my code I get the following error message
components = torch.IntTensor(torch.argmax(torch.cat( self.compose_stick_segments(v_means),1) ,1), dtype=torch.long, device=self.device)
TypeError: new() received an invalid combination of arguments - got (Tensor, device=torch.device, dtype=torch.dtype), but expected one of:
 * (*, torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, *, torch.device device)
      didn't match because some of the keywords were incorrect: dtype
 * (object data, *, torch.device device)
      didn't match because some of the keywords were incorrect: dtype

I will appreciate if someone suggests a solution for this error.

Output:
v_means is already a tensor, try to simple remove the tensor re-implementation in:
components = torch.IntTensor(torch.argmax(torch.cat( self.compose_stick_segments(v_means),1) ,1), dtype=torch.long, device=self.device)

to:
components = torch.argmax(torch.cat( self.compose_stick_segments(v_means),1) ,1)

or simply remove the dtype, since it seems to cast it to integer anyway:
components = torch.IntTensor(torch.argmax(torch.cat( self.compose_stick_segments(v_means),1) ,1), device=self.device)


Instruction: torchvision.datasets.ImageFolder is giving me a 3x3 grid of images intead of 1 image

Input:
I can't figure out why it's giving me 9 gray images in a 3x3 grid instead of just one color image (original image is not gray and has RGB channels). I have spent 5 hours on this. Thanks for the help.
Here is my code
test_path = "asl_data/test/" #path to the folder
test_data = torchvision.datasets.ImageFolder(test_path, transform=torchvision.transforms.ToTensor())
def test32():
    for x, y in test_data:
        print(x.shape)
        x = x.reshape(533,800,3)
        plt.axis("off")
        plt.imshow(x)
        plt.show()
        plt.axis("off")
        plt.imshow(x[:176,:267,:])
        break
test32()

Output:
Classic.
You reshape instead of permute.
See this thread on the crucial difference between the two.
Fix:
x = x.permute((1, 2, 0))
plt.imshow(x)

A simple visual example:
x, y = test_data[0]  # take one image
x.shape  # torch.Size([3, 223, 320])

# see the difference
fig, ax = plt.subplots(1,2)
ax[0].imshow(x.numpy().reshape(223, 320, 3))
ax[0].set_title('Wrong reshape instead of permute')

ax[1].imshow(x.permute((1,2,0)))
ax[1].set_title('correctly permuting')


Instruction: How can I calculate the F1-score and other classification metrics from a faster-RCNN? (object detection in PyTorch)

Input:
I'm trying to wrap my head around this but struggling to understand how I can compute the f1-score in an object detection task.
Ideally, I would like to know false positives, true positives, false negatives and true negatives for every target in the image (it's a binary problem with an object in the image as one class and the background as the other class).
Eventually I would also like to extract the false positive bounding boxes from the image. I'm not sure if this is efficient but I'd save the image names and bbox predictions and whether they are false positives etc. into a numpy file.
I currently have this set up with a batch size of 1 so I can apply a non-maximum suppression algorithm per image:
def apply_nms(orig_prediction, iou_thresh=0.3):

    # torchvision returns the indices of the bboxes to keep
    keep = torchvision.ops.nms(orig_prediction['boxes'], orig_prediction['scores'], iou_thresh)

    final_prediction = orig_prediction
    final_prediction['boxes'] = final_prediction['boxes'][keep]
    final_prediction['scores'] = final_prediction['scores'][keep]
    final_prediction['labels'] = final_prediction['labels'][keep]

    return final_prediction

cpu_device = torch.device("cpu")
model.eval()
with torch.no_grad():
  for images, targets in valid_data_loader:
    images = list(img.to(device) for img in images)
    outputs = model(images)
    outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]
    predictions = apply_nms(outputs[0], iou_thresh=0.3)

Any idea on how I can determine the aforementioned classification metrics and f1-score?
I've come across this line in an evaluation code provided by torchvision and wondering whether it would help me going forward:
res = {target["image_id"].item(): output for target, output in zip(targets, outputs)}

Output:
The use of the terms precision, recall, and F1 score in object detection are slightly confusing because these metrics were originally used for binary evaluation tasks (e.g. classifiation). In any case, in object detection they have slightly different meanings:
let:
TP - set of predicted objects that are successfully matched to a ground truth object (above IOU threshold for whatever dataset you're using, generally 0.5 or 0.7)
FP - set of predicted objects that were not successfully matched to a ground truth object
FN - set of ground truth objects that were not successfully matched to a predicted object
Precision: TP / (TP + FP)
Recall:    TP / (TP + FN)
F1:        2*Precision*Recall /(Precision + Recall)

You can find many an implementation of the matching step (matching ground truth and predicted objects) generally provided with an dataset for evaluation, or you can implement it yourself. I'll suggest the py-motmetrics repository.
A simple implementation of the IOU calculation might look like:
def iou(self,a,b):
    """
    Description
    -----------
    Calculates intersection over union for all sets of boxes in a and b

    Parameters
    ----------
    a : tensor of size [batch_size,4]
        bounding boxes
    b : tensor of size [batch_size,4]
        bounding boxes.

    Returns
    -------
    iou - float between [0,1]
        average iou for a and b
    """

    area_a = (a[2]-a[0]) * (a[3]-a[1])
    area_b = (b[2]-b[0]) * (b[3]-b[1])

    minx = max(a[0], b[0])
    maxx = min(a[2], b[2])
    miny = max(a[1], b[1])
    maxy = min(a[3], b[3])

    intersection = max(0, maxx-minx) * max(0,maxy-miny)
    union = area_a + area_b - intersection
    iou = intersection/union

    return iou


Instruction: Pass correct image shape to model that uses a conv1d and con2d in the same network

Input:
so I am trying to implement the VGG network, everything in the paper, but i have when i am using the architecture that has a conv1-255 as part of it network. below is my code
def _make_convo_layers(architecture) -> torch.nn.Sequential:
        """
        Create convolutional layers from the vgg architecture type passed in.
        : param architecture:
        """
        layers = []
        in_channels = 3
        for layer in architecture:
            if type(layer) == int:
                out_channels = layer
                layers += [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
                # layers.append([nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=1) + nn.ReLU()])
                in_channels = layer
            elif (layer == 'Conv1-256'):
                out_channels = 256
                layers += [nn.Conv1d(256, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
            elif (layer == 'LRN'):
                layers += [nn.LocalResponseNorm(5, alpha=0.0001, beta=0.75, k=1)]
            elif (layer == 'M'):
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        return nn.Sequential(*layers)

below is me passing some random data to the model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
vgg = VGGNet(config['vgg16-C1']).to(device)
x = torch.randn(1, 3, 224, 224).to(device)
model = vgg(x).to(device)
print(model.shape)

below is the error i received when i passed the x variable to the model
RuntimeError: Expected 3-dimensional input for 3-dimensional weight [256, 256, 3], but got 4-dimensional input of size [1, 256, 56, 56] instead

any help will do, please

Output:
just as jhso commented use are doing this wrong, looking that this VGG explanation on this VGG page, what you need to do isn't to use a 1D convolution but to perform a convolution operation of kernel size 1 instead of using the original kernel size of 3.
def _make_convo_layers(architecture) -> torch.nn.Sequential:
        """
        Create convolutional layers from the vgg architecture type passed in.
        : param architecture:
        """
        layers = []
        in_channels = 3
        for layer in architecture:
            if type(layer) == int:
                out_channels = layer
                layers += [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=1), nn.ReLU()]
                in_channels = layer
            elif (layer == 'Conv1-256'):
                out_channels = 256
                layers += [nn.Conv1d(256, out_channels, kernel_size=1, padding=1, stride=1), nn.ReLU()]
            elif (layer == 'LRN'):
                layers += [nn.LocalResponseNorm(5, alpha=0.0001, beta=0.75, k=1)]
            elif (layer == 'M'):
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        return nn.Sequential(*layers)


Instruction: Assign custom weight in pytorch

Input:
I'm trying to assign some custom weight to my PyTorch model but it doesn't work correctly.
class Mod(nn.Module):
    def __init__(self):
        super(Mod, self).__init__()

        self.linear = nn.Sequential(
            nn.Linear(1, 5)
        )
    def forward(self, x):
        x = self.linear(x)
        return x
mod = Mod()

mod.linear.weight = torch.tensor([1. ,2. ,3. ,4. ,5.], requires_grad=True)
mod.linear.bias = torch.nn.Parameter(torch.tensor(0., requires_grad=True))

print(mod.linear.weight)
>>> tensor([1., 2., 3., 4., 5.], requires_grad=True)

output = mod(torch.ones(1))
print(output)
>>> tensor([ 0.2657,  0.3220, -0.0726, -1.6987,  0.3945], grad_fn=<AddBackward0>)

The output is expected to be [1., 2., 3., 4., 5.] but it doesn't work as expected. What am I missing here?

Output:
You are not updating the weights in the right place. Your self.linear is not a nn.Linear layer, but rather a nn.Sequential container. Your nn.Linear is the first layer in the sequential. To access it you need to index self.linear:
with torch.no_grad():
  mod.linear[0].weight.data = torch.tensor([1. ,2. ,3. ,4. ,5.], requires_grad=True)[:, None]
  mod.linear[0].bias.data = torch.zeros((5, ), requires_grad=True)  # bias is not a scalar here


Instruction: Is there a PyTorch equivalent of tf.custom_gradient()?

Input:
I am new to PyTorch but have a lot of experience with TensorFlow.
I would like to modify the gradient of just a tiny piece of the graph: just the derivative of activation function of a single layer. This can be easily done in Tensorflow using tf.custom_gradient, which allows you to supply customized gradient for any functions.
I would like to do the same thing in PyTorch and I know that you can modify the backward() method, but that requires you to rewrite the derivative for the whole network defined in the forward() method, when I would just like to modify the gradient of a tiny piece of the graph. Is there something like tf.custom_gradient() in PyTorch? Thanks!

Output:
You can do this in two ways:
1. Modifying the backward() function:
As you already said in your question, pytorch also allows you to provide a custom backward implementation. However, in contrast to what you wrote, you do not need to re-write the backward() of the entire model - only the backward() of the specific layer you want to change.
Here's a simple and nice tutorial that shows how this can be done.
For example, here is a custom clip activation that instead of killing the gradients outside the [0, 1] domain, simply passes the gradients as-is:
class MyClip(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return torch.clip(x, 0., 1.)

    @staticmethod
    def backward(ctx, grad):
        return grad

Now you can use MyClip layer wherever you like in your model and you do not need to worry about the overall backward function.

2. Using a backward hook
pytorch allows you to attach hooks to different layer (=sub nn.Modules) of your network. You can register_full_backward_hook to your layer. That hook function can modify the gradients:

The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations.


Instruction: CNN: Why do we first resize the image to 256 and then center crop to 224?

Input:
The transformation for Alexnet image input is below:
transforms.Resize(256),
transforms.CenterCrop(224),

Why do we first resize the image to 256 and then center crop to 224? I know that 224x224 is the default image size of ImageNet but why we can't directly resize the image to 224x224?

Output:
Perhaps this is best illustrated visually. Consider the following image (128x128px):

Say we would resize it to 16x16px directly, we'd end up with:

But if we'd resize it to 24x24px first,

and then crop it to 16x16px, it would look like this:

As you see, it's getting rid of the border, while retains details in the center. Note the differences side by side:
The same applies to 224px vs 256px, except this is at a larger resolution.


Instruction: How to convert a list of PyTorch tensors to a list of floats?

Input:
I have a list of tensors like the following one:
[tensor(0.9757), tensor(0.9987), tensor(0.9990), tensor(0.9994), tensor(0.9994)]

How can I change the type of the entire list and obtain something like:
[0.9757, 0.9987, 0.9990, 0.9994, 0.9994]

Thanks for the help!

Output:
You can use .item() and a list comprehension, assuming that every element is a one-element tensor:
result = [tensor.item() for tensor in data]
print(type(result[0]))
print(result)

This prints the desired result, albeit with some unavoidable precision error:
<class 'float'>
[0.9757000207901001, 0.9987000226974487, 0.9990000128746033, 0.9994000196456909, 0.9994000196456909]


Instruction: When does Pytorch initialize parameters?

Input:
I’m now writing my own network with Pytorch. And I want to use a pretrained model in my net. Here is my overwriting init() code:
class Generator(nn.Module):
    def __init__(self) -> None:
        super(Generator, self).__init__()
        model_path = "somedir"
        chekpoint = torch.load(model_path)
        h_model = H_model()
        h_model.load_state_dict(chekpoint['model'])
        # 设置为测试模式
        h_model.eval()
        self.H_model = h_model
        self.unet = UNet(enc_chs=(9,64,128,256,512), dec_chs=(512, 256, 128, 64), num_class=3, retain_dim=False, out_sz=(304, 304))

Here, the h_model is loaded from checkpoint which I’ve trained it well.
My question is that after the initialization, will the parameter in h_model changed(Are the pretrained parameters vaule being modified by some function?)? And why(I mean how does Pytorch treat self-defined layer when it initializes parameters? And when does Pytorch initialize parameters?)

Output:
For the basic layers (e.g., nn.Conv, nn.Linear, etc.) the parameters are initialized by the __init__ method of the layer.
For example, look at the source code of class _ConvNd(Module) (the class from which all other convolution layers are derived). At the bottom of its __init__ it calls self.reset_parameters() which initialize the weights.
Therefore, if your nn.Module does not have any "independent" nn.Parameters, only trainable parameters inside sub-nn.Modules, when you construct your network, all weights of the sub modules are being initialized as the sub modules are constructed.
That is, once you call h_model = H_model() the weights of h_model are already initialized to their default values. Calling h_model.load_state_dict(...) overrides these values to the desired pre-trained weights.


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (5x46656 and 50176x3)

Input:
I am trying to build a cnn model and while building i am getting the run time error. I have image size of 224x224x3 and batch size of 5 and i have 3 classes to be predicted.
class CNN(nn.Module):
def __init__(self):
    super().__init__()
    #5*3*224*224
    self.conv1=nn.Conv2d(3,6,3,1)
    self.relu1=nn.ReLU()
    self.pool=nn.MaxPool2d(2)
    #112*112*6

    self.conv2=nn.Conv2d(6,16,3,1)
    self.relu2=nn.ReLU()

    self.fc1=nn.Linear(16*56*56,3)

def forward(self,x):
    x=self.pool(self.relu1(self.conv1(x)))
    x=self.pool(self.relu2(self.conv2(x)))
    x=x.flatten(1)
    x=self.fc1(x)
    return x

I get the run time error during training. How do I fix this?

Output:
Considering the size of your input, your fully connected layer should have 16*54*54 neurons, not 16*56*56.
self.fc1 = nn.Linear(16*54*54, 3)

Alternatively, you can you a lazy module that will infer the number of neurons needed: nn.LazyLinear:
self.fc1 = nn.LazyLinear(3)


Instruction: PyTorch second derivative is zero everywhere

Input:
I'm working on a custom loss function that requires me to compute the trace of the hessian matrix of a neural network with respect to its inputs.
In one dimension, this reduces to the second derivative of the network with respect to its input. Suppose my network is u and my input is x. The problem is completely one-dimensional, so the relationship I'm trying to model is u(x) where u has one input and one output, and x is one-dimensional. However, I'm running with batches, so x "comes in" as a column vector and I'm expecting a column vector as output.
If we label the samples in the batch as x_1, x_2, ..., x_n, I'm thus interested in the following vectors:

In PyTorch, I have tried the following:
u = model(x)
d = grad(u, x, grad_outputs=torch.ones_like(u), create_graph=True)[0]
dd = grad(d, x, grad_outputs=torch.ones_like(d), retain_graph=True, create_graph=False)[0]

This works well for u' but u'' comes out as being zero everywhere:

I'm a bit confused on the whole notion of needing a vector jacobian product here to begin with. Should I view the computation as u mapping from R^n to R^n, where n is the size of my batch?
Any help is appreciated!

Output:
Toy example
I have tried to devise a toy example for you:
x = torch.rand(3, requires_grad=True).reshape(-1, 1)
u = x ** 2.

# du/dx = 2 * x
dx = torch.autograd.grad(u, x, torch.ones_like(u), retain_graph=True, create_graph=True)[0]

# d^2 u/dx^2 = 2
ddx = torch.autograd.grad(dx, x, torch.ones_like(u))[0]

These are the outputs of x, dx, ddx:
tensor([[0.5217],
        [0.0400],
        [0.5509]], grad_fn=<ReshapeAliasBackward0>)
tensor([[1.0434],
        [0.0800],
        [1.1018]], grad_fn=<MulBackward0>)
tensor([[2.],
        [2.],
        [2.]])

So grad is computing the gradient of x element-wise.
Should I view the computation as u mapping from R^n to R^n, where n is the size of my batch?
My answer is no, otherwise dx should have been a Jacobian matrix instead of a vector.
This works well for u' but u'' comes out as being zero everywhere
Given the limited context, the only thing I can think of is that the second derivative of model(x) with respect to x is zero.


Instruction: Detectron2 prediction problem on my local machine with custom model

Input:
I trained a custom model with detectron2 on google colab, and ok, it's working correctly. The model was trained, the predictions were ok, this on google colab. But when I made predictions on my local machine did'nt work. Here a similar example on google colab: https://colab.research.google.com/drive/1bSlH5Am_zFEWbJ9zTRu2wFEDKDvn0LUv?usp=sharing
I exported de final model and ran with this code:
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor
from detectron2.data import MetadataCatalog
from detectron2.utils.visualizer import Visualizer, ColorMode
import matplotlib.pyplot as plt
import cv2.cv2 as cv2

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.WEIGHTS = "model_final.pth" # path for final model
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.8
predictor = DefaultPredictor(cfg)
im = cv2.imread('0.jpg')
outputs = predictor(im)
v = Visualizer(im[:, :, ::-1],
               metadata=MetadataCatalog.get(cfg.DATASETS.TRAIN[0]),
               scale=0.5,
               instance_mode=ColorMode.IMAGE_BW)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
img = cv2.cvtColor(out.get_image()[:, :, ::-1], cv2.COLOR_RGBA2RGB)
cv2.imwrite('img.jpg',img)

I supose that the cfg.merge_from_file is the problem. Is there other file? Where I find on colab?
I tested the standard models and worked well on my local machine, the problem is with the custom model.

Output:
I saved the configs with this comand and then I downloaded.
f = open('config.yml','w')
f.write(cfg.dump())
f.close()

and replaced:
cfg.merge_from_file("./detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")

by
cfg.merge_from_file("config.yml")

and worked.


Instruction: What is the opposite of torch.unique_consecutive?

Input:
How can I efficiently revert torch.unique_consecutive? I.e.:
x = torch.tensor([1, 1, 2, 2, 3, 1, 1, 2])
output, counts = torch.unique_consecutive(x, return_counts=True)

y = torch.SOMETHING(output,counts) #y equals x

Output:
Please use torch.repeat_interleave(**args) for your task
x = torch.tensor([1, 1, 2, 2, 3, 1, 1, 2])
output, counts = torch.unique_consecutive(x, return_counts=True)
y = torch.repeat_interleave(output, counts)
#>>y = [1, 1, 2, 2, 3, 1, 1, 2]


Instruction: Constrained Linear combination of learned parameters is pytorch?

Input:
I have three tensors  X,Y,Z and I want to learn the optimal convex combination of these tensors wrt to  some cost, i.e.
aX + bY + cZ such that a + b + c = 1. How can I do this easily in Pytorch?
I know that I could just concatenate along an unsqueezed  axis and then apply linear  layer as  so:
X = X.unsqueeze(-1)
Y = Y.unsqueeze(-1)
Z = Z.unsqueeze(-1)
W = torch.cat([X,Y,Z], dim = -1)   #third axis has dimension  3)
W = torch.linear(3,1)(W)

but this would not apply the convex combination constraint...

Output:
I found an answer that works well for  those who are interested this would generalize to a linear combination of N tensors you just need to change the weights dim and number of tensors you concatenate.
weights = nn.Parameter(torch.rand(1,3))
X = X.unsqueeze(-1)
Y = Y.unsqueeze(-1)
Z = Z.unsqueeze(-1)
weights_normalized = nn.functional.softmax(weights, dim=-1)
output = torch.matmul(torch.cat([X, Y, Z], dim=-1), weights_normalized.t()).squeeze()


Instruction: torchmetrics behaviour for one-hot encoded values

Input:
I am having a hard time understanding the following scenario. I have a output probability of 0.0 on each class which means value of metrics such as f1 score, accuracy and recall should be zero? However i get the following:
import torch, torchmetrics
preds = torch.tensor([[0., 0., 0.],
                      [0., 0., 0.],
                      [0., 0., 0.]])

target = torch.tensor([[1, 0, 0],
                       [0, 1, 0],
                       [0, 0, 1]])

print("F1: ", torchmetrics.functional.f1_score(preds, target))
print("Accuracy: ", torchmetrics.functional.accuracy(preds, target))
print("Recall: ", torchmetrics.functional.recall(preds, target))
print("Precision: ", torchmetrics.functional.precision(preds, target))

Output:
F1:  tensor(0.)
Accuracy:  tensor(0.6667)
Recall:  tensor(0.)
Precision:  tensor(0.)

Why is accuracy 0.6667? I would expect all outputs to be 0.0.

Output:
Your preds is a probabilities array for multi-label classification problem:
To make it simpler, I will assume the example like that:
preds = torch.tensor([[0., 0., 0.]]) # multi-labels [label1, label2, label3]

target = torch.tensor([[1, 0, 0]])

The true negatives are 2 since classifier predicts not existence for label2 and label3 while label2 and label3 indeed should not be existed.
The true positives are 0 since classifier predicts the existence of any label while a label should be existed.
The false negative is 1 since classifier predicts no existence for label1 while  label1 should be existed.
The false positives are 0 since classifier predicts any label while a label should not be existed.

According to the above equation, Accuracy = 2/3 = 0.6667
You can read here more about different metrics and their calculations.


Instruction: Pytorch: How to get mean of slices along an axis where the slices indices value are defined on a different tensor and gradients only flow into slices

Input:
I would like to take the mean along an axis of a tensor, defined by tensor which contains several slices.
So this would be my sample tensor for which I want to get mean of slices from, along the first dimension
import torch

sample = torch.arange(0,40).reshape(10,-1)
sample

tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11],
        [12, 13, 14, 15],
        [16, 17, 18, 19],
        [20, 21, 22, 23],
        [24, 25, 26, 27],
        [28, 29, 30, 31],
        [32, 33, 34, 35],
        [36, 37, 38, 39]])

And this would be the tensor which contains the start and end indices for which I would like to get the mean of
mean_slices = torch.tensor([
    [3, 5],
    [1, 8],
    [4, 8],
    [6, 9],
])

Since Pytorch doesn't have ragged tensors, I use a trick described here
https://stackoverflow.com/a/71337491/3259896
Where the cumsum is calculated through the entire axis for which I want to get the mean from, then the row of each end slice index is retrieved and subtracted from the cumsum row that is before the start slice index. Finally, the result is divided by the length of the slices.
padded = torch.nn.functional.pad(
    sample.cumsum(dim=0), (0, 0, 1, 0)
)
padded

tensor([[  0,   0,   0,   0],
        [  0,   1,   2,   3],
        [  4,   6,   8,  10],
        [ 12,  15,  18,  21],
        [ 24,  28,  32,  36],
        [ 40,  45,  50,  55],
        [ 60,  66,  72,  78],
        [ 84,  91,  98, 105],
        [112, 120, 128, 136],
        [144, 153, 162, 171],
        [180, 190, 200, 210]])

pools = torch.diff(
    padded[mean_slices], dim=1
).squeeze()/torch.diff(mean_slices, dim=1)

pools

tensor([[14., 15., 16., 17.],
        [16., 17., 18., 19.],
        [22., 23., 24., 25.],
        [28., 29., 30., 31.]])

The only issue with this solution is that originally I was only looking to get the mean of specifically the rows defined by the slices, and while my current solution does that, the calculations involve all rows before the slices indices as well. So the backwards pass may not work as intended.
Is this guess correct?
Is there a more exact and computationally efficient way to calculate the mean for the slices defined in a tensor?

Output:
Why do you think that the gradient calculation includes values of pixels outside the slices?
When you compute the sum over the slice using torch.cumsum you sum all the values outside the slice twice: once to estimate their sum, stored in the row before the slice and the second time you sum the slice and these values storing this value at the last row of the slice.
The most important thing is you subtract the row-before-first from the last row: That is, you eliminate the sum of all values outside the slice from the equation. Thus these values has no effect on the calculation nor on the gradients.
Here's a simple example:
Consider the function f(x,y,z) = x + y + z - z. What is the gradient of f w.r.t z? Once z is eliminated, it has no effect on the value of f nor on its gradient.
Bottom line: your backward pass is correct, and is not affected by values outside the slices.

regarding a more efficient implementation:
If the minimal slice starting index is high (that is, there's a large portion of sample that is ignored by all slices) you might remove it completely:
mn, mx = mean_slices.min(), mean_slices.max()  # only the relevant par of sample
padded_ef = torch.nn.functional.pad(
    sample[mn:mx, :].cumsum(dim=0), (0, 0, 1, 0)
)
# sum the slices - need to shift the index
pools_ef = torch.diff(
    padded_ef[mean_slices-mn], dim=1
).squeeze()/torch.diff(mean_slices, dim=1)

Results with the same pools, but potentially involving less elements of sample if the slices are "packed".
However, unless sample is very large w.r.t the slices, I don't believe this will give you a significant boost in run time.


Instruction: Reshape PyTorch tensor so that matrices are horizontal

Input:
I'm trying to combine n matrices in a 3-dimensional PyTorch tensor of shape (n, i, j) into a single 2-dimensional matrix of shape (i, j*n). Here's a simple example where n=2, i=2, j=2:
m = torch.tensor([[[2, 3],
                   [5, 7]],
                  [[11, 13],
                   [17, 19]]])
m.reshape(2, 4)

I was hoping this would produce:
tensor([[ 2,  3, 11, 13],
        [ 5,  7, 17, 19]])

But instead it produced:
tensor([[ 2,  3,  5,  7],
        [11, 13, 17, 19]])

How do I do this? I tried torch.cat and torch.stack, but they require tuples of tensors. I could try and create tuples, but that seems inefficient. Is there a better way?

Output:
To combine n + j with reshape you need them consequent in shape. One can fix it with swapaxes:
m = torch.tensor([[[2, 3],
               [5, 7]],
              [[11, 13],
               [17, 19]]])
m=m.swapaxes( 0,1 )
m.reshape(2, 4)

tensor([[ 2,  3, 11, 13],
        [ 5,  7, 17, 19]])


Instruction: TypeError: _open() got an unexpected keyword argument 'pilmode'

Input:
I am training a CNN model on the COCO dataset, and I am getting this error after a few iterations. The error is not consistent because I got this error in 1100 iterations, once in 4500 iterations and one time in 8900 iterations (all of them in 1 epoch).
I thought that this error might be a bug in the new version of imageio, so I changed the version to 2.3.0 but still, after 8900 iterations in 1 epoch, I am getting this error.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-4b33bec4a89e> in <module>()
     52
     53     # train for one epoch
---> 54     train_loss = train(train_loader, model, [criterion1, criterion2], optimizer)
     55     print('train_loss: ',train_loss)
     56

4 frames
/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
    432             # instantiate since we don't know how to
    433             raise RuntimeError(msg) from None
--> 434         raise exception
    435
    436

TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-34-4c8722b5b16b>", line 143, in __getitem__
    image = imageio.imread(img_path, pilmode='RGB')
  File "/usr/local/lib/python3.7/dist-packages/imageio/core/functions.py", line 206, in imread
    reader = read(uri, format, 'i', **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/imageio/core/functions.py", line 129, in get_reader
    return format.get_reader(request)
  File "/usr/local/lib/python3.7/dist-packages/imageio/core/format.py", line 168, in get_reader
    return self.Reader(self, request)
  File "/usr/local/lib/python3.7/dist-packages/imageio/core/format.py", line 217, in __init__
    self._open(**self.request.kwargs.copy())
TypeError: _open() got an unexpected keyword argument 'pilmode'

Output:
I've had this error before. The TLDR is that you can't assume all of your data is clean and able to be parsed.  You aren't loading the data in order as far as I can tell either.  You may even have data shuffling enabled.  With all of that in mind you should not expect it to fail determinisitically at iteration 100 or 102 or anything.
The issue comes down to one (or more) of the files in COCO dataset is either corrupted or is of a different format.  You can process the images in order with a batchsize of 1 and print out the file name to see which one it is.
To "fix" this issue you can do one of several things:

wrap the call to load the image in a try-except block and just skip it.
Convert the image yourself to another appropriate format.
Try a different way to load images in with pytorch.

See here as an example failure scenario when loading in images with imageio.


Instruction: Quantized model gives negative accuracy after conversion from pytorch to ONNX

Input:
I'm trying to train a quantize model in pytorch and convert it to ONNX.
I employ the quantized-aware-training technique with help of pytorch_quantization package.
I used the below code to convert my model to ONNX:
from pytorch_quantization import nn as quant_nn
from pytorch_quantization import calib
from pytorch_quantization.tensor_quant import QuantDescriptor
from pytorch_quantization import quant_modules
import onnxruntime
import torch
import torch.utils.data
from torch import nn
import torchvision

def export_onnx(model, onnx_filename, batch_onnx, per_channel_quantization):
    model.eval()
    quant_nn.TensorQuantizer.use_fb_fake_quant = True # We have to shift to pytorch's fake quant ops before exporting the model to ONNX

    if per_channel_quantization:
        opset_version = 13
    else:
        opset_version = 12

    # Export ONNX for multiple batch sizes
    print("Creating ONNX file: " + onnx_filename)
    dummy_input = torch.randn(batch_onnx, 3, 224, 224, device='cuda') #TODO: switch input dims by model
    input_names = ['input']
    output_names = ['Linear[fc]']  ### ResNet34
    dynamic_axes = {'input': {0: 'batch_size'}}

    try:
        torch.onnx.export(model, dummy_input, onnx_filename, input_names=input_names,
                          export_params=True, output_names=output_names, opset_version=opset_version,
                          verbose=True, enable_onnx_checker=False, do_constant_folding=True)

    except ValueError:
        warnings.warn(UserWarning("Per-channel quantization is not yet supported in Pytorch/ONNX RT (requires ONNX opset 13)"))
        print("Failed to export to ONNX")
        return False
    return True

After conversion, I get the following warnings:

warnings.warn("'enable_onnx_checker' is deprecated and ignored. It will be removed in "
W0305 12:39:40.472136 140018114328384 tensor_quantizer.py:280] Use Pytorch's native experimental fake quantization.

/usr/local/lib/python3.8/dist-packages/pytorch_quantization/nn/modules/tensor_quantizer.py:285: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!

Also, the accuracy is not valid for ONNX model!
Accuracy summary:
+-----------+-------+
| Stage     |  Top1 |
+-----------+-------+
| Finetuned | 38.03 |
| ONNX      | -1.00 |
+-----------+-------+

More info is here:
pytorch 1.10.2+cu102
torchvision 0.11.3+cu102
TensorRT  8.2.3-1+cuda11.4
ONNX 1.11.0
ONNX Runtime 1.10.0
cuda 11.6
python 3.8

What is the problem with ONNX conversion?

Output:
After some tries, I found that there is a version conflict. I changed the versions accordingly:
onnx == 1.9.0
onnxruntime == 1.8.1
pytorch == 1.9.0+cu111
torchvision == 0.10.0+cu111


Instruction: how to combine two trained models using PyTorch?

Input:
I'm currently working on two models that use different types of data but are connected. I'd want to create a combination model that takes in one instance of each of the data types, runs them through each of the pre-trained models independently, and then processes the combined output of the two distinct models through a few feed-forward layers at the top.
So far, I've learned that I can change forward to accept both inputs, so I've just cloned the structures of my individual models into the combined one, processed them each individually using forward(right )'s layers, and then merged the outputs as specified. What I'm having trouble with is figuring out how to achieve this.

Output:
as I understand from your question you can create two models then you need a third model that combines both the neural network with the forward and in the __main__ you can then load_state_dict
for example:
the first model
class FirstM(nn.Module):
    def __init__(self):
        super(FirstM, self).__init__()
        self.fc1 = nn.Linear(20, 2)

    def forward(self, x):
        x = self.fc1(x)
        return x

the second model
class SecondM(nn.Module):
    def __init__(self):
        super(SecondM, self).__init__()
        self.fc1 = nn.Linear(20, 2)

    def forward(self, x):
        x = self.fc1(x)
        return x

here you create a model that you can merge both two models in it as follows:
class Combined_model(nn.Module):
    def __init__(self, modelA, modelB):
        super(Combined_model, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        self.classifier = nn.Linear(4, 2)

    def forward(self, x1, x2):
        x1 = self.modelA(x1)
        x2 = self.modelB(x2)
        x = torch.cat((x1, x2), dim=1)
        x = self.classifier(F.relu(x))
        return x

and then outside the classed in the main you can do as following
# Create models and load state_dicts
modelA = FirstM()
modelB = SecondM()
# Load state dicts
modelA.load_state_dict(torch.load(PATH))
modelB.load_state_dict(torch.load(PATH))

model = Combined_model(modelA, modelB)
x1, x2 = torch.randn(1, 10), torch.randn(1, 20)
output = model(x1, x2)


Instruction: creating tensor by composition of smaller tensors

Input:
I would like to create a 4x4 tensor that is composed of four smaller 2x2 tensors in this manner:
The tensor I would like to create:
in_t = torch.tensor([[14,  7,  6,  2],
                     [ 4,  8, 11,  1],
                     [ 3,  5,  9, 10],
                     [12, 15, 16, 13]])

I would like to create this tensor composed from these four smaller tensors:
a = torch.tensor([[14, 7], [ 4,  8]])
b = torch.tensor([[6,  2], [11,  1]])
c = torch.tensor([[3,  5], [12, 15]])
d = torch.tensor([[9, 10], [16, 13]])

I have tried to use torch.cat like this:
mm_ab = torch.cat((a,b,c,d), dim=0)

but I end up with an 8x2 tensor.

Output:
You can control the layout of your tensor and achieve the desired result with a combination of torch.transpose and torch.reshape. You can perform an outer transpose followed by an inner transpose:
>>> stack = torch.stack((a,b,c,d))
tensor([[[14,  7],
         [ 4,  8]],

        [[ 6,  2],
         [11,  1]],

        [[ 3,  5],
         [12, 15]],

        [[ 9, 10],
         [16, 13]]])

Reshape-tranpose-reshape-transpose-reshape:
>>> stack.reshape(4,2,-1).transpose(0,1).reshape(-1,2,4).transpose(0,1).reshape(-1,4)
tensor([[14,  7,  6,  2],
        [ 4,  8, 11,  1],
        [ 3,  5,  9, 10],
        [12, 15, 16, 13]])

Essentially, reshapes allow you to group and view your tensor differently while transpose operation will alter its layout (it won't remain contiguous) meaning you can achieve the desired output.


Instruction: Discrepancy between log_prob and manual calculation

Input:
I want to define multivariate normal distribution with mean [1, 1, 1] and variance covariance matrix with 0.3 on diagonal. After that I want to calculate log likelihood on datapoints [2, 3, 4]
By torch distributions
import torch
import torch.distributions as td

input_x = torch.tensor([2, 3, 4])
loc = torch.ones(3)
scale = torch.eye(3) * 0.3
mvn = td.MultivariateNormal(loc = loc, scale_tril=scale)
mvn.log_prob(input_x)
tensor(-76.9227)

From scratch
By using formula for log likelihood:

We obtain tensor:
first_term = (2 * np.pi* 0.3)**(3)
first_term = -np.log(np.sqrt(first_term))
x_center = input_x - loc
tmp = torch.matmul(x_center, scale.inverse())
tmp = -1/2 * torch.matmul(tmp, x_center)
first_term + tmp
tensor(-24.2842)

where I used fact that
My question is - what's the source of this discrepancy?

Output:
You are passing the covariance matrix to the scale_tril instead of covariance_matrix. From the docs of PyTorch's Multivariate Normal

scale_tril (Tensor) – lower-triangular factor of covariance, with positive-valued diagonal

So, replacing scale_tril with covariance_matrix would yield the same results as your manual attempt.
In [1]: mvn = td.MultivariateNormal(loc = loc, covariance_matrix=scale)
In [2]: mvn.log_prob(input_x)
Out[2]: tensor(-24.2842)

However, it's more efficient to use scale_tril according to the authors:

...Using scale_tril will be more efficient:

You can calculate the lower choelsky using torch.cholesky
In [3]: mvn = td.MultivariateNormal(loc = loc, scale_tril=torch.cholesky(scale))
In [4]: mvn.log_prob(input_x)
Out[4]: tensor(-24.2842)


Instruction: Create Pytorch "Stack of Views" to save on GPU memory

Input:
I am trying to expand datasets for analysis in Pytorch such that from one 1D (or 2D) tensor two stacks of views are generated. In the following image A (green) and B (blue) are views of the original tensor that are slid from left to right, which would then be combined into single tensors for batch processing:

The motivation behind using views for this is to save on GPU memory, since for large, multi-dimensional datasets this expansion process can convert a dataset of tens of MB into tens of GB despite tremendous data reuse (if normal tensors are used). Simply returning one view at a time is not desirable since the actual processing of tensors works in large batches.
Is what I'm trying to do possible in Pytorch? Simply using torch.stack(list of views) creates a new tensor with a copy of the original data, as verified by tensor.storage().data_ptr().
Another way to phrase the question: can you create batches of tensor views?
The current steps are:

Load and pre-process all datasets
Convert datasets into tensors and expand into stacks of sliding views, as shown above
Move all stacks to GPU to avoid transfer bottleneck during training

Output:
As mentioned in the comments, Tensor.unfold can be used for this task. You provide a tensor, starting index, length value, and step size. This returns a batch of views exactly like I was describing, though you have to unfold tensors one at a time for A and B.
The following code can be used to generate A and B:
A = source_tensor[:-B_length].unfold(0, A_length, 1)
B = source_tensor[A_length:].unfold(0, B_length, 1)

A.storage().data_ptr() == source_tensor.storage().data_ptr() returns True

Since the data pointers are the same it is correctly returning views of the original tensor instead of copies, which saves on memory.