Instruction: creating a dictionary of large file names in test dataloader and assigning the prediction of all 512x512 patches in it as a list for its values

Input:
I am not sure why making a dictionary as the following is not creating the desired output. Instead of ending up with a dictionary with 887large filenames, I ended up with a dictionary with only 2 large filenames.
Quick intro to my test set. I have large images and I have tiled them into 512x512 patches. Below you can see number of large images and 512x512 patches for each positive and negative label:
--test
---pos_label 14, 11051
---neg_label 74, 45230

sample_fnames_labels = dataloaders_dict['test'].dataset.samples

test_large_images = {}
test_loss = 0.0
test_acc = 0

with torch.no_grad():

    test_running_loss = 0.0
    test_running_corrects = 0
    print(len(dataloaders_dict['test']))
    for i, (inputs, labels) in enumerate(dataloaders_dict['test']):

        patch_name = sample_fname.split('/')[-1]
        large_image_name = patch_name.split('_')[0]

        test_inputs = inputs.to(device)
        test_labels = labels.to(device)

        test_outputs = saved_model_ft(test_inputs)

        _, test_preds = torch.max(test_outputs, 1)

        max_bs = len(test_preds)

        for j in range(max_bs):

            sample_file_name = sample_fnames_labels[i+j][0]
            patch_name = sample_file_name.split('/')[-1]
            large_image_name = patch_name.split('_')[0]

            if large_image_name not in test_large_images.keys():

                test_large_images[large_image_name] = list()
                test_large_images[large_image_name].append(test_preds[j].item())

            else:
                test_large_images[large_image_name].append(test_preds[j].item())

        #test_running_loss += test_loss.item() * test_inputs.size(0)
        test_running_corrects += torch.sum(test_preds == test_labels.data)

    #test_loss = test_running_loss / len(dataloaders_dict['test'].dataset)
    test_acc = test_running_corrects / len(dataloaders_dict['test'].dataset)

here test_large_images dictionary only has two large images as the key instead of 88 test large images. Thanks for having a look.
Essentially I want to collect all the labels of 512x512 patches of each large image as a list into a dictionary with the large_image_filename as the key. So, I could do a majority voting later on.
Here's the used dataloader from PyTorch and batch size is 512.
# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['train', 'val', 'test']}
# Create training and validation dataloaders
print('batch size: ', batch_size)
dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['train', 'val', 'test']}

Ultimately, I am hoping to get something like:
{large_image_1: [0, 1, 1, 0], large_image_2: [1, 1, 1, 0, 0 , 0 , 0, 0, 0], large_image_3: [0, 0], ...}
Please note that my large images are of different sizes in terms of number of 512x512 patches.
I do actually see 87 unique large image filenames below. Not sure why in the dictionary only two of them gets updated:
fnames = set()
for i in range(len(sample_fnames_labels)):
    fname = sample_fnames_labels[i][0].split('/')[-1][:23]
    fnames.add(fname)

print(len(fnames))

87

Output:
fixed the problem by setting the batch size to 1 in dataloader of test
# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['test']}
# Create training and validation dataloaders

dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=1, shuffle=True, num_workers=4) for x in ['test']}

test_large_images = {}
test_loss = 0.0
test_acc = 0

with torch.no_grad():

    test_running_loss = 0.0
    test_running_corrects = 0
    print(len(dataloaders_dict['test']))
    for i, (inputs, labels) in enumerate(dataloaders_dict['test']):

        print(i)
        test_input = inputs.to(device)
        test_label = labels.to(device)
        test_output = saved_model_ft(test_input)
        _, test_pred = torch.max(test_output, 1)
        sample_fname, label = dataloaders_dict['test'].dataset.samples[i]
        patch_name = sample_fname.split('/')[-1]
        large_image_name = patch_name.split('_')[0]
        if large_image_name not in test_large_images.keys():
            test_large_images[large_image_name] = list()
            test_large_images[large_image_name].append(test_pred.item())
        else:
            test_large_images[large_image_name].append(test_pred.item())

        #print('test_large_images.keys(): ', test_large_images.keys())
        test_running_corrects += torch.sum(test_preds == test_labels.data)

    test_acc = test_running_corrects / len(dataloaders_dict['test'].dataset)

print(test_acc)


Instruction: The size of tensor a (3) must match the size of tensor b (32) at non-singleton dimension 1

Input:
I was training a deep learning model but i am encountering the error like The size of tensor a (3) must match the size of tensor b (32) at non-singleton dimension 1.And also while training the data the accuracy is above 1 that means i am getting the accuracy like 1.04,1.06 like that.
The Below is the Training Code
def train(model,criterion,optimizer,iters):
    epoch = iters
    train_loss = []
    validaion_loss = []
    train_acc = []
    validation_acc = []
    states = ['Train','Valid']
    for epoch in range(epochs):
        print("epoch : {}/{}".format(epoch+1,epochs))
        for phase in states:
            if phase == 'Train':
                model.train()
                dataload = train_data_loader
            else:
                model.eval()
                dataload = valid_data_loader

            run_loss,run_acc = 0,0
            for data in dataload:
                inputs,labels = data
                #print("Inputs:",inputs.shape)
                #print("Labels:",labels.shape)
                inputs = inputs.to(device)
                labels = labels.to(device)
                labels = labels.byte()
                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'Train'):
                    outputs = model(inputs)
                    print("Outputs",outputs.shape)
                    loss = criterion(outputs,labels)

                    predict = outputs>=0.5
                    #print("Predict",predict.shape)
                    if phase == 'Train':
                        loss.backward()
                        optimizer.step()

                    acc = torch.sum(predict == labels.data)

                run_loss+=loss.item()
                #print("Running_Loss",run_loss)
                run_acc+=acc.item()/len(labels)
                #print("Running_Acc",run_acc)
            if phase == 'Train':
                epoch_loss = run_loss/len(train_data_loader)
                train_loss.append(epoch_loss)
                epoch_acc = run_acc/len(train_data_loader)
                train_acc.append(epoch_acc)
            else:
                epoch_loss = run_loss/len(valid_data_loader)
                validaion_loss.append(epoch_loss)
                epoch_acc = run_acc/len(valid_data_loader)
                validation_acc.append(epoch_acc)

            print("{}, loss :{},accuracy:{}".format(phase,epoch_loss,epoch_acc))

    history = {'Train_loss':train_loss,'Train_accuracy':train_acc,
               'Validation_loss':validaion_loss,'Validation_Accuracy':validation_acc}
    return model,history

Below is the code of the base model
model = models.resnet34(pretrained = True)

for param in model.parameters():
  param.requires_grad = False

model.fc = nn.Sequential(nn.Linear(model.fc.in_features,out_features = 1024),nn.ReLU(),
                         nn.Linear(in_features = 1024,out_features = 512),nn.ReLU(),
                         nn.Dropout(0.3),
                         nn.Linear(in_features=512,out_features=256),nn.ReLU(),
                         nn.Linear(in_features = 256,out_features = 3),nn.LogSoftmax(dim = 1))

device = torch.device("cuda" if cuda.is_available() else "cpu")
print(device)
model.to(device)

optimizer = optim.Adam(model.parameters(),lr = 0.00001)
criterion = nn.CrossEntropyLoss()

I tried with predict == labels.unsqueeze(1) it didn't raise any error but the accuracy is going over 1. May I know where i had to change the code.

Output:
Your output tensor of the size [32,3] 32 is the number of mini-batches and 3 is the output of your neural network e.g.
[[0.25, 0.45, 0.3],
 [0.45, 0.15, 0.4],
      ....
      ....
 [0.2, 0.15, 0.65]]

When you compare whether output >= 0.5 the result is predict tensor but it is bool tensor with the same size of output [32,3] like that:
[[False, False, False],
 [False, False, False],
      ....
      ....
 [False, False, True]]

and Labels is 0D tensor with 32 values e.g.
[0,2,...,0]

The cause of the problem is here: to compare between predicts and labels, you should select the index of the maximum probability from each row in predicts tensor like that:
predicts = predicts.argmax(1)
# output
[0,0,...,2]

But predicts is bool tensor, and you cannot apply argmax to bool tensor directly. Therefore, you got the error message as you indicated in the comment. To solve this problem, you need only to do the following:
 predicts = (output >= 0.5)*1

Now you can compare between the two tensors predicts, labels, because both have the same size.
In brief, you should use:
predicts = (output >= 0.5)*1
acc = torch.sum(predicts.argmax(1) == Labels)

Your problem is solved, but the accuracy is logically not correct.
Therefore, be careful if you want to use sigmoid with multi-classification problem because you use output >= 0.5 however, you have 3 classes in the output. This is not correct because imagine you have in the output [0.15,0.45,0.4]. Your predict will be [0, 0, 0] and then argmax(1) will select the first index if there are equal numbers, however, the second index should be selected in this case because it has the largest probability. The best way to do that if you have a multi-classification problem is to use softmax instead of sigmoid (>= 0.5).
By the way, if you come back to your model structure (last line), you will find that you already used nn.LogSoftmax. You need just to remove this line predicts = (outputs >= 0.5) and use directly:
#before the for loop
num_corrects  = 0

# inside the for loop
num_corrects = num_corrects  + torch.sum(outputs.argmax(1) == Labels)

#outside the loop
train_accuracy = (100. * num_corrects / len(train_loader.dataset))


Instruction: How to fed last 4 concatenated hidden layers of BERT to FC layers

Input:
I am trying to do classification task and I got last 4 layers from BERT and concatenate them.
out = model(...)
out=torch.cat([out['hidden_states'][-i] for i in range(1,5)],dim=-1)

Now the shape is (12,200,768*4) which is batch,max_length,concatenation layer but for fully connected layer we need to have two dimension. So one way is to average like torch.mean((12,200,768*4),dim=1) and get the output as (12,768*4).
But i am confused what is the original BERT approach

Output:
There is no "original" BERT approach for classification with concatenated hidden layers. You have several options to proceed and I will just describe a comment on your approach and suggest an alternative in the following.
Preliminary:
import torch.nn as nn
from transformers import BertTokenizerFast, BertModel

t = BertTokenizerFast.from_pretrained("bert-base-cased")
m = BertModel.from_pretrained("bert-base-cased")
fc = nn.Linear(768, 5)

s = ["This is a random sentence", "This is another random sentence with more words"]

i = t(s, padding=True,return_tensors="pt")

with torch.no_grad():
  o = m(**i, output_hidden_states=True)

print(i)

At first, you should look at your input:
#print(I)
{'input_ids':
tensor([[ 101, 1188, 1110,  170, 7091, 5650,  102,    0,    0,    0],
        [ 101, 1188, 1110, 1330, 7091, 5650, 1114, 1167, 1734,  102]]),
'token_type_ids':
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
'attention_mask':
tensor([[1, 1, 1, 1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
}

What you should notice here, is that the shorter sentence gets padded. That is relevant because simply pooling the mean with torch.mean, will result in different sentence embeddings for the same sentence depending on the number of padding tokens. Of course, the model will learn to handle that to some extent after sufficient training, but you should, however, use a more sophisticated mean function that removes the padding tokens right away :
def mean_pooling(model_output, attention_mask):
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(model_output.size()).float()
    return torch.sum(model_output * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

o_mean = [mean_pooling(o.hidden_states[-x],i.attention_mask) for x in range(1,5)]
#we want a tensor and not a list
o_mean = torch.stack(o_mean, dim=1)
#we want only one tensor per sequence
o_mean = torch.mean(o_mean,dim=1)

print(o_mean.shape)
with torch.no_grad():
  print(fc(o_mean))

Output:
torch.Size([2, 768])
tensor([[ 0.0677, -0.0261, -0.3602,  0.4221,  0.2251],
        [-0.0328, -0.0161, -0.5209,  0.5825,  0.2405]])

These operations are pretty expensive and people often use an approach called cls pooling as a cheaper alternative with comparable performance:
#We only use the cls token (i.e. first token of the sequence)
#id 101
o_cls = [o.hidden_states[-x][:, 0] for x in range(1,5)]
#we want a tensor and not a list
o_cls = torch.stack(o_cls, dim=1)
#we want only one tensor per sequence
o_cls = torch.mean(o_cls,dim=1)
print(o_cls.shape)
with torch.no_grad():
  print(fc(o_cls))

Output:
torch.Size([2, 768])
tensor([[-0.3731,  0.0473, -0.4472,  0.3804,  0.4057],
        [-0.3468,  0.0685, -0.5885,  0.4994,  0.4182]])


Instruction: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! on DataLore

Input:
I'm working with IntelliJ DataLore to train a basic VGG16 CNN, but when I try to do it using a GPU machine I get the following error:
Traceback (most recent call last):
  at block 20, line 1
  at /data/workspace_files/train/trainer/training.py, line 115, in train(self, max_epochs)
  at /data/workspace_files/train/trainer/training.py, line 46, in train_epoch(self, train_loader)
  at /data/workspace_files/train/trainer/training.py, line 94, in forward_to_loss(self, step_images, step_labels)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/modules/module.py, line 1102, in _call_impl(self, *input, **kwargs)
  at /data/workspace_files/models/vgg.py, line 49, in forward(self, x)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/modules/module.py, line 1102, in _call_impl(self, *input, **kwargs)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/modules/container.py, line 141, in forward(self, input)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/modules/module.py, line 1102, in _call_impl(self, *input, **kwargs)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/modules/linear.py, line 103, in forward(self, input)
  at /opt/python/envs/default/lib/python3.8/site-packages/torch/nn/functional.py, line 1848, in linear(input, weight, bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

Here is my code so you guys can review it.
use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")
model = model.to(device)

In this fragment of code I use self.device because I pass the device as parameter to the class Train
for _, (data, target) in tqdm(enumerate(train_loader, 1)):
            self.optimizer.zero_grad()
            step_images, step_labels = data.to(
                self.device), target.to(self.device)
            step_output, loss = self.forward_to_loss(step_images, step_labels)

I haven't had this issue before so I don't know if there something missing on DataLore or my code is wrong.
Hope you can help me!

Output:
can you try this
step_output, loss = self.forward_to_loss(step_images.to(self.device), step_labels.to(self.device))


Instruction: How to swap values in PyTorch tensor without using in-place operation (conserve gradient)

Input:
I have a tensor called state of shape torch.Size([N, 2**n, 2**n]), and I want to apply the following operations:
state[[0,1]] = state[[1,0]]
state[0] = -1*state[0]

Both of these are in-place operations. Are there some out-of-place operations that I can substitute them with? These lines are inside a for-loop, so it would be a bit difficult to just create new variables.

Output:
I managed to figure it out!
Replace:
state[[0,1]] = state[[1,0]] # in-place operation

with:
state = state[[1,0]] # out-of-place operation

And for the second line, we replace:
state[0] = -1*state[0] # in-place operation

with:
# out-of-place operations
temp = torch.ones(state.shape).type(state.type()).to(state.device)
temp[1] = -1*temp[1]
state = state*temp

This seems to be doing the job!


Instruction: Couldn't convert pytorch model to ONNX

Input:
I used this repo : https://github.com/Turoad/lanedet
to convert a pytorch model that use mobilenetv2 as backbone To ONNX but I didn't succeeded.
i got a Runtime error that says:

RuntimeError: Exporting the operator eye to ONNX opset version 12 is
not supported. Please open a bug to request ONNX export support for
the missing operator.

it's really disappointing, looking to the good result that this model gives and the quick performance that it provides,
is there any way that I can fix this bug? because I need to convert it to ONNX and then to TF lite model to use it in Android App
I will provide the pretrained model that I have used and the way that I follow in converting..
Thank you so much for helping!
my colab notebook:
https://colab.research.google.com/drive/18udIh8tNJvti7jKmR4jRaRO-oYDgRmvA?usp=sharing
the pretrained model that I use:
https://drive.google.com/file/d/1o3-BgLIQesurIyDCKGliqbo2inUA5cPw/view?usp=sharing

Output: Use torch>=1.7.0 to convert the model, because operation Eye is added.


Instruction: Turning list of 2D tensors with different length to one 3D tensor

Input:
I have a list of 3 tensors with the shape: (8, 2), (8, 4), (8, 6)
And I want to turn this list into this shape: (8, 3, x)
How do I do this? I know I need to use some combination of torch.cat, torch.stack and torch.transpose, but I can't figure it out.
Thanks in advance!

Output:
As you said, you need to use torch.cat, but also torch.reshape. Assume the following:
a = torch.rand(8,2)
b = torch.rand(8,4)
c = torch.rand(8,6)

And assume that it is indeed possible to reshape the tensors to a (8,3,-1) shape, where -1 stands for as long as it need to be, then:
d = torch.cat((a,b,c), dim=1)
e = torch.reshape(d, (8,3,-1))

I'll explain. Because the 1st dimension if different in a,b,c the concatenation has to be along the 1st dimension, as seen in variable d. Then, you can reshape the tensor as seen in e where the -1 stands for "as long as it needs to be".


Instruction: Pytorch GPU Out of memory on example script

Input:
I tried running the example script from official huggingface transformers repository with installed Python 3.10.2, PyTorch 1.11.0 and CUDA 11.3 for Sber GPT-3 Large.
Without any file modifications I ran this script with arguments:
--output_dir out --model_name_or_path sberbank-ai/rugpt3large_based_on_gpt2 --train_file dataset.txt --do_train --num_train_epochs 15 --overwrite_output_dir

and got
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB

also tried --block_size 32 and --per_device_train_batch_size 1 but unsuccessfully.
My GPU is RTX 2060 6GB. Maybe it's a real lack of video memory? Can it be solved without buying a new GPU?

Output:
The GPT-3 Models have an extremely large number of parameters and are therefore very memory-heavy. Just to get an idea, if I understand Sber AIs documentation right the Large model was pre-trained on 128/16 V100 GPUs (which have 32GB each) for multiple days. Model-finetuning and inference is obviously going to be much easier on memory but even that will require some serious hardware, at least for the larger models.
You can try to use the Medium and Small model and see if that works for you. Also you can always try to run it in a cloud service like Google Colab, they also have a notebook that demonstrates this. Make sure to activate GPU usage in notebook settings of Google Colab. In the free version you get some decent GPU, if you are more serious about this you can get the pro version for better hardware in their cloud. Probably a lot cheaper than buying a GPU more powerful than an RTX 2060 with the current prices. Of course there are many cloud hardware services where you can run a large model training/fine-tuning, not only Google.


Instruction: PyTorch NN not training

Input:
I have a bespoke NN model which works and wanted to move it to the PyTorch framework. However, the network is not training likely due to some misconfiguration. Please advise if you see something that is odd/wrong or could be a contributing reason.
import torch
from torch import nn, optim
import torch.nn.functional as F
X_train_t = torch.tensor(X_train).float()
X_test_t = torch.tensor(X_test).float()
y_train_t = torch.tensor(y_train).long().reshape(y_train_t.shape[0], 1)
y_test_t = torch.tensor(y_test).long().reshape(y_test_t.shape[0], 1)

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(22, 10)
        self.fc2 = nn.Linear(10, 1)

    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        x = F.relu(self.fc1(x))
        x = F.log_softmax(self.fc2(x), dim=1)

        return x

model = Classifier()
criterion = nn.BCELoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 2000
steps = 0

train_losses, test_losses = [], []
for e in range(epochs):
    # training loss
    optimizer.zero_grad()

    log_ps = model(X_train_t)
    loss = criterion(log_ps, y_train_t.type(torch.float32))
    loss.backward()
    optimizer.step()
    train_loss = loss.item()

    # test loss
    # Turn off gradients for validation, saves memory and computations
    with torch.no_grad():
        log_ps = model(X_test_t)
        test_loss = criterion(log_ps, y_test_t.to(torch.float32))
        ps = torch.exp(log_ps)

    train_losses.append(train_loss/len(X_train_t))
    test_losses.append(test_loss/len(X_test_t))

    if (e % 100 == 0):
        print("Epoch: {}/{}.. ".format(e, epochs),
          "Training Loss: {:.3f}.. ".format(train_loss/len(X_train_t)),
          "Test Loss: {:.3f}.. ".format(test_loss/len(X_test_t)))

Training is not happening:
Epoch: 0/2000..  Training Loss: 0.014..  Test Loss: 0.082..
Epoch: 100/2000..  Training Loss: 0.014..  Test Loss: 0.082..
...

Output:
The source of your problem is the fact that you apply the softmax operation on the output of self.fc2. The output of self.fc2 has a size of 1 and therfore the output of the softmax will be 1 regardless of the input. Read more on the softmax activation function in the pytorch package here. I suspect that you wanted to use the Sigmoid function to transform the output of the last linear layer to to interval [0,1] and then apply a log function of some sorts.
Because the softmax results in an output of 1 regardless of the input, the model did not train well. I do not have access to your data so i can not simulate it exactly but from the information I have, replacing the softmax activation with the sigmoid should solve this.
A better and more numerically stable approach will be to use the BCEWITHLOGITSLOSS instead of the criterion in criterion = nn.BCELoss() and remove the activation function at the end, since this criterion applies the sigmoid along with the BCE loss for a more stable numerical computation.
To summarize, my advice will be to change criterion = nn.BCELoss() to criterion = nn.BCEWithLogitsLoss() and change the forawrd function as follows:
def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        x = F.relu(self.fc1(x))
        x = self.fc2(x)


Instruction: Matplot histogram of Pytorch Tensor

Input:
I have a tensor of size 10, with only 1 values: 0 and 1.
I want to plot an histogram of the tensor above, simply using matplotlib.pyplot.hist. This is my code:
import torch
import matplotlib.pyplot as plt
t = torch.tensor([0., 1., 1., 1., 0., 1., 1., 0., 0., 0.])
print(t)
plt.hist(t, bins=2)
plt.show()

And the output:

Why are there so many values in the histogram? Where did the rest of the values come from? How can I plot a correct histogram for my tensor?

Output:
The plt.hist(t, bins=2) function is not meant to work with tensors. For this to work properly, you can try using t.numpy() or t.tolist() instead. As far as I could educate myself, the way to compute a histogramwith pytorch is through the torch.histc() function and to plot the histogram you use plt.bar() function as follows:
import torch
import matplotlib.pyplot as plt

t = torch.tensor([0., 0., 1., 1., 0., 1., 1., 0., 0., 0.])
hist = torch.histc(t, bins = 2, min = 0, max = 1)

bins = 2
x = range(bins)
plt.bar(x, hist, align='center')
plt.xlabel('Bins')

Some sources for plotting a histogram can be seen here and here . I could not find the root cause for this and if some could educate me it will be great, but as far as I am aware, this is the way to plot a tensor
I changed the tensor to have 4 '1.0' and 6 '0.0' to be able to see the difference


Instruction: Problem with data cast on the GPU in PyTorch

Input:
Im trying to do an image classifier, but im having a problem with the data cast on the GPU.
def train(train_loader, net, epoch):

  # Training mode
  net.train()

  start = time.time()

  epoch_loss  = []
  pred_list, label_list = [], []

  for batch in train_loader:

    #Batch cast on the GPU
    input, label = batch
    input.to(args['device'])
    label.to(args['device'])

    #Forward
    ypred = net(input)
    loss = criterion(ypred, label)
    epoch_loss.append(loss.cpu().data)

    _, pred = torch.max(ypred, axis=1)
    pred_list.append(pred.cpu().numpy())
    label_list.append(label.cpu().numpy())

    #Backward
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

  epoch_loss = np.asarray(epoch_loss)
  pred_list  = np.asarray(pred_list).ravel()
  label_list  = np.asarray(label_list).ravel()

  acc = accuracy_score(pred_list, label_list)

  end = time.time()
  print('#################### Train ####################')
  print('Epoch %d, Loss: %.4f +/- %.4f, Acc: %.2f, Time: %.2f' % (epoch, epoch_loss.mean(),
  epoch_loss.std(), acc*100, end-start))

  return epoch_loss.mean()

for epoch in range(args['epoch_num']):
  train(train_loader, net, epoch)
  break #Testing

Model already is in cuda, but i get error that says
Input type is torch.FloatTensor and not torch.cuda.FloatTensor

Whats the problem with input.to(args['device'])?

Output:
UPDATE: According to the OP, an aditional data.to(device) before the train loop caused this issue.

you are probably getting a string like 0 or cuda from
args['device']; you should do this:
'cpu') #pass your args['device'] ``` so then use `device` to move the
model to GPU:  ``` model.to(device) ```

then call the model with:

``` for batch,(data,label) in enumerate(train_loader):

    #Batch cast on the GPU
    data.to(device =device)
    label.to(device =device)


Instruction: Setting constant learning rates in Pytorch

Input: I am optimizing lstm networks with pytorch using the Adam optimizer. I have the feeling that my learning rate is decaying too fast, but I am not even 100% sure if Adam does that, since I can't find good documentation. If Adam decays the learning rate by default, is there a way to turn this off and set a constant learning rate?

Output:
"I can't find good documentation" - you could read the original paper, for example. Also, the documentation is here: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html.
If by "learning rate" you mean the lr parameter of torch.optim.Adam, then it remains constant - Adam itself doesn' modify it, in contrast to learning-rate schedulers. However, Adam applies extra scaling to the gradient, so the learning rate is applied to this transformation of the gradient, not the gradient itself. This can't be turned off because this is the essence of the algorithm. If you'd like to apply the learning rate directly to the gradient, use stochastic gradient descent.


Instruction: pytorch BCEWithLogitsLoss calculating pos_weight

Input:
I have a neural network as below for binary prediction. My classes are heavily imbalanced and class 1 occurs only 2% of times. Showing last few layers only
self.batch_norm2 = nn.BatchNorm1d(num_filters)

self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)

self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)

self.fc3 = nn.Linear(fc2_neurons, 1)

My loss is as below. Is this a correct way to calculate pos_weight parameter? I looked into official documentation at this link and it shows that pos_weight needs to have one value for each class for multiclass classification. Not sure if for the binary class it is a difference scenario. I tried to input 2 values and I was getting an error
My question: for binary problem, would pos_weight be a single value unlike multiclass classification where it needs to a list/array with length equal to number of classes?
BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_wts[0]/class_wts[1])

My y variable is a single variable that has 0 or 1 to represent the actual class and the neural network outputs a single value
--------------------------------------------------Update 1
based upon the answer by Shai I have below questions:

BCEWithLogitsLoss - if it is a multiclass problem then how to use pos_weigh parameter?
Is there any example of using focal loss in pytorch? I found some links but most of them were old - dating 2 or 3 or more years
For training I am oversampling my class 1. Is focal loss still appropiate?

Output:
The documentation of pos_weight is indeed a bit unclear. For BCEWithLogitsLoss pos_weight should be a torch.tensor of size=1:
BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=torch.tensor([class_wts[0]/class_wts[1]]))

However, in your case, where pos class occurs only 2% of the times, I think setting pos_weight will not be enough.
Please consider using Focal loss:
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár Focal Loss for Dense Object Detection (ICCV 2017).
Apart from describing Focal loss, this paper provides a very good explanation as to why CE loss performs so poorly in the case of imbalance. I strongly recommend reading this paper.
Other alternatives are listed here.


Instruction: How to use PyTorch's nn.MultiheadAttention

Input:
I want to use PyTorch's nn.MultiheadAttention but it doesn't work.
I just want to use the functionality of pytorch for the manual calculated example of attention
I always got an error when trying to run this example.
import torch.nn as nn

embed_dim = 4
num_heads = 1

x = [
  [1, 0, 1, 0], # Input 1
  [0, 2, 0, 2], # Input 2
  [1, 1, 1, 1]  # Input 3
 ]
x = torch.tensor(x, dtype=torch.float32)

w_key = [
  [0, 0, 1],
  [1, 1, 0],
  [0, 1, 0],
  [1, 1, 0]
]
w_query = [
  [1, 0, 1],
  [1, 0, 0],
  [0, 0, 1],
  [0, 1, 1]
]
w_value = [
  [0, 2, 0],
  [0, 3, 0],
  [1, 0, 3],
  [1, 1, 0]
]
w_key = torch.tensor(w_key, dtype=torch.float32)
w_query = torch.tensor(w_query, dtype=torch.float32)
w_value = torch.tensor(w_value, dtype=torch.float32)

keys = x @ w_key
querys = x @ w_query
values = x @ w_value

multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
attn_output, attn_output_weights = multihead_attn(querys, keys, values)

Output:
Try this.
First, your x is a (3x4) matrix. So you need a weight matrix of (4x4) instead.
Seems nn.MultiheadAttention only supports batch mode although the doc said it supports unbatch input. So let's just make your one data point in batch mode via .unsqueeze(0).
embed_dim = 4
num_heads = 1

x = [
  [1, 0, 1, 0], # Seq 1
  [0, 2, 0, 2], # Seq 2
  [1, 1, 1, 1]  # Seq 3
 ]
x = torch.tensor(x, dtype=torch.float32)

w_key = [
  [0, 0, 1, 1],
  [1, 1, 0, 1],
  [0, 1, 0, 1],
  [1, 1, 0, 1]
]
w_query = [
  [1, 0, 1, 1],
  [1, 0, 0, 1],
  [0, 0, 1, 1],
  [0, 1, 1, 1]
]
w_value = [
  [0, 2, 0, 1],
  [0, 3, 0, 1],
  [1, 0, 3, 1],
  [1, 1, 0, 1]
]
w_key = torch.tensor(w_key, dtype=torch.float32)
w_query = torch.tensor(w_query, dtype=torch.float32)
w_value = torch.tensor(w_value, dtype=torch.float32)

keys = (x @ w_key).unsqueeze(0)     # to batch mode
querys = (x @ w_query).unsqueeze(0)
values = (x @ w_value).unsqueeze(0)

multihead_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
attn_output, attn_output_weights = multihead_attn(querys, keys, values)


Instruction: Can't backward pass two losses in Classification Transformer Model

Input:
For my model I'm using a roberta transformer model and the Trainer from the Huggingface transformer library.
I calculate two losses:
lloss is a Cross Entropy Loss and dloss calculates the loss inbetween hierarchy layers.
The total loss is the sum of lloss and dloss. (Based on this)
When calling total_loss.backwards() however, I get the error:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed

Any idea why that happens? Can I force it to only call backwards once? Here is the loss calculation part:
dloss = calculate_dloss(prediction, labels, 3)
lloss = calculate_lloss(predeiction, labels, 3)
total_loss = lloss + dloss
total_loss.backward()

def calculate_lloss(predictions, true_labels, total_level):
    '''Calculates the layer loss.
    '''

    loss_fct = nn.CrossEntropyLoss()

    lloss = 0
    for l in range(total_level):

        lloss += loss_fct(predictions[l], true_labels[l])

    return self.alpha * lloss

def calculate_dloss(predictions, true_labels, total_level):
    '''Calculate the dependence loss.
    '''

    dloss = 0
    for l in range(1, total_level):

        current_lvl_pred = torch.argmax(nn.Softmax(dim=1)(predictions[l]), dim=1)
        prev_lvl_pred = torch.argmax(nn.Softmax(dim=1)(predictions[l-1]), dim=1)

        D_l = self.check_hierarchy(current_lvl_pred, prev_lvl_pred, l)  #just a boolean tensor

        l_prev = torch.where(prev_lvl_pred == true_labels[l-1], torch.FloatTensor([0]).to(self.device), torch.FloatTensor([1]).to(self.device))
        l_curr = torch.where(current_lvl_pred == true_labels[l], torch.FloatTensor([0]).to(self.device), torch.FloatTensor([1]).to(self.device))

        dloss += torch.sum(torch.pow(self.p_loss, D_l*l_prev)*torch.pow(self.p_loss, D_l*l_curr) - 1)

    return self.beta * dloss

Output:
There is nothing wrong with having a loss that is the sum of two individual losses, here is a small proof of principle adapted from the docs:
import torch
import numpy
from sklearn.datasets import make_blobs

class Feedforward(torch.nn.Module):
    def __init__(self, input_size, hidden_size):
        super(Feedforward, self).__init__()
        self.input_size = input_size
        self.hidden_size  = hidden_size
        self.fc1 = torch.nn.Linear(self.input_size, self.hidden_size)
        self.relu = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(self.hidden_size, 1)
        self.sigmoid = torch.nn.Sigmoid()
    def forward(self, x):
        hidden = self.fc1(x)
        relu = self.relu(hidden)
        output = self.fc2(relu)
        output = self.sigmoid(output)
        return output

def blob_label(y, label, loc): # assign labels
    target = numpy.copy(y)
    for l in loc:
        target[y == l] = label
    return target

x_train, y_train = make_blobs(n_samples=40, n_features=2, cluster_std=1.5, shuffle=True)
x_train = torch.FloatTensor(x_train)
y_train = torch.FloatTensor(blob_label(y_train, 0, [0]))
y_train = torch.FloatTensor(blob_label(y_train, 1, [1,2,3]))

x_test, y_test = make_blobs(n_samples=10, n_features=2, cluster_std=1.5, shuffle=True)
x_test = torch.FloatTensor(x_test)
y_test = torch.FloatTensor(blob_label(y_test, 0, [0]))
y_test = torch.FloatTensor(blob_label(y_test, 1, [1,2,3]))

model = Feedforward(2, 10)
criterion = torch.nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

model.eval()
y_pred = model(x_test)
before_train = criterion(y_pred.squeeze(), y_test)
print('Test loss before training' , before_train.item())

model.train()
epoch = 20
for epoch in range(epoch):
    optimizer.zero_grad()    # Forward pass
    y_pred = model(x_train)    # Compute Loss
    lossCE= criterion(y_pred.squeeze(), y_train)
    lossSQD = (y_pred.squeeze()-y_train).pow(2).mean()
    loss=lossCE+lossSQD
    print('Epoch {}: train loss: {}'.format(epoch, loss.item()))    # Backward pass
    loss.backward()
    optimizer.step()

There must be a real second time that you call directly or indirectly backward on some varaible that then traverses through your graph. It is a bit too much to ask for the complete code here, only you can check this or at least reduce it to a minimal example (while doing so, you might already find the issue). Apart from that, I would start checking:

Does it already occur in the first iteration of training? If not: are you reusing any calculation results for the second iteration without a detach?
When you do backward on your losses individually lloss.backward() followed by dloss.backward() (this has the same effect as adding them together first as gradients are accumulated): what happens? This will let you track down for which of the two losses the error occurs.


Instruction: Can I train my pretrained model with a totally different architecture?

Input:
I have trained a pretrained ResNet18 model with my custom dataset on Pytorch and wondered whether I could transfer my model file to train another one with a different architecture, e.g. ResNet50. I know I have to save my model accordingly (explained well on another post here) but this was a question that I have never thought before.
I was planning to use more advanced models like VisionTransformers (ViT) but I couldn't figure out whether I had to start with a pretrained ViT already or I could just take my previous model file and use it as the pretrained model to train a ViT.
Example Scenario: ResNet18 --> ResNet50 --> Inception v3 --> ViT
My best guess it that it's not possible due to number of weights, neurons and layer structures but I would love to hear that if I miss a crucial point here. Thanks!

Output:
Between models that only differ in number of layers (Resnet-18 and Resnet-50), it has been done to initialize some layers of the larger model from the weights of the smaller model's layers. Inversely, you can truncate a larger model by taking a subset of regularly spaced layers and initialize a smaller model. In both cases, you need to retrain everything at the end if you hope to achieve semi-decent performances.
The whole point of using architectures that vastly differ (vision transformers vs CNNs) is to learn different features from the inputs and unlock new levels of semantic understanding. Recent models like BeiT also use new self-supervised training schemes that have nothing to do with the classic ImageNet pretraining. Using trained weights from another model would go against the point.
Having said that,if you want to use a ViT, why not start from the available pretrained weights on HuggingFace and fine-tune it on the data you used to train your ResNet50 ?


Instruction: Pytorch: Weighting in BCEWithLogitsLoss, but with 'weight' instead of 'pos_weight'

Input:
I'm looking how to do class weighting using BCEWithLogitsLoss.
https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html
The example on how to use pos_weight seems clear to me. If there are 3x more negative samples than positive samples, then you can set pos_weight=3
Does the weight parameter do the same thing?
Say that I set it weight=torch.tensor([1, 3]). Is that the same thing as pos_weight=3
Also, is weight normalized? Is weight=torch.tensor([1, 3]) the same as weight=torch.tensor([3, 9]), or are they different in how they affect the magnitude of the loss?

Output: They are different things. pos_weight is size n_classes. weight is size batch_size. In the formula in the page you linked, weight refers to the w_n variable, whereas pos_weight refers to the p_c variable.


Instruction: Image augmentation on deep learning training data

Input:
I have a question about mean and standard deviation in image augmentation.
Are the two parameters recommended to be filled in?
If so, how could I know the number? Do I have to iterate through the data， also each channel of image, before the train to get it?
import albumentations as A
train_transform = A.Compose(
        [
            A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
            A.ColorJitter(brightness=0.3, hue=0.3, p=0.3),
            A.Rotate(limit=5, p=1.0),
            # A.HorizontalFlip(p=0.3),
            # A.VerticalFlip(p=0.2),
            A.Normalize(
                mean=[0.0, 0.0, 0.0],# <-----------this parameter
                std=[1.0, 1.0, 1.0],# <-----------this parameter
                max_pixel_value=255.0,
            ),
            ToTensorV2(),
        ],
    )

Output:
Yes it is strongly recommended to normalize your images in most of the cases, obviously you will face some situations that does not require normalization. The reason is to keep the values in a certain range. The output of the network, even if the network is 'big', is strongly influenced by the input data range. If you keep your input range out of control, your predictions will drastically change from one to another. Thus, the gradient would be out of control too and might make your training unefficient. I invite you to read this and that answers to have more details about the 'why' behind normalization and have a deeper understanding of the behaviours.
It is quite common to normalize images with imagenet mean & standard deviation : mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]. Of course you could also consider, if your dataset is enough realistic, in a production context, to use its own mean and std instead of imagenet's.
Finally keep in mind those values since, once your model will be trained, you will still need to normalize any new image to achieve a good accuracy with your future inferences.


Instruction: How am i incorrectly using SubsetRandomSampler?

Input:
I have a custom defined dataset:
rcvdataset =  rcvLSTMDataSet('foo.csv', 'foolabels.csv')
I also define the following:
batch_size = 50
validation_split = .2
shuffle_rcvdataset = True
random_seed= 42

```
rcvdataset_size = len(rcvdataset)
indices = list(range(rcvdataset_size))
split = int(np.floor(validation_split * rcvdataset_size))
if shuffle_rcvdataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(rcvdataset, batch_size=batch_size,
                                           sampler=train_sampler)
test_loader = torch.utils.data.DataLoader(rcvdataset, batch_size=batch_size,
                                                sampler=test_sampler)
```

using this training call:
```
def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")
```

But when i try to run it, i get:
    Epoch 1
    -------------------------------
    Traceback (most recent call last):
      File "lstmTrainer.py", line 94, in <module>
        train(train_sampler, model, loss_fn, optimizer)
      File "lstmTrainer.py", line 58, in train
        size = len(dataloader.dataset)
    AttributeError: 'SubsetRandomSampler' object has no attribute 'dataset'

If i instead load the dataset indirectly with:
    train(train_loader, model, loss_fn, optimizer)

it tells me:
    TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'pandas.core.series.Series'>

I'm not clear on what the first error is, at all.  Is the second error trying to tell me that somewhere in my dataset, something isn't a tensor?
Thank you.
As requested, here is rcvDataSet.py:
from __future__ import print_function, division
import os
import torch
import pandas as pd
import numpy as np
from torch.utils.data import Dataset, DataLoader
class rcvLSTMDataSet(Dataset):
    """rcv dataset."""
    TIMESTEPS = 10

    def __init__(self, csv_data_file, annotations_file):
        """
        Args:
            csv_data_file (string): Path to the csv file with the training data
            annotations_file (string): Path to the file with the annotations

        """

        self.csv_data_file = csv_data_file
        self.annotations_file = annotations_file
        self.labels = pd.read_csv(annotations_file)
        self.data = pd.read_csv(csv_data_file)

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        """
        pytorch expects whatever data is returned is in the form of a tensor.  Included, it expects the label for the data.
        Together, they make a tuple.
        """

        # convert every ten indexes and label into one observation
        Observation = []
        counter = 0
        start_pos = self.TIMESTEPS *idx
        avg_1 = 0
        avg_2 = 0
        avg_3 = 0
        while counter < self.TIMESTEPS:
            Observation.append(self.data.iloc[idx + counter])
            avg_1 += self.labels.iloc[idx + counter][2]
            avg_2 += self.labels.iloc[idx + counter][1]
            avg_3 += self.labels.iloc[idx + counter][0]
            counter += 1

        avg_1 = avg_1 / self.TIMESTEPS
        avg_2 = avg_2 / self.TIMESTEPS
        avg_3 = avg_3 / self.TIMESTEPS
        current_labels = [avg_1, avg_2, avg_3]
        print(current_labels)
        return Observation, current_labels

def main():
        loader = rcvLSTMDataSet('foo1.csv','foo2.csv')
        j = 0
        while j < len(loader.data % loader.TIMESTEPS):
            print(loader.__getitem__(j))
            j += 1

if "__main__" == __name__:
    main()

Output:
Cause: If you look to the error message, you will find that you call train function like that:
train(train_sampler, model, loss_fn, optimizer)

This is not correct, you should call train() with train_loader  not with train_sampler.
Solution: you should correct it to:
train(train_loader, model, loss_fn, optimizer)

Error message:
 Epoch 1
    -------------------------------
    Traceback (most recent call last):
      File "lstmTrainer.py", line 94, in <module>
        train(train_sampler, model, loss_fn, optimizer)   <------ look here
      File "lstmTrainer.py", line 58, in train
        size = len(dataloader.dataset)
    AttributeError: 'SubsetRandomSampler' object has no attribute 'dataset'

Second Error message:
If you look at your Dataset class rcvLSTMDataSet, you will find observations list append items with pandas.core.series.Series type, and it is not a pythonic scalar numbers because you read all columns in your csv file. You should use .iloc[....].values instead of iloc[....]. By doing this, you will ensure that your list contains char or float or int types  and it can be converted to a tensor smoothly without errors.
Final remark:
You can read here about Dataloader and Samplers, I summarized here some points:

Samplers are used to specify the sequence of indices/keys used in data
loading.
Data loader combines a dataset and a sampler, and provides an iterable
over the given dataset.
PyTorch provides two data primitives: torch.utils.data.DataLoader and
torch.utils.data.Dataset that allow you to use pre-loaded datasets as
well as your own data. Dataset stores the samples and their
corresponding labels, and DataLoader wraps an iterable around the
Dataset to enable easy access to the samples.


Instruction: TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first. Error occured using Pytorch

Input:
I've read questions about the same error, here on StackOverflow, but unfortunately they does not work.
I have a defined function:
def plot_loss(train_loss, validation_loss, title):
    plt.grid(True)
    plt.xlabel("subsequent epochs")
    plt.ylabel('average loss')
    plt.plot(range(1, len(train_loss)+1), train_loss, 'o-', label='training')
    plt.plot(range(1, len(validation_loss)+1), validation_loss, 'o-', label='validation')
    plt.legend()
    plt.title(title)
    plt.show()

and the problem is that in line
    plt.plot(range(1, len(validation_loss)+1), validation_loss, 'o-', label='validation')

this error occurs. Then it flows through gca().plot(...) in pyplot.py and eventually to return self.numpy()in tensor.py
In testing phase I have the following function defined:
with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            # calculate and sum up batch loss
            test_loss += F.nll_loss(output, target, reduction='mean')
            # get the index of class with the max log-probability
            prediction = output.argmax(dim=1)
            # item() returns value of the given tensor
            correct += prediction.eq(target).sum().item()
    test_loss /= len(test_loader)
    return test_loss

I've tried to change the line
  prediction = output.argmax(dim=1)

as it was described in another questions about the same error but unfortunately it did not help.
I've tried to run this code on Google Colab and also on my local machine with GPU (cuda is available) but unfortunately the same error occurs.
EDIT I've manage to find the solution in this link. It seems that it is connected with moving data between CUDA and CPU. I've invoked .cpu() and it's solved:
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            # calculate and sum up batch loss
            test_loss += F.nll_loss(output, target, reduction='mean')
            # get the index of class with the max log-probability
            prediction = output.argmax(dim=1)
            # item() returns value of the given tensor
            correct += prediction.eq(target).sum().item()
    test_loss /= len(test_loader)
    return test_loss.cpu()

Output:
I've manage to find the solution in this link. It seems that it is connected with moving data between CUDA and CPU. I've invoked .cpu() and it's solved:
 with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            # calculate and sum up batch loss
            test_loss += F.nll_loss(output, target, reduction='mean')
            # get the index of class with the max log-probability
            prediction = output.argmax(dim=1)
            # item() returns value of the given tensor
            correct += prediction.eq(target).sum().item()
    test_loss /= len(test_loader)
    return test_loss.cpu()


Instruction: Build confusion matrix for instance segmantation (mask r-cnn from detectron2)

Input:
I've trained a mask r-cnn on corn images (I cannot show examples because they are confidential), but they are basically pictures of corn kernels scattered over a flat surface.
There are different kinds of corn kernels I want to be able to segment and classify. I understand the AP metrics are the best way of measuring the performance of an instance segmentation algorithm and I know a confusion matrix for this kind of algorithm doesn't usually make sense.
But for his specific case, where I have 4 classes of very similar objects, I would like to be able to set a fixed AP value, like AP50/AP75 and build a confusion matrix for that.
Would it be possible? How would I do it?
I used detectron2 library to train and get predictions. Here is the code I use to load my trained model from disk, generate predictions in the validation set, and visualize the results:
import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

import numpy as np
import matplotlib.pyplot as plt
import os, json, cv2, random, gc

from detectron2 import model_zoo
from detectron2.data.datasets import register_coco_instances
from detectron2.checkpoint import DetectionCheckpointer, Checkpointer
from detectron2.data import MetadataCatalog, DatasetCatalog, build_detection_test_loader
from detectron2.engine import DefaultTrainer, DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer, ColorMode
from detectron2.modeling import build_model
from detectron2.evaluation import COCOEvaluator, inference_on_dataset

train_annotations_path = "./data/cvat-corn-train-coco-1.0/annotations/instances_default.json"
train_images_path = "./data/cvat-corn-train-coco-1.0/images"
validation_annotations_path = "./data/cvat-corn-validation-coco-1.0/annotations/instances_default.json"
validation_images_path = "./data/cvat-corn-validation-coco-1.0/images"

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train-corn",)
cfg.DATASETS.TEST = ("validation-corn",)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 10000
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4
cfg.OUTPUT_DIR = "./output"
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7

register_coco_instances(
    "train-corn",
    {},
    train_annotations_path,
    train_images_path
)
register_coco_instances(
    "validation-corn",
    {},
    validation_annotations_path,
    validation_images_path
)
metadata_train = MetadataCatalog.get("train-corn")
dataset_dicts = DatasetCatalog.get("train-corn")

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7
predictor = DefaultPredictor(cfg)

predicted_images_path = os.path.abspath("./predicted/")

dataset_dicts_validation = DatasetCatalog.get("validation-corn")
for d in dataset_dicts_validation:
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1],
                   metadata=metadata_train,
                   scale=0.5,
                   instance_mode=ColorMode.IMAGE_BW
    )
    out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    fig = plt.figure(frameon=False, dpi=1)
    fig.set_size_inches(1024,1024)
    ax = plt.Axes(fig, [0., 0., 1., 1.])
    ax.set_axis_off()
    fig.add_axes(ax)
    ax.imshow(cv2.cvtColor(out.get_image()[:, :, ::-1], cv2.COLOR_BGR2RGB), aspect='auto')
    fig.savefig(f"{predicted_images_path}/{d['file_name'].split('/')[-1]}")

That is what my output for a given image looks like:

It is a dictionary with an Instances object as its only value, the Instances object has four lists: pred_boxes, scores, pred_classes and pred_masks. And can be visualized using the detectron2 visualizer, but I can't show the visualization for confidentiality reasons.
Those are the metrics I have for the model right now:

And for each class:

And I noticed visually that some of the kernels are being confused for other classes, specially between classes ardido and fermentado, that is why I want to somehow be able to build a confusion matrix.
I expect the confusion matrix would look something like this:

EDIT:
I found this repository:
https://github.com/kaanakan/object_detection_confusion_matrix
And tried to use it:
from confusion_matrix import ConfusionMatrix
cm = ConfusionMatrix(4, CONF_THRESHOLD=0.3, IOU_THRESHOLD=0.3)

for d in dataset_dicts_validation:
    img = cv2.imread(d["file_name"])
    outputs = predictor(img)
    labels = list()
    detections = list()
    for ann in d["annotations"]:
        labels.append([ann["category_id"]] + ann["bbox"])
    for coord, conf, cls in zip(
        outputs["instances"].get("pred_boxes").tensor.cpu().numpy(),
        outputs["instances"].get("scores").cpu().numpy(),
        outputs["instances"].get("pred_classes").cpu().numpy()
    ):
        detections.append(list(coord) + [conf] + [cls])
    cm.process_batch(np.array(detections), np.array(labels))

But the matrix I got is clearly wrong, and I'm having a hard time trying to fix it.

Output:
I was able to do it, I built the confusion matrix function from scratch:
import pandas as pd
import torch
from detectron2.structures import Boxes, pairwise_iou

def coco_bbox_to_coordinates(bbox):
    out = bbox.copy().astype(float)
    out[:, 2] = bbox[:, 0] + bbox[:, 2]
    out[:, 3] = bbox[:, 1] + bbox[:, 3]
    return out

def conf_matrix_calc(labels, detections, n_classes, conf_thresh, iou_thresh):
    confusion_matrix = np.zeros([n_classes + 1, n_classes + 1])
    l_classes = np.array(labels)[:, 0].astype(int)
    l_bboxs = coco_bbox_to_coordinates((np.array(labels)[:, 1:]))
    d_confs = np.array(detections)[:, 4]
    d_bboxs = (np.array(detections)[:, :4])
    d_classes = np.array(detections)[:, -1].astype(int)
    detections = detections[np.where(d_confs > conf_thresh)]
    labels_detected = np.zeros(len(labels))
    detections_matched = np.zeros(len(detections))
    for l_idx, (l_class, l_bbox) in enumerate(zip(l_classes, l_bboxs)):
        for d_idx, (d_bbox, d_class) in enumerate(zip(d_bboxs, d_classes)):
            iou = pairwise_iou(Boxes(torch.from_numpy(np.array([l_bbox]))), Boxes(torch.from_numpy(np.array([d_bbox]))))
            if iou >= iou_thresh:
                confusion_matrix[l_class, d_class] += 1
                labels_detected[l_idx] = 1
                detections_matched[d_idx] = 1
    for i in np.where(labels_detected == 0)[0]:
        confusion_matrix[l_classes[i], -1] += 1
    for i in np.where(detections_matched == 0)[0]:
        confusion_matrix[-1, d_classes[i]] += 1
    return confusion_matrix

n_classes = 4
confusion_matrix = np.zeros([n_classes + 1, n_classes + 1])
for d in dataset_dicts_validation:
    img = cv2.imread(d["file_name"])
    outputs = predictor(img)
    labels = list()
    detections = list()
    for coord, conf, cls, ann in zip(
        outputs["instances"].get("pred_boxes").tensor.cpu().numpy(),
        outputs["instances"].get("scores").cpu().numpy(),
        outputs["instances"].get("pred_classes").cpu().numpy(),
        d["annotations"]
    ):
        labels.append([ann["category_id"]] + ann["bbox"])
        detections.append(list(coord) + [conf] + [cls])
    confusion_matrix += conf_matrix_calc(np.array(labels), np.array(detections), n_classes, conf_thresh=0.5, iou_thresh=0.5)
matrix_indexes = metadata_train.get("thing_classes") + ["null"]
pd.DataFrame(confusion_matrix, columns=matrix_indexes, index=matrix_indexes)

I built the conf_matrix_calc, that computes the conf_matrix for each image, than I executed it for every image. It took me a while to make it work, because there was a hidden problem. For some reason the labels are saved in a different format than the detections, instead of [x1, y1, x2, y2], it is saved as [x1, y2, x1-x2, y1-y2], and searching online I didn't find anywhere in detectron's or coco's documentation where that is described, but I found one format that was saved as [(x1+x2)/2, (y1+y2)/2, x1-x2, y1-y2], anyway that wasn't my case, I only found that out because I opened the images and checked the pixel coordinates of the boxes in the labels and the predictions and noticed something was wrong.
Anyway, now it works, that is my result:


Instruction: PyTorch normalization in onnx model

Input:
I am doing image classification in pytorch, in that, I used this transforms
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
and completed the training. After, I converted the .pth model file to .onnx file
Now, in inference, how should I apply this transforms in numpy array, because the onnx handles input in numpy array

Output:
You have a couple options.
Since normalize is pretty trivial to write yourself you could just do
import numpy as np
mean = np.array([0.485, 0.456, 0.406]).reshape(-1,1,1)
std = np.array([0.229, 0.224, 0.225]).reshape(-1,1,1)
x_normalized = (x - mean) / std

which doesn't require the pytorch or torchvision libraries at all.
If you are still using your pytorch dataset you could use the following transform
transforms.Compose([
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
    torch.Tensor.numpy  # or equivalently transforms.Lambda(lambda x: x.numpy())
])

which will just apply the normalization to the tensor then convert it to a numpy array.


Instruction: OpenCV Window Freezing when trying to combine with Neural Network Image Classification

Input:
I am using a NN to detect 4 types of objects (chassis, front-spoiler, hubcap, wheel) in the live feed of my webcam. When one is detected, I want to display an image with information about it (chassis.png, front-spoiler.png, hubcap.png, wheel.png).
When I run my NN and hold one of the items in front of the webcam, the opencv windows freezes and doesnt display anything. What is the reason for that?
def displayImg(path):
    img = cv2.imread(path)
    cv2.namedWindow("window", cv2.WND_PROP_FULLSCREEN)
    cv2.setWindowProperty("window",cv2.WND_PROP_FULLSCREEN,cv2.WINDOW_FULLSCREEN)
    cv2.imshow("window", img)

# ----------------LIVE DETECTIONS ---------------
imagePath = "picture.jpg"
frontSpoilerImageOpen = False
chassisImageOpen = False
hubcapImageOpen = False
wheelImageOpen = False

model = torch.hub.load('ultralytics/yolov5', 'custom', path='yolov5/runs/train/exp5/weights/last.pt', force_reload=True)
cap = cv2.VideoCapture(0)
while cap.isOpened():
    ret, frame = cap.read()
    results = model(frame)

    try:
        detectedItem = results.pandas().xyxy[0].iloc[0, 6]
        if detectedItem == "front-spoiler" and not frontSpoilerImageOpen:

            frontSpoilerImageOpen = False
            chassisImageOpen = False
            hubcapImageOpen = False
            wheelImageOpen = False

            displayImg(os.path.join("imagesToDisplay", "front-spoiler.png"))
            frontSpoilerImageOpen = True

        elif detectedItem == "chassis" and not chassisImageOpen:

            frontSpoilerImageOpen = False
            chassisImageOpen = False
            hubcapImageOpen = False
            wheelImageOpen = False

            displayImg(os.path.join("imagesToDisplay", "chassis.png"))
            chassisImageOpen = True

        elif detectedItem == "hubcap" and not hubcapImageOpen:

            frontSpoilerImageOpen = False
            chassisImageOpen = False
            hubcapImageOpen = False
            wheelImageOpen = False

            displayImg(os.path.join("imagesToDisplay", "hubcap.png"))
            hubcapImageOpen = True

        elif detectedItem == "wheel" and not wheelImageOpen:

            frontSpoilerImageOpen = False
            chassisImageOpen = False
            hubcapImageOpen = False
            wheelImageOpen = False

            displayImg(os.path.join("imagesToDisplay", "wheel.png"))
            wheelImageOpen = True
    except Exception as e:
        print(e)

Output:
Your code contains no waitKey at all.
OpenCV GUI (imshow) requires waitKey to work.
This is described in all OpenCV documentation and tutorials.

one basic tutorial: https://docs.opencv.org/4.x/db/deb/tutorial_display_image.html
documentation for imshow mentions the requirement: https://docs.opencv.org/4.x/d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563

waitKey isn't about delays or breaks. It runs the event loop that all GUI processing requires.
You can use waitKey(1) for the shortest non-zero delay of one millisecond (a little more in practice), or you can use pollKey(), which will not wait even that millisecond.


Instruction: Slice pytorch tensor using coordinates tensor without loop

Input:
I have a tensor T with dimension (d1 x d2 x d3 x ... dk) and a tensor I with dimension (p x q). Here, I contains coordinates of T but q < k, each column of I corresponds to a dimension of T. I have another tensor V of dimension p x di x ...dj where sum([di, ..., dj]) = k - q. (di, .., dj) corresponds to missing dimensions from I. I need to perform T[I] = V
A specific example of such problem using numpy array posted here[1].
The solution[2] uses fancy indexing[3] which relies on numpy.index_exp. In case of pytorch such option is not available. Is there any alternative way to mimic this in pytorch without using loops or casting tensors to numpy array?
Below is a demo:
import torch
t = torch.randn((32, 16, 60, 64)) # tensor

i0 = torch.randint(0, 32, (10, 1)).to(dtype=torch.long) # indexes for dim=0
i2 = torch.randint(0, 60, (10, 1)).to(dtype=torch.long) # indexes for dim=2

i = torch.cat((i0, i2), 1) # indexes
v = torch.randn((10, 16, 64)) # to be assigned

# t[i0, :, i2, :] = v ?? Obviously this does not work

[1] Slice numpy array using list of coordinates
[2] https://stackoverflow.com/a/42538465/6422069
[3] https://numpy.org/doc/stable/reference/generated/numpy.s_.html

Output:
After some discussion in the comments, we arrived at the following solution:
import torch
t = torch.randn((32, 16, 60, 64)) # tensor

# indices
i0 = torch.randint(0, 32, (10,)).to(dtype=torch.long) # indexes for dim=0
i2 = torch.randint(0, 60, (10,)).to(dtype=torch.long) # indexes for dim=2

v = torch.randn((10, 16, 64)) # to be assigned

t[(i0, slice(None), i2, slice(None))] = v


Instruction: How to install .whl files using dockerfile

Input:
I am trying to install .whl files using Dockerfile. Here is my dockerfile.
FROM us-docker.pkg.dev/vertex-ai/training/tf-gpu.2-8:latest
USER root
CMD python --version
COPY *.txt /
ADD ./train/ /train/
RUN pip install ./whl_files/*.whl
# RUN pip install whl_files/torchaudio-0.11.0+cu113-cp39-cp39-linux_x86_64.whl
# RUN pip install whl_files/torchvision-0.12.0+cu113-cp39-cp39-linux_x86_64.whl
# RUN pip3 install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
# RUN pip3 install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip3 install --ignore-installed -r requirements.txt --trusted-host pypi.org --trusted-host files.pythonhosted.org
COPY *.py /root/
RUN chmod +x /root/docker_fetch.py
RUN python /root/docker_fetch.py

ENTRYPOINT ["/bin/bash"]

I am getting this error:
 => ERROR [4/8] RUN pip install ./whl_files/*.whl                                                                                0.9s
------
 > [4/8] RUN pip install ./whl_files/*.whl:
#8 0.711 WARNING: Requirement './whl_files/*.whl' looks like a filename, but the file does not exist
#8 0.711 ERROR: *.whl is not a valid wheel filename.
#8 0.856 Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)'))) - skipping
------
executor failed running [/bin/bash -c pip install ./whl_files/*.whl]: exit code: 1

There is no firewall issue.
I tried to install using this also but that also didn't work.
RUN pip3 install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Output:
To expand on my comment:
You haven't copied any files to whl_files in your container, looks like. You only ADD and COPY some txt files and the train/ folder.
The whl files need to be there in the container (just like any other file that's to be used) to be installed (unless you use Buildkit's cache volumes instead, but it makes things more complicated).


Instruction: AttributeError: 'T5Config' object has no attribute 'adapters'

Input: How to solve this error? I've created the .pkl object of the T5-base model and tried to execute it but suddenly I got this error message. I wondered a bit, tried to google it but didn't get any reason why I got this error!!!!

Output:
Well, I was thinking right!!!
I did 2 experiments :

install only transformers library :
When I load the model, each layer of the model was without an adapter attribute!!

install only adapter-Transformers library :
When I load the model, each layer of the model was with an adapter attribute!!

Conclusion : install adapter-Transformers


Instruction: Batched input shows 3d, but got 2d, 2d tensor

Input:
I have this training loop
def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = torch.stack(X).to(device), torch.stack(y).to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

and this lstm:
import torch
import torch.nn as nn
import pandas as pd
import numpy as np
class BELT_LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, num_layers):
        super (BELT_LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.input_size = input_size
        self.BELT_LSTM   = nn.LSTM(input_size, hidden_size, num_layers)

    def forward(self, x):
        # receive an input, create a new hidden state, return output?
        # reset the hidden state?
        hidden = (torch.zeros(self.num_layers, self.hidden_size), torch.zeros(self.num_layers, self.hidden_size))
        x, _ = self.BELT_LSTM(x, hidden)

        #since our observation has several sequences, we only want the output after the last sequence of the observation'''
        x = x[:, -1]
        return x

and here's the dataset class:
from __future__ import print_function, division
import os
import torch
import pandas as pd
import numpy as np
import math
from torch.utils.data import Dataset, DataLoader
class rcvLSTMDataSet(Dataset):
    """rcv dataset."""
    TIMESTEPS = 10

    def __init__(self, csv_data_file, annotations_file):
        """
        Args:
            csv_data_file (string): Path to the csv file with the training data
            annotations_file (string): Path to the file with the annotations

        """

        self.csv_data_file = csv_data_file
        self.annotations_file = annotations_file
        self.labels = pd.read_csv(annotations_file)
        self.data = pd.read_csv(csv_data_file)

    def __len__(self):
        return math.floor(len(self.labels) / 10)

    def __getitem__(self, idx):
        """
        pytorch expects whatever data is returned is in the form of a tensor.  Included, it expects the label for the data.
        Together, they make a tuple.
        """

        # convert every ten indexes and label into one observation
        Observation = []
        counter = 0
        start_pos = self.TIMESTEPS *idx
        avg_avg_1 = 0
        avg_avg_2 = 0
        avg_avg_3 = 0
        while counter < self.TIMESTEPS:
            Observation.append(self.data.iloc[idx + counter].values)
            avg_avg_1 += self.labels.iloc[idx + counter][2]
            avg_avg_2 += self.labels.iloc[idx + counter][1]
            avg_avg_3 += self.labels.iloc[idx + counter][0]
            counter += 1
        #average the avg_1, avg_2, avg_3 for TIMESTEPS length
        avg_avg_1 = avg_avg_1 / self.TIMESTEPS
        avg_avg_2 = avg_avg_2 / self.TIMESTEPS
        avg_avg_3 = avg_avg_3 / self.TIMESTEPS
        current_labels = [avg_avg_1, avg_avg_2, avg_avg_3]
        print(current_labels)
        return Observation, current_labels

def main():
        loader = rcvDataSet('foo','foo2.csv')
        j = 0
        while j < len(loader.data % loader.TIMESTEPS):
            print(loader.__getitem__(j))
            j += 1

if "__main__" == __name__:
    main()

When running this, i get:
  File "module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "lstm.py", line 21, in forward
    x, _ = self.BELT_LSTM(x, hidden)
  File "module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "rnn.py", line 747, in forward
    raise RuntimeError(msg)
RuntimeError: For batched 3-D input, hx and cx should also be 3-D but got (2-D, 2-D) tensors

but as far as i can tell, i've followed the nn.LSTM instructions for both setting up the layers, and shaping the data properly.    What am i doing wrong?
For reference, the incoming data is rows from a csv file, 12 columns wide, and i serve 10 rows per observation
Thanks

Output:
Your code:
hidden = (torch.zeros(self.num_layers, self.hidden_size), torch.zeros(self.num_layers, self.hidden_size))
x, _ = self.BELT_LSTM(x, hidden)

Here hx and cx are both 2-D tensors. The correct way should be:
h_0 = torch.randn(self.num_directions * self.num_layers, self.batch_size, self.hidden_size)
c_0 = torch.randn(self.num_directions * self.num_layers, self.batch_size, self.hidden_size)
x, _ = self.BELT_LSTM(x, (h_0, c_0))


Instruction: Is there any indexing method in pytorch?

Input:
I'm studying about pytorch recently.
But this problem is so weird..
x=np.arrage(24)
ft=torch.FloatTensor(x)
print(floatT.view([@1])[@2])

Answer = tensor([[13., 16.], [19., 22.]])
Can there be indexing methods @1 and @2 that satisfy the Answer?

Output:
by viewing ft as a tensor with 6 columns:
ft.view(-1, 6)
Out[]:
tensor([[ 0.,  1.,  2.,  3.,  4.,  5.],
        [ 6.,  7.,  8.,  9., 10., 11.],
        [12., 13., 14., 15., 16., 17.],
        [18., 19., 20., 21., 22., 23.]])

you place the elements (13, 19), and (16, 22) on top of each other.
Now you only need to pick them up from the right rows/columns:
.view(-1, 6)[2:, (1, 4)]
Out[]:
tensor([[13., 16.],
        [19., 22.]])


Instruction: Custom Sampler correct use in Pytorch

Input:
I have a map-stype dataset, which is used for instance segmentation tasks.
The dataset is very imbalanced, in the sense that some images have only 10 objects while others have up to 1200.
How can I limit the number of objects per batch?
A minimal reproducible example is:
import math
import torch
import random
import numpy as np
import pandas as pd
from torch.utils.data import Dataset
from torch.utils.data.sampler import BatchSampler

np.random.seed(0)
random.seed(0)
torch.manual_seed(0)

W = 700
H = 1000

def collate_fn(batch) -> tuple:
    return tuple(zip(*batch))

class SyntheticDataset(Dataset):
    def __init__(self, image_ids):
        self.image_ids = torch.tensor(image_ids, dtype=torch.int64)
        self.num_classes = 9

    def __len__(self):
        return len(self.image_ids)

    def __getitem__(self, idx: int):
        """
            returns single sample
        """
        # print("idx: ", idx)

        # deliberately left dangling
        # id = self.image_ids[idx].item()
        # image_id = self.image_ids[idx]
        image_id = torch.as_tensor(idx)
        image = torch.randint(0, 255, (H, W))

        num_objects = random.randint(10, 1200)
        image = torch.randint(0, 255, (3, H, W))
        masks = torch.randint(0, 255, (num_objects, H, W))

        target = {}
        target["image_id"] = image_id

        areas = torch.randint(100, 20000, (1, num_objects), dtype=torch.int64)
        boxes = torch.randint(100, H * W, (num_objects, 4), dtype=torch.int64)
        labels = torch.randint(1, self.num_classes, (1, num_objects), dtype=torch.int64)
        iscrowd = torch.zeros(len(labels), dtype=torch.int64)

        target["boxes"] = boxes
        target["labels"] = labels
        target["area"] = areas
        target["iscrowd"] = iscrowd
        target["masks"] = masks

        return image, target, image_id

class BalancedObjectsSampler(BatchSampler):
    """Samples either batch_size images or batches num_objs_per_batch objects.

    Args:
        data_source (list): contains tuples of (img_id).
        batch_size (int): batch size.
        num_objs_per_batch (int): number of objects in a batch.
    Return
        yields the batch_ids/image_ids/image_indices

    """

    def __init__(self, data_source, batch_size, num_objs_per_batch, drop_last=False):
        self.data_source = data_source
        self.sampler = data_source
        self.batch_size = batch_size
        self.drop_last = drop_last
        self.num_objs_per_batch = num_objs_per_batch
        self.batch_count = math.ceil(len(self.data_source) / self.batch_size)

    def __iter__(self):

        obj_count = 0
        batch = []
        batches = []
        counter = 0
        for i, (k, s) in enumerate(self.data_source.iteritems()):
            if (
                obj_count <= obj_count + s
                and len(batch) <= self.batch_size - 1
                and obj_count + s <= self.num_objs_per_batch
                and i < len(self.data_source) - 1
            ):
                # because of https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler
                batch.append(i)
                obj_count += s
            else:
                batches.append(batch)
                yield batch
                obj_count = 0
                batch = []
            counter += 1

obj_sums = {}
batch_size = 10
workers = 4
fake_image_ids = np.random.randint(1600000, 1700000, 100)

# assigning any in-range number objects count to each image
for i, k in enumerate(fake_image_ids):
    obj_sums[k] = random.randint(10, 1200)

obj_counts = pd.Series(obj_sums)

train_dataset = SyntheticDataset(image_ids=fake_image_ids)

balanced_sampler = BalancedObjectsSampler(
    data_source=obj_counts,
    batch_size=batch_size,
    num_objs_per_batch=1500,
    drop_last=False,
)

data_loader_sampler = torch.utils.data.DataLoader(
    train_dataset,
    num_workers=workers,
    collate_fn=collate_fn,
    sampler=balanced_sampler,
)

data_loader_iter = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=batch_size,
    shuffle=False,
    num_workers=workers,
    collate_fn=collate_fn,
)

Iterating over the balanced_sampler
for i, bal_batch in enumerate(balanced_sampler):
    print(f"batch_{i}: ", bal_batch)

yields
batch_0:  [0]
batch_1:  [2, 3]
batch_2:  [5]
batch_3:  [7]
batch_4:  [9, 10]
batch_5:  [12, 13, 14, 15]
batch_6:  [17, 18]
batch_7:  [20, 21, 22]
batch_8:  [24, 25]
batch_9:  [27]
batch_10:  [29]
batch_11:  [31]
batch_12:  [33]
batch_13:  [35, 36, 37]
batch_14:  [39, 40]
batch_15:  [42, 43]
batch_16:  [45, 46]
batch_17:  [48, 49, 50]
batch_18:  [52, 53, 54]
batch_19:  [56]
batch_20:  [58, 59]
batch_21:  [61, 62]
batch_22:  [64]
batch_23:  [66]
batch_24:  [68]
batch_25:  [70, 71]
batch_26:  [73]
batch_27:  [75, 76, 77]
batch_28:  [79, 80]
batch_29:  [82, 83, 84, 85, 86, 87]
batch_30:  [89]
batch_31:  [91]
batch_32:  [93, 94]
batch_33:  [96]
batch_34:  [98]

The above displayed values are the images' indices, but could also be the batch index or even the images' ids.
By running
for i, batch in enumerate(data_loader_sampler):
    print("__sample__: ", i, len(batch[0]))

One sees that the batch contains a single sample instead of the expected amount.
__sample__:  0 1
__sample__:  1 1
__sample__:  2 1
__sample__:  3 1
__sample__:  4 1
__sample__:  5 1
__sample__:  6 1
__sample__:  7 1
__sample__:  8 1
__sample__:  9 1
__sample__:  10 1
__sample__:  11 1
__sample__:  12 1
__sample__:  13 1
__sample__:  14 1
__sample__:  15 1
__sample__:  16 1
__sample__:  17 1
__sample__:  18 1
__sample__:  19 1
__sample__:  20 1
__sample__:  21 1
__sample__:  22 1
__sample__:  23 1
__sample__:  24 1
__sample__:  25 1
__sample__:  26 1
__sample__:  27 1
__sample__:  28 1
__sample__:  29 1
__sample__:  30 1
__sample__:  31 1
__sample__:  32 1
__sample__:  33 1
__sample__:  34 1

What I am really trying to prevent is the following behavior that arises from
for i, batch in enumerate(data_loader_iter):
    print("__iter__: ", i, sum([k["masks"].shape[0] for k in batch[1]]))

which is
__iter__:  0 2510
__iter__:  1 2060
__iter__:  2 2203
__iter__:  3 2815
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/blip/venv/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 328, in reduce_storage
    fd, size = storage._share_fd_()
RuntimeError: falseINTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":300, please report a bug to PyTorch. unable to write to file </torch_431207_56>
Traceback (most recent call last):
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 107, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 431257) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "so.py", line 170, in <module>
    for i, batch in enumerate(data_loader_iter):
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
    success, data = self._try_get_data()
  File "/blip/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 431257) exited unexpectedly

which invariably happens when the number of objects per batch is greater than ~2500.
An immediate workaround would be to set the batch_size low, I just need a more optimal solution.

Output:
If what you are trying to solve really is:
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).

You could try resizing the allocated shared memory with
# mount -o remount,size=<whatever_is_enough>G /dev/shm

However, as this is not always possible, one fix to your problem would be
class SyntheticDataset(Dataset):

    def __init__(self, image_ids):
        self.image_ids = torch.tensor(image_ids, dtype=torch.int64)
        self.num_classes = 9

    def __len__(self):
        return len(self.image_ids)

    def __getitem__(self, indices):
        worker_info = torch.utils.data.get_worker_info()

        batch = []
        for i in indices:
            sample = self.get_sample(i)
            batch.append(sample)
        gc.collect()
        return batch

    def get_sample(self, idx: int):

        image_id = torch.as_tensor(idx)
        image = torch.randint(0, 255, (H, W))

        num_objects = idx
        image = torch.randint(0, 255, (3, H, W))
        masks = torch.randint(0, 255, (num_objects, H, W))

        target = {}
        target["image_id"] = image_id

        areas = torch.randint(100, 20000, (1, num_objects), dtype=torch.int64)
        boxes = torch.randint(100, H * W, (num_objects, 4), dtype=torch.int64)
        labels = torch.randint(1, self.num_classes, (1, num_objects), dtype=torch.int64)
        iscrowd = torch.zeros(len(labels), dtype=torch.int64)

        target["boxes"] = boxes
        target["labels"] = labels
        target["area"] = areas
        target["iscrowd"] = iscrowd
        target["masks"] = masks

        return image, target, image_id

and
class BalancedObjectsSampler(BatchSampler):
    """Samples either batch_size images or batches num_objs_per_batch objects.

    Args:
        data_source (list): contains tuples of (img_id).
        batch_size (int): batch size.
        num_objs_per_batch (int): number of objects in a batch.
    Return
        yields the batch_ids/image_ids/image_indices

    """

    def __init__(self, data_source, batch_size, num_objs_per_batch, drop_last=False):
        self.data_source = data_source
        self.sampler = data_source
        self.batch_size = batch_size
        self.drop_last = drop_last
        self.num_objs_per_batch = num_objs_per_batch
        self.batch_count = math.ceil(len(self.data_source) / self.batch_size)

        obj_count = 0
        batch = []
        batches = []
        batches_sums = []
        for i, (k, s) in enumerate(self.data_source.iteritems()):

            if (
                len(batch) < self.batch_size
                and obj_count + s < self.num_objs_per_batch
                and i < len(self.data_source) - 1
            ):
                batch.append(s)
                obj_count += s
            else:
                batches.append(len(batch))
                batches_sums.append(obj_count)
                obj_count = 0
                batch = []

        self.batches = batches
        self.batch_count = len(batches)

    def __iter__(self):
        batch = []
        img_counts_id = 0
        for idx, (k, s) in enumerate(self.data_source.iteritems()):
            if len(batch) < self.batches[img_counts_id] and idx < len(self.data_source):
                batch.append(s)
            elif len(batch) == self.batches[img_counts_id]:
                gc.collect()
                yield batch
                batch = []
                if img_counts_id < self.batch_count - 1:
                    img_counts_id += 1
                else:
                    break

        if len(batch) > 0 and not self.drop_last:
            yield batch

    def __len__(self) -> int:
        if self.drop_last:
            return len(self.data_source) // self.batch_size
        else:
            return (len(self.data_source) + self.batch_size - 1) // self.batch_size

As SyntheticDataset's __getitem__ was receiving a list of indices, the simplest solution would just iterate over the indices and retrieve a list of samples. You may just have to collate the output differently in order to feed it to your model.
For the BalancedObjectsSampler, I calculated the size of each batch within the __init__ and used it in __iter__ to assemble the batches.
NOTE: This will still fail if your num_workers > 0 for you are trying to pack at most 1500 objects into a batch - and usually one worker loads one batch at a time. Hence, you have to re-assess your num_objs_per_batch when considering using multiprocessing.


Instruction: How to seperate code into train, val and test functions for pytorch cnn?

Input:
I am training a cnn using pytorch and have created a training loop. As I am performing optimisation and experimenting with hyper-parameter tuning, I want to separate my training, validation and testing into different functions. I need to be able to record my accuracy and loss for each function in order to plot graphs. For this I want to create a function which returns the accuracy.
I am pretty new to coding and was wondering the best way to go about this. I feel like my code is a bit messy at the moment. I need to be able to feed in various hyper-parameters for experimentation in my training function. Could anyone offer any advice? Below is what I can so far:
def train_model(model, optimizer, data_loader,  num_epochs, criterion=criterion):
  total_epochs = notebook.tqdm(range(num_epochs))

  for epoch in total_epochs:
    model.train()

    train_correct = 0.0
    train_running_loss=0.0
    train_total=0.0

    for i, (img, label) in enumerate(data_loader['train']):
      #uploading images and labels to GPU
      img = img.to(device)
      label = label.to(device)

      #training model
      outputs = model(img)

      #computing losss
      loss = criterion(outputs, label)

      #propagating the loss backwards
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      train_running_loss += loss.item()

      _, predicted = outputs.max(1)
      train_total += label.size(0)
      train_correct += predicted.eq(label).sum().item()

    train_loss=train_running_loss/len(data_loader['train'])
    train_accu=100.*correct/total

    print('Train Loss: %.3f | Train Accuracy: %.3f'%(train_loss,train_accu))

I have also experimented with making a functions to record accuracy:
def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim = 1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

Output:
First, note that:

Unless you have some specific motivation, validation (and testing) should be performed on a different dataset than the training set, so you should use a different DataLoader. The computation time will increase because of an additional for loop at every epoch.
Always call model.eval() before validation/testing.

That said, The signature of the validation function is pretty much similar to that of train_model
# criterion is passed if you want to register the validation loss too
def validate_model(model, eval_loader, criterion):
   ...

Then, in train_model, after each epoch, you can call the function validate_model and store the returned metrics in some data structure (list, tensor, etc.) that will be used later for plotting.
At the end of the training, you can then use the same validate_model function for testing.
Instead of coding the accuracy by yourself, you can use Accuracy from TorchMetrics
Finally, if you feel the need to level up, you can use DL training frameworks like PyTorch Lightning or FastAI. Give also a look at some hyperparameter tuning library such as Ray Tune.


Instruction: loss increasing significantly in training loop

Input:
My training loss is increasing quite dramatically. Not sure why this is. I was thinking it could be something to do with how I am calculating the loss, but not sure. I think it might be because i am printing running loss instead of loss per batch. Below is my training loop:
def train_model(model, optimizer, train_loader,  num_epochs, criterion=criterion):

  total_epochs = notebook.tqdm(range(num_epochs))

  model.train()

  running_loss=0
  correct=0
  total=0

  for epoch in total_epochs:
    for i, (x_train, y_train) in enumerate(train_loader):

      x_train = x_train.to(device)
      y_train = y_train.to(device)

      y_pred = model(x_train)
      loss = criterion(y_pred, y_train)

      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      running_loss += loss.item()

      _, predicted = y_pred.max(1)
      train_loss=running_loss/len(train_loader)

      total += y_train.size(0)
      correct += predicted.eq(y_train).sum().item()

    train_loss=running_loss/len(train_loader)
    train_accu=100.*correct/total

    print('Train Loss: %.3f | Train Accuracy: %.3f'%(train_loss,train_accu))

However when i call train_model():
train_md = train_model(cnn_net, optimizer, data_loaders['train'], 10)

It returns this:
Train Loss: 1.472 | Train Accuracy: 47.949
Train Loss: 2.655 | Train Accuracy: 53.324
Train Loss: 3.732 | Train Accuracy: 56.521
Train Loss: 4.750 | Train Accuracy: 58.565
Train Loss: 5.728 | Train Accuracy: 60.130
Train Loss: 6.673 | Train Accuracy: 61.364
Train Loss: 7.590 | Train Accuracy: 62.335
Train Loss: 8.484 | Train Accuracy: 63.190
Train Loss: 9.365 | Train Accuracy: 63.934
Train Loss: 10.225 | Train Accuracy: 64.571

Output:
You keep accumulating the loss onto running_loss.
That's the reason why it's increasing every epoch!