Instruction: KeyError when iterating through pytorch dataloader

Input:
I am trying to build a model with pytorch, and I want to use a customized dataset. So, I have a dataset.py which defines a class, MyData, which is a subclass of torch.utils.data.Dataset. Here's the file.
# dataset.py
import torch
from tqdm import tqdm
import numpy as np
import re
from torch.utils.data import Dataset
from pathlib import Path

class MyDataset(Dataset):
    def __init__(self, path, size=10000):
        if not Path(path).exists():
            raise FileNotFoundError

        self.data = []
        self.load_data(path, size)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        return self.data[index]

    def load_data(self, path, size):
        # Loading data from csv files and some preparation
        # Each sample is in the format of (int_tag1, int_tag2, feature_dictionary),
        # then the sample is appended to self.data
        pass

Then I tried to test this dataset using a DataLoader in the test file dataset_test.py
from torch.utils.data import DataLoader
from dataset import MyDataset

path = 'dataset/sample_train.csv'
size = 1000
dataset = MyDataset(path, size)

dataloader = DataLoader(dataset, batch_size=1000)
for v in dataloader:
    print(v)

I got the following output
730600it [11:08, 1093.11it/s]
1000it [00:00, 20325.47it/s]
Traceback (most recent call last):
  File "dataset_test.py", line 12, in <module>
    for v in dataloader:
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 84, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 84, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/home/usr/.local/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 74, in <listcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
KeyError: '210'

The first two lines might be the output when loading data. (I'm not sure because I didn't write any output. But I am using tqdm to load data, so I assume it's tqdm's output?)
Then, I got this key error. I'm wondering which part should be modified? I think the dataset class is well-written, since there's no error when reading the data from file. Is it because the format of samples is not right, so the dataloader cannot load data from dataset properly? Is there any requirement for the format? I've read other people's code, but I didn't find any info mentioning that there's any requirement of the format of samples in Dataset class.
EDIT: A single sample looks like this
('0', '0', {'210': '9093445', '216': '9154780', '301': '9351665', '205': '4186222', '206': '8316799', '207': '8416205', '508': '9355039', '121': '3438658', '122': '3438762', '101': '31390', '124': '3438769', '125': '3438774', '127': '3438782', '128': '3864885', '129': '3864887', '150_14': '3941161', '127_14': '3812616', '109_14': '449068', '110_14': '569621'})

The first two '0's are labels, and the following dictionary contains features.

Output:
As @Shai mentioned, if they keys in feature_dictionary are not the same in a batch, then you get this error from the default collate_fn of DataLoader. As a solution, you can write a custom collate_fn as follows and it works
class MyDataset(Dataset):
    # ... your code ...

    def collate_fn(self, batch):
        tag1_batch = []
        tag2_batch = []
        feat_dict_batch = []
        for tag1, tag2, feat_dict in batch:
            tag1_batch.append(tag1)
            tag2_batch.append(tag2)
            feat_dict_batch.append(feat_dict)

        return tag1_batch, tag2_batch, feat_dict_batch

path = 'dataset/sample_train.csv'
size = 1000
dataset = MyDataset(path, size)

dataloader = DataLoader(dataset, batch_size=3, collate_fn=dataset.collate_fn)


Instruction: Re-setting learning rate while training in Pytorch

Input:
I am training a model using Learning Rate Scheduler in Pytorch to decrease the value of learning rate. By using learning rate scheduler, I reduced learning rate from 0.0001 to 1e-5, and save all the weights, parameters, learning rate values, etc at a particular checkpoint. Now, I want to resume training the model, but with different value of learning rate, while remaining all other values. How can I do this?
This is the code for saving checkpoint. I used Adam optimizer
checkpoint = {
        'epoch': epoch + 1,
        'val_loss_min': val_loss['total'].avg,
        'state_dict': model.state_dict(),
        'optimizer': optimizer.state_dict(),
        'scheduler': scheduler.state_dict(),
    }

When loading checkpoint, I used this code:
checkpoint = torch.load(args.SAVED_MODEL)
        # Load current epoch from checkpoint
        epochs = checkpoint['epoch']
        # Load state_dict from checkpoint to model
        model.load_state_dict(checkpoint['state_dict'])
        # Load optimizer from checkpoint to optimizer
        optimizer.load_state_dict(checkpoint['optimizer'])
        # Load valid_loss_min from checkpoint to valid_loss_min
        val_loss_min = checkpoint['val_loss_min']
        # Load scheduler from checkpoint to scheduler
        scheduler.load_state_dict(checkpoint['scheduler'])

Output:
You can change the learning rate of your optimizer by accessing its param_groups attribute. Depending on whether you have multiple groups or not, you can do the following (after having loaded the checkpoint onto it):
for g in optimizer.param_groups:
    g['lr'] = new_lr


Instruction: the derivative for 'target' is not implemented

Input:
I added two VAEs to the original model, so I need to add optimizer and loss. However, the following errors are reported. How can I modify them?
Traceback (most recent call last):
  File "train.py", line 320, in <module>
    main()
  File "train.py", line 315, in main
    ImgCla.TrainingData()
  File "train.py", line 201, in TrainingData
    lossv1 = self.loss_function(recon_audio, audio1, mean1, logstd1)
  File "train.py", line 135, in loss_function
    BCE = F.binary_cross_entropy(recon_x, x, reduction='sum')
  File "/home/user1/.conda/envs/tyz/lib/python3.6/site-packages/torch/nn/functional.py", line 2762, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
RuntimeError: the derivative for 'target' is not implemented

The train.py is as follows:
from torch.utils.data import DataLoader
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import dataloader
import pandas
import os
import imp
import model
import math
import time
import matplotlib.pyplot as plt
import seaborn as sn
from tqdm import tqdm
from sklearn.metrics import classification_report,accuracy_score
import training_plot
from sklearn.metrics import confusion_matrix
import torch.nn.functional as F
from model import VAE1,VAE2

config = imp.load_source("config","config/Resnet50.py").config
device_ids = config["device_ids"]
data_train_opt = config['data_train_opt']
device = torch.device("cuda:0" if torch.cuda.is_available() else 'cpu')
print("======================================")
print("Device: {}".format(device_ids))

def fix_bn(m):

 classname = m.__class__.__name__
   if classname.find('BatchNorm') != -1:
       m.eval()

class AverageMeter(object):
    """Computes and stores the average and current value"""
    def __init__(self, name, fmt=':f'):
        self.name = name
        self.fmt = fmt
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

    def __str__(self):
        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'
        return fmtstr.format(**self.__dict__)

class ProgressMeter(object):
    def __init__(self, num_batches, meters, prefix=""):
        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)
        self.meters = meters
        self.prefix = prefix

    def display(self, batch):
        entries = [self.prefix + self.batch_fmtstr.format(batch)]
        entries += [str(meter) for meter in self.meters]
        print('\t'.join(entries))

    def _get_batch_fmtstr(self, num_batches):
        num_digits = len(str(num_batches // 1))
        fmt = '{:' + str(num_digits) + 'd}'
        return '[' + fmt + '/' + fmt.format(num_batches) + ']'

def adjust_learning_rate(optimizer, epoch, args):
    """Decay the learning rate based on schedule"""
    lr = args.lr
    if args.cos:  # cosine lr schedule
        lr *= 0.5 * (1. + math.cos(math.pi * epoch / args.epochs))
    else:  # stepwise lr schedule
        for milestone in args.schedule:
            lr *= 0.1 if epoch >= milestone else 1.
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

def accuracy(output, target, topk=(1,)):
    """Computes the accuracy over the k top predictions for the specified values of k"""
    with torch.no_grad():
        maxk = max(topk)
        batch_size = target.size(0)

        _, pred = output.topk(maxk, 1, True, True)
        pred = pred.t()
        correct = pred.eq(target.view(1, -1).expand_as(pred))

        res = []
        for k in topk:
            correct_k = correct[:k]
            correct_k = torch.sum(correct_k).float()
            res.append(correct_k.mul_(100.0 / batch_size))
        return res

class ImageClassify(object):
    def __init__(self):
        self.name_list = []
        self.model = model.Mixed_model(data_train_opt["dim"])
        self.model = torch.nn.DataParallel(self.model, device_ids=device_ids)
        self.model = self.model.cuda(device=device_ids[0])
        self.save = data_train_opt["final_model_file"]
        self.training_save = data_train_opt["feat_training_file"]
        self.training_log = data_train_opt["training_log"]
        self.loss = 9999
        self.best = 0
        self.train_dataset = dataloader.Load_Data(config["data_dir"],"train")
        self.trainloader = DataLoader(self.train_dataset, batch_size=data_train_opt['batch_size']*len(device_ids),num_workers=8,shuffle=True,drop_last=False)

        self.valid_dataset = dataloader.Load_Data(config["data_dir"],"val")
        self.validloader = DataLoader(self.valid_dataset,batch_size=data_train_opt['batch_size']*len(device_ids),num_workers=8,shuffle=True)
        self.LossFun()
        print("Trainloader: {}".format(len(self.trainloader)))
        print("Validloader: {}".format(len(self.validloader)))
        self.vae1 = VAE1().cuda()
        self.vae2 = VAE2().cuda()

    def loss_function(self,recon_x, x, mean, std):
        BCE = F.binary_cross_entropy(recon_x, x, reduction='sum')
        var = torch.pow(torch.exp(std), 2)
        KLD = -0.5 * torch.sum(1 + torch.log(var) - torch.pow(mean, 2) - var)
        return BCE+KLD

    def loss_function2(self,recon_x, x, mean, std):
        BCE = F.binary_cross_entropy(recon_x, x, reduction='sum')
        var = torch.pow(torch.exp(std), 2)
        KLD = -0.5 * torch.sum(1 + torch.log(var) - torch.pow(mean, 2) - var)
        return BCE + KLD

    def LossFun(self):
        print("lossing...")
        self.criterion = nn.CrossEntropyLoss()
        self.optimizer = optim.Adam(self.model.parameters(), lr=data_train_opt['lr'])

VAE needs to introduce reconstruction error, which is added to my previous model, so I first updated the parameters of the model and trained the previous model. Update the parameters of VAE and train VAE. When training VAE, I want to fix the parameters of other parts, so I add this part:
        for name,param in model.Mixed_model().named_parameters():
            if 'video' in name:
                param.requires_grad=False
            if 'audio_net' in name:
                param.requires_grad=False
            if 'classifier' in name:
                param.requires_grad=False
        self.optimizer2 = optim.Adam(filter(lambda param:param.requires_grad,model.Mixed_model().parameters()), lr=data_train_opt['lr'])

    def TrainingData(self):
        self.model.train()
        log = []
        for epoch in range(data_train_opt['epoch']):
            if (epoch+1) % data_train_opt["decay_epoch"] == 0 :
                for param_group in self.optimizer.param_groups:
                    param_group['lr'] = param_group['lr']*data_train_opt["decay_rate"]

            batch_time = AverageMeter('Time', ':6.3f')
            data_time = AverageMeter('Data', ':6.3f')
            losses = AverageMeter('Loss', ':.4e')
            top1 = AverageMeter('Acc@1', ':6.2f')
            progress = ProgressMeter(
                len(self.trainloader),
                [batch_time, data_time, losses,top1],
                prefix="Epoch: [{}]".format(epoch+1))

            # switch to train mode
            self.model.train()
            end = time.time()
            for i, (img,audio, class_id) in enumerate(self.trainloader):
                # measure data loading time
                data_time.update(time.time() - end)
                img,audio,class_id = img.cuda(device=device_ids[0]),audio.cuda(device=device_ids[0]),class_id.cuda(device=device_ids[0])
                predict,audio1,img1= self.model(img,audio)
                loss = self.criterion(predict, class_id)

                # acc1/acc5 are (K+1)-way contrast classifier accuracy
                # measure accuracy and record loss
                acc1= accuracy(predict, class_id, topk=(1,))
                losses.update(loss.item(), img.size(0))
                top1.update(acc1[0], img.size(0))
                self.optimizer.zero_grad()
                loss.backward(retain_graph=True)
                self.optimizer.step()

                z1, logstd1, mean1, eps1,recon_audio = self.vae1(audio1)
                z2, logstd2, mean2, eps2,recon_img = self.vae2(img1)

                lossv1 = self.loss_function(recon_audio, audio1, mean1, logstd1)
                lossv2 = self.loss_function2(recon_img, img1, mean2, logstd2)
                lossv = lossv2 + lossv1
                lossv.backward()
                self.optimizer2.zero_grad()
                lossv.backward()
                self.optimizer2.step()

                # measure elapsed time
                batch_time.update(time.time() - end)
                end = time.time()
                if (i+1) % data_train_opt["log_step"] == 0:
                    loss_avg = losses.avg
                    acc_avg = top1.avg
                    log.append([epoch, i + 1, loss.item(), acc1[0], loss_avg, acc_avg])
                    progress.display(i+1)

            if (epoch+1) % data_train_opt["save_epoch"] == 0:

                acc, a = self.ValidingData(epoch+1)
                if losses.avg <self.loss:
                    self.loss = losses.avg
                    a = 1
                np.save(data_train_opt["training_log"], log)
                if a == 1:
                    self.save_checkpoint({
                        'epoch': epoch + 1,
                        'state_dict': self.model.state_dict(),
                        'optimizer' : self.optimizer.state_dict(),
                        'acc':acc
                    }, filename=os.path.join(data_train_opt["feat_training_file"],'Epoch_{}_acc_{}_loss_{}.pth'.format(epoch+1,acc,losses.avg)))
                    # }, filename=os.path.join(data_train_opt["feat_training_file"],'checkpoint_{:04d}.pth'.format(epoch+1)))
                    # }, filename=os.path.join(data_train_opt["feat_training_file"],'best.pth'))

    def save_checkpoint(self,state,filename='checkpoint.pth.tar'):
        torch.save(state, filename)
    def ValidingData(self,epoch):

        self.model.eval()
        a = 0
        with torch.no_grad():
            y_pre = []
            y_true = []
            with tqdm(total=len(self.validloader), desc='Example', leave=True, ncols=100, unit='batch', unit_scale=True) as pbar:
                for i, (img,audio,class_id) in enumerate(self.validloader):
                    img,audio, class_id = img.cuda(device=device_ids[0]),audio.cuda(device=device_ids[0]), class_id.cuda(device=device_ids[0])
                    predict = self.model(img, audio)
                    _, pre = torch.max(predict,dim=1)
                    y_pre.append(pre.cpu())
                    y_true.append(class_id.cpu())
                    pbar.update(1)

            y_pre = torch.cat(y_pre).cpu().detach().numpy()
            y_true = torch.cat(y_true).cpu().detach().numpy()
            cm = confusion_matrix(y_true, y_pre)
            cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
            cm.diagonal()
            keys = ['airport',
                    'bus',
                    'metro',
                    'metro_station',
                    'park',
                    'public_square',
                    'shopping_mall',
                    'street_pedestrian',
                    'street_traffic',
                    'tram']
            values = [np.round(i, decimals=3) for i in list(cm.diagonal())]
            df_cm = pandas.DataFrame(cm, index=[i for i in keys], columns=[i for i in keys])
            plt.figure(figsize=(15, 12))
            sn.heatmap(df_cm, annot=True)
            plt.savefig('confusion.png')
            report = classification_report(y_true, y_pre, target_names=
                        ['airport',
                       'bus',
                       'metro',
                       'metro_station',
                       'park',
                       'public_square',
                       'shopping_mall',
                       'street_pedestrian',
                       'street_traffic',
                       'tram'], digits=4)
            acc = accuracy_score(y_true, y_pre)
            if acc>self.best:
                a = 1
                self.best=acc
            print(report)

            print("==================")
            with open(data_train_opt["txt"],"a") as f:
                f.write("========= {} =======\n".format(epoch))
                f.write("classification_report".format(epoch))
                f.write(report)
                f.write("\n")
        self.model.train()

        if a ==1:
            with open(data_train_opt["best"], "a") as f:
                f.write("========= {} =======\n".format(epoch))
                f.write("classification_report".format(epoch))
                f.write(report)
                f.write("================\n")

        return acc,a

def main():

    ImgCla = ImageClassify()
    ImgCla.TrainingData()
    training_plot.draw(data_train_opt["training_log"])
    acc, a = ImgCla.ValidingData(epoch=0)

if __name__ == '__main__':
    main()

Output:
The error message refers to the fact you are requiring gradient computation on the target tensor which is not supported by nn.functional.binary_cross_entropy. In other words you need to detach the target before computing the loss term:
BCE = F.binary_cross_entropy(recon_x, x.detach(), reduction='sum')

In both loss_function and loss_function2.


Instruction: Error while reading csv file: converting a column from string to float

Input:
I am trying to read a csv file that contains a column, SpType, in which there are String values. My variable is being converted into an object, but I need it to be float type.
Here's the snippet:
data = pd.read_csv("/content/Star3642_balanced.csv")

X_orig = data[["Vmag", "Plx", "e_Plx", "B-V", "SpType", "Amag"]].to_numpy()

Here's what's giving me the error:
X = torch.tensor(X_orig, dtype=torch.float32)

The error reads "can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool."
I tried doing this after reading the csv file, but it didn't help:
data["SpType"] = data.SpType.astype(float)

Can someone please tell me what can be done about this?

Output:
Strings should be encoded into numeric values. The easiest way would be using Pandas one-hot encoding (that will create lots of extra columns in this case, but a neural network should process those without much effort):
ohe = pd.get_dummies(data["SpType"], drop_first=True)
data[ohe.columns] = ohe
data = data.drop(["SpType"], axis=1)

Alternatively, you may use sklearn encoders or category_encoders library - more complex encoding might require to process the test set separately to avoid the target leakage.


Instruction: Retraining pytorch model (augmented learning)

Input:
I have the following code:
import torch
from facenet_pytorch import InceptionResnetV1, MTCNN
from torch.utils.data import DataLoader
from torchvision import datasets
import numpy as np
import pandas as pd
import os

workers = 0 if os.name == 'nt' else 4
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('Running on device: {}'.format(device))

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

def collate_fn(x):
    return x[0]

dataset = datasets.ImageFolder('data/images/')
dataset.idx_to_class = {i:c for c, i in dataset.class_to_idx.items()}
loader = DataLoader(dataset, collate_fn=collate_fn, num_workers=workers)
#print(dataset.idx_to_class)

aligned = []
names = []
i = 0
for x, y in loader:
    x_aligned, prob = mtcnn(x, return_prob=True)
    if x_aligned is not None:
        print('Face detected with probability: {:8f}'.format(prob))
        aligned.append(x_aligned)
        names.append(dataset.idx_to_class[y])
        i += 1
#print(i)

for name, param in mtcnn.named_parameters(): #Freezing everything but last layer
    #print(name)
    if name != "onet.dense6_3.bias":
        param.require_grad = False
    else:
        param.require_grad = True

And now I would like to retrain this model to predict three classes (Now it only predicts the probability of a face). Let say that I have inside data/images/ three folders, faces1, faces2 and faces3. How could I retrain this model with these three folders? I would like to have a tensor like [prob1, prob2, prob3] with the probability of an image for each class. Thanks.

Output:
MTCN: This class loads pretrained P-, R-, and O-nets and returns images cropped to include the face only, given raw input images.
I am assuming that you are trying to use InceptionResnetV1 for classification on your dataset. To retrain the Inception model you just load the model with the number of classes you need and then train it.
resnet = InceptionResnetV1(
    classify=True,
    pretrained='vggface2',
    num_classes=3
)

Complete finetuning example is here https://github.com/timesler/facenet-pytorch/blob/master/examples/finetune.ipynb


Instruction: How to train network on images of different sizes Pytorch

Input:
I  am trying to feed the Neural network  dataset of images and I am getting this error
I don't know what might be the cause as all the images have different sizes
I have also tried to change batch sizes and kernels but I had no success with this.
 File "c:\Users\david\Desktop\cs_agent\main.py", line 49, in <module>
    for i, data in enumerate(train_loader, 0):
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\dataloader.py", line 530, in __next__
    data = self._next_data()
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\_utils\collate.py", line 172, in default_collate
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\_utils\collate.py", line 172, in <listcomp>
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "C:\Users\david\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\data\_utils\collate.py", line 138, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 300, 535] at entry 0 and [3, 1080, 1920] at entry 23

this is my main file

import numpy as np
import matplotlib.pyplot as plt
import torch
import dataset
import os
from torch.utils.data import  DataLoader
import torch.nn as nn

import torchvision
import check_device

import neural_network
import torch.optim as optim

EPS = 1.e-7
LR=0.5
WEIGHT_DECAY=0.5
batch_size =50
#DATA LOADING ###################################################################################################################

test_dataset =dataset.csHeadBody(csv_file="images\\test_labels.csv",root_dir="images\\test")
train_dataset =dataset.csHeadBody(csv_file="images\\train_labels.csv",root_dir="images\\train")
train_loader =DataLoader(dataset =train_dataset,batch_size=batch_size,shuffle=True)
test_loader =DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=True)

#DATA LOADING ###################################################################################################################END

#NEURAL NET #####################################################################################################################################################

net=neural_network.Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

#NEURAL NET END ######################################################################################

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        print(data)
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')

and this is my  dataset file
class csHeadBody(Dataset):
    def __init__(self, csv_file, root_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.root_dir, self.img_labels.iloc[idx, 0])

        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label

this is my neural network architecture
import torch.nn.functional as F
import torch.nn as nn
import torch

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 535, 535)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Output:
You need to adjust the parameters of your convolutional and linear layers. The first argument is the number of input channels (3 for standard RGB images in conv1), then the number of output channels and then the convolution kernel size. To clarify, I've used named arguments in the code below. The code works for images of a square input size of 224x224 pixels (standard imagenet size, adjust if needed). If you want image size agnostic code you could use something like global average pooling (mean of each channel in the last conv layer). The net below supports both:
class Net(nn.Module):
    def __init__(self, use_global_average_pooling: bool = False):
        super().__init__()
        self.use_global_average_pooling = use_global_average_pooling
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3)
        self.pool = nn.MaxPool2d(kernel_size=(2, 2))
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3)
        if use_global_average_pooling:
            self.fc_gap = nn.Linear(64, 10)
        else:
            self.fc_1 = nn.Linear(54 * 54 * 64, 84)  # 54 img side times 64 out channels from conv2
            self.fc_2 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))  # img side: (224 - 2) // 2 = 111
        x = self.pool(F.relu(self.conv2(x)))  # img side: (111 - 2) // 2 =  54
        if self.use_global_average_pooling:
            # mean for global average pooling (mean over channel dimension)
            x = x.mean(dim=(-1, -2))
            x = F.relu(self.fc_gap(x))
        else:  # use all features
            x = torch.flatten(x, 1)
            x = F.relu(self.fc_1(x))
            x = self.fc_2(x)
        return x

Additionally, the torchvision.io.read_image function used in your Dataset returns an uint8 tensor with integer values from 0 to 255. You'll want floating point values for your network, so you have to divide the result by 255 to get values in the [0, 1] range. Furthermore, neural networks work best with normalized inputs (subtracting the mean and then dividing by the standard error of your training dataset). I've added normalization to the image transforms below. For convenience, it is using the imagenet mean and standard error, which should work fine if your images are similar to imagenet images (otherwise you can calculate them on your own images).
Note that the resizing might distort your images (doesn't keep the original aspect ratio). Often this is no problem, but if it is you might want to pad your images with a constant color (e.g. black) to resize them to the required dimensions (there are also transforms for this in the torchvision library).
IMAGENET_MEAN = [0.485, 0.456, 0.406]
IMAGENET_STD = [0.229, 0.224, 0.225]
transforms = torchvision.transforms.Compose([
    torchvision.transforms.Lambda(lambda x: x / 255.),
    torchvision.transforms.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
    torchvision.transforms.Resize((224, 224)),
])

You might also need to adjust the code in your Dataset to load images as an RGB image (if they also have an alpha channel). This can be done like this:
image = read_image(img_path, mode=torchvision.io.image.ImageReadMode.RGB)

You can then initialise your Dataset using:
test_dataset = dataset.csHeadBody(csv_file="images\\test_labels.csv", root_dir="images\\test", transform=transforms)
train_dataset = dataset.csHeadBody(csv_file="images\\train_labels.csv", root_dir="images\\train", transform=transforms)

I haven't tested the code, let me know if it doesn't work!


Instruction: why Heroku: slug size so large after installing Pytorch?

Input:
I'm trying to deploy python to heroku, but it's too big to deploy.
I've been getting the slug size too large warning (Compiled slug size: 789.8M is too large (max is 500M)) from Heroku
Compiled slug size: 826.6M is too large (max is 500M).
remote:  !     See: http://devcenter.heroku.com/articles/slug-size
remote:
remote:  !     Push failed
remote: Verifying deploy....
remote:
fastapi==0.78.0
gunicorn==20.1.0
numpy==1.22.4
opencv-python==4.5.4.58
Pillow==8.4.0
retina-face==0.0.12
uvicorn==0.17.6
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.11.0+cpu; python_full_version >= "3.6.2"
torchvision==0.12.0+cpu
tensorflow-cpu == 2.8.0

remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Building on the Heroku-20 stack
remote: -----> Determining which buildpack to use for this app
remote: -----> Python app detected
remote: -----> No Python version was specified. Using the buildpack default: python-3.10.5
remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
remote: -----> Installing python-3.10.5
remote: -----> Installing pip 22.1.2, setuptools 60.10.0 and wheel 0.37.1
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Looking in links: https://download.pytorch.org/whl/torch_stable.html
remote:        Collecting fastapi==0.78.0
remote:          Downloading fastapi-0.78.0-py3-none-any.whl (54 kB)
remote:        Collecting gunicorn==20.1.0
remote:          Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
remote:        Collecting numpy==1.22.4
remote:          Downloading numpy-1.22.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
remote:        Collecting opencv-python==4.5.4.58
remote:          Downloading opencv_python-4.5.4.58-cp310-cp310-manylinux2014_x86_64.whl (60.3 MB)
remote:        Collecting Pillow==8.4.0
remote:          Downloading Pillow-8.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
remote:        Collecting retina-face==0.0.12
remote:          Downloading retina_face-0.0.12-py3-none-any.whl (15 kB)
remote:        Collecting uvicorn==0.17.6
remote:          Downloading uvicorn-0.17.6-py3-none-any.whl (53 kB)
remote:        Collecting torch==1.11.0+cpu
remote:          Downloading https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp310-cp310-linux_x86_64.whl (169.2 MB)
remote:        Collecting torchvision==0.12.0+cpu
remote:          Downloading https://download.pytorch.org/whl/cpu/torchvision-0.12.0%2Bcpu-cp310-cp310-linux_x86_64.whl (14.7 MB)
remote:        Collecting tensorflow-cpu==2.8.0
remote:          Downloading tensorflow_cpu-2.8.0-cp310-cp310-manylinux2010_x86_64.whl (190.6 MB)
remote:        Collecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2
remote:          Downloading pydantic-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB)
remote:        Collecting starlette==0.19.1
remote:          Downloading starlette-0.19.1-py3-none-any.whl (63 kB)
remote:        Collecting gdown>=3.10.1
remote:          Downloading gdown-4.4.0.tar.gz (14 kB)
remote:          Installing build dependencies: started
remote:          Installing build dependencies: finished with status 'done'
remote:          Getting requirements to build wheel: started
remote:          Getting requirements to build wheel: finished with status 'done'
remote:          Preparing metadata (pyproject.toml): started
remote:          Preparing metadata (pyproject.toml): finished with status 'done'
remote:        Collecting tensorflow>=1.9.0
remote:          Downloading tensorflow-2.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
remote:        Collecting h11>=0.8
remote:          Downloading h11-0.13.0-py3-none-any.whl (58 kB)
remote:        Collecting click>=7.0
remote:          Downloading click-8.1.3-py3-none-any.whl (96 kB)
remote:        Collecting asgiref>=3.4.0
remote:          Downloading asgiref-3.5.2-py3-none-any.whl (22 kB)
remote:        Collecting typing-extensions
remote:          Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB)
remote:        Collecting requests
remote:          Downloading requests-2.28.0-py3-none-any.whl (62 kB)
remote:        Collecting opt-einsum>=2.3.2
remote:          Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
remote:        Collecting flatbuffers>=1.12
remote:          Downloading flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
remote:        Collecting gast>=0.2.1
remote:          Downloading gast-0.5.3-py3-none-any.whl (19 kB)
remote:        Collecting keras-preprocessing>=1.1.1
remote:          Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
remote:        Collecting libclang>=9.0.1
remote:          Downloading libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)
remote:        Collecting grpcio<2.0,>=1.24.3
remote:          Downloading grpcio-1.46.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB)
remote:        Collecting tensorboard<2.9,>=2.8
remote:          Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
remote:        Collecting astunparse>=1.6.0
remote:          Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
remote:        Collecting protobuf>=3.9.2
remote:          Downloading protobuf-4.21.1-cp37-abi3-manylinux2014_x86_64.whl (407 kB)
remote:        Collecting keras<2.9,>=2.8.0rc0
remote:          Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)
remote:        Collecting six>=1.12.0
remote:          Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
remote:        Collecting termcolor>=1.1.0
remote:          Downloading termcolor-1.1.0.tar.gz (3.9 kB)
remote:          Preparing metadata (setup.py): started
remote:          Preparing metadata (setup.py): finished with status 'done'
remote:        Collecting absl-py>=0.4.0
remote:          Downloading absl_py-1.1.0-py3-none-any.whl (123 kB)
remote:        Collecting google-pasta>=0.1.1
remote:          Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
remote:        Collecting tensorflow-io-gcs-filesystem>=0.23.1
remote:          Downloading tensorflow_io_gcs_filesystem-0.26.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
remote:        Collecting h5py>=2.9.0
remote:          Downloading h5py-3.7.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
remote:        Collecting tf-estimator-nightly==2.8.0.dev2021122109
remote:          Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
remote:        Collecting wrapt>=1.11.0
remote:          Downloading wrapt-1.14.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
remote:        Collecting anyio<5,>=3.4.0
remote:          Downloading anyio-3.6.1-py3-none-any.whl (80 kB)
remote:        Collecting tqdm
remote:          Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
remote:        Collecting filelock
remote:          Downloading filelock-3.7.1-py3-none-any.whl (10 kB)
remote:        Collecting beautifulsoup4
remote:          Downloading beautifulsoup4-4.11.1-py3-none-any.whl (128 kB)
remote:        Collecting werkzeug>=0.11.15
remote:          Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
remote:        Collecting google-auth-oauthlib<0.5,>=0.4.1
remote:          Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
remote:        Collecting markdown>=2.6.8
remote:          Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
remote:        Collecting google-auth<3,>=1.6.3
remote:          Downloading google_auth-2.7.0-py2.py3-none-any.whl (160 kB)
remote:        Collecting tensorboard-data-server<0.7.0,>=0.6.0
remote:          Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
remote:        Collecting tensorboard-plugin-wit>=1.6.0
remote:          Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
remote:        Collecting charset-normalizer~=2.0.0
remote:          Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
remote:        Collecting urllib3<1.27,>=1.21.1
remote:          Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
remote:        Collecting idna<4,>=2.5
remote:          Downloading idna-3.3-py3-none-any.whl (61 kB)
remote:        Collecting certifi>=2017.4.17
remote:          Downloading certifi-2022.5.18.1-py3-none-any.whl (155 kB)
remote:        Collecting gast>=0.2.1
remote:          Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
remote:        Collecting tensorflow-estimator<2.10.0,>=2.9.0rc0
remote:          Downloading tensorflow_estimator-2.9.0-py2.py3-none-any.whl (438 kB)
remote:        Collecting tensorflow>=1.9.0
remote:          Downloading tensorflow-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
remote:          Downloading tensorflow-2.8.2-cp310-cp310-manylinux2010_x86_64.whl (498.0 MB)
remote:        Collecting tensorflow-estimator<2.9,>=2.8
remote:          Downloading tensorflow_estimator-2.8.0-py2.py3-none-any.whl (462 kB)
remote:        Collecting protobuf>=3.9.2
remote:          Downloading protobuf-3.19.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
remote:        Collecting sniffio>=1.1
remote:          Downloading sniffio-1.2.0-py3-none-any.whl (10 kB)
remote:        Collecting rsa<5,>=3.1.4
remote:          Downloading rsa-4.8-py3-none-any.whl (39 kB)
remote:        Collecting cachetools<6.0,>=2.0.0
remote:          Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
remote:        Collecting pyasn1-modules>=0.2.1
remote:          Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
remote:        Collecting requests-oauthlib>=0.7.0
remote:          Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
remote:        Collecting soupsieve>1.2
remote:          Downloading soupsieve-2.3.2.post1-py3-none-any.whl (37 kB)
remote:        Collecting PySocks!=1.5.7,>=1.5.6
remote:          Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)
remote:        Collecting pyasn1<0.5.0,>=0.4.6
remote:          Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
remote:        Collecting oauthlib>=3.0.0
remote:          Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
remote:        Building wheels for collected packages: gdown, termcolor
remote:          Building wheel for gdown (pyproject.toml): started
remote:          Building wheel for gdown (pyproject.toml): finished with status 'done'
remote:          Created wheel for gdown: filename=gdown-4.4.0-py3-none-any.whl size=14759 sha256=7708c4f7156a089f3fc5fc4488a952096a5a55641ba3eb077fbe30557bad88b9
remote:          Stored in directory: /tmp/pip-ephem-wheel-cache-gs7e9o2q/wheels/03/0b/3f/6ddf67a417a5b400b213b0bb772a50276c199a386b12c06bfc
remote:          Building wheel for termcolor (setup.py): started
remote:          Building wheel for termcolor (setup.py): finished with status 'done'
remote:          Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4848 sha256=1d47aa236c647ae87a51f34c01fb6c879088c2d0af3e04cd7a949a4b8b0ae977
remote:          Stored in directory: /tmp/pip-ephem-wheel-cache-gs7e9o2q/wheels/a1/49/46/1b13a65d8da11238af9616b00fdde6d45b0f95d9291bac8452
remote:        Successfully built gdown termcolor
remote:        Installing collected packages: tf-estimator-nightly, termcolor, tensorflow-estimator, tensorboard-plugin-wit, pyasn1, libclang, keras, flatbuffers, wrapt, werkzeug, urllib3, typing-extensions, tqdm, tensorflow-io-gcs-filesystem, tensorboard-data-server, soupsieve, sniffio, six, rsa, PySocks, pyasn1-modules, protobuf, Pillow, oauthlib, numpy, markdown, idna, h11, gunicorn, gast, filelock, click, charset-normalizer, certifi, cachetools, asgiref, absl-py, uvicorn, torch, requests, pydantic, opt-einsum, opencv-python, keras-preprocessing, h5py, grpcio, google-pasta, google-auth, beautifulsoup4, astunparse, anyio, torchvision, starlette, requests-oauthlib, google-auth-oauthlib, gdown, fastapi, tensorboard, tensorflow-cpu, tensorflow, retina-face
remote:        Successfully installed Pillow-8.4.0 PySocks-1.7.1 absl-py-1.1.0 anyio-3.6.1 asgiref-3.5.2 astunparse-1.6.3 beautifulsoup4-4.11.1 cachetools-5.2.0 certifi-2022.5.18.1 charset-normalizer-2.0.12 click-8.1.3 fastapi-0.78.0 filelock-3.7.1 flatbuffers-2.0 gast-0.5.3 gdown-4.4.0 google-auth-2.7.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.46.3 gunicorn-20.1.0 h11-0.13.0 h5py-3.7.0 idna-3.3 keras-2.8.0 keras-preprocessing-1.1.2 libclang-14.0.1 markdown-3.3.7 numpy-1.22.4 oauthlib-3.2.0 opencv-python-4.5.4.58 opt-einsum-3.3.0 protobuf-3.19.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydantic-1.9.1 requests-2.28.0 requests-oauthlib-1.3.1 retina-face-0.0.12 rsa-4.8 six-1.16.0 sniffio-1.2.0 soupsieve-2.3.2.post1 starlette-0.19.1 tensorboard-2.8.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.8.2 tensorflow-cpu-2.8.0 tensorflow-estimator-2.8.0 tensorflow-io-gcs-filesystem-0.26.0 termcolor-1.1.0 tf-estimator-nightly-2.8.0.dev2021122109 torch-1.11.0+cpu torchvision-0.12.0+cpu tqdm-4.64.0 typing-extensions-4.2.0 urllib3-1.26.9 uvicorn-0.17.6 werkzeug-2.1.2 wrapt-1.14.1
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote:
remote: -----> Compressing...
remote:  !     Compiled slug size: 826.6M is too large (max is 500M).
remote:  !     See: http://devcenter.heroku.com/articles/slug-size
remote:
remote:  !     Push failed
remote: Verifying deploy....
remote:
remote: !       Push rejected to lit-castle-78959.
remote:

Output:
You are installing both pytorch and tensorflow in the same slug.
Both libraries are known to be large and they are rarely deployed in the same slug.
Since they both aim to solve the same problem (neural networks/tensor computation), do you really need both? If yes, try to split your application and create two different slugs.
I am used to deploy slugs with either torch==1.11.0+cpu or tensorflow-cpu==2.1.0.


Instruction: AttributeError: 'MaskedLMOutput' object has no attribute 'view'

Input:
Sorry to bother,  I met this error when I evaluate some models,  and I didn't find a good method to fix it.
What does 'MaskedLMOutput'means?Could somebody tell me How to fix this please? Thank you.
(AttributeError: 'MaskedLMOutput' object has no attribute 'view')
from transformers import BertForMaskedLM

class BertPunc(nn.Module):

    def __init__(self, segment_size, output_size, dropout):
        super(BertPunc, self).__init__()
        self.bert = BertForMaskedLM.from_pretrained('cl-tohoku/bert-base-japanese')
        self.bert_vocab_size = 32000
        self.bn = nn.BatchNorm1d(segment_size*self.bert_vocab_size)
        self.fc = nn.Linear(segment_size*self.bert_vocab_size, output_size)
        self.dropout = nn.Dropout(dropout)

    def forward(self, input):

        x = self.bert(input)
        x = x.view(x.shape[0], -1)                   # wrong thing here
        x = self.fc(self.dropout(self.bn(x)))
        return x

I ran this in jupyter notebook:
    def predictions(data_loader):
        y_pred = []
        y_true = []
        for inputs, labels in tqdm(data_loader, total=len(data_loader)):
            with torch.no_grad():
                inputs, labels = inputs.cuda(), labels.cuda()
                output = bert_punc(inputs)
                y_pred += list(output.argmax(dim=1).cpu().data.numpy().flatten())
                y_true += list(labels.cpu().data.numpy().flatten())
        return y_pred, y_true

def evaluation(y_pred, y_test):
    precision, recall, f1, _ = metrics.precision_recall_fscore_support(
        y_test, y_pred, average=None, labels=[1, 2, 3])
    overall = metrics.precision_recall_fscore_support(
        y_test, y_pred, average='macro', labels=[1, 2, 3])
    result = pd.DataFrame(
        np.array([precision, recall, f1]),
        columns=list(punctuation_enc.keys())[1:],
        index=['Precision', 'Recall', 'F1']
    )
    result['OVERALL'] = overall[:3]
    return result

y_pred_test, y_true_test = predictions(data_loader_test)
eval_test = evaluation(y_pred_test, y_true_test)
eval_test

wrong here:
   ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Input In [12], in <cell line: 1>()
    ----> 1 y_pred_test, y_true_test = predictions(data_loader_test)
          2 eval_test = evaluation(y_pred_test, y_true_test)
          3 eval_test

    Input In [10], in predictions(data_loader)
          5 with torch.no_grad():
          6     inputs, labels = inputs.cuda(), labels.cuda()
    ----> 7     output = bert_punc(inputs)
          8     y_pred += list(output.argmax(dim=1).cpu().data.numpy().flatten())
          9     y_true += list(labels.cpu().data.numpy().flatten())

    File ~\anaconda3\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
       1106 # If we don't have any hooks, we want to skip the rest of the logic in
       1107 # this function, and just call forward.
       1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
       1109         or _global_forward_hooks or _global_forward_pre_hooks):
    -> 1110     return forward_call(*input, **kwargs)
       1111 # Do not call functions when jit is used
       1112 full_backward_hooks, non_full_backward_hooks = [], []

    File ~\anaconda3\lib\site-packages\torch\nn\parallel\data_parallel.py:166, in DataParallel.forward(self, *inputs, **kwargs)
        163     kwargs = ({},)
        165 if len(self.device_ids) == 1:
    --> 166     return self.module(*inputs[0], **kwargs[0])
        167 replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
        168 outputs = self.parallel_apply(replicas, inputs, kwargs)

    File ~\anaconda3\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
       1106 # If we don't have any hooks, we want to skip the rest of the logic in
       1107 # this function, and just call forward.
       1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
       1109         or _global_forward_hooks or _global_forward_pre_hooks):
    -> 1110     return forward_call(*input, **kwargs)
       1111 # Do not call functions when jit is used
       1112 full_backward_hooks, non_full_backward_hooks = [], []

    File F:\BertPunc-master\model.py:19, in BertPunc.forward(self, input)
         16 def forward(self, input):
         18     x = self.bert(input)
    ---> 19     x = x.view(x.shape[0], -1)
         20     x = self.fc(self.dropout(self.bn(x)))
         21     return x

    AttributeError: 'MaskedLMOutput' object has no attribute 'view'

Output:
You can refer to the documentation of MaskedLMOutput. Basically, it is an object holding the loss, logits, hidden_states and attentions. It is not a tensor so you are getting this error. I think you are interested in logits, i.e., score for each token before applying softmax. Then, in the forward function, you can just access to the logits tensor like this
x = self.bert(input).logits


Instruction: PyTorch CNN Different Input Size

Input:
Hello Guys I have a question about different Input Sizes.
My training set and validation dataset have an input Size of 256 and for my prediction (with an unseen Test Dataset) I have an input size of 496.
class Net(nn.Module):
    def __init__(self, shape):
        super(Net,self).__init__()
        self.conv1 = nn.Conv1d(shape,1,1)
        self.batch1 = nn.BatchNorm1d(1)
        self.avgpl1 = nn.AvgPool1d(1, stride=1)
        self.fc1 = nn.Linear(1,3)

    #forward method
    def forward(self,x):
        x = self.conv1(x)
        x = self.batch1(x)
        x = F.relu(x)
        x = self.avgpl1(x)
        x = torch.flatten(x,1)
        x = F.log_softmax(self.fc1(x))
        return x

I saved the model and wanna use it also for my prediction.
Error Message is:
Input In [244], in predict_data(prediction_data, model_path, data_config, context)
     25 new_model = Net(shape_preprocessed_data)
     26 # load the previously saved state_dict
---> 27 new_model.load_state_dict(torch.load("NetModel.pth"))
     29 # check if predictions of models are equal
     30
     31 # generate random input of size (N,C,H,W)
     32
     33 # switch to eval mode for both models
     34 model = model.eval()

    RuntimeError: Error(s) in loading state_dict for Net:
    size mismatch for conv1.weight: copying a param with shape
    torch.Size([1, 256, 1]) from checkpoint, the shape in current model is torch.Size([1, 494, 1]).

How can I solve this?

Output:
You could reshape/downsample the input as the first step of the forward pass in your model. This can be done using the torch.nn.functional.interpolate function.
For example:
class Net(nn.Module):
def __init__(self, shape):
    super(Net,self).__init__()
    self.input_shape = shape
    self.conv1 = nn.Conv1d(shape,1,1)
    self.batch1 = nn.BatchNorm1d(1)
    self.avgpl1 = nn.AvgPool1d(1, stride=1)
    self.fc1 = nn.Linear(1,3)

#forward method
def forward(self,x):
    x = torch.nn.functional.interpolate(x, size=self.input_shape)
    x = self.conv1(x)
    x = self.batch1(x)
    x = F.relu(x)
    x = self.avgpl1(x)
    x = torch.flatten(x,1)
    x = F.log_softmax(self.fc1(x))
    return x

Your test images would then be downsampled to size 256 in order to be compatible with the model.


Instruction: Running Detectron2 locally - windows - [Pytorch Config error]

Input:
I am trying to run this code locally:
https://gist.github.com/shashank524/74d8f46d5de633b84e2265fcc34774de#file-tabledetection-ipynb
After installing required packages, when I am trying to run this line:
import layoutparser as lp

# PubLayNet
model = layoutparser.Detectron2LayoutModel('lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config',extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST", 0.81],label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"})

I receive this error:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\username/.torch/iopath_cache\\s/f3b12qc4hc0yh4m\\config.yml?dl=1.lock'

I looked into the directory and there was no config file available.
I tried to download the config file from here (https://layout-parser.readthedocs.io/en/latest/notes/modelzoo.html) and put it in the directory but it didn't solve the issue!

Output:
Even I got a similar error. I tried out manually some work around in Windows.
I am using your case as example: OSError: [Errno 22] Invalid argument: 'C:\Users\username/.torch/iopath_cache\s/f3b12qc4hc0yh4m\config.yml?dl=1.lock'
Please follow the following process.

Navigate to C:\Users\username/.torch/iopath_cache\s/f3b12qc4hc0yh4m\config.yml
Open that config.yaml file

Scroll down to WEIGHTS: https://www.dropbox.com/s/h7th27jfv19rxiy/model_final.pth?dl=1 should be around 265 line.

Copy that link and paste it in your browser, a 'model_final.pth' will be downloaded. Copy this file to your desired folder.

Now replace the path to WEIGHTS: your_desired_folder\model_final.pth

Save it and run the code it works!

But there is a small work around I think before you do this (if you have not done)
iopath work around
https://github.com/Layout-Parser/layout-parser/issues/15 (Github link to the issue)
Hope this helps!


Instruction: How to check if any of the gradients in a PyTorch model is nan?

Input:
I have a toy model:
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Model()

opt = optim.Adam(net.parameters())

The training loop is
features = torch.rand((3,1))
for i in range(10):
    opt.zero_grad()
    out = net(features)
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss.backward()
    opt.step()

How can I check if any of the gradients is nan? That is, if just 1 of the gradients is nan print something/break
pseudocode:
for i in range(10):
    opt.zero_grad()
    out = net(features)
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss.backward()

    if_gradients_nan:
        print("NAN")

    opt.step()

Output:
You can check as below. This approach only checks for the gradients with respect to the model parameters. It does not look at intermediate gradients, actually, those intermediate gradients do not exist after loss.backward() is called without retain_graph=True argument. For the demonstration purposes, I have multiplied output of first torch.relu(x) with float("inf") so that some of the gradients become nan.
...
    def forward(self, x):
        x = self.fc1(x)
        x = torch.relu(x) * float("inf")
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x
...

loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
loss.backward()

for name, param in net.named_parameters():
    print(name, torch.isnan(param.grad))

opt.step()

This prints
fc1.weight tensor([[False],
        [False]])
fc1.bias tensor([False, False])
fc2.weight tensor([[True, True],
        [True, True],
        [True, True]])
fc2.bias tensor([True, True, True])
fc3.weight tensor([[True, True, True]])
fc3.bias tensor([True])
fc1.weight tensor([[False],
        [False]])
fc1.bias tensor([False, False])
fc2.weight tensor([[True, True],
        [True, True],
        [True, True]])
...

To check if any of the gradients is nan, you can use
for name, param in net.named_parameters():
    if torch.isnan(param.grad).any():
        print("nan gradient found")
        raise SystemExit


Instruction: Mobilevit Binary classification ValueError: `logits` and `labels` must have the same shape, received ((None, 2) vs (None, 1))

Input:
I am using the colab notebook(https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/mobilevit.ipynb) for mobilevit to train on a dataset I have of 25k pictures for 2 classes. Since it's a binary classification, I have used keras.losses.BinaryCrossentropy and Sigmoid as activation function at the last layer:-
def create_mobilevit(num_classes=2):
inputs = keras.Input((image_size, image_size, 3))
x = layers.Rescaling(scale=1.0 / 255)(inputs)

# Initial conv-stem -> MV2 block.
x = conv_block(x, filters=16)
x = inverted_residual_block(
    x, expanded_channels=16 * expansion_factor, output_channels=16
)

# Downsampling with MV2 block.
x = inverted_residual_block(
    x, expanded_channels=16 * expansion_factor, output_channels=24, strides=2
)
x = inverted_residual_block(
    x, expanded_channels=24 * expansion_factor, output_channels=24
)
x = inverted_residual_block(
    x, expanded_channels=24 * expansion_factor, output_channels=24
)

# First MV2 -> MobileViT block.
x = inverted_residual_block(
    x, expanded_channels=24 * expansion_factor, output_channels=48, strides=2
)
x = mobilevit_block(x, num_blocks=2, projection_dim=64)

# Second MV2 -> MobileViT block.
x = inverted_residual_block(
    x, expanded_channels=64 * expansion_factor, output_channels=64, strides=2
)
x = mobilevit_block(x, num_blocks=4, projection_dim=80)

# Third MV2 -> MobileViT block.
x = inverted_residual_block(
    x, expanded_channels=80 * expansion_factor, output_channels=80, strides=2
)
x = mobilevit_block(x, num_blocks=3, projection_dim=96)
x = conv_block(x, filters=320, kernel_size=1, strides=1)

# Classification head.
x = layers.GlobalAvgPool2D()(x)
outputs = layers.Dense(num_classes, activation="sigmoid")(x)

return keras.Model(inputs, outputs)

And here's my dataset preparation cell:-
batch_size = 64
auto = tf.data.AUTOTUNE
resize_bigger = 512
num_classes = 2

def preprocess_dataset(is_training=True):
    def _pp(image, label):
        if is_training:
            # Resize to a bigger spatial resolution and take the random
            # crops.
            image = tf.image.resize(image, (resize_bigger, resize_bigger))
            image = tf.image.random_crop(image, (image_size, image_size, 3))
            image = tf.image.random_flip_left_right(image)
        else:
            image = tf.image.resize(image, (image_size, image_size))
        label = tf.one_hot(label, depth=num_classes)
        return image, label

    return _pp

def prepare_dataset(dataset, is_training=True):
    if is_training:
        dataset = dataset.shuffle(batch_size * 10)
    dataset = dataset.map(preprocess_dataset(is_training), num_parallel_calls=auto)
    return dataset.batch(batch_size).prefetch(auto)

And this is the cell for training the model:-
learning_rate = 0.002
label_smoothing_factor = 0.1
epochs = 30

optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
loss_fn = keras.losses.BinaryCrossentropy(label_smoothing=label_smoothing_factor)

def run_experiment(epochs=epochs):
    mobilevit_xxs = create_mobilevit(num_classes=num_classes)
    mobilevit_xxs.compile(optimizer=optimizer, loss=loss_fn, metrics=["accuracy"])

    checkpoint_filepath = "/tmp/checkpoint"
    checkpoint_callback = keras.callbacks.ModelCheckpoint(
        checkpoint_filepath,
        monitor="val_accuracy",
        save_best_only=True,
        save_weights_only=True,
    )

    mobilevit_xxs.fit(
        train_ds,
        validation_data=val_ds,
        epochs=epochs,
        callbacks=[checkpoint_callback],
    )
    mobilevit_xxs.load_weights(checkpoint_filepath)
    _, accuracy = mobilevit_xxs.evaluate(val_ds)
    print(f"Validation accuracy: {round(accuracy * 100, 2)}%")
    return mobilevit_xxs

mobilevit_xxs = run_experiment()

Basically the code is identical to https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/mobilevit.ipynb except for the change in BinaryCrossEntropy loss and Sigmoid as actv. func. I don't understand why I am getting this even though I am explicitly ont-hot-coded my class labels -
ValueError: `logits` and `labels` must have the same shape, received ((None, 2) vs (None, 1)).

Output:
You need to change the num_classes = 1 instead of num_classes = 2 as you have used Sigmoid activation function which returns the values between 0 to 1 for binary classification(0,1).
The values <0.5 will be considered as class 0 and values >0.5 will be as class 1 in between two binary classes (0,1).
Please refer to the replicated gist for your reference.


Instruction: RuntimeError: Trying to backward through the graph a second time. Saved intermediate values of the graph are freed when you call .backward()

Input:
I am trying to train SRGAN from scratch. I have read solutions for this type of problem, but it would be great if someone could help me debug my code. The exact error is: "RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad()" Here is the snippet I am trying to train:
gen_model = Generator().to(device, non_blocking=True)
disc_model  = Discriminator().to(device, non_blocking=True)
opt_gen = optim.Adam(gen_model.parameters(), lr=0.01)
opt_disc = optim.Adam(disc_model.parameters(), lr=0.01)
from torch.nn.modules.loss import BCELoss

def train_model(gen, disc):
  for epoch in range(20):
    run_loss_disc = 0
    run_loss_gen = 0
    for data in train:
      low_res, high_res = data[0].to(device, non_blocking=True, dtype=torch.float).permute(0, 3, 1, 2),data[1].to(device, non_blocking=True, dtype=torch.float).permute(0, 3, 1, 2)
      #--------Discriminator-----------------

      gen_image = gen(low_res)
      gen_image = gen_image.detach()
      disc_gen = disc(gen_image)
      disc_real = disc(high_res)
      p=nn.BCEWithLogitsLoss()
      loss_gen = p(disc_real, torch.ones_like(disc_real))

      loss_real = p(disc_gen, torch.zeros_like(disc_gen))
      loss_disc = loss_gen + loss_real
      opt_disc.zero_grad()
      loss_disc.backward()

      run_loss_disc+=loss_disc
      #---------Generator--------------------
      cont_loss = vgg_loss(high_res, gen_image)
      adv_loss = 1e-3*p(disc_gen, torch.ones_like(disc_gen))
      gen_loss = cont_loss+(10^-3)*adv_loss
      opt_gen.zero_grad()
      gen_loss.backward()
      opt_disc.step()
      opt_gen.step()
      run_loss_gen+=gen_loss
    print("Run Loss Discriminator: %d", run_loss_disc)
    print("Run Loss Generator: %d", run_loss_gen)

train_model(gen_model, disc_model)

Output:
Apparently your disc_gen value was discarded by the first backward() call, as it says.
It should work if you change the discriminator part a bit:
gen_image = gen(low_res)
disc_gen = disc(gen_image.detach())

and add this at the start of the generator part:
disc_gen = disc(gen_image)


Instruction: In the latest version of PyTorch, what is best practice to get all tensors to use a particular device by default?

Input:
In pytorch, if I do something like
import torch
x = torch.randn(3)
y = x + 5

all tensors correspond to the "cpu" device by default. Is there some way to make to make it so, by default, all tensors are on another device (e.g. "cuda:0")?
I know I can always be careful to add .cuda() or specify cuda whenever creating a tensor, but it would be great if I could just change the default device directly at the beginning of the program and be done with it, so that torch.randn(3) comes from the desired device without having to specify it every time.
Or would that be a bad thing to do for some reason? E.g. is there any reason I wouldn't want every tensor/operation to be done on cuda by default?

Output:
Pytorch has an optional function to change the default type of tensor set_default_tensor_type. Applying the default type on the main script:
>>> import torch
>>>
>>> if __name__ == '__main__':
...     cuda = torch.cuda.is_available()
...     if cuda:
...         torch.set_default_tensor_type('torch.cuda.FloatTensor')
...     a = torch.randn(3,3)
...     print(a.device)
...
cuda:0

Or would that be a bad thing to do for some reason? E.g. is there any reason I wouldn't want every tensor/operation to be done on cuda by default?

I couldn't find any reference or any document to answer this question. However, in my opinion, it's to avoid the memory fragmentation in GPU memory.
I'm not an expert, but the data in memory should be arranged in an efficient way, if not, the redundant space will cause OOM. That's why, in default, Tensorflow will take all of your GPU's memory no matter how many parameters your model has. You can improve the space and speed just by setting the tensor shape multiples of 8 amp documents.

In practice, higher performance is achieved when A and B dimensions are multiples of 8.

In conclusion, I think it's better to control the device of tensor manually instead of setting it gpu as default.


Instruction: How to reshape a dataset in order to make it sequential?

Input:
I have a classic dataset of images and labels.
Here is a simple representation of the __getitem__ function :
def __getitem__(self, index):
        (img_path, label) = df.iloc[index].values
        img = Image.open(img_path).convert("RGB")
        y = torch.tensor(labels))

        return (img, y)

I have :
dataset = ClassDataset()
train_set, validation_set = random_split(dataset)
train_loader = DataLoader(dataset=train_set)

The size of one batch of the train loader would be : [32,3,256,256]
With 32 being the batch size, 3 the number of channels and 256 the width and height of my image.
I want to modify the shape of one batch so that it is sequential [8,4,3,256,256] with 8 being the batch size and 4 the length of one sequence.
I know that it could be easily done with torch.view() or torch.reshape() knowing that my data are already in the right order (they can be grouped directly into sequences).
But I want to know where is the most intelligent place to make this change, in the dataset class, in the dataloader class or in the train loop.
I already tried passing sequences into the getitem :
(img_path, coords) = df.iloc[4*(index-1):4*index].values

(assuming that sequence length is 4), but it didn't work.

Output:
It is more relevant to do this kind of processing in the dataset layer. Indeed, what you are looking to implement there is "given a dataset index index return the corresponding input and its label". In your case you are dealing with a sequence as input, so something like this makes sense for your __getitem__ to return a sequence of images.
The data loader will automatically collate the data such that you get (batch_size, seq_len, channel, height, width) for your input, and (batch_size, seq_len) for your label (or (batch_size,) if there is meant to be a single label per sequence).


Instruction: Why is the Tensorflow and Pytorch CrossEntropy loss returns different values for same example

Input:
I have tried getting Tensorflow and Pytorch CrossEntropyLoss but it returns different values and I don't know why. Please find the below code and results. Thanks for your inputs and help.
import tensorflow as tf
import numpy as np

y_true = [3, 3, 1]
y_pred = [
    [0.3377, 0.4867, 0.8842, 0.0854, 0.2147],
    [0.4853, 0.0468, 0.6769, 0.5482, 0.1570],
    [0.0976, 0.9899, 0.6903, 0.0828, 0.0647]
]

scce3 = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.AUTO)
loss3 = scce3(y_true, y_pred).numpy()
print(loss3)

The result for above is : 1.69
Pytorch loss:
from torch import nn
import torch
loss = nn.CrossEntropyLoss()
y_true = torch.Tensor([3, 3, 1]).long()
y_pred = torch.Tensor([
    [0.3377, 0.4867, 0.8842, 0.0854, 0.2147],
    [0.4853, 0.0468, 0.6769, 0.5482, 0.1570],
    [0.0976, 0.9899, 0.6903, 0.0828, 0.0647]
])
loss2 = loss(y_pred, y_true)
print(loss2)

The loss value for above is: 1.5

Output:
Tensorflow's CrossEntropy expects probabilities as inputs (i.e. values after a tf.nn.softmax operation), whereas PyTorch's CrossEntropyLoss expects raw inputs, or more commonly named, logits. If you use the softmax operation, the values should be the same:
import tensorflow as tf
import numpy as np

y_true = [3, 3, 1]
y_pred = [
    [0.3377, 0.4867, 0.8842, 0.0854, 0.2147],
    [0.4853, 0.0468, 0.6769, 0.5482, 0.1570],
    [0.0976, 0.9899, 0.6903, 0.0828, 0.0647]
]

scce3 = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.AUTO)
loss3 = scce3(y_true, tf.nn.softmax(y_pred)).numpy()
print(loss3)

>>> 1.5067214

from torch import nn
import torch
loss = nn.CrossEntropyLoss()
y_true = torch.Tensor([3, 3, 1]).long()
y_pred = torch.Tensor([
    [0.3377, 0.4867, 0.8842, 0.0854, 0.2147],
    [0.4853, 0.0468, 0.6769, 0.5482, 0.1570],
    [0.0976, 0.9899, 0.6903, 0.0828, 0.0647]
])
loss2 = loss(y_pred, y_true)
print(loss2)

>>> tensor(1.5067)

Using the raw inputs (logits) is usually advised due to the LogSumExp trick for numerical stability. If you are using Tensorflow, I'd suggest using the tf.nn.softmax_cross_entropy_with_logits function instead, or its sparse counterpart. Edit: The SparseCategoricalCrossentropy class also has a keyword argument from_logits=False that can be set to True to the same effect.


Instruction: PyTorch TransformerEncoderLayer different input order gets different results

Input:
Before I start, I’m very new to Transformers, and sorry for by bad sentence structure, I have a fever right now.
Any time I use nn.TransformerEncoderLayer in anyway with a saved model if the data is in a different order I get different results. Is there a way to save the Encode table (or whatever this would be), this would be in the MultiheadAttention part of the TransformerEncoderLayer right?
Just using TransformerEncoderLayer and save the model and then use np.random.permutation() to shuffle the input data. And running the input data through the model after calling model.eval. This always gives me different results unless I use the same order every time.
I have this layer in my model like this
self.transformer = nn.TransformerEncoderLayer()

I save the model like so
torch.save(model, path)

Does this not save the nn.TransformerEncoderLayer() or something?

Output:
Got it, what is the shape of your input, and are you using batch_first=True or not? Basically one thing to just make sure of is that you don’t have your batch and the sequence dimensions mixed up in your implementation.
batch_first – If True, then the input and output tensors are provided as (batch, seq, feature). Default: False (seq, batch, feature).


Instruction: timm: typeError: 'builtin_function_or_method' object is not subscriptable

Input:
I am implementing timm tutorials on data augmentation to increase the number of images of my dataset. According to their tutorials, I have implemented same code but it did not work.
Code
import numpy as np
import torch
from PIL import Image
from timm.data.transforms_factory import create_transform
a = create_transform(224, is_training=True)
print(a)
pets_image_paths = './download.png'
image = Image.open(pets_image_paths)
# We can convert this into a tensor, and transpose the channels into the format that PyTorch expects:
np_image = np.array(image, dtype=np.float32)
image = torch.as_tensor(np_image).transpose(2, 0)[None]
from timm.data.transforms import RandomResizedCropAndInterpolation
tfm = RandomResizedCropAndInterpolation(size=350, interpolation='random')
import matplotlib.pyplot as plt
fig, ax = plt.subplots(2, 4, figsize=(10, 5))
for idx, im in enumerate([tfm(image) for i in range(4)]):
    ax[0, idx].imshow(im)
for idx, im in enumerate([tfm(image) for i in range(4)]):
    ax[1, idx].imshow(im)
fig.tight_layout()
plt.show()

Traceback
Traceback (most recent call last):
  File "/home/cvpr/PycharmProjects/timm_tutorials/9_augmentation.py", line 24, in <module>
    for idx, im in enumerate([tfm(image) for i in range(4)]):
  File "/home/cvpr/PycharmProjects/timm_tutorials/9_augmentation.py", line 24, in <listcomp>
    for idx, im in enumerate([tfm(image) for i in range(4)]):
  File "/home/cvpr/anaconda3/envs/timm_tutorials/lib/python3.8/site-packages/timm/data/transforms.py", line 181, in __call__
    i, j, h, w = self.get_params(img, self.scale, self.ratio)
  File "/home/cvpr/anaconda3/envs/timm_tutorials/lib/python3.8/site-packages/timm/data/transforms.py", line 143, in get_params
    area = img.size[0] * img.size[1]
TypeError: 'builtin_function_or_method' object is not subscriptable

Output:
As the previous answer mentions RandomResizedCropAndInterpolation expects PIL.Image.
You can look at timm docs in the Note:

Note: RandomResizedCropAndInterpolation expects the input to be an instance of PIL.Image and not torch.tensor.

So you can remove the Tensor convert:
import numpy as np
import torch
from PIL import Image
from timm.data.transforms_factory import create_transform
a = create_transform(224, is_training=True)
print(a)
pets_image_paths = './download.png'
image = Image.open(pets_image_paths)

from timm.data.transforms import RandomResizedCropAndInterpolation
tfm = RandomResizedCropAndInterpolation(size=350, interpolation='random')
import matplotlib.pyplot as plt
fig, ax = plt.subplots(2, 4, figsize=(10, 5))
for idx, im in enumerate([tfm(image) for i in range(4)]):
    ax[0, idx].imshow(im)
for idx, im in enumerate([tfm(image) for i in range(4)]):
    ax[1, idx].imshow(im)
fig.tight_layout()
plt.show()


Instruction: PyTorch indexing by argmax

Input:
Dear community I have a challenge with regard to tensor indexing in PyTorch. The problem is very simple. Given a tensor create an index tensor to index its maximum values per column.
x = T.tensor([[0, 3, 0, 5, 9, 8, 2, 0],
              [0, 4, 9, 6, 7, 9, 1, 0]])

Given this tensor I would like to build a boolean mask for indexing its maximum values per colum. To be specific I do not need its maximum values, torch.max(x, dim=0), nor its indices, torch.argmax(x, dim=0), but a boolean mask for indexing other tensor based on this tensor max values. My ideal output would be:
# Input tensor
x
tensor([[0, 3, 0, 5, 9, 8, 2, 0],
        [0, 4, 9, 6, 7, 9, 1, 0]])

# Ideal output bool mask tensor
idx
tensor([[1, 0, 0, 0, 1, 0, 1, 1],
        [0, 1, 1, 1, 0, 1, 0, 0]])

I know that values_max = x[idx] and values_max = x.max(dim=0) are equivalent but I am not looking for values_max but for idx.
I have built a solution around it but it just seem to complex and I am sure torch have an optimized way to do this. I have tried to use torch.index_select with the output of x.argmax(dim=0) but failed so I built a custom solution that seems to cumbersome to me so I am asking for help to do this in a vectorized / tensorial / torch way.

Output:
You can perform this operation by first extracting the index of the maximum value column-wise of your tensor with torch.argmax, setting keepdim to True
>>> x.argmax(0, keepdim=True)
tensor([[0, 1, 1, 1, 0, 1, 0, 0]])

Then you can use torch.scatter to place 1s in a zero tensor at the designated indices:
>>> torch.zeros_like(x).scatter(0, x.argmax(0,True), value=1)
tensor([[1, 0, 0, 0, 1, 0, 1, 1],
        [0, 1, 1, 1, 0, 1, 0, 0]])


Instruction: 'MaskedLMOutput' object has no attribute 'view'

Input:
I wrote this:
def forward(self, x):
    x = self.bert(x)

    x = x.view(x.shape[0], -1)
    x = self.fc(self.dropout(self.bn(x)))
    return x

but it doesn't work well, and the error is 'MaskedLMOutput' object has no attribute 'view'.
I'm considering the input might not be 'tensor' type, so I change it as below:
def forward(self, x):
        x = torch.tensor(x)     # this part
        x = self.bert(x)

        x = x.view(x.shape[0], -1)
        x = self.fc(self.dropout(self.bn(x)))
        return x

but it still gets wrong, same error 'MaskedLMOutput' object has no attribute 'view'.
Could someone tell me how to fix this?  Much thanks.
Whole error information here:
 ------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Input In [5], in <cell line: 8>()
          6 optimizer = optim.Adam(bert_punc.parameters(), lr=learning_rate_top)
          7 criterion = nn.CrossEntropyLoss()
    ----> 8 bert_punc, optimizer, best_val_loss = train(bert_punc, optimizer, criterion, epochs_top,
          9     data_loader_train, data_loader_valid, save_path, punctuation_enc, iterations_top, best_val_loss=1e9)

    Input In [3], in train(model, optimizer, criterion, epochs, data_loader_train, data_loader_valid, save_path, punctuation_enc, iterations, best_val_loss)
         17 inputs.requires_grad = False
         18 labels.requires_grad = False
    ---> 19 output = model(inputs)
         20 loss = criterion(output, labels)
         21 loss.backward()

    File ~\anaconda3\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
       1106 # If we don't have any hooks, we want to skip the rest of the logic in
       1107 # this function, and just call forward.
       1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
       1109         or _global_forward_hooks or _global_forward_pre_hooks):
    -> 1110     return forward_call(*input, **kwargs)
       1111 # Do not call functions when jit is used
       1112 full_backward_hooks, non_full_backward_hooks = [], []

    File ~\anaconda3\lib\site-packages\torch\nn\parallel\data_parallel.py:166, in DataParallel.forward(self, *inputs, **kwargs)
        163     kwargs = ({},)
        165 if len(self.device_ids) == 1:
    --> 166     return self.module(*inputs[0], **kwargs[0])
        167 replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
        168 outputs = self.parallel_apply(replicas, inputs, kwargs)

    File ~\anaconda3\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
       1106 # If we don't have any hooks, we want to skip the rest of the logic in
       1107 # this function, and just call forward.
       1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
       1109         or _global_forward_hooks or _global_forward_pre_hooks):
    -> 1110     return forward_call(*input, **kwargs)
       1111 # Do not call functions when jit is used
       1112 full_backward_hooks, non_full_backward_hooks = [], []

    File D:\BertPunc-original\model.py:21, in BertPunc.forward(self, x)
         18 x = torch.tensor(x)
         19 x = self.bert(x)
    ---> 21 x = x.view(x.shape[0], -1)
         22 x = self.fc(self.dropout(self.bn(x)))
         23 return x

    AttributeError: 'MaskedLMOutput' object has no attribute 'view'

Output:
I think so this should help you solve the error. https://stackoverflow.com/a/72601533/13748930
The output after self.bert(x) is an object of the class MaskedLMOutput.


Instruction: Error in pytorch data loader batch cycles

Input:
I keep getting this error in sagemaker when iterating through pytorch dataloader batch cycles:
Traceback (most recent call last):
  File "main.py", line 371, in <module>
    g_scaler=g_scaler, d_scaler=d_scaler, runtime_log_folder=runtime_log_folder, runtime_log_file_name=runtime_log_file_name)
  File "main.py", line 78, in train_fn
    for idx, (x, y) in enumerate(loop):
  File "/opt/conda/lib/python3.6/site-packages/tqdm/std.py", line 1171, in __iter__
    for obj in iterable:
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 525, in __next__
    (data, worker_id) = self._next_data()
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1252, in _next_data
    return (self._process_data(data), w_id)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1299, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.6/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
  File "/opt/conda/lib/python3.6/site-packages/botocore/exceptions.py", line 84, in __init__
    super(HTTPClientError, self).__init__(**kwargs)
  File "/opt/conda/lib/python3.6/site-packages/botocore/exceptions.py", line 40, in __init__
    msg = self.fmt.format(**kwargs)
KeyError: 'error'

---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<ipython-input-1-81655136a841> in <module>
     58                             py_version='py3')
     59
---> 60 pytorch_estimator.fit({'train': Runtime.dataset_path}, job_name=Runtime.job_name)
     61
     62 #print(pytorch_estimator.latest_job_tensorboard_artifacts_path())

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    955         self.jobs.append(self.latest_training_job)
    956         if wait:
--> 957             self.latest_training_job.wait(logs=logs)
    958
    959     def _compilation_job_name(self):

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
   1954         # If logs are requested, call logs_for_jobs.
   1955         if logs != "None":
-> 1956             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   1957         else:
   1958             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3751
   3752         if wait:
-> 3753             self._check_job_status(job_name, description, "TrainingJobStatus")
   3754             if dot:
   3755                 print()

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3304                 ),
   3305                 allowed_statuses=["Completed", "Stopped"],
-> 3306                 actual_status=status,
   3307             )
   3308

UnexpectedStatusException: Error for Training job 2022-06-03-05-16-49-pix2pix-U12239-2022-05-09-14-39-18-training: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 main.py --runtime_var dataset_name=U12239-2022-05-09-14-39-18,job_name=2022-06-03-05-16-49-pix2pix-U12239-2022-05-09-14-39-18-training,model_name=pix2pix"

  0%|          | 0/248 [00:00<?, ?it/s]
  0%|          | 1/248 [00:30<2:07:28, 30.97s/it]
  0%|          | 1/248 [00:30<2:07:28, 30.97s/it]
Traceback (most recent call last):
  File "main.py", line 371, in <module>
    g_scaler=g_scaler, d_scaler=d_scaler, runtime_log_folder=runtime_log_folder, runtime_log_file_name=runtime_log_file_name)
  File "main.py", line 78, in train_fn
    for idx, (x, y) in enumerate(loop):
  File "/opt/conda/lib/python3.6/site-packages/tqdm/std.py", line 1171, in __iter__
    for obj in iterable:
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 525, in __next__
    (data, worker_id) = self._next_data()
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1252, in _next_data
    return (self

Here is the code which results in the error:
def train_fn(disc, gen, loader, opt_disc, opt_gen, l1, bce, g_scaler, d_scaler,runtime_log_folder,runtime_log_file_name):

    total_output=''

    loop = tqdm(loader, leave=True)
    device = "cuda" if torch.cuda.is_available() else "cpu"

    print("Loop")
    print(loop)
    print("Length loop")
    print(len(loop))
    for idx, (x, y) in enumerate(loop): #<--error happens here
        print("Loop index")
        print(idx)
        print("Loop item")
        print(x,y)
        x = x.to(device)
        y = y.to(device)

        # train discriminator
        with torch.cuda.amp.autocast():
            y_fake = gen(x)

            D_real = disc(x, y)
            D_fake = disc(x, y_fake.detach())
            # use detach so as to avoid breaking computational graph when do optimizer.step on discriminator
            # can use detach, or when do loss.backward put loss.backward(retain_graph = True)

            D_real_loss = bce(D_real, torch.ones_like(D_real))
            D_fake_loss = bce(D_fake, torch.ones_like(D_fake))

            D_loss = (D_real_loss + D_fake_loss) / 2

            # log tensorboard

        disc.zero_grad()
        d_scaler.scale(D_loss).backward()
        d_scaler.step(opt_disc)
        d_scaler.update()

        # train generator
        with torch.cuda.amp.autocast():

            D_fake = disc(x, y_fake)

            # compute fake loss
            # trick discriminator to believe these are real, hence send in torch.oneslikedfake
            G_fake_loss = bce(D_fake, torch.ones_like(D_fake))

            # compute L1 loss
            L1 = l1(y_fake, y) * args.l1_lambda

            G_loss = G_fake_loss + L1

            # log tensorboard

        opt_gen.zero_grad()
        g_scaler.scale(G_loss).backward()
        g_scaler.step(opt_gen)
        g_scaler.update()

        # print epoch, generator loss, discriminator loss
        print(f'[Epoch {epoch}/{args.num_epochs} (b: {idx})] [D loss: {D_loss}, D real loss: {D_real_loss}, D fake loss: {D_fake_loss}] [G loss: ##{G_loss}, G fake loss: {G_fake_loss}, L1 loss: {L1}]')
        output = f'[Epoch {epoch}/{args.num_epochs} (b: {idx})] [D loss: {D_loss}, D real loss: {D_real_loss}, D fake loss: {D_fake_loss}] [G loss: ##{G_loss}, G fake loss: {G_fake_loss}, L1 loss: {L1}]\n'
        total_output+=output

    runtime_log = get_json_file_from_s3(runtime_log_folder, runtime_log_file_name)
    runtime_log += total_output
    upload_json_file_to_s3(runtime_log_folder,runtime_log_file_name,json.dumps(runtime_log))

def __getitem__(self, index):
    print("Index ",index)
    pair_key = self.list_files[index]
    print("Pair key ",pair_key)
    pair = Boto.s3_client.list_objects(Bucket=Boto.bucket_name, Prefix=pair_key, Delimiter='/')

    input_image_key = pair.get('Contents')[1].get('Key')
    input_image_path = f's3://{Boto.bucket_name}/{input_image_key}'
    print("Input image path ",input_image_path)
    input_image_s3_source = get_file_from_filepath(input_image_path)
    input_image = np.array(Image.open(input_image_s3_source))

    target_image_key = pair.get('Contents')[0].get('Key')
    target_image_path = f's3://{Boto.bucket_name}/{target_image_key}'
    print("Target image path ",target_image_path)
    target_image_s3_source = get_file_from_filepath(target_image_path)
    target_image = np.array(Image.open(target_image_s3_source))

    augmentations = config.both_transform(image=input_image, image0=target_image)

    # get input image and target image by doing augmentations of images
    input_image, target_image = augmentations['image'], augmentations['image0']

    input_image = config.transform_only_input(image=input_image)['image']
    target_image = config.transform_only_mask(image=target_image)['image']

    print("Input image size ",input_image.size())
    print("Target image size ",target_image.size())

    return input_image, target_image

I did multiple runs and here are the traces of the failure points
i) 2022-06-03-05-00-04-pix2pix-U12239-2022-05-09-14-39-18-training
No index shown
[Epoch 0/100 (b: 0)]

ii) 2022-06-03-05-16-49-pix2pix-U12239-2022-05-09-14-39-18-training
Index  160
[Epoch 0/100 (b: 0)]

iii) 2022-06-03-05-44-46-pix2pix-U12239-2022-05-09-14-39-18-training
Index  160
[Epoch 0/100 (b: 0)]

iv) 2022-06-03-06-08-33-pix2pix-U12239-2022-05-09-14-39-18-training
Index  160
[Epoch 1/100 (b: 0)]

v) 2022-06-15-02-49-20-pix2pix-U12239-2022-05-09-14-39-18-training
Index  160
Pair key  datasets/training-data/testing/2022-05-09-14-39-18/match-raws-finals/U12239/P423712/Pair_71/
[Epoch 0/100 (b: 0)

vi) 2022-06-15-02-59-43-pix2pix-U12239-2022-05-09-14-39-18-training
Index  64
Pair key  datasets/training-data/testing/2022-05-09-14-39-18/match-raws-finals/U12239/P425642/Pair_27/
[Epoch 0/100 (b: 247)]

vii) 2022-06-15-04-49-33-pix2pix-U12239-2022-05-09-14-39-18-training
Index  64
Pair key  datasets/training-data/testing/2022-05-09-14-39-18/match-raws-finals/U12239/P415414/Pair_124/
No specific epoch

My batch size is 248, so as you can see it seems to fail either at the start of the batch (0) or at the end (247). Also there are some common Indexes in the get item which seems to cause it to fail, namely Index 64 and Index 160. However there doesn't seem to be a common data point in the dataset that causes it to fail, as can be seen from the pair key all 3 data points in the datasets are different.
Does anyone have any idea why this error happens please?

Output:
Try to run the same training script outside of a SageMaker training job and see what happens.
If the error doesn't happen on a standalone script, try to run it as a Local SageMaker training job, so you can reproduce it in seconds instead of minutes, and potentially use a debugger to figure out what is the problem.


Instruction: Is there any shuffle mode in training Pytorch neural network model?

Input:
I'm using the code below to train a simple neural net to learn a harmonic wave by PyTorch. But I want to turn the shuffle mode on to improve the model. Is there any syntax to this aim?
model = FCN(1,1,50,4)
optimizer = torch.optim.Adam(model.parameters(),lr=15e-3, weight_decay=15e-3/4000)

for i in range(4000):
    optimizer.zero_grad()
    yhh = model(x_data)
    loss = torch.mean((yhh-y_data)**2)
    loss.backward()
    optimizer.step()

Also, I used the code below alternatively to reorder the learning set randomly, but the result was awful.
yhh = model(x_data[[np.random.choice(range(len(x_data)), len(x_data), replace=False)]])

Output:
Assuming your x_data is plain Torch tensor of size, say [100], you can use torch.utils.data.DataLoader with shuffle=True to shuffle x_data after each epoch:
dataset = torch.utils.data.TensorDataset(x_data)  # first create a dataset wrapping your tensor
dataloader = torch.utils.data.DataLoader(dataset, batch_size=bs, shuffle=True)  # specify your required batch size

dataloader object is now an iterable and can be used as:
for data in dataloader:
    model(data[0])  #data[0] is a tensor of size(bs)