Instruction: Custom data loader is returning list in pytorch

Input:
I want to get 3 batches of images from 3 different folders. I have written custom data loader in pytorch. but it is returning list that has all the batches instead of single batch at a time.(running in google colab)

#custom data loader
class set(Dataset):
    def __init__(self, dataset_input, dataset_expertA, dataset_expertB):
        self.dataset1 = dataset_input
        self.dataset2 = dataset_expertA
        self.dataset3 = dataset_expertB

    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]
        x3 = self.dataset3[index]

        return x1, x2, x3

    def __len__(self):
        return len(self.dataset1)

input_path = "/content/gdrive/My Drive/project/input/"

dataset = datasets.ImageFolder(root= input_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

expertA_path = "/content/gdrive/My Drive/project/expertA/"

datasetA = datasets.ImageFolder(root= expertA_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

expertB_path = "/content/gdrive/My Drive/project/expertB/"

datasetB = datasets.ImageFolder(root= expertB_path, transform=transforms.Compose([
                               transforms.Resize([64,64]),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ]))

data = set(dataset, datasetA, datasetB)
dataloader = torch.utils.data.DataLoader(data, batch_size=64,
                                         shuffle=True, num_workers=2)

for i, (inp, expA, expB) in enumerate(dataloader):

  print(inp.shape)
  break

this prints error that inp is list and when i print(inp[0].shape) i get proper shape i think inp contains all batches ie inp[0], inp[1]...

what mistake am i doing in data loader code?

Output:
datasets.ImageFolder returns a tuple of (image, label), hence inp is also a tuple, where inp[0] are the images and inp[1] their corresponding labels. The same applies to expA and expB.

If you only want the images without the labels, you can ignore the labels and just return the images when accessing the data in your custom dataset:

def __getitem__(self, index):
    image1, label1 = self.dataset1[index]
    image2, label2 = self.dataset2[index]
    image3, label3 = self.dataset3[index]

    return image1, image2, image3


Instruction: Pytorch: Dimensions for cross entropy is correct but somehow wrong for MSE?

Input:
I was creating a program that would take in as input the Fashion MNIST set and I was tweaking around with my model to see how different parameters would change the accuracy.

One of the tweaks I made to my model was to change my model's loss function from cross entropy to MSE.

# The code above is miscellaneous training data import code

trainloader = torch.utils.data.DataLoader(trainset, batch_size = 64, shuffle = True, num_workers=4)
testloader = torch.utils.data.DataLoader(testset, batch_size = 64, shuffle = True, num_workers=4)

dataiter = iter(trainloader)
images, labels = dataiter.next()
from torch import nn, optim
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim = 1)
                     )
model.to(device)

# Define the loss
criterion = nn.MSELoss()

# Define the optimizer
optimizer = optim.Adam(model.parameters(), lr = 0.001)

# Define the epochs
epochs = 5

train_losses, test_losses = [], []

for e in range(epochs):
  running_loss = 0
  for images, labels in trainloader:
    # Flatten Fashion-MNIST images into a 784 long vector
    images = images.to(device)
    labels = labels.to(device)
    images = images.view(images.shape[0], -1)

    # Training pass
    optimizer.zero_grad()

    output = model.forward(images)

    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()

My model worked without any problems when using cross entropy loss, but when I changed to MSE loss, the interpreter complained and said that my tensors were different sizes and thus could not be computed.

<class 'torch.Tensor'>
torch.Size([64, 1, 28, 28])
torch.Size([64])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-62-ec6942122f02> in <module>
     44     output = model.forward(images)
     45
---> 46     loss = criterion(output, labels)
     47     loss.backward()
     48     optimizer.step()

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    429
    430     def forward(self, input, target):
--> 431         return F.mse_loss(input, target, reduction=self.reduction)
    432
    433

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2213             ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
   2214     else:
-> 2215         expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   2216         ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   2217     return ret

/opt/conda/lib/python3.7/site-packages/torch/functional.py in broadcast_tensors(*tensors)
     50                 [0, 1, 2]])
     51     """
---> 52     return torch._C._VariableFunctions.broadcast_tensors(tensors)
     53
     54

RuntimeError: The size of tensor a (10) must match the size of tensor b (64) at non-singleton dimension 1

I tried reshaping my tensors and creating new arrays as placeholders for my output array, yet seem to be getting nowhere.

Why cross entropy loss works without any errors yet MSE does not?

Output:
nn.CrossEntropyLoss and nn.MSELoss are completely different loss functions with fundamentally different rationale behind them.

nn.CrossEntropyLoss is a loss function for discrete labeling tasks. Therefore it expects as inputs a prediction of label probabilities and targets as ground-truth discrete labels: x shape is nxc (where c is the number of labels) and y is of shape n of type integer, each target takes values in the range {0,...,c-1}.

In contrast, nn.MSELoss is a loss function for regression tasks. Therefore it expects both predictions and targets to be of the same shape and data type. That is, if your prediction is of shape nxc the target should also be of shape nxc (and not just n as in the cross-entropy case).

If you are insisting on using MSE loss instead of cross entropy, you will need to convert the target integer labels you currently have (of shape n) into 1-hot vectors of shape nxc and only then compute the MSE loss between your predictions and the generated one-hot targets.


Instruction: Why does output shape in a simple Elman RNN depend on the sequence length(while hidden state shape doesn't)?

Input:
I am learning about RNNs, and am trying to code one up using PyTorch.
I have some trouble understanding the output dimensions

Here is some code for a simple RNN architecture

class RNN(nn.Module):
    def __init__(self, input_size, hidden_dim, n_layers):
        super(RNN, self).__init__()
        self.hidden_dim=hidden_dim
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)

    def forward(self, x, hidden):
        r_out, hidden = self.rnn(x, hidden)

        return r_out, hidden

So, what I understand is the hidden_dim is the number blocks I will have in my hidden layer, and essentially the number of features in the output and in the hidden state.

I create some dummy data, to test it

test_rnn = RNN(input_size=1, hidden_dim=4, n_layers=1)

# generate evenly spaced, test data pts
time_steps = np.linspace(0, 6, 3)
data = np.sin(time_steps)
data.resize((3, 1))

test_input = torch.Tensor(data).unsqueeze(0) # give it a batch_size of 1 as first dimension
print('Input size: ', test_input.size())

# test out rnn sizes
test_out, test_h = test_rnn(test_input, None)
print('Hidden state size: ', test_h.size())
print('Output size: ', test_out.size())

What I get is

Input size:  torch.Size([1, 3, 1])
Hidden state size:  torch.Size([1, 1, 4])
Output size:  torch.Size([1, 3, 4])

So I understand that the shape of x is determined like so
x = (batch_size, seq_length, input_size).. so 1 bath size, and input of 1 feature and 3 time steps(sequence length).

For hidden state, like so hidden = (n_layers, batch_size, hidden_dim).. so I had 1 layer, batch size 1, and 4 blocks in my hidden layer.

What I don't get is the RNN output. From the documentation, r_out = (batch_size, time_step, hidden_size).. Wasn't the output supposed to be the same as the hidden state that was output from the hidden units? That is, if I have 4 units in my hidden layer, I would expect it to output 4 numbers for the hidden state, and 4 numbers for the output. Why is the time step a dimension of the output? Because, each hidden unit, takes in some numbers, outputs a state S and output Y, and both of these are equal, yes? I tried a diagram, this is what I came up with. Help me understand what part of it I'm doing wrong.

So TL;DR
Why does output shape in a simple Elman RNN depend on the sequence length(while hidden state shape doesn't)? For in the diagram I drew, I see both of them being the same.

Output:
In the PyTorch API, the output is a sequence of hidden states during the RNN computation, i.e., there is one hidden state vector per input vector. The hidden state is the last hidden state, the state the RNN ends with after processing the input, so test_out[:, -1, :] = test_h.

Vector y in your diagrams is the same as a hidden state Ht, it indeed has 4 numbers, but the state is different for every time step, so you have 4 number for every time step.

The reason why PyTorch separates the sequence of outputs = hidden states (it's not the same in LSTMs, though) is that you can have a batch of sequences of different lengths. In that case, the final state is not simply test_out[:, -1, :], because you need to select final states based on the lengths of individual sequences.


Instruction: RuntimeError: Expected 4-dimensional input for 4-dimensional weight X, but got 3-dimensional input of size Y instead

Input:
I am building a CNN to do image classification on the EMNIST dataset.

To do so, I have the following datasets:

import scipy .io
emnist = scipy.io.loadmat(DIRECTORY + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

X_test = data ['test'][0, 0]['images'][0, 0]
X_test = X_test.reshape((-1,28,28), order = 'F')

y_test = data ['test'][0, 0]['labels'][0, 0]

With shape:

X_train = (124800, 28, 28)
y_train = (124800, 1)
X_test = (20800, 28, 28)
y_test = (20800, 1)

Note that the pictures are grayscale, so the colors are represented with only one number.

Which I prepare further as follows:

train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
test_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

My model looks as follows:

class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()

        self.cnn_layers = Sequential(
            # Defining a 2D convolution layer
            Conv2d(1, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
            # Defining another 2D convolution layer
            Conv2d(4, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
        )

        self.linear_layers = Sequential(
            Linear(4 * 7 * 7, 10)
        )

    # Defining the forward pass
    def forward(self, x):
        x = self.cnn_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

model = CNNModel()

The code below is part of the code that I use to train my model:

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

    images = Variable(images)
    labels = Variable(labels)

    # Forward pass to get output/logits
    outputs = model(images)

However, by excuting my code, I get the following error:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [4, 1, 3, 3], but got 3-dimensional input of size [100, 28, 28] instead

So a 4D input is expected while my input is 3D. What can I do so a 3D model is expected rather than a 4D model?

Here a similar question is asked, however I dont see how I can translate this to my code

Output:
The convolution expects the input to have size [batch_size, channels, height, width], but your images have size [batch_size, height, width], the channel dimension is missing. Greyscale is represented with a single channel and you have correctly set the in_channels of the first convolutions to 1, but your images don't have the matching dimension.

You can easily add the singular dimension with torch.unsqueeze.

Also, please don't use Variable, it was deprecated with PyTorch 0.4.0, which was released over 2 years ago, and all of its functionality has been merged into the tensors.

for i, (images, labels) in enumerate(train_loader):
    # Add a single channel dimension
    # From: [batch_size, height, width]
    # To: [batch_size, 1, height, width]
    images = images.unsqueeze(1)

    # Forward pass to get output/logits
    outputs = model(images)


Instruction: RuntimeError: value cannot be converted to type uint8_t without overflow: -0.192746

Input:
I am new to Pytorch and am aiming to do an image classification task using a CNN based on the EMNIST dataset.
I read my data in as follows:
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

X_test = data ['test'][0, 0]['images'][0, 0]
X_test = X_test.reshape((-1,28,28), order = 'F')

y_test = data ['test'][0, 0]['labels'][0, 0]

train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
test_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))

batch_size = 128
n_iters = 3000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

Then, I found the following configurations (that I still have to adjust to fit to my data):
class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()

        # Convolution 1
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)
        self.relu1 = nn.ReLU()

        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(kernel_size=2)

        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)
        self.relu2 = nn.ReLU()

        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        # Fully connected 1 (readout)
        self.fc1 = nn.Linear(32 * 4 * 4, 10)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x)
        out = self.relu1(out)

        # Max pool 1
        out = self.maxpool1(out)

        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)

        # Max pool 2
        out = self.maxpool2(out)

        # Resize
        # Original size: (100, 32, 7, 7)
        # out.size(0): 100
        # New out size: (100, 32*7*7)
        out = out.view(out.size(0), -1)

        # Linear function (readout)
        out = self.fc1(out)

        return out

model = CNNModel()

criterion = nn.CrossEntropyLoss()

To train the model, I use the following code:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, when I run this, I get the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-1fbdd53d1194> in <module>()
     12
     13         # Forward pass to get output/logits
---> 14         outputs = model(images)
     15
     16         # Clear gradients w.r.t. parameters

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    348                             _pair(0), self.dilation, self.groups)
    349         return F.conv2d(input, weight, self.bias, self.stride,
--> 350                         self.padding, self.dilation, self.groups)
    351
    352     def forward(self, input):

RuntimeError: value cannot be converted to type uint8_t without overflow: -0.0510302

I found this question already and think that the solution might work for me as well. However, I don't understand where in my code I can implement this.
What can I do to overcome this problem?
Ps.
I have used the following import statements:
    import scipy .io
    import torch
    import torch.nn as nn
    import torchvision.transforms as transforms
    import torchvision.datasets as dsets
    from torch.autograd import Variable
    import cv2
    import torch
    import torch.nn as nn
    import torch.optim as optim
    import torch.nn.functional as F
    import numpy as np
    import os
    from PIL import Image
    from PIL import ImageOps
    from torchvision import datasets, transforms
    from torch.autograd import Variable
    import matplotlib.pyplot as plt
    from torchvision.datasets import ImageFolder
    from torch.utils.data import DataLoader
    from torchvision.transforms import ToTensor
    from torch.nn import Sequential
    from torch.nn import Conv2d
    from torch.nn import BatchNorm2d
    from torch.nn import MaxPool2d
    from torch.nn import ReLU
    from torch.nn import Linear

Output: What fixed my problem was replacing out = self.cnn1(x) with out = self.cnn1(x.float())


Instruction: Python, class dataset, how to concatenate images with their respective labels in pytorch

Input:
I am new to PyTorch, and in the last couple of days I have been struggling with the class Dataset that lets you build your custom dataset.

I am working with this dataset (https://www.kaggle.com/ianmoone0617/flower-goggle-tpu-classification/kernels) , the problem is that it has the images and their labels in separate folders, and I can’t figure out how to concatenate them.

This is the code I am using:

class MyDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        self.labels = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.tolist()

        image_name = os.path.join(self.root_dir, self.labels.iloc[index, 0])
        image = io.imread(image_name)

        if self.transform:
            image = self.transform(image)

        return (image, labels)

While the structure of the folders is the following one:

I really want to understand this so thank you in advance guys!!

Output:
Seems like you're nearly there. There are many ways to deal with this. For example, you could read both csv files during initialization to build a dictionary which maps the label string in the flowers_idx.csv to the label index specified in flowers_label.csv.

import os
import pandas as pd
import torch
from torchvision.datasets.folder import default_loader
from torch.utils.data import Dataset

class MyDataset(Dataset):
    def __init__(self, data_csv, label_csv, root_dir, transform=None):
        self.data_entries = pd.read_csv(data_csv)
        self.root_dir = root_dir
        self.transform = transform

        label_map = pd.read_csv(label_csv)
        self.label_str_to_idx = {label_str: label_idx for label_idx, label_str in label_map.iloc}

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.item()

        label = self.label_str_to_idx[self.data_entries.iloc[index, 1]]
        image_path = os.path.join(self.root_dir, f'{self.data_entries.iloc[index, 0]}.jpeg')

        # torchvision datasets generally return PIL image rather than numpy ndarray
        image = default_loader(image_path)

        # alternative to load ndarray using skimage.io
        # image = io.imread(image_path)

        if self.transform:
            image = self.transform(image)

        return (image, label)

Note that this returns PIL images rather than ndarrays since that's generally what is returned by torchvision datasets. This is also nice since many of the torchvision transforms can only be appled to PIL images.

For now a simple use case could be:

import torchvision.transforms as tt

dataset_dir = '/home/jodag/datasets/527293_966816_bundle_archive'
# TODO add more transforms/data-augmentation etc...
transform = tt.Compose((
    tt.ToTensor(),
))

dataset = MyDataset(
    os.path.join(dataset_dir, 'flowers_idx.csv'),
    os.path.join(dataset_dir, 'flowers_label.csv'),
    os.path.join(dataset_dir, 'flower_tpu/flower_tpu/flowers_google/flowers_google'),
    transform)

image, label = dataset[0]

During training or validation you would probably use a DataLoader to sample the dataset.


Instruction: Getting started: Huggingface Model Cards

Input:
I just recently started looking into the huggingface transformer library.
When I tried to get started using the model card code at e.g. community model

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

However, I got the following error:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
  File "/Users/Lukas/miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/tokenization_auto.py", line 124, in from_pretrained
    "'xlm', 'roberta', 'ctrl'".format(pretrained_model_name_or_path))
ValueError: Unrecognized model identifier in emilyalsentzer/Bio_ClinicalBERT. Should contains one of 'bert', 'openai-gpt', 'gpt2', 'transfo-xl', 'xlnet', 'xlm', 'roberta', 'ctrl'

If I try a different tokenizer such as "baykenney/bert-base-gpt2detector-topp92" I get the following error:

OSError: Model name 'baykenney/bert-base-gpt2detector-topp92' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'baykenney/bert-base-gpt2detector-topp92' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

Did I miss anything to get started? I feel like the model cards indicate that these three lines of code should should be enough to get started.

I am using Python 3.7 and the transformer library version 2.1.1 and pytorch 1.5.

Output: Please update your transformers library to at least 2.4.0. You should create a new conda environment and install all your packages directly from pypi with pip to get the most recent version (currently 2.11.0).


Instruction: Where in the code of pytorch or huggingface/transformer label gets "renamed" into labels?

Input:
My question concerns the example, available in the great huggingface/transformers library.
I am using a notebook, provided by library creators as a starting point for my pipeline. It presents a pipeline of finetuning a BERT for Sentence Classification on Glue dataset.
When getting into the code, I noticed a very weird thing, which I cannot explain.
In the example, input data is introduced to the model as the instances of the InputFeatures class from here:
This class has 4 attributes, including the label attribute:
class InputFeatures:
    ...
    input_ids: List[int]
    attention_mask: Optional[List[int]] = None
    token_type_ids: Optional[List[int]] = None
    label: Optional[Union[int, float]] = None

which are later passed as a dictionary of inputs to the forward() method of the model. This is done by the Trainer class, for example in the lines 573-576 here:
    def _training_step(
        self, model: nn.Module, inputs: Dict[str, torch.Tensor], optimizer: torch.optim.Optimizer
    ) -> float:
        model.train()
        for k, v in inputs.items():
            inputs[k] = v.to(self.args.device)

        outputs = model(**inputs)

However, the forward() method expects labels (note the plural form) input parameter (taken from here):
    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
    ):

So my question is where does the label become labels in this pipeline?
To give some extra info on the issue, I created my own pipeline, which uses nothing, related, with Glue data and pipe, basically it relies only on the Trainer class of transformers. I even use another model (Flaubert). I replicated the InputFeature class and my code works for both cases below:
class InputFeature:
    def __init__(self, text, label):
        self.input_ids = text
        self.label = label

class InputFeaturePlural:
    def __init__(self, text, label):
        self.input_ids = text
        self.labels = label

But it does not work if I name the second attribute as self.labe or by any other names. Why is it possible to use both attribute names?
It's not like it is extremely important in my case, but I feel uncomfortable passing around the data in the variable, which "changes name" somewhere along the way.

Output:
The rename happens in the collator. In the trainer init, when data_collator is None, a default one is used:

class Trainer:
    # ...
    def __init__(...):
        # ...
        self.data_collator = data_collator if data_collator is not None else default_data_collator
        # ...

FYI, the self.data_collator is later used when you get the dataloader:

data_loader = DataLoader(
    self.train_dataset,
    batch_size=self.args.train_batch_size,
    sampler=train_sampler,
    collate_fn=self.data_collator,              # <-- here
    drop_last=self.args.dataloader_drop_last,
)

The default collator has a special handling for labels, which does this renaming, if needed:

# Special handling for labels.
# Ensure that tensor is created with the correct type
# (it should be automatically the case, but let's make sure of it.)
if hasattr(first, "label") and first.label is not None:
    if type(first.label) is int:
        labels = torch.tensor([f.label for f in features], dtype=torch.long)
    else:
        labels = torch.tensor([f.label for f in features], dtype=torch.float)
    batch = {"labels": labels}  # <-- here is where it happens
elif hasattr(first, "label_ids") and first.label_ids is not None:
    if type(first.label_ids[0]) is int:
        labels = torch.tensor([f.label_ids for f in features], dtype=torch.long)
    else:
        labels = torch.tensor([f.label_ids for f in features], dtype=torch.float)
    batch = {"labels": labels}
else:
    batch = {}


Instruction: PyTorch extracting tensor elements with boolean mask (retaining dimensions)

Input:
Say, I have a PyTorch 2x2 tensor, and I also generated a boolean tensor of the same dimension (2x2). I want to use this as a mask.

For example, if I have a tensor:

tensor([[1, 3],
        [4, 7]])

And if my mask is:

tensor([[ True, False],
        [False,  True]])

I want to use that mask to get a tensor where elements corresponding to True from my original tensor are retained, whereas elements corresponding to False are set to zero in the output tensor.

Expected Output:

tensor([[1, 0],
        [0, 7]])

Any help is appreciated. Thanks!

Output:
Assume you have :

t = torch.Tensor([[1,2], [3,4]])
mask = torch.Tensor([[True,False], [False,True]])

You can use the mask by:

masked_t = t * mask

and the output will be:

tensor([[1., 0.],
        [0., 4.]])


Instruction: Extending PyTorch nn.Sequential class

Input:
I'm pretty new to OOP in Python, and rusty in general. I'd like to extend PyTorch's 'nn.Sequential' object in such a way that passing it a tuple of containing the number of node in each layer automatically generates an OrderedDict according to those nodes. For a functional example:

layers = (784, 392, 196, 98, 10)
n_layers = len(layers)
modules = OrderedDict()

# Layer definitions for inner layers:
for i in range(n_layers - 2):
    modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
    modules[f'relu{i}'] = nn.ReLU()

# Definition for output layer:
modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
modules['smax_out'] = nn.LogSoftmax(dim=1)

# Define model and check attributes:
model = nn.Sequential(modules)

So, rather than pass the 'OrderedDict' object when initializing nn.Sequential, I want my class to take the tuple instead.

class Network(nn.Sequential):
   def__init__(self, n_nodes):
      super().__init__()

      **** INSERT LOGIC FROM LAST SNIPPET ***

So it seems like this won't work, because when my Network class calls super().__init__(), it's going to want to see the dictionary of layer activations. How might I go about writing my own network in such a way that it circumvents this problem, but still has all the functionality of PyTorche's sequential object?

I was thinking along the lines of something like:

class Network(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))

    def init_modules(self, layers):
        n_layers = len(layers)
        modules = OrderedDict()

        # Layer definitions for inner layers:
        for i in range(n_layers - 2):
            modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
            modules[f'relu{i}'] = nn.ReLU()

        # Definition for output layer:
        modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
        modules['smax_out'] = nn.LogSoftmax(dim=1)

        return modules

I'm not sure if this sort of thing is allowed and/or good practice in Python.

Output:
Your implementation is allowed and good.

And, you can also initilize super().__init__() vacant, then use self.add_module(key, module) in a loop to attach Linear or Relu or whatever else subsequently. In this way the function __init__ may cover the job of init_modules.


Instruction: When extending this class, one of its methods now requires an extra argument. Why?

Input:
When I want to do a forward pass with an initialized model = nn.Sequential object, I simply use:

out = model(X)
# OR
out = model.forward(X)

However, I have tried extending the Sequential class, and now both of these methods suddenly require a second argument. For example, note in the following method my call to self(x):

def train(self, trainloader, epochs):
        for e in range(epochs):
            for x, y in trainloader:
                x = x.view(x.shape[0], -1)
                self.optimizer.zero_grad()
                loss = self.criterion(self(x), y) # CALL OCCURS HERE
                loss.backward()
                self.optimizer.step()

This code now gives me TypeError: forward() missing 1 required positional argument: 'target'.

My Question: Since I have done nothing but extend the class, why is this?

Code for full class below:

class Network(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))
        self.criterion = nn.NLLLoss()
        self.optimizer = optim.Adam(self.parameters(), lr=0.003)

    def init_modules(self, layers):

        n_layers = len(layers)
        modules = OrderedDict()

        # Layer definitions for input and inner layers:
        for i in range(n_layers - 2):
            modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
            modules[f'relu{i}'] = nn.ReLU()

        # Definition for output layer:
        modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
        modules['smax_out'] = nn.LogSoftmax(dim=1)

        return modules

    def train(self, trainloader, epochs):
        for e in range(epochs):
            for x, y in trainloader:
                x = x.view(x.shape[0], -1)
                self.optimizer.zero_grad()
                loss = self.criterion(self(x), y)
                loss.backward()
                self.optimizer.step()

Full stack trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-63-490e0b9eef22> in <module>
----> 1 model2.train(trainloader, 5, plot_loss=True)

<ipython-input-61-e173e5672f18> in train(self, trainloader, epochs, plot_loss)
     32                 x = x.view(x.shape[0], -1)
     33                 self.optimizer.zero_grad()
---> 34                 loss = self.criterion(self(x), y)
     35                 loss.backward()
     36                 self.optimizer.step()

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\program files\python38\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'target'

Output:
Q: Since I have done nothing but extend the class, why is this?

Actually, you have. You chose the "wrong" base class. The forward of the nn.Sequential simply goes through all modules, and when you defined:

self.criterion = nn.NLLLoss()

you registered the loss as a module. Therefore, when you call self(x), you're actually calling self.criterion(x) at some point, hence the TypeError.


Instruction: Detectron2 - Extract region features at a threshold for object detection

Input:
I am trying to extract region features where class detection is higher than some threshold using the detectron2 framework. I will be using these features later in my pipeline (similar to: VilBert section 3.1 Training ViLBERT) So far I have trained a Mask R-CNN with this config and fine-tuned it on some custom data. It performs well. What I would like to do is extract the features from my trained model for the produced bounding box.

EDIT: I looked at what the users who closed my post wrote and tried to refine it. Although the reader needs context as to what I am doing. If you have any idea on how I can make the question better or if you have some insight as to how to do what I am trying to do your feedback is welcome!

I have a question:

Why am I only getting one prediction instance, but when I look
at the prediction CLS scores there are more than 1 which passes the
threshold?

I believe this is the correct way of producing the ROI features:

images = ImageList.from_tensors(lst[:1], size_divisibility=32).to("cuda")  # preprocessed input tensor
#setup config
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (pnumonia)
#Just run these lines if you have the trained model im memory
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set the testing threshold for this model
#build model
model = build_model(cfg)
DetectionCheckpointer(model).load("output/model_final.pth")
model.eval()#make sure its in eval mode

#run model
with torch.no_grad():
    features = model.backbone(images.tensor.float())
    proposals, _ = model.proposal_generator(images, features)
    instances = model.roi_heads._forward_box(features, proposals)

Then

pred_boxes = [x.pred_boxes for x in instances]
rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], pred_boxes)

This should be my ROI features.

What I am very confused about is instead of using the bounding boxes produced at inference I could use the proposals and the proposal_boxes with their class scores to get the top n features for this image. Cool so I have tried the following:

proposal_boxes = [x.proposal_boxes for x in proposals]
proposal_rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], proposal_boxes)
#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
box_features = model.roi_heads.box_head(proposal_rois)
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, losses = model.roi_heads.box_predictor.inference(predictions, proposals)

Where I should be getting my proposal box features and its cls in my predictions object. Inspecting this predictions object I see the scores for each box:

CLS Scores in Predictions object

(tensor([[ 0.6308, -0.4926],
         [-1.6662,  1.5430],
         [-0.2080,  0.4856],
         ...,
         [-6.9698,  6.6695],
         [-5.6361,  5.4046],
         [-4.4918,  4.3899]], device='cuda:0', grad_fn=<AddmmBackward>),

After softmaxing and placing these cls scores in a dataframe and setting a threshold of 0.6 I get:

pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
pred_df[pred_df[0] > 0.6]
    0           1
0   0.754618    0.245382
6   0.686816    0.313184
38  0.722627    0.277373

and in my predictions object I get the same top score, but only 1 instance rather than 2 (I set cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7):

Prediction Instances:

[Instances(num_instances=1, image_height=800, image_width=800, fields=[pred_boxes: Boxes(tensor([[548.5992, 341.7193, 756.9728, 438.0507]], device='cuda:0',
        grad_fn=<IndexBackward>)), scores: tensor([0.7546], device='cuda:0', grad_fn=<IndexBackward>), pred_classes: tensor([0], device='cuda:0')])]

The predictions also contain Tensor: Nx4 or Nx(Kx4) bounding box regression deltas. which I don't exactly know what they do and look like:

Bounding box regression deltas in Predictions object

tensor([[ 0.2502,  0.2461, -0.4559, -0.3304],
        [-0.1359, -0.1563, -0.2821,  0.0557],
        [ 0.7802,  0.5719, -1.0790, -1.3001],
        ...,
        [-0.8594,  0.0632,  0.2024, -0.6000],
        [-0.2020, -3.3195,  0.6745,  0.5456],
        [-0.5542,  1.1727,  1.9679, -2.3912]], device='cuda:0',
       grad_fn=<AddmmBackward>)

Something else strange is that my proposal boxes and my prediction boxes are different but similar:

Proposal bounding boxes

[Boxes(tensor([[532.9427, 335.8969, 761.2068, 438.8086],#this box vs the instance box
         [102.7041, 352.5067, 329.4510, 440.7240],
         [499.2719, 317.9529, 764.1958, 448.1386],
         ...,
         [ 25.2890, 379.3329,  28.6030, 429.9694],
         [127.1215, 392.6055, 328.6081, 489.0793],
         [164.5633, 275.6021, 295.0134, 462.7395]], device='cuda:0'))]

Output:
You are almost there. Looking at roi_heads.box_predictor.inference() you will see that it doesn't simply sort the scores of the box candidates. First, it applies box deltas to readjust the proposal boxes. Then it computes Non-Maximum Suppression to remove non-overlapping boxes (while also applying other hyper-settings such as score threshold). Finally, it ranks top-k boxes according to their scores. That probably explains why your method produces the same box scores but different number of output boxes and its coordinates.
Back to your original question, here is the way to extract the features of the proposed boxes in one inference pass:
image = cv2.imread('my_image.jpg')
height, width = image.shape[:2]
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))
inputs = [{"image": image, "height": height, "width": width}]
with torch.no_grad():
    images = model.preprocess_image(inputs)  # don't forget to preprocess
    features = model.backbone(images.tensor)  # set of cnn features
    proposals, _ = model.proposal_generator(images, features, None)  # RPN

    features_ = [features[f] for f in model.roi_heads.box_in_features]
    box_features = model.roi_heads.box_pooler(features_, [x.proposal_boxes for x in proposals])
    box_features = model.roi_heads.box_head(box_features)  # features of all 1k candidates
    predictions = model.roi_heads.box_predictor(box_features)
    pred_instances, pred_inds = model.roi_heads.box_predictor.inference(predictions, proposals)
    pred_instances = model.roi_heads.forward_with_given_boxes(features, pred_instances)

    # output boxes, masks, scores, etc
    pred_instances = model._postprocess(pred_instances, inputs, images.image_sizes)  # scale box to orig size
    # features of the proposed boxes
    feats = box_features[pred_inds]


Instruction: Dequantize values to their original prior to quantization

Input:
The paper "Natural Language Processing with Small Feed-Forward Networks" https://arxiv.org/pdf/1708.00214.pdf states:

I've implemented quantization as per the above equations in python:
b = 128

embedding_matrix = [[20000,3000,1000],[1999999,20000,1999999], [20000,3000,1000]]

scaled = [ abs(round( (1 / (b - 1) * max(e)) , 3)) for e in embedding_matrix]

print(scaled)

i = 0

quantized = []
for e in embedding_matrix :
    for v in e :
        quantized.append((v , math.floor(.5 + ( (v / scaled[i]) + b) )))
    i = i + 1

quantized

Running this code quantized is set to :
[(20000, 255),
 (3000, 147),
 (1000, 134),
 (1999999, 255),
 (20000, 129),
 (1999999, 255),
 (20000, 255),
 (3000, 147),
 (1000, 134)]

How to de-quantize back to the original values prior to quantization ?
Reading https://www.tensorflow.org/api_docs/python/tf/quantization/dequantize describes :
tf.quantization.dequantize(
    input, min_range, max_range, mode='MIN_COMBINED', name=None, axis=None,
    narrow_range=False, dtype=tf.dtypes.float32
)

[min_range, max_range] are scalar floats that specify the range for the output. The 'mode' attribute controls exactly which calculations are used to convert the float values to their quantized equivalents.

and the PyTorch docs: https://pytorch.org/docs/stable/quantization.html
Seems to implement quantize differently to above implementation ?

Output:
What they are doing in the paper is roughly this:
import numpy as np

b = 128

embedding_matrix = np.array([[20000,3000,1000,1000],[1999999,20000,1999999,1999999], [20000,3000,1000,1000]])
scales = (np.abs(embedding_matrix).max(axis=1) / (b-1)).reshape(-1, 1)
quantized = (embedding_matrix / scales + b + 0.5).astype(np.uint8)
dequantized = (quantized - b) * scales
print(quantized)
print(dequantized)

Output:
[[255 147 134 134]
 [255 129 255 255]
 [255 147 134 134]]
[[2.00000000e+04 2.99212598e+03 9.44881890e+02 9.44881890e+02]
 [1.99999900e+06 1.57480236e+04 1.99999900e+06 1.99999900e+06]
 [2.00000000e+04 2.99212598e+03 9.44881890e+02 9.44881890e+02]]

In short they just have q_ij = round(e_ij / s_i + b), so after you just have quantized value q_ij your best approximation is to say that q_ij = dequantized_ij / s_i + b, so dequantized_ij = (q_ij - b) * s_i
As to pytorch - similar functionality is available with torch.quantize_per_channel e.g the following code is doing pretty much the same:
import torch
t = torch.tensor(embedding_matrix, dtype=torch.float32)
zero_point = torch.tensor([b]).repeat(t.shape[0], 1).reshape(-1)
quantized_tensor = torch.quantize_per_channel(t, t.abs().max(axis=1)[0] / (b-1), zero_point, 0, torch.quint8)
print(quantized_tensor)
print(quantized_tensor.int_repr())

Output:
tensor([[2.0000e+04, 2.9921e+03, 9.4488e+02, 9.4488e+02],
        [2.0000e+06, 1.5748e+04, 2.0000e+06, 2.0000e+06],
        [2.0000e+04, 2.9921e+03, 9.4488e+02, 9.4488e+02]], size=(3, 4),
       dtype=torch.quint8, quantization_scheme=torch.per_channel_affine,
       scale=tensor([  157.4803, 15748.0234,   157.4803], dtype=torch.float64),
       zero_point=tensor([128, 128, 128]), axis=0)
tensor([[255, 147, 134, 134],
        [255, 129, 255, 255],
        [255, 147, 134, 134]], dtype=torch.uint8)

If quantized per channel like this in pytorch you can only apply .dequantize() on the full tensor rather then the sliced which wouldn't be a good thing for embeddings, but you can do it manually very easy using repr_int, q_per_channel_zero_points, and q_per_channel_scales.
Does this answer your question?


Instruction: TypeError: forward() missing 1 required positional argument with Tensorboard PyTorch

Input:
I am trying to write my model to tensorboard with the following code:

model = SimpleLSTM(4, HIDDEN_DIM, HIDDEN_LAYERS, 1, BATCH_SIZE, device)
writer = tb.SummaryWriter(log_dir=tb_path)
sample_data = iter(trainloader).next()[0]
writer.add_graph(model, sample_data.to(device))

I get the error: TypeError: forward() missing 1 required positional argument: 'batch_size'

My model looks like this:

class SimpleLSTM(nn.Module):

    def __init__(self, input_dims, hidden_units, hidden_layers, out, batch_size, device):
        super(SimpleLSTM, self).__init__()
        self.input_dims = input_dims
        self.hidden_units = hidden_units
        self.hidden_layers = hidden_layers
        self.batch_size = batch_size
        self.device = device
        self.lstm = nn.LSTM(self.input_dims, self.hidden_units, self.hidden_layers,
                            batch_first=True, bidirectional=False)
        self.output_layer = nn.Linear(self.hidden_units, out)

    def init_hidden(self, batch_size):

        hidden = torch.rand(self.hidden_layers, batch_size, self.hidden_units, device=self.device, dtype=torch.float32)
        cell = torch.rand(self.hidden_layers, batch_size, self.hidden_units, device=self.device, dtype=torch.float32)
        hidden = nn.init.xavier_normal_(hidden)
        cell = nn.init.xavier_normal_(cell)
        return (hidden, cell)

    def forward(self, input, batch_size):
        hidden = self.init_hidden(batch_size)  incomplete batch
        lstm_out, (h_n, c_n) = self.lstm(input, hidden)
        raw_out = self.output_layer(h_n[-1])
        return raw_out

How can I write this model to TensorBoard?

Output:
Your model takes two arguments input and batch_size, but you only provide one argument for add_graph to call your model with.

The inputs (second argument to add_graph) should be a tuple with the input and the batch_size:

writer.add_graph(model, (sample_data.to(device), BATCH_SIZE))

You don't really need to provide the batch size to the forward method, because you can infer it from the input. As your LSTM uses batch_first=True, it means that the input is required to have size [batch_size, seq_len, num_features], therefore the size of the first dimension is the current batch size.

def forward(self, input):
    batch_size = input.size(0)
    # ...


Instruction: Problems using pretrained ResNet50 in PyTorch to solve CIFAR10 Dataset

Input:
I got the following error using a pretrained ResNet50 in PyTorch:

RuntimeError
Traceback (most recent call last)
<ipython-input-14-8f0d0641ef12> in <module>()
      28         # Update parameters
      29         optimizer.zero_grad()
 ---> 30         loss.backward()
      31         optimizer.step()
      32

 1 frames
 /usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in
 backward(tensors, grad_tensors, retain_graph, create_graph,
 grad_variables)
      98     Variable._execution_engine.run_backward(
      99         tensors, grad_tensors, retain_graph, create_graph,
 --> 100         allow_unreachable=True)  # allow_unreachable flag
     101
     102

 RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Notebook is in this link: https://colab.research.google.com/drive/1k40NNulSIS6ANagopSPBH4Xty_Cw39qC?usp=sharing

Output:
The problem is that you're setting a new attribute model.classifier, while you actually want to replace the current "classifier", i.e., change the model.fc.

It is beyond the scope of your question, but you'll find another problem later on. Your new classifier has a LogSoftmax() module and you're using the nn.CrossEntropyLoss(). As you can see here, you should not do this.


Instruction: RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

Input:
Can someone kindly help me trace the root of the following error? I don't understand where the switching between GPU and CPU is taking place as from the beginning I have instructed collab to use GPU.
Also following the error stack trace, it points to labels, what could be potentially wrong here?

Thanks in advance!

Output:
I would recommend to take a look at this youtube series to understand how pytorch works.

For your issue specifically I think you'll find your answer in this video

The idea is that you need to specify that you want to place your data and your model on your GPU. Using the method .to(device), device being either cuda if your GPU is available otherwise your cpu, and you need to do the same for your data.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

You also need to do it your data, I assume you have a for loop to iterate over your batch so you could do it like:

    for batch in train_loader:
        ***
        x, y = batch[0].to(device), batch[1].to(device)
        ***


Instruction: Is One-Hot Encoding required for using PyTorch's Cross Entropy Loss Function?

Input: For example, if I want to solve the MNIST classification problem, we have 10 output classes. With PyTorch, I would like to use the torch.nn.CrossEntropyLoss function. Do I have to format the targets so that they are one-hot encoded or can I simply use their class labels that come with the dataset?

Output:
nn.CrossEntropyLoss expects integer labels.  What it does internally is that it doesn't end up one-hot encoding the class label at all, but uses the label to index into the output probability vector to calculate the loss should you decide to use this class as the final label. This small but important detail makes computing the loss easier and is the equivalent operation to performing one-hot encoding, measuring the output loss per output neuron as every value in the output layer would be zero with the exception of the neuron indexed at the target class.  Therefore, there's no need to one-hot encode your data if you have the labels already provided.
The documentation has some more insight on this: https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html.  In the documentation you'll see targets which serves as part of the input parameters.  These are your labels and they are described as:

This clearly shows how the input should be shaped and what is expected.  If you in fact wanted to one-hot encode your data, you would need to use torch.nn.functional.one_hot. To best replicate what the cross entropy loss is doing under the hood, you'd also need nn.functional.log_softmax as the final output and you'd have to additionally write your own loss layer since none of the PyTorch layers use log softmax inputs and one-hot encoded targets. However, nn.CrossEntropyLoss combines both of these operations together and is preferred if your outputs are simply class labels so there is no need to do the conversion.


Instruction: BERT:Question-Answering - Total number of permissible words/tokens for training

Input:
Let's say I want to train BERT with 2 sentences (query-answer) pair against a certain binary label (1,0) for the correctness of the answer,  will BERT let me use 512 words/tokens each for the query and the answer or together(query+answer combined) they should be 512? [510 upon ignoring the [start] and [sep] token]

Thanks in advance!

Output:
Together, and actually it's together they should be 509 since there are two [SEP], one after question and another after answer:

[CLS] q_word1 q_word2 ... [SEP] a_word1 a_word2 ... [SEP]

where q_word refers to words in the question and a_word refers to words in the answer


Instruction: How to improve my image classifier to recognize real world images

Input:
I would like to train a hand gesture classifier with pytorch.
  The dataset images looks like this.

I tried to use resnet34 and some kind of data augmentation. I got a high accuracy on test set, but low accuracy when trying to recognize my own gesture in the real world. It works fine when the background is white, goes crazy when other things(my face, chair, bed etc.) appeared in the background. Maybe that's because test images have a pure background, so how can I improve my classifier?
Also I want to add a 'non-gesture' category in my claasifier as well. How can I do that?
This is my data augmentation transforms:

transform = torchvision.transforms.Compose([
         torchvision.transforms.Grayscale(3),
         torchvision.transforms.RandomHorizontalFlip(),
         torchvision.transforms.RandomRotation(20),
         torchvision.transforms.RandomResizedCrop(64, (0.6, 1.2)),
         torchvision.transforms.ToTensor(),
         torchvision.transforms.Normalize((0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
    ])

Any advice is appreciated. Thanks!

Output:
Aside from changes to your classifier, you should look at your training data:

Ask yourself:

Is my sample size big enough?
If you have too few images to train on, no data augmentation in the world will make up for that. Aim to acquire a large, heterogenous dataset with even label distribution.
Does your training data accurately reflect the circumstances you want to use your classifier in. The images you supplied seem to have a light background, maybe try to get images of hand gestures with different backgrounds.

After that you should take a look at your classifier and improve it. Since you didn't include your model I can't comment on that.


Instruction: Shall we lower case input data for (pre) training a BERT uncased model using huggingface?

Input:
Shall we lower case input data for (pre) training a BERT uncased model using huggingface? I looked into this response from Thomas Wolf (https://github.com/huggingface/transformers/issues/92#issuecomment-444677920) but not entirely sure if he meant that.

What happens if we lowercase the text ?

Output:
Tokenizer will take care of that.
A simple example:
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

Out:
tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])
tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])

But in case of cased,
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

tensor([[ 101, 1142, 1110,  170, 5855,  102,    0,    0,    0,    0]])

tensor([[ 101, 1188, 1110,  170, 8572,  102,    0,    0,    0,    0]])


Instruction: model.parameters() does not produce an iterable of Tensors

Input:
I am trying to use torch.nn.utils.clip_grad_norm_() which requires an iterable of Tensors.  See below

for epoch in progress_bar(range(num_epochs)):
    lstm.train()
    outputs = lstm(trainX.to(device))
    optimizer.zero_grad()
    torch.nn.utils.clip_grad_norm_(lstm.parameters(), 1)

My code errors with:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-168-4cd34e6fd44d> in <module>
     28     lstm.train()
     29     outputs = lstm(trainX.to(device))
---> 30     torch.nn.utils.clip_grad_norm_(lstm.parameters(), 1)
     31
     32

/opt/conda/lib/python3.6/site-packages/torch/nn/utils/clip_grad.py in clip_grad_norm_(parameters, max_norm, norm_type)
     28         total_norm = max(p.grad.detach().abs().max() for p in parameters)
     29     else:
---> 30         total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type) for p in parameters]), norm_type)
     31     clip_coef = max_norm / (total_norm + 1e-6)
     32     if clip_coef < 1:

RuntimeError: stack expects a non-empty TensorList

If I example lstm.parameters() I get a list of Parameters, instead of a list of Tensors:

<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 1])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([1, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([1])

Looking at the first Parameter, it is a list of Tensors:

<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
.
.
.

Does anyone know what is going on here?

Output:
PyTorch's clip_grad_norm, as the name suggests, operates on gradients.
You have to calculate your loss from output, use loss.backward() and perform gradient clipping afterwards.

Also, you should use optimizer.step() after this operation.

Something like this:

for epoch in progress_bar(range(num_epochs)):
    lstm.train()
    for batch in dataloader:
        optimizer.zero_grad()
        outputs = lstm(trainX.to(device))
        loss = my_loss(outputs, targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(lstm.parameters(), 1)
        optimizer.step()

You don't have parameter.grad calculated (it's value is None) and that's the reason of your error.


Instruction: Checking the contents of python dataloader

Input:
This is probably a simple question, but how see how the contents of this standard data loader looks like:

from torchtext import datasets
import random

train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
train_data, valid_data = train_data.split(random_state = random.seed(SEED))

I can't use .head() and:

print(test_data)

just gives me:

<torchtext.datasets.imdb.IMDB object at 0x7f0b42e8c240>

I'm probably just missing .values or [0] or something similar...

Output: Datasets are iterables, you can get the first element with next(iter(test_data)).


Instruction: Understanding nn.Sequential in convolutional layers

Input:
I am new to PyTorch/Deep learning and I am trying to understand the use of the following line to define a convolutional layer:

self.layer1 = nn.Sequential(nn.Conv1d(input_dim, n_conv_filters, kernel_size=7, padding=0), nn.ReLU(), nn.MaxPool1d(3))

I understand that that it is creating a 1d convolutional layer to the network with max pooling 3 wide. However, I don't understand the function of the sequential module or RelU. How do these function in creating a layer?

For reference, the rest of the code can be found here: https://github.com/ArdalanM/nlp-benchmarks/blob/master/src/cnn/net.py

Output:
As per the description provided it seems you are in the process of developing a convolutional architecture for a problem (More likely a Computer Vision one as CNNs are usually targeted for solving CV problems).

Now talking about the code by using Sequential module you are telling the PyTorch that you are developing an architecture that will work in a sequential manner and by specifying ReLU you are bringing the concept of Non-Linearity in the picture (ReLU is one of the widely used activation functions in the Deep learning framework). Non-Linearity helps CNNs to generalize to complex decision boundaries and ultimately helps them to perform better.

PS: I recommend reviewing the https://towardsdatascience.com/convolutional-neural-network-for-image-classification-with-implementation-on-python-using-pytorch-7b88342c9ca9 for getting better idea from a coder perspective.


Instruction: AttributeError: module 'torch.optim' has no attribute 'RMSProp'

Input:
Getting the following error when trying to use the RMSProp Optimizer with PyTorch:

AttributeError: module 'torch.optim' has no attribute 'RMSProp'

Code:

import torch as T
import torch.nn as nn
import torch.optim as optim

class DeepQNetwork(nn.Module):

  def __init__(self, alpha, ...):
    super(DeepQNetwork, self).__init__()
    ...
    self.optimizer = optim.RMSProp(self.parameters(), lr=alpha)
    ...

PyTorch version is 1.5.1 with Python version 3.6. There's a documentation for torch.optim and its optimizers including RMSProp, but PyCharm only suggests Adam and SGD and it really seems like all other optimizers are missing.

Does anyone have an idea? I did not find a single thing on the internet and it starts driving me crazy.

Suggesstion from PyCharm

Output: RMSprop (as seen in the documentation) instead of RMSProp. So, it's just a typo.


Instruction: With the HuggingFace transformer, how can I return multiple samples when generating text?

Input:
I'm going off of https://github.com/cortexlabs/cortex/blob/master/examples/pytorch/text-generator/predictor.py

But if I pass num_samples=5, I get:

    generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Got 5 and 1 in dimension 0

the code is:

def sample_sequence(
    model,
    length,
    context,
    num_samples=1,
    temperature=1,
    top_k=0,
    top_p=0.9,
    repetition_penalty=1.0,
    device="cpu",
):
    context = torch.tensor(context, dtype=torch.long, device=device)
    context = context.unsqueeze(0).repeat(num_samples, 1)
    print('context_size', context.shape)
    generated = context
    print('context', context)
    with torch.no_grad():
        for _ in trange(length):
            inputs = {"input_ids": generated}
            outputs = model(
                **inputs
            )  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
            next_token_logits = outputs[0][0, -1, :] / (temperature if temperature > 0 else 1.0)

            # reptition penalty from CTRL (https://arxiv.org/abs/1909.05858)
            for _ in set(generated.view(-1).tolist()):
                next_token_logits[_] /= repetition_penalty

            filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
            if temperature == 0:  # greedy sampling:
                next_token = torch.argmax(filtered_logits).unsqueeze(0)
            else:
                next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)
            generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
    return generated

Output:
As far as I can see this code doesn't provide multiple samples, but you can adjust it with a some adjustments.
This line uses already multinomial but returns only 1:
next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)

change it to:
next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=num_samples)

Now you also need to change the result construction. This concatenates line the next_token with the sentence. You get now num_samples of next_tokens and you need unsqueeze all of them:
generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)

change it to:
generated = torch.cat((generated, next_token.unsqueeze(1)), dim=1)

The whole function should look like this now:
def sample_sequence(
    model,
    length,
    context,
    num_samples=1,
    temperature=1,
    top_k=0,
    top_p=0.9,
    repetition_penalty=1.0,
    device="cpu",
):
    context = torch.tensor(context, dtype=torch.long, device=device)
    context = context.unsqueeze(0).repeat(num_samples, 1)
    generated = context
    with torch.no_grad():
        for _ in trange(length):
            inputs = {"input_ids": generated}
            outputs = model(
                **inputs
            )  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
            next_token_logits = outputs[0][0, -1, :] / (temperature if temperature > 0 else 1.0)

            # reptition penalty from CTRL (https://arxiv.org/abs/1909.05858)
            for _ in set(generated.view(-1).tolist()):
                next_token_logits[_] /= repetition_penalty

            filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
            if temperature == 0:  # greedy sampling:
                next_token = torch.argmax(filtered_logits).unsqueeze(0)
            else:
                next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=num_samples)
            generated = torch.cat((generated, next_token.unsqueeze(1)), dim=1)
    return generated

Last but not least you have to change your tokenizer.decode call to tokenizer.batch_decode as the return value contains now multiple samples:
tokenizer.batch_decode(output.tolist(), clean_up_tokenization_spaces=True, skip_special_tokens=True)

Something you have to think of byt yourself, is what you want to do when there is no valide next_token. Currently you will receive an error message like:

RuntimeError: invalid multinomial distribution (with replacement=False, not enough non-negative category to sample)

Another thing you have to think of, is if their code is even correct. During the few test I have conducted, it felt like that the quality of created sentences decreased with an increasing number of num_samples (i.e. Maybe the quality is better when you use a simple loop to call sample_sequence multiple times?). I haven't worked with GPT2 yet and can't help you here.


Instruction: Identical random crop on two images Pytorch transforms

Input: I am trying to feed two images into a network and I want to do identical transform between these two images. transforms.Compose() takes one image at a time and produces output independent to each other but I want same transformation. I did my own coding for hflip() now I am interested to get the random crop. Is there any way to do that without writing custom functions?

Output:
I would use workaround like this - make my own crop class inherited from RandomCrop, redefining call with
    …
        if self.call_is_even :
            self.ijhw = self.get_params(img, self.size)
        i, j, h, w = self.ijhw
        self.call_is_even = not self.call_is_even

instead of
    i, j, h, w = self.get_params(img, self.size)

The idea is to suppress randomizer on odd calls


Instruction: Pytorch: Add input normalization to model (division layer)

Input:
I want to add the image normalization to an existing pytorch model, so that I don't have to normalize the input image anymore.
Say I have an existing model
model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True)
model.eval()

Now I can add new layers (for example a relu) using torch.nn.Sequential:
new_model = nn.Sequential(
    model,
    nn.ReLU()
)

However I couldn't find a layer to do perform just a division or subtraction as needed for the input normalization here shown in numpy:
import cv2
import numpy as np
img = cv2.imread("my_img.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img.astype(np.float32)
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
img = img / 255.0
img = img - mean
img = img / std
img = np.transpose(img, (2, 0, 1))
img = np.expand_dims(img, axis=0)

The goal is that normalization is eventually done on GPU to save time during inference. Also I cannot use torchvision transforms as those operation are not stored inside the model itself. For example, if I want to save the model to disk (in order to convert it to tflite using onnx) the torchvision transform operations will not be saved along with the model. Is there an elegant way of doing this?
(preferably without using a linear layer, which would fix my model input size, which should be flexible as my real model is fully convolutional)

Output:
Untested code which hopefully you can vet yourself.
import torch.nn as nn

cuda0 = torch.device('cuda:0')

class Normalize(nn.Module):
    def __init__(self, mean, std):
        super(Normlize, self).__init__()
        self.mean = torch.tensor(mean, device=cuda0)
        self.std = torch.tensor(std, device=cuda0)

    def forward(self, input):
        x = input / 255.0
        x = x - self.mean
        x = x / self.std
        return x

In your model you can do
new_model = nn.Sequential(
    Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
    model,
    nn.ReLU()
)


Instruction: How to run a pre-trained pytorch model on the GPU?

Input:
Here I am trying to use the mobilenetv2 mobile to train on a custom dataset. I can get it to work on the CPU, but I would prefer to run it on the GPU. Instead, I am getting errors like these:

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #4 'mat1

So like my Post asks how can I get the pre-trained model to run on the GPU?

MobileNet = models.mobilenet_v2(pretrained = True)
if torch.cuda.is_available():
    MobileNet.cuda()

for param in MobileNet.parameters():
    param.requires_grad = False

    torch.manual_seed(50)

MobileNet.classifier = nn.Sequential(nn.Linear(1280, 1000), nn.ReLU(), nn.Dropout(0.5), nn.Linear(1000,3), nn.LogSoftmax(dim=1))

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(MobileNet.classifier.parameters(), lr=0.001)

train_transform = transforms.Compose([
        transforms.RandomRotation(10),      # rotate +/- 10 degrees
        transforms.RandomHorizontalFlip(),  # reverse 50% of images
        transforms.Resize(224),             # resize shortest side to 224 pixels
        transforms.CenterCrop(224),         # crop longest side to 224 pixels at center
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])

test_transform = transforms.Compose([
        transforms.Resize(224),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])

train_data = datasets.ImageFolder('C:/Users/mixv/Pictures/Summer/datasets/train', transform=train_transform)
test_data = datasets.ImageFolder('C:/Users/mix/Pictures/Summer/datasets/test', transform=test_transform)

torch.manual_seed(42)
batch=64
train_loader = DataLoader(train_data, batch_size=batch, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch, shuffle=True)

if torch.cuda.is_available():
    train_loader = DataLoader(train_data, batch_size=batch, shuffle=True, pin_memory = True)
    test_loader = DataLoader(test_data, batch_size=batch, shuffle=True, pin_memory = True)

epochs = 10

train_losses = []
test_losses = []
train_correct = []
test_correct = []
start_time =time.time()
for i in range(epochs):
    trn_corr = 0
    tst_corr = 0

    # Run the training batches

    for b, (images, labels) in enumerate(train_loader):

        if torch.cuda.is_available():
            images = images.cuda()
            labels = labels.cuda()

        b+=1

        # Apply the model
        y_pred = MobileNet(images)
        loss = criterion(y_pred, labels)

        # Tally the number of correct predictions
        predicted = torch.max(y_pred.data, 1)[1]
        batch_corr = (predicted == labels).sum()
        trn_corr += batch_corr

        accuracy = trn_corr.item()*100/(b*batch)
        # Update parameters
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Output: As the RuntimeError said, some weights are still in cpu. One possible flaw I suspect is MobileNet.classifier = nn.Sequential(nn.Linear(1280, 1000), nn.ReLU(), nn.Dropout(0.5), nn.Linear(1000,3), nn.LogSoftmax(dim=1)) is done after MobileNet.cuda(), which mean these new created weight probably did not be sent to gpu. Try reverse these two's sequence and see


Instruction: How is data augmentation done in each epoch?

Input:
I'm new to PyTorch and want to apply data augmentation to the datasets on each epoch. I
train_transform = Compose([
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomCrop(32, padding=4),
    transforms.ToTensor(),
    transforms.Normalize([0, 0, 0], [1, 1, 1])
])

test_transform = Compose([
    transforms.ToTensor(),
    transforms.Normalize([0, 0, 0], [1, 1, 1])
])

cifar10_train = CIFAR10(root = "/data", train=True, download = True, transform=train_transform)
train_loader = torch.utils.data.DataLoader(cifar10_train, batch_size=128, shuffle=True)

cifar10_test = CIFAR10(root = "/data", train=False, download = True, transform=test_transform)
test_loader = torch.utils.data.DataLoader(cifar10_test, batch_size=128, shuffle=True)

I got the code from an online tutorial. So from what I understand train_transform and test_transform is the augmentation code while cifar10_train and cifar10_test are where data is loaded and augmentation is done at the same time. Does this mean data augmentation is only done once before training? What if I want to do data augmentation for each epoch.

Output: I think you have some misunderstandings in your code. The cifar10_train and cifar10_test actually load the dataset into python (this data is not augmented and is the raw data), then the data goes through the transforms. In most cases, the training set is where the data augmentation is done, and the testing set is not augmented because it is supposed to replicate real-world data. The transforms (train_transform and test_transforms) are what decide how the data is augmented, normalized, and converted into PyTorch Tensors, you can think of it as a set of guidelines/rules for the dataset to follow. As mentioned before, the training set only gets augmented, which is why the train_transform has RandomHorizontalFlip and RandomCrop (which does augmentation), and why test_transforms does not have RandomHorizontalFlip and RandomCrop. The loaders (train_loader and test_loader) is what splits the data into batches (groups of data), and applies the transforms to the cifar10 dataset.


Instruction: Pytorch for loop inefficience

Input:
I have a efficient issue with some tensor for loop.

I’m extracting the features from the last layer of a CNN through a image data loader (I’m using batch size 8). Im getting the euclidean distance of the batch tensor and a table with previous features.

I want to add to the table a tensor every time all the tensors in the table are above a threshole. I have implemented a successful running code but the loop i use its not efficient and im wondering how i could do something similar using something more efficient rather than this secuential way.

for i, data in enumerate(dataloader, 0):
  input, label = data
  input, label = input.to(device), label.to(device)
  n,c h,w = input.size()
  outputs = model(input)
  if (i == 0):
    features_list = torch.cat( (features_list, outputs[0].view(1,-1)), 0)
  dist_tensores = torch.cdist(outputs, features_list, p=2.0)
  activation = torch.gt(dist_tensores, AVG, out=torch.cuda.FloatTensor(len(outputs), len(features_list)))
  counter = len(features_list)
  activation_list = torch.sum(activation, dim=0)
  for x in range(len(activation)):
    if (torch.sum(activation[x], dim=0) == counter):
      features_list = torch.cat( (features_list, outputs[x].view(1,-1)), 0)

The last loop is the part i want to change but i really don´t know how to assign and add the tensor i want if it's not by creating a loop where i can control the tensor to add.

Output:
idx = activation.sum(1) == counter
features_list = torch.cat((features_list, outputs[idx]), 0)

This would replace the loop and save computational and inefficiences issues.


Instruction: torch.einsum 'RuntimeError: dimension mismatch for operand 0: equation 4 tensor 2'

Input:
I'm trying to manually calculate a gradient of a matrix and I can do it by using numpy but I don't know to do the same thing in pytorch.
the equation in NumPy is

def grad(A, W0, W1, X):
    dim = A.shape
    assert len(dim) == 2
    A_rows = dim[0]
    A_cols = dim[1]
    gradient = (np.einsum('ik, jl', np.eye(A_cols, A_rows), (((A).dot(X)).dot(W0)).dot(W1).T) + np.einsum('ik, jl', A, ((X).dot(W0)).dot(W1).T))
    return gradient

I wrote a function in pytorch but it's giving me an error saying 'RuntimeError: dimension mismatch for operand 0: equation 4 tensor 2'

The function I wrote using pytorch is

def torch_grad(A, W0, W1, X):
    dim = A.shape
    A_rows = dim[0]
    A_cols = dim[1]
    W0W1 = torch.mm(W0, W1)
    AX = torch.mm(A, X)
    AXW0W1 = torch.mm(AX, W0W1)
    XW0W1 = torch.mm(X, W0W1)
    print(torch.eye(A_cols, A_rows).shape, torch.t(AXW0W1).shape)
    e1 = torch.einsum('ik jl', torch.eye(A_cols, A_rows), torch.t(AXW0W1))
    e2 = torch.einsum('ik, jl', A, torch.t(XW0W1))
    return e1 + e2

I would appreciate if someone can show me how to implement the numpy code in pytorch.
Thanks!

Output:
You are missing a comma in the first torch.einsum call.
e1 = torch.einsum('ik, jl', torch.eye(A_cols, A_rows), torch.t(AXW0W1))

Besides the typo, that's not how the gradients with respect to A are calculated, and it fails when A and AX have different sizes, which would be otherwise valid for the forward pass. For the e1, that should be a matrix multiplication, maybe that was your intention, in which case the torch.einsum should be 'ik, kl', but that's just an overly complicated way to perform a matrix multiplication and using torch.mm is simpler and more efficient. And the e2 is not involved in anything calculation that was performed with respect to A, therefore it is not part of gradients.
def torch_grad(A, W0, W1, X):
    # Forward
    W0W1 = torch.mm(W0, W1)
    AX = torch.mm(A, X)
    AXW0W1 = torch.mm(AX, W0W1)
    XW0W1 = torch.mm(X, W0W1)

    # Backward / Gradients
    rows, cols = AXW0W1.size()
    grad_AX = torch.mm(torch.eye(rows, cols), W0W1.t())
    grad_A = torch.mm(grad_AX, X.t())
    return grad_A

# Autograd version to verify that the gradients are correct
def torch_autograd(A, W0, W1, X):
    # Forward
    W0W1 = torch.mm(W0, W1)
    AX = torch.mm(A, X)
    AXW0W1 = torch.mm(AX, W0W1)
    XW0W1 = torch.mm(X, W0W1)

    # Backward / Gradients
    rows, cols = AXW0W1.size()
    AXW0W1.backward(torch.eye(rows, cols))
    return A.grad

# requires_grad=True for the autograd version to track
# gradients with respect to A
A = torch.randn(3, 4, requires_grad=True)
X = torch.randn(4, 5)
W0 = torch.randn(5, 6)
W1 = torch.randn(6, 5)

grad_result = torch_grad(A, W0, W1, X)
autograd_result = torch_autograd(A, W0, W1, X)

torch.equal(grad_result, autograd_result) # => True


Instruction: Classification with pretrained pytorch vgg16 model and its classes

Input:
I wrote a image vgg classification model with pytorch's pretrained vgg16 model.
import matplotlib.pyplot as plt
import numpy as np
import torch
from PIL import Image
import urllib
from skimage.transform import resize
from skimage import io
import yaml

# Downloading imagenet 1000 classes list
file = urllib. request. urlopen("https://gist.githubusercontent.com/yrevar/942d3a0ac09ec9e5eb3a/raw/238f720ff059c1f82f368259d1ca4ffa5dd8f9f5/imagenet1000_clsidx_to_labels.txt")
classes = ''
for f in file:
  classes = classes +  f.decode("utf-8")
classes = yaml.load(classes)

# Downloading pretrained vgg16 model
model = torch.hub.load('pytorch/vision:v0.6.0', 'vgg16', pretrained=True)

print(model)

for param in model.parameters():
    param.requires_grad = False

url, filename = ("https://raw.githubusercontent.com/pytorch/hub/master/dog.jpg", "dog.jpg")

image=io.imread(url)

plt.imshow(image)
plt.show()

# resize to 224x224x3
img = resize(image,(224,224,3))

plt.imshow(img)
plt.show()
# Normalizing input for vgg16
mean = [0.485, 0.456, 0.406]
std = [0.229, 0.224, 0.225]
img1 = mean*img+std
img1 = np.clip(img1,0,1)

img1 = torch.from_numpy(img1).unsqueeze(0)
img1 = img1.permute(0,3,2,1) # batch_size x channels x height x width

model.eval()
pred = model(img1.float())
print(classes[torch.argmax(pred).numpy().tolist()])

The code works fine but its outputting wrong classes. I am not sure where I did wrong but If I have to guess it might be the imagenet yaml classes list or at the normalizing input image. Can anyone tell me where I am making the mistakes?

Output:
There are some issues with the image preprocessing. Firstly, the normalisation is calculated as (value - mean) / std), not value * mean + std. Secondly, the values should not be clipped to [0, 1], the normalisation purposely shifts the values away from [0, 1]. Secondly, the image as NumPy array has shape [height, width, 3], when you permute the dimensions you swap the height and width dimension, creating a tensor with shape [batch_size, channels, width, height].
img = resize(image,(224,224,3))

# Normalizing input for vgg16
mean = [0.485, 0.456, 0.406]
std = [0.229, 0.224, 0.225]
img1 = (img1 - mean) / std

img1 = torch.from_numpy(img1).unsqueeze(0)
img1 = img1.permute(0, 3, 1, 2) # batch_size x channels x height x width

Instead of doing that manually, you can use torchvision.transforms.
from torchvision import transforms

preprocess = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

img = resize(image,(224,224,3))
img1 = preprocess(img)
img1 = img1.unsqueeze(0)

If you use PIL to load the images, you could also resize the images by adding transforms.Resize((224, 224)) to the preprocessing pipeline, or you could even add transforms.ToPILImage() to first convert the image to a PIL image (transforms.Resize requires a PIL image).


Instruction: Sampling data batch wise from tensor Pytorch

Input:
I have train_x and valid_x splited from trainX ,train_y and valid_y splited from trainY and they are having shapes as per below. i want to classify images of labels LABELS = set(["Faces", "Leopards", "Motorbikes", "airplanes"]).
print(train_x.shape, len(train_y))
torch.Size([1339, 96, 96, 3]) 1339

print(valid_x.shape, len(valid_y))
torch.Size([335, 96, 96, 3]) 335

print(testX.shape, len(testY))
torch.Size([559, 96, 96, 3]) 559

so i want to use regular train/valid on data batch-wise code as per below :
#train the network

n_epochs = 20

valid_loss = []
train_loss = []

for epoch in range(1,n_epochs+1):

    cur_train_loss = 0.0
    cur_valid_loss = 0.0

    #####################
    #### Train model ####
    #####################
    cnn_model.train()

    for data, target in trainLoader:
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        optimizer.zero_grad()
        output = cnn_model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        cur_train_loss += loss.item() * data.size(0)

    ########################
    #### Validate model ####
    ########################

    cnn_model.eval()

    for data, target in validLoader:
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        output = cnn_model(data)
        loss = criterion(output, target)
        cur_valid_loss += loss.item() * data.size(0)

    # calculate avg loss
    avg_train_loss = cur_train_loss / len(trainLoader.sampler)
    avg_valid_loss = cur_valid_loss / len(validLoader.sampler)

    train_loss.append(avg_train_loss)
    valid_loss.append(avg_valid_loss)

    print('Epoch: {} \t train_loss: {:.6f} \t valid_loss: {:.6f}'.format(epoch, avg_train_loss, avg_valid_loss))

so what i have to do for that ?
i have search for that but nothing specific i found out. i want to use pytorch for this. i have built model for another problem same like this but in that i have used DataLoader for loading one batch of data at a time for training and validation.

Output:
You can create a dataset with torch.utils.data.TensorDataset, where each sample of train_x is associated with its corresponding label in train_y, such that the DataLoader can create batches as you are used to.
from torch.utils.data import DataLoader, TensorDataset

train_dataset = TensorDataset(train_x, train_y)
train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)

valid_dataset = TensorDataset(valid_x, valid_y)
valid_dataloader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=False)

test_dataset = TensorDataset(testX, testY)
test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False)


Instruction: Why pytorch transformer src_mask doesn't block positions from attending?

Input:
I am trying to train word embedding with transformer encoder by masking the word itself with diagonal src_mask:
def _generate_square_subsequent_mask(self, sz):
    mask = torch.diag(torch.full((sz,),float('-inf')))
    return mask

def forward(self, src):

    if self.src_mask is None or self.src_mask.size(0) != len(src):
        device = src.device
        mask = self._generate_square_subsequent_mask(len(src)).to(device)
        self.src_mask = mask

    src = self.embedding(src) * math.sqrt(self.ninp)
    src = self.dropout(src)
    src = self.pos_encoder(src)
    src = self.transformer_encoder(src, self.src_mask)
    output = self.decoder(src) # Linear layer
    return output

After training the model predicts exactly the same sentence from the input. If I change any word in the input - it predict the new word. So the model doesn't block according to the mask.
Why is it ?
I understand that there is a mistake in my logic because BERT would probably be much simpler if it worked. But where am I wrong ?
Edit:
I am using the a sequence of word indices as input. Output is the same sequence as input.

Output: As far as I understand - the model doesn't prevent each word to indirectly “see itself” in multylayer context. I tried to use one layer - it looks like the model works. But training is too slow.


Instruction: How to use a layer with gradient but without weight adjustment?

Input:
Is it possible to mark part of the forward pass to only backpropagate the gradient but not to adjust weights?
In the following example code I have a Module that uses only one layer (one set of parameters) but it is used twice in the forward step. During the optimization I would expect the weights to be adjusted twice as well. If I want to only adjust the weights for one of the layer usages, what can I do?
import torch

class ExampleModel(torch.nn.Module):

    def __init__(self, dim) -> None:
        super(ExampleModel, self).__init__()
        self.linear = torch.nn.Linear(dim, dim)

    def forward(self, x):
        out1 = self.linear(x)  # backprop gradients and adjust weights here
        out2 = self.linear(out1)  # only backprop gradients here
        return out2

# Random input output data for this example
N, D = 64, 100
x = torch.randn(N, D)
y = torch.randn(N, D)

model = ExampleModel(D)

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters())

y_pred = model(x)
loss = criterion(y_pred, y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

The following will not work since with torch.no_grad() no gradient at all is backpropagated:
def forward(self, x):
    out1 = self.linear(x)  # backprop gradients and adjust weights here
    with torch.no_grad():
        out2 = self.linear(out1)  # only backprop gradients here
    return out2

I can not simply exclude the parameters from the optimization since they need to be optimized in the first part (i.e. out1 = self.linear(x)).
For the same reason I can also not set a learning rate of 0 for these parameters.
What else can I do to achieve this?

Output:
One way to do it is to use requires_grad_ to temporarily disable gradients on the layer's parameters:
def forward(self, x):
    out1 = self.linear(x)  # backprop gradients and adjust weights here
    self.linear.requires_grad_(False)
    out2 = self.linear(out1)  # only backprop gradients here
    self.linear.requires_grad_(True)
    return out2

This still lets gradients flow through the activations; it merely stops them from reaching the parameters.
You could also consider manipulating the weight tensors manually and calling .detach():
import torch.nn.functional as F
def forward(self, x):
    out1 = self.linear(x)
    out2 = F.linear(out1, self.linear.weight.detach(), self.linear.bias.detach())
    return out2


Instruction: DeepFix - Location Biased Convolution

Input:
I got a question regarding this paper: https://arxiv.org/pdf/1510.02927.pdf
In the Network Architecture they implement something called location biased convolution.
Basically it is 16 2d-gausians appended to the 512 filters of the convolutional layer (See figure 5 from the paper)
Picture of Location Biased Convolution.
I want to implement this in PyTorch, but have no clue how to add fixed filters to a convolutional block. The weights should be trained as discussed in the paper.
Can anyone give a hint of what to do or has done this before?

Output:
From what it looks like in the figure you provided, they append the location priors to the data, i.e.
location_priors = generate_gaussians(positions, variances, data.size())
data_w_loc_priors = T.cat((data, location_priors), dim=1)

Now, the number of in_channels for your convolution just needs to be adjusted accordingly: If you had 512 in_channels before, you now have 512 + number of location priors.


Instruction: How to do Class Activation Mapping in pytorch vgg16 model?

Input:
I wrote a pretrained vgg16 model for image classification and its layers are
VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

After some initial hick-up its now working fine. I want to use this model for class activation mapping (CAM) for visualizing CNN outputs. I know that in order to do that first we have to get the activations of last convolutional layer in vgg16 then the weight matrix of the last fully connected layer and lastly take the dot product of the two.
First I got the class index for the query image using this code
model.eval()
pred = model(img1.float())
class_idx = torch.argmax(pred).detach().numpy().tolist()
classes[class_idx]

Then I fetched the input images last convolutional layer activations which is of the size torch.Size([1, 512, 14, 14])
last_conv_feat = torch.nn.Sequential(*list(model.features)[:30])
pred_a = last_conv_feat(img1.float())
print(pred_a.shape)

After this I extracted the weights of the fully connected layers of vgg16 classifier and it has a shape of torch.Size([1000, 4096])
model.classifier[6].weight.shape

From this weight matrix then I recovered the weight parameters for the relevant class index
w_idx = model.classifier[6].weight[class_idx] # torch.Size([4096])

The problem is the shape of the convolutional activation matrix and the fully connected layer doest match, one is [1, 512, 14, 14] and the other is [4096]. How do I take dot product of these two matrix and get the CAM output?

Output:
This particular model isn't suitable for the simple approach you've pointed out. The CAM you refer to are extracted from models that have only one linear layer at the end, preceeded by a global average pooling layer, like this
features = MyConvolutions(x)
pooled_features = AveragePool(features)
predictions = Linear(pooled_features)

This typically works with ResNet architectures or one of their many derivates. Hence, my recommendation would be that unless there's a specific reason to use VGG you adopt a ResNet architecture.
------- EDIT -------
If you want to go with VGG, there are two options:

The easy one: cut off the VGG's last three (linear) layers, replace them by AveragePooling and a single Linear layer and finetune for ImageNet or whatever dataset you are using.
Approximate a CAM by converting VGG's last three layers into convolutional layers (i.e. 4096x512x7x7 with no padding and then 4096x4096x1x1 and 1000x4096x1x1), and reorgnise the parameters. This whole thing has only convolutional layers now and you can operate it like a huge convolutional filter. Only problem: It's output is still size 1x1. Therefore, you will need to enlarge your image (try maybe 2x) and then convolve it with the newly created fully-convolutional network. This gives you an approximate CAM.