Instruction: Whence MaskRCNN's segm IoU metrics = 0?

Input:
When training a MaskRCNN on my multi-class instance segmentation custom data set, given an input formatted as:
image   -)  shape: torch.Size([3, 850, 600]),   dtype: torch.float32, min: tensor(0.0431),               max: tensor(0.9137)
boxes   -)  shape: torch.Size([4, 4]),          dtype: torch.float32, min: tensor(47.),                  max: tensor(807.)
masks   -)  shape: torch.Size([850, 600, 600]), dtype: torch.uint8,   min: tensor(0, dtype=torch.uint8), max: tensor(1, dtype=torch.uint8)
areas   -)  shape: torch.Size([4]),             dtype: torch.float32, min: tensor(1479.),                max: tensor(8014.)
labels  -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(1),                    max: tensor(1)
iscrowd -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(0),                    max: tensor(0)

I consistently obtain all segmentation IoU metrics as shown below:
DONE (t=0.03s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.010
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
IoU metric: segm
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

How can I think, debug and fix this?

Output:
As your input image size is (850, 600) (H, W) and considering that for this given image you have 4 objects, not 850 with (600, 600) masks.
your masks tensor should have dimension (number of objects, 850, 600), thus your input should be:
image   -)  shape: torch.Size([3, 850, 600]),   dtype: torch.float32, min: tensor(0.0431),               max: tensor(0.9137)
boxes   -)  shape: torch.Size([4, 4]),          dtype: torch.float32, min: tensor(47.),                  max: tensor(807.)
masks   -)  shape: torch.Size([4, 850, 600]), dtype: torch.uint8,   min: tensor(0, dtype=torch.uint8), max: tensor(1, dtype=torch.uint8)
areas   -)  shape: torch.Size([4]),             dtype: torch.float32, min: tensor(1479.),                max: tensor(8014.)
labels  -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(1),                    max: tensor(1)
iscrowd -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(0),                    max: tensor(0)

How to fix it
Because you are trying to solve an instance segmentation problem, ensure that each of your (850, 600) masks are stacked so as to yield a tensor in the (number of masks, 850, 600) shape.


Instruction: How can I change the NN weights without affecting the gradients?

Input:
Say I have a simple NN:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.nn.utils import parameters_to_vector

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Model()

opt = optim.Adam(net.parameters())

And also some features
features = torch.rand((3,1))

I can train it normally using:
for i in range(10):
    opt.zero_grad()
    out = net(features)
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss.backward()
    opt.step()

However, I'm interested in updating the weights of each layer after each example in the batch. That is, updating the actual weight values by some amount that will be different for each layer.
I can print the parameters of each layer with:
for i in range(1):
    opt.zero_grad()
    out = net(features)
    print(parameters_to_vector(net.fc1.parameters()))
    print(parameters_to_vector(net.fc2.parameters()))
    print(parameters_to_vector(net.fc3.parameters()))
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss.backward()
    opt.step()

How can I change the values of the weights before the backprop without affecting the gradient?
Say that I want the layers weights' to be updated according to the following functions:
def first_layer_update(weight):
    return weight + 1e-3*weight

def second_layer_update(weight):
    return 1e-2*weight

def third_layer_update(weight):
    return weight - 1e-1*weight

Output:
- Using the torch.no_grad context manager.
This allows you to perform (in-place or out-of-place) operations on your tensors without Autograd keeping track of those changes. As @user3474165 explained:
def first_layer_update(weight):
    with torch.no_grad():
        return weight + 1e-3*weight

def second_layer_update(weight):
    with torch_no_grad():
        return 1e-2*weight

def third_layer_update(weight):
    with torch.no_grad():
        return weight - 1e-1*weight

Or differently without altering your functions using the context manager when calling them:
with torch.no_grad():
    first_layer_update(net.fc1.weight)
    second_layer_update(net.fc2.weight)
    third_layer_update(net.fc3.weight)

- Using the @torch.no_grad decorator.
A variant is to use the @torch.no_grad decorator:
@torch.no_grad()
def first_layer_update(weight):
    return weight + 1e-3*weight

@torch.no_grad():
def second_layer_update(weight):
    return 1e-2*weight

@torch.no_grad():
def third_layer_update(weight):
    return weight - 1e-1*weight

And call these with: first_layer_update(net.fc1.weight), second_layer_update(net.fc2.weight), etc...

- Mutating torch.Tensor.data.
An alternative to wrapping your operations with the torch.no_grad context is to mutate the weights using their data attribute. This means calling your functions with:
>>> first_layer_update(net.fc1.weight.data)
>>> second_layer_update(net.fc2.weight.data)
>>> third_layer_update(net.fc3.weight.data)

Which would mutate the weights (not the biases) of the three layers with their respective update policies.

In a nutshell, if you want to mutate all parameters of a nn.Module you can either do:
>>> with torch.no_grad():
...     update_policy(parameters_to_vector(net.layer.parameters()))

Or
>>> update_policy(parameters_to_vector(net.layer.parameters().data))


Instruction: How to np.concatenate list with tensors?

Input:
I have a list with tensors:
[tensor([[0.4839, 0.3282, 0.1773,  ..., 0.2931, 1.2194, 1.3533],
        [0.4395, 0.3462, 0.1832,  ..., 0.7184, 0.4948, 0.3998]],
        device='cuda:0'),
 tensor([[1.0586, 0.2390, 0.2315,  ..., 0.9662, 0.1495, 0.7092],
        [0.6403, 0.0527, 0.1832,  ..., 0.1467, 0.8238, 0.4422]],
        device='cuda:0')]

I want to stack all [1xfeatures] matrices into one by np.concatenate(X). but this error appears:
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

How to fix it?

Output:
Your tensors are still on the GPU and numpy operations happen on CPU. You can either send both tensors back to cpu first numpy.concatenate((a.cpu(), b.cpu()), as the error message indicates.
Or you can avoid moving off the GPU and use a torch.cat()
a = torch.ones((6),)
b = torch.zeros((6),)

torch.cat([a,b], dim=0)
# tensor([1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0.])


Instruction: Why the accuracy of my neural network does not increase?

Input:
I tried to implement in python using pytorch from scratch a convolutional neural network based on the structure of AlexNet using the CIFAR10 dataset but my accuracy is very very low (10%). How can I improve my accuracy? Is there a structural problem or I have only to change the hyperparameters?I am sorry to there are trivial errors but I am a beginner in neural networks.
import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision

from collections import OrderedDict

import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10
import os

class Config:
  def __init__(self):
    self.random_seed = 42
    self.n_epochs = 50
    self.batch_size_train = 256
    self.batch_size_test = 1000
    self.learning_rate = 0.0001
    self.momentum = 0.5
    self.log_interval = 500
    self.dropout_probability = 0.5

conf = Config()

torch.manual_seed(conf.random_seed)
torch.cuda.manual_seed(conf.random_seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(conf.random_seed)

train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.CIFAR10('/files/', train=True, download=True,
                               transform=torchvision.transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])),
                               batch_size=conf.batch_size_train,
                               shuffle=True
)

test_loader = torch.utils.data.DataLoader(
    torchvision.datasets.CIFAR10('/files/', train=False, download=True,
                               transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)
                             )
    ])),
                               batch_size=conf.batch_size_test,
                               shuffle=True
)
examples = enumerate(test_loader)
batch_idx, (example_data, example_targets) = next(examples)

example_data.shape
class AlexNet(nn.Module):
  def __init__(self):
    super(AlexNet,self).__init__()
    self.feature_extraction = nn.Sequential(
        nn.Conv2d(3, 96, kernel_size=11,stride = 4,padding = 2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=3,stride=2),
        nn.Conv2d(96, 256, kernel_size = 5, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=3,stride=2),
        nn.Conv2d(256, 384, kernel_size = 3, padding=1),
        nn.ReLU(),
        nn.Conv2d(384, 384, kernel_size = 3, padding=1),
        nn.ReLU(),
        nn.Conv2d(384, 256, kernel_size = 3, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=3,stride=2)
        )
    self.classifier = nn.Sequential (
        nn.Dropout(p=0.5, inplace=True),
        nn.Linear(3072, out_features=4096),
        nn.ReLU(),
        nn.Dropout(p=0.5, inplace=True),
        nn.Linear(in_features=4096, out_features=4096),
        nn.ReLU(),
        nn.Linear(in_features=4096, out_features=1000)
    )
  def forward(self,x):
      x = x.view(x.size(0),-1)
      return self.classifier(x)
network = AlexNet()

model_parameters = filter(lambda p: p.requires_grad, network.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print("The model has {} parameters.".format(f'{params:,}'))

optimizer = optim.Adam(
    network.parameters(),
    lr=conf.learning_rate
)

train_losses = []
train_counter = []
test_losses = []
test_counter = [i*len(train_loader.dataset) for i in range(conf.n_epochs + 1)]

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = network(inputs)
        loss = F.nll_loss(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        if i % 256 == 255:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 256))
            running_loss = 0.0

print('Finished Training')

correct = 0
total = 0
# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data in test_loader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = network(images)
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))

Output:
As you may have noticed there are plenty of loss functions in any deep learning package. You must choose the appropriate one based on problem criteria,  such as multiclass/binary, multilabel/simple, log_logits, already softmaxed logits, and ... . nll_loss often used in with log_softmax logits, but you have used it with raw logits. Based on what was said adding log_softmax to forward path would do the work. So the model would change to this:
  def forward(self,x):
      x = x.view(x.size(0),-1)
      x = self.classifier(x)
      return torch.nn.functional.log_softmax(x, 1)

In this way I have got Accuracy of the network on the 10000 test images: 43 % after one epoch.


Instruction: Mismatching dims in GRU for classification

Input:
I'm trying to complete a task and write simple RNN. Here's the class:
class RNNBaseline(nn.Module):
def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,
             bidirectional, dropout, pad_idx):

    super().__init__()

    self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)

    self.rnn = nn.GRU(input_size=embedding_dim, hidden_size=hidden_dim) #RNN(embedding_dim, hidden_dim)

    self.fc = nn.Linear(hidden_dim, output_dim)  # YOUR CODE GOES HERE

    self.dropout = nn.Dropout(dropout)

def forward(self, text, text_lengths, hidden = None):

    #text = [sent len, batch size]

    embedded = self.embedding(text)

    #embedded = [sent len, batch size, emb dim]

    #pack sequence
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)

    # cell arg for LSTM, remove for GRU
    # packed_output, (hidden, cell) = self.rnn(packed_embedded)
    # unpack sequence
    # output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)

    #output = [sent len, batch size, hid dim * num directions]
    #output over padding tokens are zero tensors

    #hidden = [num layers * num directions, batch size, hid dim]
    #cell = [num layers * num directions, batch size, hid dim]

    #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
    #and apply dropout
    output, hidden = self.rnn(packed_embedded, hidden)
    #hidden = None  # concatenate

    #hidden = [batch size, hid dim * num directions] or [batch_size, hid dim * num directions]

    return self.fc(hidden)

For now I'm not using LSTM or trying to do bidirectional RNN, I just want simple GRU to train without errors. This is the training function:
import numpy as np

min_loss = np.inf

cur_patience = 0

for epoch in range(1, max_epochs + 1):
train_loss = 0.0
model.train()
pbar = tqdm(enumerate(train_iter), total=len(train_iter), leave=False)
pbar.set_description(f"Epoch {epoch}")
for it, ((text, txt_len), label) in pbar:
    #YOUR CODE GOES HERE
    opt.zero_grad()
    input = text.to(device)
    labels = label.to(device)
    output = model(input, txt_len.type(torch.int64).cpu())
    train_loss = loss_func(output, labels)
    train_loss.backward()
    opt.step()

train_loss /= len(train_iter)
val_loss = 0.0
model.eval()
pbar = tqdm(enumerate(valid_iter), total=len(valid_iter), leave=False)
pbar.set_description(f"Epoch {epoch}")
for it, ((text, txt_len), label) in pbar:
    # YOUR CODE GOES HERE
    input = text.to(device)
    labels = label.to(device)
    output = model(input, txt_len.type(torch.int64).cpu())
    val_loss = loss_func(output, labels)

val_loss /= len(valid_iter)
if val_loss < min_loss:
    min_loss = val_loss
    best_model = model.state_dict()
else:
    cur_patience += 1
    if cur_patience == patience:
        cur_patience = 0
        break

print('Epoch: {}, Training Loss: {}, Validation Loss: {}'.format(epoch, train_loss, val_loss))
model.load_state_dict(best_model)

And some variables:
vocab_size = len(TEXT.vocab)
emb_dim = 100
hidden_dim = 256
output_dim = 1
n_layers = 2
bidirectional = False
dropout = 0.2
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]
patience=3

opt = torch.optim.Adam(model.parameters())
loss_func = nn.BCEWithLogitsLoss()

max_epochs = 1

But I get this error:

ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([1, 64, 1]))

... in this line:

---> 18         train_loss = loss_func(output, labels)

What am I doing wrong?

Output:
nn.BCEWithLogitsLoss expects both outputs and targets (or in your case labels) to be of size [b,d] where b is the batch size and d is the number of classes (or dimension of whatever you are trying to predict). Currently, your outputs are of size [b,d,1] and your targets are of size [d]. Two fixes are necessary, and both are very simple:

Add a batch dimension to your targets (labels). This is a common error when using a dataset that returns data elements because it generally does not add a batch dimension. Encapsulating your dataset class within a pytorch dataloader, but if you don't want to do this simply add an unsqueeze() operation. Note that the unsqueeze operation only works with a batch size of 1, otherwise using dataloader is probably a better bet.

Your output has an empty  3rd dimension, which can easily be flattened with a squeeze() operation. Both unsqueeze and squeeze are differentiable so shouldn't present problems for backpropagation.

... code before here

for it, ((text, txt_len), label) in pbar:
    # YOUR CODE GOES HERE
    input = text.to(device)
    labels = label.to(device).unsqueeze(0)                 # added unsqueeze operation
    output = model(input, txt_len.type(torch.int64).cpu())
    output = output.squeeze(-1)                            # added squeeze on last dim
    val_loss = loss_func(output, labels)

... code after here


Instruction: pipenv is unable to find the right package version using pytorch index

Input:
I'm confused by how pipenv resolves the available package versions. I specified the index and I clearly see version 0.8.1 in the list on their download site

This is my pipenv version. I'm running on Mac OSX and installed pipenv through brew.
~ pipenv --version
pipenv, version 2021.5.29

Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/torch_stable.html"
verify_ssl = false

[packages]
...
torch = {index="pytorch", version="==1.7.1"}
torchtext = {index="pytorch", version="==0.8.1"}
...

[dev-packages]

[requires]
python_version = "3.7"

The error when I run pipenv install:
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
CRITICAL:pipenv.patched.notpip._internal.index.package_finder:Could not find a version that satisfies the requirement torchtext==0.8.1 (from -r /var/folders/0q/10lqt8nn6_l87q71lx_d233r0000gn/T/pipenv2ix9if8erequirements/pipenv-qm980cdf-constraints.txt (line 7)) (from versions: 0.1.1, 0.2.0, 0.2.1, 0.2.3, 0.3.1, 0.4.0, 0.5.0, 0.6.0)
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2021.5.29/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
...

The error tells me it can't find 0.8.1 and that I can only choose from a super limited set. Why is this? What can I do to fix this?

Output:
You are using the wrong version of Python. It's really that simple.
Read very carefully in the list of library versions in that list:
Torchtext 0.8.1 exists for:

cp36m = CPython 3.6m
cp37m = CPython 3.7m
cp38 = CPython 3.8
cp39 = CPython 3.9

You are using: CPython 3.7.
CPython 3.7 (yours) and CPython 3.7m (theirs) is not the same thing.
The "M" stands for a special malloc version of Python which is faster. You can't install an "M" library on a non-"M" Python.
Upgrade to CPython 3.8 or 3.9. I recommend using Pipenv + Pyenv together, which makes it trivial to install different versions of CPython.


Instruction: Pytorch cifar10 images are not normalized

Input:
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

trainset.data[0]

I am using the above code and expect that data will be normalized. But it is not, below is the result. I need to access data using the data method to do some more processing. Output is below.
array([[[ 59,  62,  63],
        [ 43,  46,  45],
        [ 50,  48,  43],
        ...,
        [158, 132, 108],
        [152, 125, 102],
        [148, 124, 103]],

Output:
The torchvision.transforms.Normalize is merely a shift-scale operator. Given parameters mean (the "shift") and std (the "scale"), it will map the input to (input - shift) / scale.
Since you are using mean=0.5 and std=0.5 on all three channels, the results with be (input - 0.5) / 0.5 which is only normalizing your data if its statistic is in fact mean=0.5 and std=0.5 which is of course not the case.
With that in mind, what you should be doing is providing the actual dataset's statistics. For CIFAR10, these can be for example found here:
mean = [0.4914, 0.4822, 0.4465]
std = [0.2470, 0.2435, 0.2616]

With those values, you will be able to normalize your data properly to mean=0 and std=1.
I've written a more general, long-form answer here.


Instruction: copy segment from image tensor

Input:
I have three tensors:
A - (1, 3, 256, 256)
B - (1, 3, 256, 256) - this is a white image tensor
C - (256, 256) - this is the segment tensor
For instance C would look like:
tensor([[ 337,  337,  337,  ...,  340,  340,  340],
        [ 337,  337,  337,  ...,  340,  340,  340],
        [ 337,  337,  337,  ...,  340,  340,  340],
        ...,
        [1022, 1022, 1022,  ..., 1010, 1010, 1010],
        [1022, 1022, 1022,  ..., 1010, 1010, 1010],
        [1022, 1022, 1022,  ..., 1010, 1010, 1010]], device='cuda:0')

where 37 could indicate a building etc.
Tensor C gives the location of the segment shape. What I want is to copy the same segment based on the location from tensor A onto tensor B. This would be photoshopping the segment onto a white image tensor.
This is similar to masking and I looked into mask_select (https://pytorch.org/docs/stable/generated/torch.masked_select.html) but that only returns 1D tensor back.

Output:
You do not need to select the pixels in C, only to mask them:
select = 337  # which segment to select
select_mask = (C == select)[None, None, ...]  # create binary mask and add singleton dimensions
# this is the part where you select the right part of A
B = B * (1 - select_mask) + A * select_mask


Instruction: Differentiable affine transformation on patches of images in pytorch

Input:
I have a tensor of object bounding boxes, e.g. with the shape of [10,4] which correspond to a batch of images e.g. with shape [2,3,64,64] and transformation matrices for each object with shape [10,6] and a vector that defines which object index belongs to which image.
I would like to apply the affine transformations on patches of the images and replace those patches after applying the transformations. I am doing this using a for loop now, but the way I am doing it is not differntiable (I get the in place operation error from pytorch). I wanted to know if there is a differntiable way to do this. e.g. via grid_sample?
Here is my current code:
for obj_num in range(obj_vecs.shape[0]): #batch_size
    im_id = obj_to_img[obj_num]
    x1, y1, x2, y2 = boxes_pred[obj_num]
    im_patch = img[im_id, :, x1:x2, y1:y2]
    im_patch = im_patch[None, :, :, :]
    img[im_id, :, x1:x2, y1:y2] = self.VITAE.stn(im_patch, theta_mean[obj_num], inverse=False)[0]

Output:
There are a few ways to perform differentiable crops in PyTorch.
Let's take a minimal example in 2D:
>>> x1, y1, x2, y2 = torch.randint(0, 9, (4,))
(tensor(7), tensor(3), tensor(5), tensor(6))

>>> x = torch.randint(0, 100, (9,9), dtype=float, requires_grad=True)
tensor([[18., 34., 28., 41.,  1., 14., 77., 75., 23.],
        [62., 33., 64., 41., 16., 70., 47., 45., 19.],
        [20., 69.,  5., 51.,  1., 16., 20., 63., 52.],
        [51., 25.,  8., 30., 40., 67., 41., 27., 33.],
        [36.,  6., 95., 53., 69., 84., 51., 42., 71.],
        [46., 72., 88., 82., 71., 75., 86., 36., 15.],
        [66., 19., 58., 50., 91., 28.,  7., 83.,  4.],
        [94., 50., 34., 34., 92., 45., 48., 97., 76.],
        [80., 34., 19., 13., 77., 77., 51., 15., 13.]], dtype=torch.float64,
       requires_grad=True)

Given x1, x2 (resp. y1, y2 the patch index boundaries on the height dimension (resp. width dimension). You can get the grid of coordinates corresponding do you patch using a combination of torch.arange and torch.meshgrid:
>>> sorted_range = lambda a, b: torch.arange(a, b) if b >= a else torch.arange(b, a)
>>> xi, yi = sorted_range(x1, x2), sorted_range(y1, y2)
(tensor([3, 4, 5, 6]), tensor([5]))

>>> i, j = torch.meshgrid(xi, yi)
(tensor([[3],
         [4],
         [5],
         [6]]),
 tensor([[5],
         [5],
         [5],
         [5]]))

With that setup, you can extract and replace patches of x.

You can extract the patch by indexing x directly:
>>> patch = x[i, j].reshape(len(xi), len(yi))
tensor([[67.],
        [84.],
        [75.],
        [28.]], dtype=torch.float64, grad_fn=<ViewBackward>)

Here is the mask for illustration purposes:
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=torch.float64,
grad_fn=<IndexPutBackward>)

You can replace the values in x with the result from some transformation on the patch using torch.Tensor.index_put:
>>> values = 2*patch
 tensor([[134.],
         [168.],
         [150.],
         [ 56.]], dtype=torch.float64, grad_fn=<MulBackward0>)

>>> x.index_put(indices=(i, j), values=values)
tensor([[ 18.,  34.,  28.,  41.,   1.,  14.,  77.,  75.,  23.],
        [ 62.,  33.,  64.,  41.,  16.,  70.,  47.,  45.,  19.],
        [ 20.,  69.,   5.,  51.,   1.,  16.,  20.,  63.,  52.],
        [ 51.,  25.,   8.,  30.,  40., 134.,  41.,  27.,  33.],
        [ 36.,   6.,  95.,  53.,  69., 168.,  51.,  42.,  71.],
        [ 46.,  72.,  88.,  82.,  71., 150.,  86.,  36.,  15.],
        [ 66.,  19.,  58.,  50.,  91.,  56.,   7.,  83.,   4.],
        [ 94.,  50.,  34.,  34.,  92.,  45.,  48.,  97.,  76.],
        [ 80.,  34.,  19.,  13.,  77.,  77.,  51.,  15.,  13.]],
    dtype=torch.float64, grad_fn=<IndexPutBackward>)


Instruction: Dataset not found or corrupted. You can use download=True to download it

Input:
Recently I downloaded CelebA dataset from this page. I want to apply some transformations to this data set:
To do it firstly let's define transformations:
from torchvision import transforms
from torchvision.datasets CelebA
celeba_transforms = transforms.Compose([
    transforms.CenterCrop(130),
    transforms.Resize([64, 64]),
    transforms.ToTensor()
    ])

And now execute it:
CelebA(root='img_align_celeba',
       split='train',
       download=False,
       transform=celeba_transforms)

However result of this code is an error:
Dataset not found or corrupted. You can use download=True to download it

Setting download=True is also not working. Could you please help me with applying those transformations to this data set?

Output:
Finally I resolved the issue. I'm posting my solution:
Problem number one
There is a problem with downloading zip file img_align_celeba.zip due to reaching daily quota. Solution to this problem is simply downloading this file from internet e.g. Kaggle.
Problem number two
When using function CelebA function with download=True program will think for a while and then return error mentioned in a question title. Cause of the problem are broken .txt files which I listed below (those files are also downloaded via CelebA function):

For correct working of this function you have to download those .txt files directly from internet. I found them here. When you download all of those and replace old ones function CelebA should work without any problems.


Instruction: PIL to numpy and PIL to tensor is different

Input:
I have image and sum torch tensor and numpy array is diffrent, why?
How to torch_img.sum() = numpy_float_img.sum()?
from PIL import Image
from torchvision import transforms as T

# Read image with PIL
img = Image.open(img_path).resize((224,224))

torch_img = T.ToTensor()(img)
numpy_img = np.asarray(img)
numpy_img_float = np.asarray(img).astype(np.float32)

print(torch_img.sum(), numpy_img.sum(), numpy_img_float.sum())
->56914.496, 14513196, 14513196.0

Does anyone have any idea why?

Output:
Notice how torch_img is in the [0,1] range while numpy_img and numpy_img_float are both in the [0, 255] range. Looking at the documentation for torchvision.transforms.ToTensor, if the provided input is a PIL image, then the values will be mapped to [0, 1]. In contrast, numpy.array will have the values remain in the [0, 255] range.
Other than that the small variations in results are caused by different floating-point precisions.


Instruction: Why my trained model output is same for each random input?

Input:
I trained my model on the Python platform. after training, I faced up with same output for each random input. I solved this problem by deactivating BatchNorm layers with the model.eval() method. but when I tried to load my trained model in C++ with Pytorch C++ API, this problem showed up again, and model.eval() not helping me at this time. I faced the same output for each random input again.
This is my C++ model loading function:
std::vector<torch::jit::script::Module> module_loader(std::string file_addr) {
    std::vector<torch::jit::script::Module> modul;
    torch::jit::script::Module model = torch::jit::load(file_addr);
    model.eval();
    modul.push_back(model);
    return modul;
}

And this is my testing function:
void test(std::vector<torch::jit::script::Module> &model) {
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(torch::rand({1, 2, 64, 172}));
    torch::Tensor output = model[0].forward(inputs).toTensor();
    std::cout << output << std::endl;
}

After all, I put it all together in main() like this:
int main() {
    auto modul = module_loader(MODEL_ADDRESS);
    test(modul);
}

MODEL_ADDRESS is a macro for the address of the trained model on my local disk.
The output of the program is this for every run:

0.3231 [ CPUFloatType{1,1} ]

Output: I debugged multiple times my code and retry to save the model. Finally, I found the answer. I used a server to train my model, and for exporting the model to C++ I load my model weight in Python Shell into a model object. The problem is here, I should go to eval() before exporting the model for C++. C++ eval() does not work for the model is trained in Python. This solved my problem.


Instruction: Pyhon: the difference of "from XX import * " when XX is file created by myself or XX is python library file(like pytorch)

Input:
When i use pytorch,i am wondering why i can't import partial functions or variables from torchvision.models.vgg  using "from torchvision.models.vgg import *"
Like this:
from torchvision.models.vgg import *

if __name__=="__main__":
    print(cfgs["A"])

and
from torchvision.models.vgg import *

from typing import Union, List, Dict, Any, cast
cfgs: Dict[str, List[Union[str, int]]] = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

if __name__=="__main__":
    print(make_layers(cfgs["A"]))

variables "cfgs" and function "make_layers" can't be found
But i could use function like "vgg16_bn()" which is in the same file as "make_layers"
from torchvision.models.vgg import *

if __name__=="__main__":
    print(vgg16_bn())

I need to import like this:
from torchvision.models.vgg import cfgs,make_layers

if __name__=="__main__":
    print(make_layers(cfgs["A"]))

But if the file i import is created by myself like this:
b.py:
url="http"

def a ():
    return True

main.py could find variable "url" and function "a()"
from b import *

if __name__=="__main__":
    print(url)
    print(a())

I don't know why it happens like this,could someone explain it.Thanks!

Output:
torchvision/models/vgg.py sets this particular __all__ indicating their exported symbols:
__all__ = [
    "VGG",
    "vgg11",
    "vgg11_bn",
    "vgg13",
    "vgg13_bn",
    "vgg16",
    "vgg16_bn",
    "vgg19_bn",
    "vgg19",
]

* imports (while generally discouraged) will by default expose the names in __all__ (or if __all__ is missing, any names which do not start with _)


Instruction: Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU (while checking arguments for addmm

Input:
So I have received this error when running my simple regression code:
class linear_regression(torch.nn.Module):
def __init__(self, inputSize, outputSize):
    super(linear_regression, self).__init__()
    self.linear = torch.nn.Linear(inputSize, outputSize)

def forward(self, X):
    out = self.linear(X)
    return out

#fit_linear_reg(train_ds, train_X_ds, train_y_ds, test_X_ds, which_case, fold_no, p_t)
def fit_linear_reg(train_X_torch, train_y_torch, test_X_torch, case_type, fold_no, p_t):
    size_input = train_X_torch.shape[1]
    size_output = train_y_torch.shape[1]

model = linear_regression(size_input, size_output)
model.to(torch.device(device_name))
learningRate = 0.01
epochs = 1
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learningRate)
training_loss_list = []

for epoch in range(epochs):
    # Converting inputs and labels to Variable
    if torch.cuda.is_available():
        inputs = Variable(train_X_torch.cuda().float())
        labels = Variable(train_y_torch.cuda().float())
    else:
        inputs = Variable(train_X_torch.float())
        labels = Variable(train_y_torch.float())

    # Clear gradient buffers because we don't want any gradient from previous epoch to carry forward, dont want to cummulate gradients
    optimizer.zero_grad()

    # get output from the model, given the inputs
    outputs = 0
    if torch.cuda.is_available():
        outputs = model(inputs.to(device))
    else:
        outputs = model(inputs)

    # get loss for the predicted output
    loss = criterion(outputs, labels.)
    print(loss)
    # get gradients w.r.t to parameters
    loss.backward()

    # update parameters
    optimizer.step()

    training_loss_list.append(loss.item())
    #print('epoch {}, loss {}'.format(epoch, loss.item()))

torch.save(model.state_dict(), './results/weights_' + case_type + '_' +  str(fold_no) + '_' + p_t)

return (model(test_X_torch.float()), training_loss_list)

I have tried to pass my variabls to cuda, however, I am still receiving this error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-48-ccf21586daaf> in <module>
     40
     41         #print("khodesh shekl", test_X_ds.shape)
---> 42         preds, training_loss_list_pt = fit_general(train_X_ds, train_y_ds, test_X_ds, test_y_ds, which_case, k_no, p_t)
     43         k_no += 1
     44         #print("final shape", preds.shape)

<ipython-input-46-abaeea73fcec> in fit_general(train_X_ds, train_y_ds, test_X_ds, test_y_torch, which_case, fold_no, p_t)
      2
      3     if(which_case == "reg_simple"): #ok
----> 4         a, b = fit_linear_reg(train_X_ds, train_y_ds, test_X_ds, which_case, fold_no, p_t)
      5         return a, b
      6

<ipython-input-45-35542c8a0f30> in fit_linear_reg(train_X_torch, train_y_torch, test_X_torch, case_type, fold_no, p_t)
     54     torch.save(model.state_dict(), './results/weights_' + case_type + '_' +  str(fold_no) + '_' + p_t)
     55
---> 56     return (model(test_X_torch.float()), training_loss_list)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-45-35542c8a0f30> in forward(self, X)
      5
      6     def forward(self, X):
----> 7         out = self.linear(X)
      8         return out
      9

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU (while checking arguments for addmm)

Probably I have missed a variable to pass it to CUDA, but what can it be here?
Here is the code that I pass my data to the fit_linear_reg function:
#simple model

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device_name = device
print("running device: ", device)

#fixme: k-fold cross validation
n_crossVal = 2
kf = KFold(n_splits = n_crossVal) #, random_state=1, shuffle=True fixme

for p_t in key_set_1:
    print(p_t)
    cur_ds = []
    for i, roi in enumerate(key_set_2):
        if(i==0):
            cur_ds = brain_ds[p_t + '_' + roi]
        else:
            cur_ds = np.hstack((cur_ds, brain_ds[p_t + '_' + roi]))

    print(cur_ds.shape)
    print(n_train)
    size_input = cur_ds.shape[1]

    preds_case = np.zeros(glove_ds.shape)
    k_no = 0
    training_loss_list_pt = []

    # Linear regression
    for k_train_index, k_test_index in kf.split(cur_ds):

        train_X_ds = torch.from_numpy(cur_ds[k_train_index, :])
        train_y_ds = torch.from_numpy(glove_ds[k_train_index, :])
        train_ds   = TensorDataset(train_X_ds, train_y_ds)

        test_X_ds  = torch.from_numpy(cur_ds[k_test_index, :])
        test_y_ds = torch.from_numpy(glove_ds[k_test_index, :])
        test_ds   =  TensorDataset(test_X_ds, test_y_ds)

        #print("khodesh shekl", test_X_ds.shape)
        preds, training_loss_list_pt = fit_linear_reg(train_X_ds, train_y_ds, test_X_ds, test_y_ds, which_case, k_no, p_t)
        k_no += 1
        #print("final shape", preds.shape)
        preds_case[k_test_index, :] = preds.detach().numpy()

        plot_the_training_loss_in_each_fold(training_loss_list_pt, which_case, k_no, p_t) #todo
        print(training_loss_list_pt)

    print("prediction results (correlation) for " + p_t + " ")
    print(np.corrcoef(preds_case, torch.from_numpy(glove_ds))) #todo

    #calculate the shuffled data
    #compare the shuffled with normal ones

    #statistical test

    #Saving the results #todo

    #if(which_case == "MM")
    ## For MM:
    # Model ke train shod, create the updated glove using each fold,
    # do linear prediction using the new updated gloves
    # Compare
    # Put ifs for cases
    """
    train_X_ds = torch.from_numpy(cur_ds[:n_train, :])
    train_y_ds = torch.from_numpy(glove_ds[:n_train, :])
    train_ds   = TensorDataset(train_X_ds, train_y_ds)

    valid_X_ds = torch.from_numpy(cur_ds[n_train:n_train+n_val, :])
    valid_y_ds = torch.from_numpy(glove_ds[n_train:n_train+n_val, :])
    valid_ds   = TensorDataset(valid_X_ds, valid_y_ds)

    test_X_ds  = torch.from_numpy(cur_ds[n_train+n_val:, :])
    test_y_ds = torch.from_numpy(glove_ds[n_train+n_val:, :])
    test_ds   =  TensorDataset(test_X_ds, test_y_ds)

    print("analyzing", p_t, ' ', roi, ' :')

    fit_reg(train_ds, train_X_ds, train_y_ds)
    #model = MM_Net(train_X_ds, train_y_ds)

    print(train_X_ds.shape, valid_X_ds.shape, test_X_ds.shape)

    """

Output:
You haven't transferred your test data on the GPU:
model(test_X_torch.float().cuda())


Instruction: Error in 'from torchtext.data import Field, TabularDataset, BucketIterator, Iterator'

Input:
I am trying to implement this article https://towardsdatascience.com/bert-text-classification-using-pytorch-723dfb8b6b5b, but I have the following problem.
# Preliminaries
from torchtext.data import Field, TabularDataset, BucketIterator, Iterator

Error
ImportError: cannot import name 'Field' from 'torchtext.data' (/usr/local/lib/python3.7/dist-packages/torchtext/data/__init__.py)

OSError: /usr/local/lib/python3.7/dist-packages/torchtext/_torchtext.so: undefined symbol: _ZNK3c104Type14isSubtypeOfExtESt10shared_ptrIS0_EPSo

Output:
Try
from torchtext.legacy.data import Field, TabularDataset, BucketIterator, Iterator

Field is a legacy functionality of Torchtext since the 0.9 release. That article you linked was from before that release. If you've got the newest torchtext, but are trying to use legacy functionality, you need to use torchtext.legacy.*
Check out this tutorial for more info.
Also check this for full torchtext release notes.


Instruction: Jax/Flax (very) slow RNN-forward-pass compared to pyTorch?

Input:
I recently implemented a two-layer GRU network in Jax and was disappointed by its performance (it was unusable).
So, i tried a little speed comparison with Pytorch.
Minimal working example
This is my minimal working example and the output was created on Google Colab with GPU-runtime. notebook in colab
import flax.linen as jnn
import jax
import torch
import torch.nn as tnn
import numpy as np
import jax.numpy as jnp

def keyGen(seed):
    key1 = jax.random.PRNGKey(seed)
    while True:
        key1, key2 = jax.random.split(key1)
        yield key2
key = keyGen(1)

hidden_size=200
seq_length = 1000
in_features = 6
out_features = 4
batch_size = 8

class RNN_jax(jnn.Module):

    @jnn.compact
    def __call__(self, x, carry_gru1, carry_gru2):
        carry_gru1, x = jnn.GRUCell()(carry_gru1, x)
        carry_gru2, x = jnn.GRUCell()(carry_gru2, x)
        x = jnn.Dense(4)(x)
        x = x/jnp.linalg.norm(x)
        return x, carry_gru1, carry_gru2

class RNN_torch(tnn.Module):
    def __init__(self, batch_size, hidden_size, in_features, out_features):
        super().__init__()

        self.gru = tnn.GRU(
            input_size=in_features,
            hidden_size=hidden_size,
            num_layers=2
            )

        self.dense = tnn.Linear(hidden_size, out_features)

        self.init_carry = torch.zeros((2, batch_size, hidden_size))

    def forward(self, X):
        X, final_carry = self.gru(X, self.init_carry)
        X = self.dense(X)
        return X/X.norm(dim=-1).unsqueeze(-1).repeat((1, 1, 4))

rnn_jax = RNN_jax()
rnn_torch = RNN_torch(batch_size, hidden_size, in_features, out_features)

Xj = jax.random.normal(next(key), (seq_length, batch_size, in_features))
Yj = jax.random.normal(next(key), (seq_length, batch_size, out_features))
Xt = torch.from_numpy(np.array(Xj))
Yt = torch.from_numpy(np.array(Yj))

initial_carry_gru1 = jnp.zeros((batch_size, hidden_size))
initial_carry_gru2 = jnp.zeros((batch_size, hidden_size))

params = rnn_jax.init(next(key), Xj[0], initial_carry_gru1, initial_carry_gru2)

def forward(params, X):

    carry_gru1, carry_gru2 = initial_carry_gru1, initial_carry_gru2

    Yhat = []
    for x in X: # x.shape = (batch_size, in_features)
        yhat, carry_gru1, carry_gru2 = rnn_jax.apply(params, x, carry_gru1, carry_gru2)
        Yhat.append(yhat) # y.shape = (batch_size, out_features)

    #return jnp.concatenate(Y, axis=0)

jitted_forward = jax.jit(forward)

Results
# uncompiled jax version
%time forward(params, Xj)

CPU times: user 7min 17s, sys: 8.18 s, total: 7min 25s Wall time: 7min 17s
# time for compiling
%time jitted_forward(params, Xj)

CPU times: user 8min 9s, sys: 4.46 s, total: 8min 13s Wall time: 8min 12s
# compiled jax version
%timeit jitted_forward(params, Xj)

The slowest run took 204.20 times longer than the fastest. This could mean that an intermediate result is being cached. 10000 loops, best of 5: 115 µs per loop
# torch version
%timeit lambda: rnn_torch(Xt)

10000000 loops, best of 5: 65.7 ns per loop
Questions
Why is my Jax-implementation so slow? What am i doing wrong?
Also, why is compiling taking so long? The sequence is not that long..
Thank you :)

Output:
The reason the JAX code compiles slowly is that during JIT compilation JAX unrolls loops. So in terms of XLA compilation, your function is actually very large: you call rnn_jax.apply() 1000 times, and compilation times tend to be roughly quadratic in the number of statements.
By contrast, your pytorch function uses no Python loops, and so under the hood it is relying on vectorized operations that run much faster.
Any time you use a for loop over data in Python, a good bet is that your code will be slow: this is true whether you're using JAX, torch, numpy, pandas, etc. I'd suggest finding an approach to the problem in JAX that relies on vectorized operations rather than relying on slow Python looping.


Instruction: How to add weight normalisation to PyTorch's pretrained VGG16?

Input:
I want to add weight normalization to PyTorch pre-trained VGG-16.
One possible solution which I can think of is as follows,
from torch.nn.utils import weight_norm as wn
import torchvision.models as models

class ResnetEncoder(nn.Module):
    def __init__(self):
        super(ResnetEncoder, self).__init__()
        ...
        self.encoder = models.vgg16(pretrained=True).features
        ...
    def forward(self, input_image):
        self.features = []
        x = (input_image - self.mean) / self.std

        self.features.append(self.encoder(x))
        ...

        return self.features

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.encoder = ResnetEncoder() # this is basically VGG16
        self.decoder = DepthDecoder(self.encoder.num_ch_enc)
        for k,m in self.encoder.encoder._modules.items():
            if isinstance(m,nn.Conv2d):
                m = wn(m)

    def forward(self,x):
        return self.decoder(self.encoder(x))

vgg_backbone_model = Net()
vgg_backbone_model.train()
...

But I do not know if this is the correct way to add weight normalization to pre-trained VGG16.

Output:
You should be using nn.Module.modules instead of accessing the _modules attribute.
Doing m = wn(m) won't update the parameters of the layer but instead make a copy and overwrite the local variable m. Instead, you should override the layer itself from the nn.Module, one way to do such thing is to use setattr:
for k, v in model.named_modules():
    if isinstance(v, nn.Conv2d):
        setattr(model, k, weight_norm(v))


Instruction: AttributeError: 'GPT2Model' object has no attribute 'gradient_checkpointing'

Input:
I am trying to load a GPT2 fine tuned model in flask initially. The model is being loaded during the init functions using:
app.modelgpt2 = torch.load('models/model_gpt2.pt', map_location=torch.device('cpu'))
app.modelgpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

But while performing the prediction task as followed in the snippet below:
from flask import current_app
input_ids = current_app.modelgpt2tokenizer.encode("sample sentence here", return_tensors='pt')
sample_outputs = current_app.modelgpt2.generate(input_ids,
                                                do_sample=True,
                                                top_k=50,
                                                min_length=30,
                                                max_length=300,
                                                top_p=0.95,
                                                temperature=0.7,
                                                num_return_sequences=1)

It throws the following error as mentioned in the question:
AttributeError: 'GPT2Model' object has no attribute 'gradient_checkpointing'

The error trace is listed starting from the model.generate function:
File "/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)

File "/venv/lib/python3.8/site-packages/transformers/generation_utils.py", line 1017, in generate
return self.sample(

File "/venv/lib/python3.8/site-packages/transformers/generation_utils.py", line 1531, in sample
outputs = self(

File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)

File "/venv/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1044, in forward
transformer_outputs = self.transformer(

File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)

File "/venv/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 861, in forward
print(self.gradient_checkpointing)

File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(

AttributeError: 'GPT2Model' object has no attribute 'gradient_checkpointing'

Checked with modeling_gpt2.py, by default self.gradient_checkpointing is set False in the constructor of the class.

Output:
This issue is found to be occurring only if the framework is run using venv or deployment frameworks like uWSGI or gunicorn.
It is resolved when transformers version 4.10.0 is used instead of the latest package.


Instruction: Constructing parameter groups in pytorch

Input:
In the torch.optim documentation, it is stated that model parameters can be grouped and optimized with different optimization hyperparameters. It says that

For example, this is very useful when one wants to specify per-layer
learning rates:
optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)

This means that model.base’s parameters will use the default
learning rate of 1e-2, model.classifier’s parameters will use a
learning rate of 1e-3, and a momentum of 0.9 will be used for all
parameters.

I was wondering how to define such groups that have parameters() attribute. What came to my mind was something in the form of
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.base()
        self.classifier()

        self.relu = nn.ReLU()

    def base(self):
        self.fc1 = nn.Linear(1, 512)
        self.fc2 = nn.Linear(512, 264)

    def classifier(self):
        self.fc3 = nn.Linear(264, 128)
        self.fc4 = nn.Linear(128, 964)

    def forward(self, y0):

        y1 = self.relu(self.fc1(y0))
        y2 = self.relu(self.fc2(y1))
        y3 = self.relu(self.fc3(y2))

        return self.fc4(y3)

How should I modify the snippet above to be able to get model.base.parameters()? Is the only way to define a nn.ParameterList and explicitly add weights and biases of the desired layers to that list? What is the best practice?

Output:
I will show three approaches to solving this. In the end though, it comes down to personal preference.

- Grouping parameters with nn.ModuleDict.
I noticed here an answer using nn.Sequential to group the layers which allow to
target different sections of the model using the parameters attribute of nn.Sequential. Indeed base and classifier might be more than sequential layers. I believe a more general approach is to leave the module as is, but instead, initialize an additional nn.ModuleDict module which will contain all parameters ordered by the optimization group in separate nn.ModuleLists:
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(1, 512)
        self.fc2 = nn.Linear(512, 264)
        self.fc3 = nn.Linear(264, 128)
        self.fc4 = nn.Linear(128, 964)

        self.params = nn.ModuleDict({
            'base': nn.ModuleList([self.fc1, self.fc2]),
            'classifier': nn.ModuleList([self.fc3, self.fc4])})

    def forward(self, y0):
        y1 = self.relu(self.fc1(y0))
        y2 = self.relu(self.fc2(y1))
        y3 = self.relu(self.fc3(y2))
        return self.fc4(y3)

Then you can define your optimizer with:
optim.SGD([
    {'params': model.params.base.parameters()},
    {'params': model.params.classifier.parameters(), 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

Do note MyModel's parameters' generator won't contain duplicate parameters.

- Creating an interface for accessing parameter groups.
A different solution is to provide an interface in the nn.Module to separate the parameters into groups:
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(1, 512)
        self.fc2 = nn.Linear(512, 264)
        self.fc3 = nn.Linear(264, 128)
        self.fc4 = nn.Linear(128, 964)

    def forward(self, y0):
        y1 = self.relu(self.fc1(y0))
        y2 = self.relu(self.fc2(y1))
        y3 = self.relu(self.fc3(y2))
        return self.fc4(y3)

    def base_params(self):
        return chain(m.parameters() for m in [self.fc1, self.fc2])

    def classifier_params(self):
        return chain(m.parameters() for m in [self.fc3, self.fc4])

Having imported itertools.chain as chain.
Then define your optimizer with:
optim.SGD([
    {'params': model.base_params()},
    {'params': model.classifier_params(), 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

- Using child nn.Modules.
Lastly, you can define your module sections as submodules (here it comes down as the method as the nn.Sequential one, yet you can generalize this to any submodules).
class Base(nn.Sequential):
    def __init__(self):
        super().__init__(nn.Linear(1, 512),
                         nn.ReLU(),
                         nn.Linear(512, 264),
                         nn.ReLU())

class Classifier(nn.Sequential):
    def __init__(self):
        super().__init__(nn.Linear(264, 128),
                         nn.ReLU(),
                         nn.Linear(128, 964))

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.base = Base()
        self.classifier = Classifier()

    def forward(self, y0):
        features = self.base(y0)
        out = self.classifier(features)
        return out

Here again you can use the same interface as the first method:
optim.SGD([
    {'params': model.base.parameters()},
    {'params': model.classifier.parameters(), 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

I would argue this is the best practice. However, it forces you to define each of your components into separate nn.Module, which can be a hassle when experimenting with more complex models.


Instruction: What is the formal definition for "view" in this context?

Input:
Consider the following paragraph from the sub-section named The essence of tensors from the section named Tensors: Multidimensional arrays of the chapter named It starts with a tensor from the book titled Deep Learning with PyTorch by Eli Stevens et al.

Python lists or tuples of numbers are collections of Python objects
that are individually allocated in memory,  as shown on the left
in figure  3.3.  PyTorch tensors or  NumPy arrays, on the other
hand, are views over (typically) contiguous memory blocks
containing unboxed C numeric types rather than Python objects. Each
element is a 32-bit (4-byte) float in this case, as we can see on the
right side of figure 3.3. This means storing a 1D tensor of 1,000,000
float numbers will require exactly 4,000,000 contiguous bytes, plus a
small overhead for the metadata (such as dimensions and numeric type).

And the figure they are referring to is shown below, taken from the book

The above paragraph is saying that tensors are views over contiguous memory blocks. What exactly is meant by a view in this context?

Output:
A "view" is how you interpret this data, or more precisely, the shape of the tensor. For example, given a memory block with 40 contiguous bytes (10 contiguous floats), you can either view it as a 2x5 tensor, or a 5x2 tensor.
In pytorch, the API to change the view of a tensor is view(). Some examples:
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> x = torch.randn(10, dtype=torch.float32)
>>> x.shape
torch.Size([10])
>>>
>>> x = x.view(2, 5)
>>> x.shape
torch.Size([2, 5])
>>>
>>> x = x.view(5, 2)
>>> x.shape
torch.Size([5, 2])

Of course, some views are forbidden for 10 floats:
>>> x = x.view(3, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: shape '[3, 3]' is invalid for input of size 10

view does not change the data in the underlying memory. It merely changes how you "view" the tensor.


Instruction: Iterable pytorch dataset with multiple workers

Input:
So I have a text file bigger than my ram memory, I would like to create a dataset in PyTorch that reads line by line, so I don't have to load it all at once in memory. I found pytorch IterableDataset as potential solution for my problem. It only works as expected when using 1 worker, if using more than one worker it will create duplicate recods. Let me show you an example:
Having a testfile.txt containing:
0 - Dummy line
1 - Dummy line
2 - Dummy line
3 - Dummy line
4 - Dummy line
5 - Dummy line
6 - Dummy line
7 - Dummy line
8 - Dummy line
9 - Dummy line

Defining a IterableDataset:
class CustomIterableDatasetv1(IterableDataset):

    def __init__(self, filename):

        #Store the filename in object's memory
        self.filename = filename

    def preprocess(self, text):

        ### Do something with text here
        text_pp = text.lower().strip()
        ###

        return text_pp

    def line_mapper(self, line):

        #Splits the line into text and label and applies preprocessing to the text
        text, label = line.split('-')
        text = self.preprocess(text)

        return text, label

    def __iter__(self):

        #Create an iterator
        file_itr = open(self.filename)

        #Map each element using the line_mapper
        mapped_itr = map(self.line_mapper, file_itr)

        return mapped_itr

We can now test it:
base_dataset = CustomIterableDatasetv1("testfile.txt")
#Wrap it around a dataloader
dataloader = DataLoader(base_dataset, batch_size = 1, num_workers = 1)
for X, y in dataloader:
    print(X,y)

It outputs:

('0',) (' Dummy line\n',)
('1',) (' Dummy line\n',)
('2',) (' Dummy line\n',)
('3',) (' Dummy line\n',)
('4',) (' Dummy line\n',)
('5',) (' Dummy line\n',)
('6',) (' Dummy line\n',)
('7',) (' Dummy line\n',)
('8',) (' Dummy line\n',)
('9',) (' Dummy line',)

That is correct. But If I change the number of workers to 2 the output becomes
('0',) (' Dummy line\n',)
('0',) (' Dummy line\n',)
('1',) (' Dummy line\n',)
('1',) (' Dummy line\n',)
('2',) (' Dummy line\n',)
('2',) (' Dummy line\n',)
('3',) (' Dummy line\n',)
('3',) (' Dummy line\n',)
('4',) (' Dummy line\n',)
('4',) (' Dummy line\n',)
('5',) (' Dummy line\n',)
('5',) (' Dummy line\n',)
('6',) (' Dummy line\n',)
('6',) (' Dummy line\n',)
('7',) (' Dummy line\n',)
('7',) (' Dummy line\n',)
('8',) (' Dummy line\n',)
('8',) (' Dummy line\n',)
('9',) (' Dummy line',)
('9',) (' Dummy line',)

Which is incorrect, as is creating duplicates of each sample per worker in the data loader.
Is there a way to solve this issue with pytorch? So a dataloader can be created to not load all file in memory with support for multiple workers.

Output:
So I found an answer within the torch discuss forum https://discuss.pytorch.org/t/iterable-pytorch-dataset-with-multiple-workers/135475/3 where they pointed out I should be using the worker info to slice consecutively to the batch size.
The new dataset would look like this:
class CustomIterableDatasetv1(IterableDataset):

    def __init__(self, filename):

        #Store the filename in object's memory
        self.filename = filename

    def preprocess(self, text):

        ### Do something with text here
        text_pp = text.lower().strip()
        ###

        return text_pp

    def line_mapper(self, line):

        #Splits the line into text and label and applies preprocessing to the text
        text, label = line.split('-')
        text = self.preprocess(text)

        return text, label

    def __iter__(self):
        worker_total_num = torch.utils.data.get_worker_info().num_workers
        worker_id = torch.utils.data.get_worker_info().id
        #Create an iterator
        file_itr = open(self.filename)

        #Map each element using the line_mapper
        mapped_itr = map(self.line_mapper, file_itr)

        #Add multiworker functionality
        mapped_itr = itertools.islice(mapped_itr, worker_id, None, worker_total_num)

        return mapped_itr

Special thanks to @Ivan who also pointed out the slicing solution.
With two workers it returns the same data as 1 worker only


Instruction: How to normalize pytorch model output to be in range [0,1]

Input:
lets say I have model called UNet
output = UNet(input)

that output is a vector of grayscale images shape: (batch_size,1,128,128)
What I want to do is to normalize each image to be in range [0,1].
I did it like this:
for i in range(batch_size):
   output[i,:,:,:] = output[i,:,:,:]/torch.amax(output,dim=(1,2,3))[i]

now every image in the output is normalized, but when I'm training such model, pytorch claim it cannot calculate the gradients in this procedure, and I understand why.
my question is what is the right way to normalize image without killing the backpropogation flow?
something like
output = UNet(input)
output = output.normalize
output2 = some_model(output)
loss = ..
loss.backward()
optimize.step()

my only option right now is adding a sigmoid activation at the end of the UNet but i dont think its a good idea..
update - code (gen2,disc = unet,discriminator models. est_bias is some output):

update 2x code:
with torch.no_grad():
            est_bias_for_disc = gen2(input_img)

            est_bias_for_disc /= est_bias_for_disc.amax(dim=(1,2,3), keepdim=True)
        disc_fake_hat = disc(est_bias_for_disc.detach())
        disc_fake_loss = BCE(disc_fake_hat, torch.zeros_like(disc_fake_hat))

        disc_real_hat = disc(bias_ref)
        disc_real_loss = BCE(disc_real_hat, torch.ones_like(disc_real_hat))
        disc_loss = (disc_fake_loss + disc_real_loss) / 2
        if epoch<=epochs_till_gen2_stop:
            disc_loss.backward(retain_graph=True) # Update gradients
            opt_disc.step() # Update optimizer

then theres seperate training:
 opt_gen2.zero_grad()
 est_bias = gen2(input_img)
 est_bias /= est_bias.amax(dim=(1,2,3), keepdim=True)
 disc_fake = disc(est_bias)
 ADV_loss = BCE(disc_fake, torch.ones_like(disc_fake))
 gen2_loss = ADV_loss
 gen2_loss.backward()
 opt_gen2.step()

Output:
You are overwriting the tensor's value because of the indexing on the batch dimension. Instead, you can perform the operation in vectorized form:
output = output / output.amax(dim=(1,2,3), keepdim=True)

The keepdim=True argument keeps the shape of torch.Tensor.amax's output equal to that of its inputs allowing you to perform an in-place operation with it.


Instruction: Understanding the PyTorch implementation of Conv2DTranspose

Input:
I am trying to understand an example snippet that makes use of the PyTorch transposed convolution function, with documentation here, where in the docs the author writes:

"The padding argument effectively adds dilation * (kernel_size - 1) -
padding amount of zero padding to both sizes of the input."

Consider the snippet below where a sample image of shape [1, 1, 4, 4] containing all ones is input to a ConvTranspose2D operation with arguments stride=2 and padding=1 with a weight matrix of shape (1, 1, 4, 4) that has entries from a range between 1 and 16 (in this case dilation=1 and added_padding = 1*(4-1)-1 = 2)
sample_im = torch.ones(1, 1, 4, 4).cuda()
sample_deconv = nn.ConvTranspose2d(1, 1, 4, 2, 1, bias=False).cuda()
sample_deconv.weight = torch.nn.Parameter(
    torch.tensor([[[[ 1.,  2.,  3.,  4.],
                    [ 5.,  6.,  7.,  8.],
                    [ 9., 10., 11., 12.],
                    [13., 14., 15., 16.]]]]).cuda())

Which yields:
>>> sample_deconv(sample_im)
tensor([[[[ 6., 12., 14., 12., 14., 12., 14.,  7.],
          [12., 24., 28., 24., 28., 24., 28., 14.],
          [20., 40., 44., 40., 44., 40., 44., 22.],
          [12., 24., 28., 24., 28., 24., 28., 14.],
          [20., 40., 44., 40., 44., 40., 44., 22.],
          [12., 24., 28., 24., 28., 24., 28., 14.],
          [20., 40., 44., 40., 44., 40., 44., 22.],
          [10., 20., 22., 20., 22., 20., 22., 11.]]]], device='cuda:0',
       grad_fn=<CudnnConvolutionTransposeBackward>)

Now I have seen simple examples of transposed convolution without stride and padding. For instance, if the input is a 2x2 image  [[2, 4], [0, 1]], and the convolutional filter with one output channel is [[3, 1], [1, 5]], then the resulting tensor of shape (1, 1, 3, 3) can be seen as the sum of the four colored matrices in the image below:

The problem is I can't seem to find examples that use strides and/or padding in the same visualization. As per my snippet, I am having a very difficult time understanding how the padding is applied to the sample image, or how the stride works to get this output. Any insights appreciated, even just understanding how the 6 in the (0,0) entry or the 12 in the (0,1) entry of the resulting matrix are computed would be very helpful.

Output:
The output spatial dimensions of nn.ConvTranspose2d are given by:
out = (x - 1)s - 2p + d(k - 1) + op + 1

where x is the input spatial dimension and out the corresponding output size, s is the stride, d the dilation, p the padding, k the kernel size, and op the output padding.

If we keep the following operands:

For each value of the input, we compute a buffer (of the corresponding color) by calculating the product with each element of the kernel.
Here are the visualizations for s=1, p=0, s=1, p=1, s=2, p=0, and s=2, p=1:

s=1, p=0: output is 3x3

For the blue buffer, we have (1) 2*k_top-left = 2*3 = 6; (2) 2*k_top-right = 2*1 = 2; (3) 2*k_bottom-left = 2*1 = 2; (4) 2*k_bottom-right = 2*5 = 10.

s=1, p=1: output is 1x1

s=2, p=0: output is 4x4

s=2, p=2: output is 2x2


Instruction: Convolution and convolution transposed do not cancel each other

Input:
I'm trying to implement an autoencoder CNN. However, I have the following problem:
The last convolutional layer of my encoder is defined as follows:
Conv2d(128, 256, 3, padding=1, stride=2)

The input of this layer has shape (1, 128, 24, 24). Thus, the output has shape (1, 256, 12, 12).
After this layer, I have ReLU activation and BatchNorm. Neither of these changes the shape of the output.
Then I have a first ConvTranspose2d layer defined as:
ConvTranspose2d(256, 128, 3, padding=1, stride=2)

But the output of this layer has shape (1, 128, 23, 23).
As far as I know, if we use the same kernel size, stride, and padding in ConvTrapnpose2d as in the preceding Conv2d layer, then the output of this 2 layers block must have the same shape as its input.
So, my question is: what is wrong with my understanding? And how can I fix this issue?

Output:
I would first like to note that the nn.ConvTranspose2d layer is not the inverse of nn.Conv2d as explained in its documentation page:

it is not an actual deconvolution operation as it does not compute a true inverse of convolution

As far as I know, if we use the same kernel size, stride, and padding in ConvTranspose2d as in the preceding Conv2d layer, then the output of this 2 layers block must have the same shape as its input.

This is not always true! It depends on the input spatial dimensions.
In terms of spatial dimensions the 2D convolution will output:
out = [(x + 2p - d(k - 1) - 1)/s + 1]

where [x] is the whole part of x.
while the 2D transpose convolution will output:
out = (x - 1)s - 2p + d(k - 1) + op + 1

where x = input_dimension, out = output_dimension, k = kernel_size, s = stride, d = dilation, p = padding, and op = output_padding.
If you look at the convT o conv operator (i.e. convT(conv(x))) then you have:
out = (out_conv - 1)s - 2p + d(k - 1) + op + 1
    = ([(x + 2p - d(k - 1) - 1)/s + 1] - 1)s - 2p + d(k - 1) + op + 1

Which equals to x only if we have [(x + 2p - d(k - 1) - 1)/s + 1] = (x + 2p - d(k - 1) - 1)/s + 1, that is: if x is odd, in this case:
out = ((x + 2p - d(k - 1) - 1)/s + 1 - 1)s - 2p + d(k - 1) + op + 1
    = x + op

And out = x when op = 0.
Otherwise if x is even then:
out = x - 1 + op

And setting op = 1 gives out = x.

Here is an example:
>>> conv = nn.Conv2d(1, 1, 3, stride=2, padding=1)

>>> convT = nn.ConvTranspose2d(1, 1, 3, stride=2, padding=1)
>>> convT(conv(torch.rand(1, 1, 25, 25))).shape # x even
(1, 1, 25, 25) #<- out = x

>>> convT = nn.ConvTranspose2d(1, 1, 3, stride=2, padding=1, output_padding=1)
>>> convT(conv(torch.rand(1, 1, 24, 24))).shape # x odd
(1, 1, 24, 24) #<- out = x - 1 + op


Instruction: module 'torch' has no attribute 'linalg'

Input:
I am getting this error AttributeError: module 'torch' has no attribute 'linalg' when updating the parameters using optimizer.step(model.closure).
I am using Pytorch version 1.4.0.

Output: linalg was introduced to pytorch only on a later version (1.7.0). Update pytorch and try again.


Instruction: pytorch save and load model

Input:
Is there any difference between original model and saved then loaded model?
Before training, I just saved model and then loaded because I wanted to know if there is any changes during saving and loading.
Here's my code
just model for test
class test_model(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 50, kernel_size = 3, stride=1, padding=1, bias = False)
        self.maxpool1 = nn.MaxPool2d(2, 2)
        self.bn1 = nn.BatchNorm2d(50)

        self.conv2_ = nn.Conv2d(in_channels = 50, out_channels = 10, kernel_size = 1, stride=1, padding=0, bias = False)
        self.conv2 = nn.Conv2d(in_channels = 10, out_channels = 50, kernel_size = 3, stride=1, padding=1, bias = False)
        self.maxpool2 = nn.MaxPool2d(2, 2)
        self.bn2 = nn.BatchNorm2d(50)

        self.conv3_ = nn.Conv2d(in_channels = 50, out_channels = 10, kernel_size = 1, stride=1, padding=0, bias = False)
        self.conv3 = nn.Conv2d(in_channels = 10, out_channels = 50, kernel_size = 3, stride=1, padding=1, bias = False)
        self.maxpool3 = nn.MaxPool2d(2, 2)
        self.bn3 = nn.BatchNorm2d(50)

        self.conv4_ = nn.Conv2d(in_channels = 50, out_channels = 20, kernel_size = 1, stride=1, padding=0, bias = False)
        self.conv4 = nn.Conv2d(in_channels =20, out_channels = 100, kernel_size = 3, stride=1, padding=1, bias = False)
        self.maxpool4 = nn.MaxPool2d(2, 2)
        self.bn4 = nn.BatchNorm2d(100)

        self.conv5_ = nn.Conv2d(in_channels = 100, out_channels = 10, kernel_size = 1, stride=1, padding=0, bias = False)
        self.conv5 = nn.Conv2d(in_channels = 10, out_channels = 100, kernel_size = 3, stride=1, padding=1, bias = False)
        self.maxpool5 = nn.MaxPool2d(2, 2)
        self.bn5 = nn.BatchNorm2d(100)

        self.fc = nn.Sequential(Flatten(),
                               nn.Linear(100*7*7, 100),
                                nn.ReLU(),
                               nn.Linear(100,100))

    def forward(self, inputs):
        feature_map1 = self.conv1(inputs)
        feature_map1 = self.maxpool1(feature_map1)
        feature_map1 = self.bn1(feature_map1)

        feature_map2 = self.conv2_(feature_map1)
        feature_map2 = self.conv2(feature_map2)
        feature_map2 = self.maxpool2(feature_map2)
        feature_map2 = self.bn2(feature_map2)

        feature_map3 = self.conv3_(feature_map2)
        feature_map3 = self.conv3(feature_map3)
        feature_map3 = self.maxpool3(feature_map3)
        feature_map3 = self.bn3(feature_map3)

        feature_map4 = self.conv4_(feature_map3)
        feature_map4 = self.conv4(feature_map4)
        feature_map4 = self.maxpool4(feature_map4)
        feature_map4 = self.bn4(feature_map4)

        feature_map5 = self.conv5_(feature_map4)
        feature_map5 = self.conv5(feature_map5)
        feature_map5 = self.maxpool5(feature_map5)
        feature_map5 = self.bn5(feature_map5)

        output = self.fc(feature_map5)

        return output

then
model_cpu = test_model()

save and load
torch.save(model_cpu, '/home/mskang/hyeokjong/model_cpu.pt')
model_load = torch.load('/home/mskang/hyeokjong/model_cpu.pt')

and
model_load  == model_cpu
------------------------------------
False

However
print(model_load)
print(model_cpu)

are seemed same
furthermore
I also trained two models(model_load, model_cpu) and results looks same too.
So I think those tow models are same and should be same
But why False

Output: They have the same underlying model but are different Python objects. That is why __eq__ returns False when trying model_load == model_cpu. You can see model_load and model_cpu as two copies of the same nn.Module.


Instruction: Why does PATH differ when I connect to my Docker container with ssh or with exec/attach?

Input:
I build a Docker image based on the following Dockerfile:
ARG PYTORCH="1.6.0"
ARG CUDA="10.1"
ARG CUDNN="7"

FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel

ENV TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0+PTX"
ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"

# SSH SERVER
RUN apt-get update && apt-get install openssh-server sudo -y
RUN echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
RUN echo 'root:root' | chpasswd

WORKDIR /

EXPOSE 22
CMD ["service ssh start"]

I launch the Docker container with
docker run -it -d -p 7220:22 --name ssh-server-test ssh-server-image /bin/bash

If I connect to the container with docker exec -it ssh-server-test /bin/bash or docker attach ssh-server-test, I get the PATH I expect:
root@9264667daf83:/# echo $PATH
/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

However, if I start an ssh server with
root@9264667daf83:/# service ssh start
 * Starting OpenBSD Secure Shell server sshd                                                                                [ OK ]
root@9264667daf83:/#

and I connect to the Docker container through ssh as root, then the PATH is completely different!
root@9264667daf83:~# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Why? And how can I get the right PATH variable also when I connect to the container through ssh?

Output:
The environment variable you are asking of has been set by docker itself, using ENV clause. Check the source code of the initial Dockerfile.
Variables that are set by ENV exist on the build stage, run, and when you exec into a running container. More here.
But when you SSH into the container, the usual Linux path for sourcing files like ~/.bashrc is working. But there is no PATH with conda, nvidia, etc in these files.
As a workaround, you can patch /root/.bashrc on the build stage with the corresponding export PATH. For example, you can add to the Dockerfile
RUN echo 'export PATH=/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:$PATH' >> /root/.bashrc

UPD
I case if you want to use exactly the same PATH as on the build stage, you can use
RUN echo "export PATH=${PATH}" >> /root/.bashrc


Instruction: Combing two torchvision.dataset objects into a single DataLoader in PyTorch

Input:
I am training a GANS on the Cifar-10 dataset in PyTorch (and hence don't need train/val/test splits), and I want to be able to combine the torchvision.datasets.CIFAR10 in the snippet below to form one single torch.utils.data.DataLoader iterator. My current solution is something like :
import torchvision
import torch
batch_size = 128
cifar_trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=False)
cifar_testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=False)
cifar_dl1 = torch.utils.data.DataLoader(cifar_trainset, batch_size=batch_size, num_workers=12, persistent_workers=True,
                                          shuffle=True, pin_memory=True)
cifar_dl2 = torch.utils.data.DataLoader(cifar_testset, batch_size=batch_size, num_workers=12, persistent_workers=True,
                                          shuffle=True, pin_memory=True)

And then in my training loop I have something like:
for dl in [cifar_dl1, cifar_l2]:
   for data in dl:
      # training

The problem with this approach in a multi-threaded context, where I have found for my setup and this task that the optimal number of workers is 12, is that now I am declaring 24 workers in total which is clearly too many, not to mention the start-up time costs associated with re-iterating over each dataloader in spite of the benefits of the persistent workers flag for each.
Any solutions to this problem much appreciated.

Output:
You can use ConcatDataset from torch.utils.data module.
Code Snippet:
import torch
import torchvision

batch_size = 128

cifar_trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=False)
cifar_testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=False)

cifar_dataset = torch.utils.data.ConcatDataset([cifar_trainset, cifar_testset])

cifar_dataloader = torch.utils.data.DataLoader(cifar_dataset, batch_size=batch_size, num_workers=12, persistent_workers=True,
                                          shuffle=True, pin_memory=True)

for data in cifar_dataloader:
    # training


Instruction: RuntimeError: shape '[10, 3, 150, 150]' is invalid for input of size 472500

Input:
I'm trying to perform a Convolutional operation on the covid CT Dataset and constantly getting this error. My image size in the train loader was (10, 150, 150, 3) and I reshaped it into [10, 3, 150, 150], using torch.reshape(). Can anybody help me with problem
    My CNN Code
class BConv(nn.Module):
def __init__(self, out=3):
    super(BConv, self).__init__()
    #(10, 150, 150, 3)
    self.conv1=nn.Conv2d(in_channels=3,out_channels=12,kernel_size=3,stride=1,padding=1)
    self.bn1=nn.BatchNorm2d(num_features=12)
    self.relu1=nn.ReLU()
    self.pool=nn.MaxPool2d(kernel_size=2)
    self.conv2=nn.Conv2d(in_channels=12,out_channels=20,kernel_size=3,stride=1,padding=1)
    self.relu2=nn.ReLU()
    # self.conv3=nn.Conv2d(in_channels=20,out_channels=32,kernel_size=3,stride=1,padding=1)
    # self.bn3=nn.BatchNorm2d(num_features=32)
    # self.relu3=nn.ReLU()
    self.fc=nn.Linear(in_features= 20*75*75, out_features=3)

def forward(self,input):
    output=self.conv1(input)
    #print("output 1", output.shape)
    output=self.bn1(output)
    #print("output 1", output.shape)
    output=self.relu1(output)
    #print("output 1", output.shape)
    output=self.pool(output)
    #print("output 1", output.shape)
    output=self.conv2(output)
    #print("output 1", output.shape)
    output=self.relu2(output)
    #print("output 1", output.shape)
    # output=self.conv3(output)
    # output=self.bn3(output)
    # output=self.relu3(output)
    print(output.shape)

        #Above output will be in matrix form, with shape (256,32,75,75)

    output=output.view(output.size(0), -1)

    output=self.fc(output)

    return output

Data Preprocessing
class Ctdataset(Dataset):
    def __init__(self, path):

        self.data= pd.read_csv(path, delimiter=" ")
        data= self.data.values.tolist()
        self.image= []
        self.labels=[]
        for i in data:
            self.image.append(i[0])
            self.labels.append(i[1])

        #print(len(self.image), len(self.labels))
        #self.class_map = {"0": 0, "1":1 , "2": 2}

    def __len__(self):
        return len(self.image)

    def __getitem__(self, idx):
        img_path = os.path.join("2A_images", self.image[idx])
        img= Image.open(img_path).convert("RGB")
        img= img.resize((150, 150))
        img= np.array(img)
        img= img.astype(float)
        return img, label

Output:
Here I'm considering your whole model including the third block consisting of conv3, bn3, and relu3. There are a few things to note:

Reshaping is substantially different from permuting the axes. When you say you have an input shape of (batch_size, 150, 150, 3), it means the channel axis is last. Since PyTorch 2D builtin layers work in the NHW format you need to permute the axes: you can do so with torch.Tensor.permute:
>>> x = torch.rand(10, 150, 150, 3)
>>> x.permute(0, 3, 1, 2).shape
(10, 3, 150, 150)

Assuming your input is shaped (batch_size, 3, 150, 150), then the output shape of relu3 will be (32, 75, 75). As such the following fully connected layer must have exactly 32*75*75 input features.

However you need to flatten this tensor as you did in your code with a view: output = output.view(output.size(0), -1). Another approach is to define a self.flatten = nn.Flatten() layer and call it with output = self.flatten(output).

As of PyTorch v1.8.0, an alternative to setting the in_features in your fully connected layer is to use nn.LazyLinear which will initialize it for you based on the first inference:
>>> self.fc = nn.LazyLinear(out_features=3)

Side note: you don't need to define separate ReLU layers with relu1, relu2, and relu3 as they're non-parametric functions:
>>> self.relu = nn.ReLU()

Here is the full code for reference:
class BConv(nn.Module):
    def __init__(self, out=3):
        super().__init__()

        # input shape (10, 150, 150, 3)
        self.conv1 = nn.Conv2d(3, 12,kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(num_features=12)
        self.pool = nn.MaxPool2d(kernel_size=2)

        self.conv2 = nn.Conv2d(12, 20,kernel_size=3, stride=1, padding=1)

        self.conv3 = nn.Conv2d(20, 32, kernel_size=3, stride=1, padding=1)
        self.bn3 = nn.BatchNorm2d(num_features=32)

        self.relu = nn.ReLU()
        self.flatten = nn.Flatten()
        self.fc = nn.Linear(in_features=32*75*75, out_features=out)

    def forward(self,input):
        output = input.permute(0, 3, 1, 2)

        output = self.conv1(output)
        output = self.bn1(output)
        output = self.relu(output)
        output = self.pool(output)

        output = self.conv2(output)
        output = self.relu(output)

        output = self.conv3(output)
        output = self.bn3(output)
        output = self.relu(output)

        output = self.flatten(output)
        output = self.fc(output)

        return output


Instruction: Augmentation in torch vision transform is not working as expected

Input:
I'm developing a CNN using pytorch. my model gives good accuracy on both training and test set without augmentation but I wanted to learn augmentation so I have used torchvision transforms for the augmentation and after applying the augmentation model started doing worst and loss is not at all decreasing. so I tried to debug and observed that the augmented image looks distorted/unexpected can somebody please help me solve this.
custom datset
class traindataset(Dataset):
    def __init__(self,data,train_end_idx,augmentation = None):
        '''
        data: data is a pandas dataframe generated from csv file where it has columns-> [name,labels,col 1,col2,...,col784]. shape of data->(10000, 786)

        '''
        self.data=data
        self.augmentation=augmentation
        self.train_end=train_end_idx
        self.target=self.data.iloc[:self.train_end,1].values
        self.image=self.data.iloc[:self.train_end,2:].values#contains full data

    def __len__(self):
        return len(self.target);
    def __getitem__(self,idx):

        self.target=self.target
        self.ima=self.image[idx].reshape(1,784) #only takes the selected index
        if self.augmentation is not None:
            self.ima = self.augmentation(self.ima)

        return torch.tensor(self.target[idx]),self.ima

Augmentation used
torchvision_transform = transforms.Compose([
    np.uint8,
    transforms.ToPILImage(),
    transforms.Resize((28,28)),
    transforms.RandomRotation([45,135]),
    transforms.ToTensor()
    ])

Augmented image(PFA for the picture)
transformed=torchvision_transform(x)
plt.imshow(transformed.squeeze().numpy(), interpolation='nearest')
plt.show()

Normal image
x=data.iloc[:1,2:].values
plt.imshow(x.reshape(28,28), interpolation='nearest')
plt.show()

The first image is with augmentation and the second image is without augmentation.
if you want you can play with the code here without downloading anything.

Output:
It seems like the transforms.Resize() function did not correctly reshape the tensor. Reshaping first seems to fix the issue and produce correct images (you did this step for the albumentations section).
transformed = torchvision_transform(x.reshape(28,28))


Instruction: Indexing a tensor with None in PyTorch

Input:
I've seen this syntax to index a tensor in PyTorch, not sure what it means:
v = torch.div(t, n[:, None])

where v, t, and n are tensors.
What is the role of "None" here? I can't seem to find it in the documentation.

Output:
Similar to NumPy you can insert a singleton dimension ("unsqueeze" a dimension) by indexing this dimension with None. In turn n[:, None] will have the effect of inserting a new dimension on dim=1. This is equivalent to n.unsqueeze(dim=1):
>>> n = torch.rand(3, 100, 100)

>>> n[:, None].shape
(3, 1, 100, 100)

>>> n.unsqueeze(1).shape
(3, 1, 100, 100)

Here are some other types of None indexings.
In the example above : is was used as a placeholder to designate the first dimension dim=0. If you want to insert a dimension on dim=2, you can add a second : as n[:, :, None].
You can also place None with respect to the last dimension instead. To do so you can use the ellipsis syntax ...:

n[..., None] will insert a dimension last, i.e. n.unsqueeze(dim=-1).

n[..., None, :] on the before last dimension, i.e. n.unsqueeze(dim=-2).


Instruction: Install PyTorch without installing Python 3.9 on Win7

Input: Got clean installation of Anaconda (Python 3.8.8), tried to install PyTorch by running conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch as suggested by the docs. The thing is, conda tries to install Python 3.9.7 in the process, which does not support Windows 7 (the famous "api-ms-win-core-path-l1-1-0.dll missing error"). Is there a way to make the installation of PyTorch without switching to the latest Python?

Output: If you start with a clean package with a specific python version you could use the --freeze-installed flag to prevent the installer from making any changes to the installed packages, see documentation.


Instruction: Prevent Updating for Specific Element of Convolutional Weight Matrix

Input: I’m trying to set one element of weight to 1 and then hold it the same until the end of learning (prevent it from updating in the next epochs). I know I can set requires_grad = False but I just want this process for one element not all of the elements.

Output:
You can attach a backward hook on your nn.Module such that during backpropagation you can overwrite the element of interest to 0. This makes sure its value never changes without preventing backpropagation of the gradient to the input.
The new API for backward hooks is nn.Module.register_full_backward_hook. First construct a callback function that will be used as the layer hook:
def freeze_single(index):
    def callback(module, grad_input, grad_output):
        module.weight.grad.data[index] = 0
    return callback

Then, we can attach this hook to any nn.Module. For instance, here I've decided to freeze component [0, 1, 2, 1] of the convolutional layer:
>>> conv = nn.Conv2d(3, 1, 3)
>>> conv.weight.data[0, 1, 2, 1] = 1

>>> conv.register_full_backward_hook(freeze_single((0, 1, 2, 1)))

Everything is set correctly, let us try:
>>> x = torch.rand(1, 3, 10, 10, requires_grad=True)
>>> conv(x).mean().backward()

Here we can verify the gradient of component [0, 1, 2, 1] is indeed equal to 0:
>>> conv.weight.grad
tensor([[[[0.4954, 0.4776, 0.4639],
          [0.5179, 0.4992, 0.4856],
          [0.5271, 0.5219, 0.5124]],

         [[0.5367, 0.5035, 0.5009],
          [0.5703, 0.5390, 0.5207],
          [0.5422, 0.0000, 0.5109]], # <-

         [[0.4937, 0.5150, 0.5200],
          [0.4817, 0.5070, 0.5241],
          [0.5039, 0.5295, 0.5445]]]])

You can detach/reattach the hook anytime with:
>>> hook = conv.register_full_backward_hook(freeze_single((0, 1, 2, 1)))
>>> hook.remove()

Don't forget if you remove the hook, the value of that component will change when you update your weights. You will have to reset it to 1 if you so desire. Otherwise, you can implement a second hook - a register_forward_pre_hook hook this time - to handle that.


Instruction: Is there any efficient way to calculate covariance matrix using PyTorch?

Input:
I want to calculate covariance matrix from vectors a and b, like k[i][j] = exp( -(a[i]-b[j])**2 ).
In numpy, I can write as follows,
import numpy as np

r = np.subtract.outer(a, b)
k = np.exp(-r*r)

In PyTorch, I can write naive code, but it's slower than numpy.
import torch

for i in range(len(a)):
    for j in range(len(b)):
        k[i][j] = torch.exp( -(a[i]-b[j])**2 )

How should I write efficient code using PyTorch?

Output:
You can use broadcasting:
r = a[:, None] - b[None, :]
k = torch.exp(-r**2)


Instruction: How to create a PyTorch hook with conditions?

Input:
I'm learning about hooks and working with binarized neural network. The issue is that sometimes my gradients are 0 in the backwards pass. I'm trying to replace those gradients with a certain value.
Say I have the following network
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Model()

opt = optim.Adam(net.parameters())

And also some features
features = torch.rand((3,1))

I can train it normally using:
for i in range(10):
    opt.zero_grad()
    out = net(features)
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss.backward()
    opt.step()

How can I attach a hook function that will have the following conditions for the backwards pass (for each layer):

If all the gradients in a single layer are 0, change them to 1.0.

If one of the gradients is 0 but the there's at least one gradient that is not 0, change it to 0.5.

Output:
You can attach a callback function on your nn.Module with nn.Module.register_full_backward_hook:
You will have to handle both cases: if all elements are equal to zero using torch.all, else (i.e. at least one is non zero) if at least one is equal to zero using torch.any.
def grad_mod(module, grad_inputs, grad_outputs):
    if module.weight.grad is None: # safety measure for last layer
        return None                # and layers w/ require_grad=False

    flat = module.weight.grad.view(-1)
    if torch.all(flat == 0):
        flat.data.fill_(1.)
    elif torch.any(flat == 0):
        flat.data.scatter_(0, (flat == 0).nonzero()[:,0], value=.5)

The instruction in the first clause will fill all values to 1. while the instruction in the second will only replace zero values with .5.
Attach the hook on an nn.Module:
>>> net.fc3.register_full_backward_hook(grad_mod)

Here I use print statements before and after mutating flat to showcase the effect of the hook:
>>> net(torch.rand((3,1))).backward(torch.tensor([[0],[1],[2]]))
>>> tensor([0.0947, 0.0000, 0.0000]) # before
>>> tensor([0.0947, 0.5000, 0.5000]) # after

>>> net(torch.rand((3,1))).backward(torch.tensor([[0],[1],[2]]))
>>> tensor([0., 0., 0.])             # before
>>> tensor([1., 1., 1.])             # after

In order to apply this hook to multiple layers you can wrap grad_mod and utilize nn.Module.apply recursive behavior:
>>> def apply_grad_mod(module):
...     if hasattr(module, 'weight'):
...         module.register_full_backward_hook(grad_mod)

Then the following will apply the hook on all layer weights.
>>> net.apply(apply_grad_mod)

Note: you will have to extend this behavior if you wish to also affect the biases!


Instruction: Is there a pytorch function to find unique tuples in a given tensor (of size N*h*w*2)?

Input:
I am trying to extract the unique tuples in a (N * h * w * 2) tensor.
For example, an 1 * 2 * 3 * 2 tensor where there are 6 tuples: a = torch.tensor([[[[1,2], [2,3], [3,4]],  [[4,5], [1,2], [3,4]]]])
and I am trying to find the indices of the unique tuples (i.e., indices of [1,2], [2,3], [3,4], [4,5], where duplicates are removed).
I've already checked out torch.unique(), but it seems not working.

Output:
You compute the difference between all pairs:
d = torch.abs(a.view(-1, 1, 2) - a.view(1, -1, 2)).sum(dim=-1)

Then you can find pairs with zero difference (masking non-unique pairs using triu):
i, j = torch.where((d + torch.triu(torch.ones_like(d))) == 0)

Resulting with:
i,j
(tensor([4, 5]), tensor([0, 2]))

That is the 4th pair in a is identical to the 0th, and the 5th is identical to the second.


Instruction: Predicting Sentiment of Raw Text using Trained BERT Model, Hugging Face

Input:
I'm predicting sentiment analysis of Tweets with positive, negative, and neutral classes. I've trained a BERT model using Hugging Face. Now I'd like to make predictions on a dataframe of unlabeled Twitter text and I'm having difficulty.
I've followed the following tutorial (https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/) and was able to train a BERT model using Hugging Face.
Here's an example of predicting on raw text however it's only one sentence and I would like to use a column of Tweets. https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/#predicting-on-raw-text
review_text = "I love completing my todos! Best app ever!!!"

encoded_review = tokenizer.encode_plus(
  review_text,
  max_length=MAX_LEN,
  add_special_tokens=True,
  return_token_type_ids=False,
  pad_to_max_length=True,
  return_attention_mask=True,
  return_tensors='pt',
)

input_ids = encoded_review['input_ids'].to(device)
attention_mask = encoded_review['attention_mask'].to(device)
output = model(input_ids, attention_mask)
_, prediction = torch.max(output, dim=1)
print(f'Review text: {review_text}')
print(f'Sentiment  : {class_names[prediction]}')

Review text: I love completing my todos! Best app ever!!!
Sentiment  : positive

Bill's response works. Here's the solution.
def predictionPipeline(text):
  encoded_review = tokenizer.encode_plus(
      text,
      max_length=MAX_LEN,
      add_special_tokens=True,
      return_token_type_ids=False,
      pad_to_max_length=True,
      return_attention_mask=True,
      return_tensors='pt',
    )

  input_ids = encoded_review['input_ids'].to(device)
  attention_mask = encoded_review['attention_mask'].to(device)

  output = model(input_ids, attention_mask)
  _, prediction = torch.max(output, dim=1)

  return(class_names[prediction])

df2['prediction']=df2['cleaned_tweet'].apply(predictionPipeline)

Output:
You can use the same code to predict texts from the dataframe column.
model = ...
tokenizer = ...

def predict(review_text):
    encoded_review = tokenizer.encode_plus(
    review_text,
    max_length=MAX_LEN,
    add_special_tokens=True,
    return_token_type_ids=False,
    pad_to_max_length=True,
    return_attention_mask=True,
    return_tensors='pt',
    )

    input_ids = encoded_review['input_ids'].to(device)
    attention_mask = encoded_review['attention_mask'].to(device)
    output = model(input_ids, attention_mask)
    _, prediction = torch.max(output, dim=1)
    print(f'Review text: {review_text}')
    print(f'Sentiment  : {class_names[prediction]}')
    return class_names[prediction]

df = pd.DataFrame({
            'texts': ["text1", "text2", "...."]
        })

df_dataset["sentiments"] = df.apply(lambda l: predict(l.texts), axis=1)