Instruction: Boolean value of Tensor with more than one value is ambiguous

Input:
I have this class of NN:
class Block(nn.Module):
    def __init__(self, in_planes, out_planes, stride=1):
        super(Block, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, in_planes, kernel_size=3, stride=stride,
             padding=1, groups=in_planes, bias=False)
        self.bn1 = nn.BatchNorm2d(in_planes)

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = nn.ReLU(out)
        return out

I create a model and pass it the random input, and it shows the error:
model = Block(3,3, 1)
x = torch.rand(64, 3, 100, 100)
model(x)

I received this error:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

Output:
The issue is with the nn.ReLU() in the feedforward(). I was printing it which is not possible in ipynb file.
class Block(nn.Module):

    def __init__(self, in_planes, out_planes, stride=1):
        super(Block, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, in_planes, kernel_size=3, stride=stride,
             padding=1, groups=in_planes, bias=False)
        self.bn1 = nn.BatchNorm2d(in_planes)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        return out


Instruction: forward() using Pytorch Lightning not giving consistent binary classification results for single VS multiple images

Input:
I have trained a Variational Autoencoder (VAE) with an additional fully connected layer after the encoder for binary image classification. It is setup using PyTorch Lightning. The encoder / decoder is resnet18 from PyTorch Lightning Bolts repo.
from pl_bolts.models.autoencoders.components import (
    resnet18_encoder,
    resnet18_decoder
)

class VariationalAutoencoder(LightningModule):

...

    self.first_conv: bool = False
    self.maxpool1: bool = False
    self.enc_out_dim: int = 512
    self.encoder = resnet18_encoder(first_conv, maxpool1)
    self.fc_object_identity = nn.Linear(self.enc_out_dim, 1)

    def forward(self, x):
        x_encoded = self.encoder(x)
        mu = self.fc_mu(x_encoded)
        log_var = self.fc_var(x_encoded)
        p, q, z = self.sample(mu, log_var)

        x_classification_score = torch.sigmoid(self.fc_object_identity(x_encoded))

        return self.decoder(z), x_classification_score

variational_autoencoder = VariationalAutoencoder.load_from_checkpoint(
        checkpoint_path=str(checkpoint_file_path)
    )

with torch.no_grad():
    predicted_images, classification_score = variational_autoencoder(test_images)

The reconstructions work well for single images and multiple images when passed through forward(). However, when I pass multiple images to forward() I get different results for the classification score than if I pass a single image tensor:
# Image 1 (class=1) [1, 3, 64, 64]
x_classification_score = 0.9857

# Image 2 (class=0) [1, 3, 64, 64]
x_classification_score = 0.0175

# Image 1 and 2 [2, 3, 64, 64]
x_classification_score =[[0.8943],
                         [0.1736]]

Why is this happening?

Output:
You are using resnet18 which has a torch.nn.BatchNorm2d layer.
Its behavior changes whether it is in train or eval mode. It calculates mean and variance across batch during training and hence its output is dependent on examples in this batch.
In evaluation mode mean and variance gathered during training via moving average are used which is batch independent, hence results are the same.


Instruction: PyTorch: why does running output = model(images) use so much GPU memory?

Input:
In trying to understand why my maximum batch size is limited for my PyTorch model, I noticed that it's not the model itself nor loading the tensors onto the GPU that uses the most memory. Most memory is used up when generating a prediction for the first time, e.g. with the following line in the training loop:
output = model(images)

where images is some input tensor, and model is my PyTorch model. Before running the line, I have something like 9GB of GPU memory available, and afterwards I'm down to 2.5GB (it then further drops to 1GB available after running loss = criterion(outputs, labels).
Two questions:

Is this normal?
Why is it happening? What is all that memory being used for? From what I understand the model is already loaded in, and the actual input tensors are already on the GPU before making that call. The output tensors themselves can't be that big. Does it have something to do with storing the computational graph?

Output:
This is normal: The key here is that all intermediate tensors (the whole computation graph) have to be stored if you want to compute the gradient via backward-mode differentiation.
You can aviod that by using the .no_grad context manager:
with torch.no_grad():
    output = model(images)

You will observe that a lot less memory is used, because no computation graph will be stored. But this also means that you can't compute the derivatives anymore. It is however the standard way if you just want to evaluate the model without the need of any optimization.
There is one way to reduce the memory cosumption if you still want to optimize, and it is called checkpointing. Whenever you need an intermediate tensor in the backward pass, it will be computed again from the input (or actually from the last "checkpoint"), without storing an intermediate tensor up to that tensor. But this is just computationally more expensive. You're trading memory against computational time.


Instruction: Why does huggingface tokenizer return only 1 `input_ids` instead of 3?

Input:
I'm trying to tokenize the squad dataset following the huggingface tutorial:
from datasets import load_dataset
from transformers import RobertaTokenizer
from transformers import logging
logging.set_verbosity_error()

dataset = load_dataset('squad')
checkpoint = 'roberta-base'
tokenizer = RobertaTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
    return tokenizer(example['question'], example['context'], [d['text'][0] for d in example['answers']], truncation=True)

tokenized_datasets = dataset['train'].map(tokenize_function, batched=True)

But when I print
tokenized_datasets

I get
Dataset({
    features: ['id', 'title', 'context', 'question', 'answers', 'input_ids', 'attention_mask'],
    num_rows: 87599
})

But shouldn't this return 3 input_ids, one for the question one for the context and one for the answer?

Output:
Is that line of code:
tokenizer(example['question'], example['context'], [d['text'][0] for d in example['answers']], truncation=True)

shown in the course?
A Tokenizer accepts plenty of parameters with its __call__ method (documentation). Since you have only specified truncation by its name, the other parameter values are determined by their position. That means, you are executing:
tokenizer(text=example['question'], text_pair=example['context'], add_special_tokens=[d['text'][0] for d in example['answers']], truncation=True)

After you execute your code the sample with the id 5733be284776f41900661182 becomes:
{'id': '5733be284776f41900661182',
'title': 'University_of_Notre_Dame',
'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
'answers': {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]},
'input_ids': [0, 3972, 2661, 222, 5, 9880, 2708, 2346, 2082, 11, 504, 4432, 11, 226, 2126, 10067, 1470, 116, 2, 2, 37848, 37471, 28108, 6, 5, 334, 34, 10, 4019, 2048, 4, 497, 1517, 5, 4326, 6919, 18, 1637, 31346, 16, 10, 9030, 9577, 9, 5, 9880, 2708, 4, 29261, 11, 760, 9, 5, 4326, 6919, 8, 2114, 24, 6, 16, 10, 7621, 9577, 9, 4845, 19, 3701, 62, 33161, 19, 5, 7875, 22, 39043, 1459, 1614, 1464, 13292, 4977, 845, 4130, 7, 5, 4326, 6919, 16, 5, 26429, 2426, 9, 5, 25095, 6924, 4, 29261, 639, 5, 32394, 2426, 16, 5, 7461, 26187, 6, 10, 19035, 317, 9, 9621, 8, 12456, 4, 85, 16, 10, 24633, 9, 5, 11491, 26187, 23, 226, 2126, 10067, 6, 1470, 147, 5, 9880, 2708, 2851, 13735, 352, 1382, 7, 6130, 6552, 625, 3398, 208, 22895, 853, 1827, 11, 504, 4432, 4, 497, 5, 253, 9, 5, 1049, 1305, 36, 463, 11, 10, 2228, 516, 14, 15230, 149, 155, 19638, 8, 5, 2610, 25336, 238, 16, 10, 2007, 6, 2297, 7326, 9577, 9, 2708, 4, 2],
'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

The input_ids are the concatenation of text and text_pair:
tokenizer.decode([0, 3972, 2661, 222, 5, 9880, 2708, 2346, 2082, 11, 504, 4432, 11, 226, 2126, 10067, 1470, 116, 2, 2, 37848, 37471, 28108, 6, 5, 334, 34, 10, 4019, 2048, 4, 497, 1517, 5, 4326, 6919, 18, 1637, 31346, 16, 10, 9030, 9577, 9, 5, 9880, 2708, 4, 29261, 11, 760, 9, 5, 4326, 6919, 8, 2114, 24, 6, 16, 10, 7621, 9577, 9, 4845, 19, 3701, 62, 33161, 19, 5, 7875, 22, 39043, 1459, 1614, 1464, 13292, 4977, 845, 4130, 7, 5, 4326, 6919, 16, 5, 26429, 2426, 9, 5, 25095, 6924, 4, 29261, 639, 5, 32394, 2426, 16, 5, 7461, 26187, 6, 10, 19035, 317, 9, 9621, 8, 12456, 4, 85, 16, 10, 24633, 9, 5, 11491, 26187, 23, 226, 2126, 10067, 6, 1470, 147, 5, 9880, 2708, 2851, 13735, 352, 1382, 7, 6130, 6552, 625, 3398, 208, 22895, 853, 1827, 11, 504, 4432, 4, 497, 5, 253, 9, 5, 1049, 1305, 36, 463, 11, 10, 2228, 516, 14, 15230, 149, 155, 19638, 8, 5, 2610, 25336, 238, 16, 10, 2007, 6, 2297, 7326, 9577, 9, 2708, 4, 2])

Output:
<s>To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?</s></s>Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.</s>

That is a common approach to handling extractive questions-answering tasks. In this, the answers are not seen as input but are only needed as a target (i.e. predicting start and end position).
Edit:
The OP specified the question in the comments and wants to know how the input_ids of the three text entities: question, context, and answer can be returned. All that needs to be changed is that the tokenize_function encodes the entities independently and returns a dict:
from datasets import load_dataset
from transformers import RobertaTokenizer

dataset = load_dataset('squad')
checkpoint = 'roberta-base'
tokenizer = RobertaTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
    question_o = tokenizer(example['question'], truncation=True)
    context_o = tokenizer(example['context'], truncation=True)
    answer_o = tokenizer([d['text'][0] for d in example['answers']], truncation=True)

    return {"question_input_ids": question_o.input_ids, "question_attention_mask": question_o.attention_mask, "context_input_ids": context_o.input_ids, "context_attention_mask": context_o.attention_mask, "answer_input_ids": answer_o.input_ids, "answer_attention_mask": answer_o.attention_mask}

tokenized_datasets = dataset['train'].map(tokenize_function, batched=True)


Instruction: How to move PyTorch model to GPU on Apple M1 chips?

Input:
On 18th May 2022, PyTorch announced support for GPU-accelerated PyTorch training on Mac.
I followed the following process to set up PyTorch on my Macbook Air M1 (using miniconda).
conda create -n torch-nightly python=3.8

$ conda activate torch-nightly

$ pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

I am trying to execute a script from Udacity's Deep Learning Course available here.
The script moves the models to GPU using the following code:
G.cuda()
D.cuda()

However, this will not work on M1 chips, since there is no CUDA.
If we want to move models to M1 GPU and our tensors to M1 GPU, and train entirely on M1 GPU, what should we be doing?

If Relevant: G and D are Discriminator and Generators for GAN's.
class Discriminator(nn.Module):

    def __init__(self, conv_dim=32):
        super(Discriminator, self).__init__()
        self.conv_dim = conv_dim
        # complete init function
        self.cv1 = conv(in_channels=3, out_channels=conv_dim, kernel_size=4, stride=2, padding=1, batch_norm=False)            # 32*32*3  -> 16*16*32
        self.cv2 = conv(in_channels=conv_dim, out_channels=conv_dim*2, kernel_size=4, stride=2, padding=1, batch_norm=True)    # 16*16*32 -> 8*8*64
        self.cv3 = conv(in_channels=conv_dim*2, out_channels=conv_dim*4, kernel_size=4, stride=2, padding=1, batch_norm=True)  # 8*8*64   -> 4*4*128
        self.fc1 = nn.Linear(in_features = 4*4*conv_dim*4, out_features = 1, bias=True)

    def forward(self, x):
        # complete forward function
        out = F.leaky_relu(self.cv1(x), 0.2)
        out = F.leaky_relu(self.cv2(x), 0.2)
        out = F.leaky_relu(self.cv3(x), 0.2)
        out = out.view(-1, 4*4*conv_dim*4)
        out = self.fc1(out)
        return out

D = Discriminator(conv_dim)

class Generator(nn.Module):
    def __init__(self, z_size, conv_dim=32):
        super(Generator, self).__init__()
        self.conv_dim = conv_dim
        self.z_size = z_size
        # complete init function
        self.fc1 = nn.Linear(in_features = z_size, out_features = 4*4*conv_dim*4)
        self.dc1 = deconv(in_channels = conv_dim*4, out_channels = conv_dim*2, kernel_size=4, stride=2, padding=1, batch_norm=True)
        self.dc2 = deconv(in_channels = conv_dim*2, out_channels = conv_dim, kernel_size=4, stride=2, padding=1, batch_norm=True)
        self.dc3 = deconv(in_channels = conv_dim, out_channels = 3, kernel_size=4, stride=2, padding=1, batch_norm=False)

    def forward(self, x):
        # complete forward function
        x = self.fc1(x)
        x = x.view(-1, conv_dim*4, 4, 4)
        x = F.relu(self.dc1(x))
        x = F.relu(self.dc2(x))
        x = F.tanh(self.dc3(x))
        return x

G = Generator(z_size=z_size, conv_dim=conv_dim)

Output:
This is what I used:
if torch.backends.mps.is_available():
    mps_device = torch.device("mps")
    G.to(mps_device)
    D.to(mps_device)

Similarly for all tensors that I want to move to M1 GPU, I used:
tensor_ = tensor_(mps_device)

Some operations are ot yet implemented using MPS, and we might need to set a few environment variables to use CPU fall back instead:
One error that I faced during executing the script was
# NotImplementedError: The operator 'aten::_slow_conv2d_forward' is not current implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

To solve it I set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1
conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1
conda activate <test-env>

References:

https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/
https://pytorch.org/docs/master/notes/mps.html
https://sebastianraschka.com/blog/2022/pytorch-m1-gpu.html
https://sebastianraschka.com/blog/2022/pytorch-m1-gpu.html
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables


Instruction: PyTorch- call to backward() on two different functions in sequence gives unexpected output second time

Input:
I am trying to differentiate two functions z and y which are based on same input x. While doing so, I run y.backward() and z.backward().  As per my understanding, I have created two functions before doing any operation on the input so both y and z should be independent and give independent results of differentiation.
However, the output of second call to backward() gives incorrect output. It can be called as  y.backward() then z.backward(). In this case, z.backward() gives 14 as output instead of 12. If I run z.backward() and then y.backward(), y.backward() gives 14 as output. Only first output is correct in both cases. I cannot understand how is it giving 14 in second time.
import torch
x = torch.tensor(2.0, requires_grad=True)
y = 2 * x + 3
z = x**3 + 1

y.backward()
print('grad attribute of the tensor::',x.grad)

z.backward()
print('grad attribute of the tensor::',x.grad)

Output:
grad attribute of the tensor:: tensor(2.)
grad attribute of the tensor:: tensor(14.)

Output:
backward() does not overwrite tensor grad attributes; it accumulates them. If you don't zero out the gradients in between backpropagations, the resulting gradients will be the sum of the gradients from each backpropagation. The reason that it's implemented like this is to better support recurrent neural networks.
The most conventional way to zero out the gradients is by calling torch.optim.Optimizer.zero_grad(); this will zero out the gradients of all of the parameters passed to the optimizer on construction. This works well when you're just using the gradients for an optimizer step. There's also torch.nn.Module.zero_grad(), which zeroes out the module's parameters (I believe it's recursive, so it should also zero out the sub-modules' parameters, and so on).
Also note that if you need to reuse intermediate results (gradients of non-leaf tensors), then you need to pass retain_graph=True when calling backward(); this is usually the case when working with recurrent neural networks. Otherwise, PyTorch will free them to conserve memory.


Instruction: PyTorch's custom dataset class should inherit from torch.utils.data.Dataset or not - why both work?

Input:
I was studying the PyTorch's Dataset class. From what I knew beforehand, we need to inherit from torch.utils.data.Dataset everytime we create a CustomDataset class of our own; and further we need to override the __len__ and __getitem__ methods as per need.
But, I got to know that it isn't always necessary to inherit and we could go on to create our CustomDataset class with __len__ and __getitem__ methods without inheriting from torch.utils.data.Dataset and even then the behaviour of an instance of custom dataset remains pretty much same (I tested it myself).
That to say, len(cust_data) would return the length of the dataset we pass while creating our cust_data instance, and we could even index cust_data like cust_data[0] and it would return what's returned by __getitem__ method in our CustomDataset class.
My questions are -

What is the need of inheriting when we are just as fine without inheriting - and if we are not, what functionality do we miss on if we do not inherit? When is inheriting recommended and when is it not? (While the official docs recommend to inherit, always)

When not inheriting, how did the instance know it needs to call the __getitem__ method when it is indexed?

Any answers appreciated.

Output:
Take a look at the source code for torch.utils.data.Dataset - it is an abstract class, which guarantees that every class inherting it must implement __getitem__. In other words, you don't "need" to inherit Dataset: as long as __getitem__ is properly implemented your dataset class will work fine. The reason of why doing so has become common practice is that it indicates to a third-party (e.g. some other code that uses your dataset class, someone else reading your code) that the class in question has __getitem__ implemented. It provides a common interface for PyTorch datasets.

Executing someClass[i] will automatically call someClass.__getitem__ with parameter i (and will throw an error if __getitem__ is not implemented). This is a Python built-in feature and has nothing to do with whichever base class you are inheriting. You can Google "dunder methods" to learn more about these special behaviors.


Instruction: How to run a GNN example with Pytorch, on a CPU without CUDA?

Input:
I am trying to code a GNN example problem as shown in the given link: https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8
I am using a Macbook Pro 2016 edition, without a Nvidia graphic card!
The example problem is implementing 'CUDA' toolkit. Can I somehow modify the code and run in on my current laptop? I have made the dataset sufficiently small, such that it does not requires high computation and can run on my PC!
The part of the code which is giving an error is as follows!
def train():
model.train()

loss_all = 0
for data in train_loader:
    data = data.to(device)
    optimizer.zero_grad()
    output = model(data)
    label = data.y.to(device)
    loss = crit(output, label)
    loss.backward()
    loss_all += data.num_graphs * loss.item()
    optimizer.step()
return loss_all / len(train_dataset)

device = torch.device('cuda')
model = Net().to(device) # Net = A class inherited from torch.nn.Module

optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
crit = torch.nn.BCELoss()
train_loader = DataLoader(train_dataset, batch_size=batch_size)
for epoch in range(num_epochs):
    train()

The error is as follows
AssertionError: Torch not compiled with CUDA enabled

Output:
You are using:
device = torch.device('cuda')

If you like to use cpu please change to:
device = torch.device('cpu')


Instruction: Fusing the features of two classes and feeding them with the fused features

Input:
In the main class, there are two classes and firstly the first class is recalled and after that the second class is recalled. I want to use a module that receives the features from these two classes and does some calculations and finally each of the mentioned classes needs to receive the outcome of the module.
The idea that comes in my mind is that importing the feature of the first class in to the second class and after that applying the module in the second class but my question is that in this scenario it is not possible to import the outcome of the module into the first class.
For example for these two classes and the module class:
class first(nn.Module):
    def __init__(self, in_planes=128, out_planes=64, kernel_size=3, stride=1, padding=0):
        super(first, self).__init__()
        self.conv_s = nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, bias=False, groups=in_planes)
        self.bn_s = nn.BatchNorm2d(out_planes)
        self.relu_s = nn.ReLU()

    def forward(self, x):
        x = self.conv_s(x)
        y1 = self.bn_s(x)
        x = self.relu_s(x)
        return x

class second(nn.Module):
    def __init__(self, in_planes=128, out_planes=64, kernel_size=3, stride=1, padding=0):
        super(second, self).__init__()
        self.conv_s = nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, bias=False, groups=in_planes)
        self.bn_s = nn.BatchNorm2d(out_planes)
        self.relu_s = nn.ReLU()
 def forward(self, x):
        x = self.conv_s(x)
        y2 = self.bn_s(x)
        x = self.relu_s(x)
        return x

The Module class:
class module(nn.Module):
def __init__(self):
    super(module, self).__init__()
    self.conv1h = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn1h   = nn.BatchNorm2d(64)
    self.conv2h = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn2h   = nn.BatchNorm2d(64)
    self.conv3h = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn3h   = nn.BatchNorm2d(64)
    self.conv4h = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn4h   = nn.BatchNorm2d(64)

    self.conv1v = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn1v   = nn.BatchNorm2d(64)
    self.conv2v = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn2v   = nn.BatchNorm2d(64)
    self.conv3v = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn3v   = nn.BatchNorm2d(64)
    self.conv4v = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
    self.bn4v   = nn.BatchNorm2d(64)

def forward(self, left, down):
    if down.size()[2:] != left.size()[2:]:
        down = F.interpolate(down, size=left.size()[2:], mode='bilinear')
    out1h = F.relu(self.bn1h(self.conv1h(left )), inplace=True)
    out2h = F.relu(self.bn2h(self.conv2h(out1h)), inplace=True)
    out1v = F.relu(self.bn1v(self.conv1v(down )), inplace=True)
    out2v = F.relu(self.bn2v(self.conv2v(out1v)), inplace=True)
    fuse  = out2h*out2v
    out3h = F.relu(self.bn3h(self.conv3h(fuse )), inplace=True)+out1h
    out4h = F.relu(self.bn4h(self.conv4h(out3h)), inplace=True)
    out3v = F.relu(self.bn3v(self.conv3v(fuse )), inplace=True)+out1v
    out4v = F.relu(self.bn4v(self.conv4v(out3v)), inplace=True)
    return out4h, out4v

The order of the classes in the main class is as follows:
class Main(nn.Module):
def __init__(self):
    super(Main, self).__init__()
self.first=first(the required arguments)
self.second=second(the required arguments)
self.features = feature_extractor()

def forward(self, x):

   x1, x2 = self.features(x) # as self.features, you can produce 128 convolutional channels
   x1 = self.first(x1)
   x2 = self.first(x2)
return x1, x2

My question is that how it is possible to import the outcome of the module into the first class. To be clearer, after importing the y1 and y2 variable of the first and second classes into the module class, how can I multiply one of the the outcomes of the module class with y1 in the first class and another outcome of the module class with the y2 variable in the second class.This is because if I integrate the module in the second class, although I can import the y1 into the second class and the module, I cannot import back the outcome of the module to the first class and multiply it with y1.

Update:
I want that the module class receive the y1 and y2 from the first class and second class respectively but I have not idea how I should  integrate the module class inside the code. The image below can show clearer information about the idea:

Output:
Since the program you've added is pretty convoluted, I've created similar classes to demonstrate the principle that will solve the problem.
class first:
    # this class gives y1 in your example
    def __init__(self, x = 0):
        self.x = x
    def get_y1(self):
        # do any computations if needed
        print(f"x (y1) has current value {self.x}")
        return self.x
    def calc(self, y):
        self.x += y
        print(f"x (y1) updated to {self.x}")

class second:
    # this class gives y2 in your example
    def __init__(self, x = 0):
        self.x = x
    def get_y2(self):
        # do any computations if needed
        print(f"x (y2) has current value {self.x}")
        return self.x
    def calc(self, y):
        self.x *= y
        print(f"x (y2) updated to {self.x}")

class module:
    # this class takes y1 and y2 for computation and returns results
    def __init__(self):
        pass
    def calc(self, x, y):
        return x+1,y+1

class main:
    def __init__(self):
        self.first = first(x = 5)
        self.second = second(x = 3)
        self.module = module()

    def calc(self):
        y1 = self.first.get_y1()
        y2 = self.second.get_y2()
        result1, result2 = self.module.calc(y1, y2)
        self.first.calc(result1)
        self.second.calc(result2)

obj = main()
obj.calc()

'''
Output for this:
x (y1) has current value 5
x (y2) has current value 3
x (y1) updated to 11
x (y2) updated to 12
'''

This basically does the same thing as you're requesting: main class holds objects for first, second and module; a computation in module uses y1 and y2 from first and second. The returned values are then used to update y1 and y2 themselves.


Instruction: PyTorch CUDA version is always 10.2

Input:
I've installed a handful of PyTorch versions (CUDA 11.7 nightly, CUDA 11.6 nightly, 11.3), but every time, torch.version.cuda returns 10.2.
I'd like to run PyTorch on CUDA 11.7. My graphics card has CUDA capability sm_86.
[me@legion imagen-test]$ sudo pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
...
[me@legion imagen-test]$ python
>>> import torch
>>> print(torch.version.cuda)
10.2

When I actually try to use PyTorch, I get an error saying the PyTorch version I have installed doesn't support the newer version of CUDA my graphics card requires.
>>> torch.Tensor([1,2,3]).cuda()
...
NVIDIA GeForce RTX 3060 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
...
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I'm completely stumped, and unsure where to go from here. I'd appreciate any help.

Output:
You've probably installed PyTorch with CUDA 10.2 among your different installed versions. This may be taking priority over the versions of PyTorch. To fix this, simply uninstall all versions of PyTorch with pip uninstall torch -y and reinstall PyTorch with CUDA 11.7.
Source: https://discuss.pytorch.org/t/cuda-version-is-always-10-2/152876


Instruction: Copy tensor elements of certain indices in PyTorch

Input:
The desired operation is similar in spirit to torch.Tensor.index_copy, but a little different.
It's best explained with an example.
Tensor A has original values that we will copy:
[10, 20, 30]
Tensor B has indices of A:
[0, 1, 0, 1, 2, 1]
Tensor C has same length as B, containing the indexed values of A:
[10, 20, 10, 20, 30, 20]
What's a good way to make C from A and B in PyTorch, without using loops?

Output:
Have you tried just indexing by A?
In [1]: import torch

In [2]: a = torch.tensor([20,30,40])

In [3]: b = torch.tensor([0,1,2,1,1,2,0,0,1,2])

In [4]: a[b]
Out[4]: tensor([20, 30, 40, 30, 30, 40, 20, 20, 30, 40])


Instruction: TypeError: 'numpy.float32' object is not iterable when logging in mlflow

Input:
I am trying a machine learning model and logging metrics using mlflow. But I am getting TypeError: 'numpy.float32' object is not iterable. I have tried using .tolist() and dict() but nothing seems to work.
def train(max_epochs, model, optimizer, scheduler, train_loader, valid_loader, project_name):
    best_val_loss = 100
    for epoch in range(max_epochs):
        model.train()
        running_loss = []
        tq_loader = tqdm(train_loader)
        o = {}
        for samples in tq_loader:
            optimizer.zero_grad()
            outputs, interaction_map = model(
                [samples[0].to(device), samples[1].to(device), torch.tensor(samples[2]).to(device),
                 torch.tensor(samples[3]).to(device)])
            l1_norm = torch.norm(interaction_map, p=2) * 1e-4
            loss = loss_fn(outputs, torch.tensor(samples[4]).to(device).float()) + l1_norm
            loss.backward()
            optimizer.step()
            loss = loss - l1_norm
            running_loss.append(loss.cpu().detach())
            tq_loader.set_description(
                "Epoch: " + str(epoch + 1) + "  Training loss: " + str(np.mean(np.array(running_loss))))
        model.eval()
        val_loss, mae_loss = get_metrics(model, valid_loader)
        scheduler.step(val_loss)

        #metrics mlflow
        mlflow.log_metrics('train_loss',(np.mean(np.array(running_loss))).tolist())
        mlflow.log_metrics('validation_loss',(val_loss).tolist())
        mlflow.log_metrics('MAE Val_loss', (mae_loss).tolist())

        print("Epoch: " + str(epoch + 1) + "  train_loss " + str(np.mean(np.array(running_loss))) + " Val_loss " + str(
            val_loss) + " MAE Val_loss " + str(mae_loss))
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            torch.save(model.state_dict(), "./runs/run-" + str(project_name) + "/models/best_model.tar")

mlflow.set_experiment('CIGIN_V2')
mlflow.start_run(nested=True)
train(max_epochs, model, optimizer, scheduler, train_loader, valid_loader, project_name)
mlflow.end_run()

Error
Epoch: 1  Training loss: 6770.575: 100%|██████████| 1/1 [00:04<00:00,  4.35s/it]
100%|██████████| 1/1 [00:03<00:00,  3.86s/it]

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-96-8c3a6eb822c3> in <module>()
      1 mlflow.set_experiment('CIGIN_V2')
      2 mlflow.start_run(nested=True)
----> 3 train(max_epochs, model, optimizer, scheduler, train_loader, valid_loader, project_name)
      4 mlflow.end_run()

<ipython-input-95-ab0a6c80b65b> in train(max_epochs, model, optimizer, scheduler, train_loader, valid_loader, project_name)
     55
     56         #metrics mlflow
---> 57         mlflow.log_metrics('train_loss',dict(np.mean(np.array(running_loss))).tolist())
     58         mlflow.log_metrics('validation_loss',dict(val_loss).tolist())
     59         mlflow.log_metrics('MAE Val_loss', dict(mae_loss).tolist())

TypeError: 'numpy.float32' object is not iterable

Output:
Youre logging a single value into log_metrics and i dont think thats correct based on the implementation of log_metric and log_metrics in the documentation:
https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_metric and
https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_metrics
So i would suggest to maybe change the "log_metrics" to "log_metric" and leave the tolist out


Instruction: RMSProp in TF VS Pytorch

Input:
In Pytorch's RMSProp implementation we are given the parameter alpha which according to the documentation:
alpha (float, optional) – smoothing constant (default: 0.99)
On the other hand, TF's implementation has the parameter rho (Formally named decay):
rho    Discounting factor for the history/coming gradient. Defaults to 0.9.
Are those parameters the same with different names or are they different?
I couldn't find any information regarding the differences.

Output:
If you compare the source code of PyTorch (here) and that of Tensorflow (on a forked build), you will see that alpha and rho are indeed the same.
Although, as opposed to Tensorflow, PyTorch is clear about the underlying logic for its module:


Instruction: GPU available in Tensorflow but not in Torch

Input:
I'm currently working on a server and I would like to be able the GPUs for PyTorch network training. I am not able to detect GPU by using torch but, if I use TensorFlow, I can detect both of the GPUs I am supposed to have. I suppose it's a problem with versions within PyTorch/TensorFlow and the CUDA versions on it.
However, after trying different versions of Pytorch, I am not still able to use them...
I am attaching the specificities of the GPUs and the current version of Tensorflow and Pytorch I am using. Does anyone have any hint on it? Would be very helpful.
| NVIDIA-SMI 4--.--.--    Driver Version: 465.19.01    CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------|
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:02:00.0 Off |                  N/A |
| 27%   39C    P8    17W / 250W |      1MiB / 11176MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:81:00.0 Off |                  N/A |
| 28%   45C    P8    11W / 250W |      1MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Torch version: 1.10.2
Tensorflow Version: 2.6.2
Cuda toolkit: 11.3.1
>>> print('Number of GPUs: %d' % len(tf.config.list_physical_devices('GPU')))
Number of GPUs: 2

>>> torch.cuda.is_available()
False

I am so lost... Thank you in advance!

Output:
I finally could resolve this problem by specifying the cuda version of pytorch... The combination of those specific versions was installing the CPU based version.
After installing the correct one, I have been able to use the GPU server without any problem.


Instruction: LBFGS Giving Tensor Object not Callable Error when using Optimizer.step

Input:
I am trying to use sgd, adam, and LBFGS optimizer.
The part of the code is:
for batch_idx, (inputs, targets) in enumerate(trainloader):
            batch_size = inputs.size(0)
            total += batch_size
            one_hot_targets = torch.FloatTensor(batch_size, 10).zero_()
            one_hot_targets = one_hot_targets.scatter_(1, targets.view(batch_size, 1), 1.0)
            one_hot_targets = one_hot_targets.float()
            if use_cuda:
                inputs, one_hot_targets = inputs.cuda(), one_hot_targets.cuda()
            inputs, one_hot_targets = Variable(inputs), Variable(one_hot_targets)

            if optimizer_val=='sgd' or optimizer_val=='adam':
              outputs = F.softmax(net(inputs))
              loss = criterion(outputs, one_hot_targets)

              loss.backward()
              optimizer.step()

            else:
              def closure():
                optimizer.zero_grad()
                outputs = F.softmax(net(inputs))
                loss = criterion(outputs, one_hot_targets)
                loss.backward()
                return loss

              optimizer.step(closure())

In the optimizer.step(closure()) part in LBFGS (running in else) I am getting this error:
TypeError: 'Tensor' object is not callable

I checked, the loss is tensor type.
How to make it work?

Output:
You need to pass a function callback to the optimizer.step function, don't call it:
optimizer.step(closure)


Instruction: Reshaping or "concatenating" a tensor along an axis

Input:
I have a tensor, t, of the following shape: torch.Size([280, 4, 768]).
What I want is to achieve, effectively, concatenation along the second axis, resulting in torch.Size([280, 3072]).
I know that I can for instance, do:
torch.cat((x[:, -4, :], x[:, -3, :], x[:, -2, :], x[:, -1, :]), dim=1)
but is there a nicer way of writing this?
How do I achieve reshaping along the second axis without messing up my values?

Output:
Yes you can apply a straight forward reshape:
>>> x.reshape(len(x), -1)


Instruction: Using the embedding layer as the input for an encoder

Input:
I want to use the embedding layer as the input for encoder, however I got an error as follow.  My input y is a time series data with shape of 1*84. Could you please help me with that?
import numpy
import torch.nn as nn

r_input    = torch.nn.Embedding(84, 10)
activation  = nn.functional.relu
mu_r      = nn.Linear(10, 6)
log_var_r = nn.Linear(10, 6)

y = np.random.rand(1,84)

def encode_r(y):
    y         = torch.reshape(y, (-1, 1, 84)) # torch.Size([batch_size, 1, 84])
    hidden    = torch.flatten(activation(r_input(y)), start_dim = 1)
    z_mu      = mu_r(hidden)
    z_log_var = log_var_r(hidden)
    return z_mu, z_log_var```

Error: RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

Output: According to this thread: https://discuss.pytorch.org/t/expected-tensor-for-argument-1-indices-to-have-scalar-type-long-but-got-cpufloattensor-instead-while-checking-arguments-for-embedding/32441/4, it seems that one possible solution would be to ensure embeddings have integer values and not float values in them (by embeddings we mean the lookup table not an actual embedding vector).


Instruction: How to input embeddings directly to a huggingface model instead of tokens?

Input:
I'm going over the huggingface tutorial where they showed how tokens can be fed into a model to generate hidden representations:
import torch
from transformers import RobertaTokenizer
from transformers import RobertaModel

checkpoint = 'roberta-base'
tokenizer = RobertaTokenizer.from_pretrained(checkpoint)
model = RobertaModel.from_pretrained(checkpoint)

sequences = ["I've been waiting for a HuggingFace course my whole life."]

tokens = tokenizer(sequences, padding=True)
out = model(torch.tensor(tokens['input_ids']))
out.last_hidden_state

But how can I input word embeddings directly instead of tokens? That is, I have another model that generates word embeddings and I need to feed those into the model

Output:
Most (every?) huggingface encoder model supports that with the parameter inputs_embeds:
import torch
from transformers import RobertaModel

m = RobertaModel.from_pretrained("roberta-base")

my_input = torch.rand(2,5,768)

outputs = m(inputs_embeds=my_input)

P.S.: Don't forget the attention mask in case this is required.


Instruction: pytorch optimizer TypeError 'collections.OrderedDict' object is not callable

Input:
I used python3.8, pytorch suddenly reported an error optimizer TypeError, but the program was still running two weeks ago.
net = Net(num_classes=7)
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=1e-4)

It seems to be net.parameters() cannot return the correct result?
Traceback (most recent call last):
   File "C:/Users/usr/Desktop/jaffeAttention/jaffe.py", line 190, in <module>
       main()
   File "C:/Users/usr/Desktop/jaffeAttention/jaffe.py", line 87, in main
      optimizer = optim.Adam(net.parameters(), lr=1e-4)
TypeError: 'collections.OrderedDict' object is not callable

enter image description here

Output:
You get the error because net.parameters is an OrderedDict, and you then try to call it with net.parameters(), but that is not possible. So you get an error.
To find this out, could try the commands
net = Net(num_classes=7)
net.to(device)
criterion = nn.CrossEntropyLoss()
params= net.parameters
print(type(params)) # verify this is indeed an OrderedDict
optimizer = optim.Adam(params, lr=1e-4) # notice there is no () when passing the params to Adam

You can read about OrderedDict in the python documentation. https://docs.python.org/3/library/collections.html#collections.OrderedDict


Instruction: Select items from a matrix or tensor using 1-D array of indexes to get 1D or 2D tensors in torch

Input:
I have a tensor of n-sampled predictions. In this example I sample 4 times a (10,2) result, where this results represents a batch of graphs with each graph having 3+4+3=10 nodes with 2 coordinates. I have a way to select the best sample for each graph, and I have an index telling me this (idx_test). I want to select in an efficient way the elements of the sampled tensor. The best way I have found is like this (toy code example):
rand_test = torch.randint(3, 10, (4, 10, 2))
idx_test = torch.LongTensor([0, 3, 1])

n_nodes = [3, 4, 3]

final_tensor = torch.zeros((10, 2))
accum_nodes = torch.cumsum(torch.LongTensor(n_nodes), 0)
# First iteration is done outside the for loop
final_tensor[:accum_nodes[0]] = rand_test[idx_test[0],:accum_nodes[0],:]
for i, idx in enumerate(idx_test):
    final_tensor[accum_nodes[i-1]:accum_nodes[i]] = rand_test[idx,accum_nodes[i-1] :accum_nodes[i], :]
print(final_tensor)

The objective is to obtain the same final tensor without using for loops since in my model this array is large
Thanks!

Output:
You could create a helper tensor containing the ranges of node indices you want to index rand_test with. In your example, you have: i=0 you have 3 nodes with index value 0, i=1 has 4 with value 3, while i=2 has 3 with index value 1.
You can do so using repeat_interleave:
>>> idx_test.repeat_interleave(torch.tensor(n_nodes))
tensor([0, 0, 0, 3, 3, 3, 3, 1, 1, 1])

Then index rand_test using idx and a range:
>>> final_tensor = rand_test[idx, range(len(idx))]
tensor([[5, 5],
        [7, 3],
        [8, 4],
        [7, 5],
        [7, 6],
        [7, 8],
        [9, 9],
        [7, 7],
        [8, 7],
        [3, 7]])


Instruction: way for multiplication of these tensors with gradients

Input:
I have a function with two inputs: heat maps and feature maps.
The heatmaps have a shape of (20, 14, 64, 64) and the feature maps have a shape of (20, 64, 64, 64). Where 20 is the batch size and 14 is the number of key points. Both heatmaps and feature maps have spatial dimensions of 64x64 and the featuremaps have 64 channels (on the second dimension).
Now I need to multiply each heatmap by each channel of the feature maps. So the first heatmap has to be multiplied by all 64 channels of the feature maps. The second with all channels, and so on.
After that, I should have a tensor of shape (20, 14, 64, 64, 64) on which I need to apply global max-pooling.
The problem is now that I can't create a new tensor to do that, because the gradients of the heatmaps and feature maps must be preserved.
My actual (slow and not-gradient-keeping) code is:
def get_keypoint_representation(self, heatmaps, features):
    heatmaps = heatmaps[0]
    pool = torch.nn.MaxPool2d(features.shape[2])
    features = features[:, None, :, :, :]
    features = features.expand(-1, 14, -1, -1, -1).clone()

    for i in range(self.cfg.SINGLE_GPU_BATCH_SIZE):
        for j in range(self.cfg.NUM_JOINTS):
            for k in range(features.shape[2]):
                features[i][j][k] = torch.matmul(heatmaps[i][j], features[i][j][k])

    gmp = features.amax(dim=(-1, -2))
    return gmp

Overview of the task:

Output:
Given a tensor of heatmaps hm shaped (b, k, h, w) and a feature tensor fm shaped (b, c, h, w).
You can perform such an operation with a single einsum operator
>>> z = torch.einsum('bkhw,bchw->bkchw', hm, FM)
>>> z.shape
torch.Size([20, 14, 64, 64, 64])

Then follow with a max-pooling operation over the spatial dimensions using amax:
 >>> gmp = z.amax(dim=(-1,-2))
 >>> gmp.shape
 torch.Size([20, 14, 64])


Instruction: Reading files with .h5 format and using it in dataset

Input:
I have two folders( one for train and one for test) and each one has around 10 files in h5 format. I want to read them and use them in a dataset. I have a function to read them, but I don't know how I can use it to read the file in my class.
def read_h5(path):
    data = h5py.File(path, 'r')
    image = data['image'][:]
    label = data['label'][:]
    return image, label

class Myclass(Dataset):
    def __init__(self, split='train', transform=None):
        raise NotImplementedError

    def __len__(self):
        raise NotImplementedError

    def __getitem__(self, index):
        raise NotImplementedError

Do you have a suggestion?
Thank you in advance

Output:
This might be a start for what you want to do. I implemented the __init__(), but not __len__() or __get_item__(). User provides the path, and the init function calls the class method read_h5() to get the arrays of image and label data. There is a short main to create a class objects from 2 different H5 files. Modify the paths list with folder and filenames for all of your training and testing data.
class H5_data():
    def __init__(self, path): #split='train', transform=None):
        self.path = path
        self.image, self.label = H5_data.read_h5(path)

    @classmethod
    def read_h5(cls,path):
        with h5py.File(path, 'r') as data:
            image = data['image'][()]
            label = data['label'][()]
            return image, label

paths = ['train_0.h5', 'test_0.h5']
for path in paths:
    h5_test = H5_data(path)
    print(f'For HDF5 file: {path}')
    print(f'image data, shape: {h5_test.image.shape}; dtype: {h5_test.image.dtype}')
    print(f'label data, shape: {h5_test.label.shape}; dtype: {h5_test.label.dtype}')

IMHO, creating a class with the array data is overkill (and could lead to memory problems if you have really large datasets). It is more memory efficient to create h5py dataset objects, and access the data when you need it. Example below does the same as code above, without creating a class object with numpy arrays.
paths = ['train_0.h5', 'test_0.h5']
for path in paths:
    with h5py.File(path, 'r') as data:
        image = data['image']
        label = data['label']
        print(f'For HDF5 file: {path}')
        print(f'image data, shape: {image.shape}; dtype: {image.dtype}')
        print(f'label data, shape: {label.shape}; dtype: {label.dtype}')


Instruction: Train multiple connected neural networks with a single optimizer

Input:
How can I jointly optimize the parameters of a model comprising two distinct neural networks with a single optimizer? What I've tried is the following, after having initialized an optimizer:
optim_global = optim.Adam(zip(model1.parameters(), model2.parameters()))

but I get this error
TypeError: optimizer can only optimize Tensors, but one of the params is tuple

Output:
These are generator you can control either with the unpacking operator *:
>>> optim.Adam([*model1.parameters(), *model2.parameters()])

Or using itertools.chain
>>> optim.Adam(chain(model1.parameters(), model2.parameters()))


Instruction: Concatenate N pytorch tensors (of the same shape) generated from within loop

Input:
Tensors of the same shape are being returned from within a loop and I want to concatenate them succinctly and as pythonically / pytorchly as possible.
Current solution:
import torch

for object_id in object_ids:

    dataset = Dataset(object_id)

    image_tensor = dataset.get_random_image_tensor()

    if 'concatenated_image_tensors' in locals():
        concatenated_image_tensors = torch.cat((merged_image_tensors, image_tensor))
    else:
        concatenated_image_tensors = image_tensor

Is there a better way?

Output:
A good approach is to first append to a python list, then concatenate at the end the whole list. Otherwise you'll end up moving data around in memory each time the torch.cat is called.
all_img = []
for object_id in object_ids:
    dataset = Dataset(object_id)
    image_tensor = dataset.get_random_image_tensor()
    all_img.append(image_tensor)

all_img = torch.cat(all_img)


Instruction: PyTorch: How to sample from a tensor where each value in the tensor has a different likelihood of being selected?

Input:
Given tensor
A = torch.tensor([0.0316, 0.2338, 0.2338, 0.2338, 0.0316, 0.0316, 0.0860, 0.0316, 0.0860]) containing probabilities which sum to 1 (I removed some decimals but it's safe to assume it'll always sum to 1), I want to sample a value from A where the value itself is the likelihood of getting sampled. For instance, the likelihood of sampling 0.0316 from A is 0.0316. The output of the value sampled should still be a tensor.
I tried using WeightedRandomSampler but it doesn't allow the value selected to be a tensor anymore, instead it detaches.
One caveat that makes this tricky is that I want to also know the index of the sampled value as it appears in the tensor. That is, say I sample 0.2338, I want to know if it's index 1, 2 or 3 of tensor A.

Output:
Selecting with the expected probabilities can be achieved by accumulating the weights and selecting the insertion index of a random float [0,1). The example array A is slightly adjusted to sum up to 1.
import torch

A = torch.tensor([0.0316, 0.2338, 0.2338, 0.2338, 0.0316, 0.0316, 0.0860, 0.0316, 0.0862], requires_grad=True)

p = A.cumsum(0)
#tensor([0.0316, 0.2654, 0.4992, 0.7330, 0.7646, 0.7962, 0.8822, 0.9138, 1.0000], grad_fn=<CumsumBackward0>))

idx = torch.searchsorted(p, torch.rand(1))
A[idx], idx

Output
(tensor([0.2338], grad_fn=<IndexBackward0>), tensor([3]))

This is faster than the more common approach with A.multinomial(1).
Sampling 10000 times one element to check that the distribution conforms to the probabilities
from collections import Counter

Counter(int(A.multinomial(1)) for _ in range(10000))
#1 loop, best of 5: 233 ms per loop

# vs @HatemAli's solution
dist=torch.distributions.categorical.Categorical(probs=A)
Counter(int(dist.sample()) for _ in range(10000))
# 10 loops, best of 5: 107 ms per loop

Counter(int(torch.searchsorted(p, torch.rand(1))) for _ in range(10000))
# 10 loops, best of 5: 53.2 ms per loop

Output
Counter({0: 319,
         1: 2360,
         2: 2321,
         3: 2319,
         4: 330,
         5: 299,
         6: 903,
         7: 298,
         8: 851})


Instruction: Retrain facenet neural network

Input:
I want to use the pretained MTCNN model to train in a subset of images, and keep the layers that are important to get the features of the images. Also, I want to use another loss function.
import torch
from facenet_pytorch import InceptionResnetV1, MTCNN
from torch.utils.data import DataLoader
from torchvision import datasets
import numpy as np
import pandas as pd
import os

workers = 0 if os.name == 'nt' else 4
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('Running on device: {}'.format(device))

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

The problem is that I don't know how to face the problem of "freezing" some layers of the neural network that I don't want to tune and tune only the last layers. Also, how to apply the new criterion to this pretrained network.

Output:
You can first print out or visualize the MTCNN model, so you will know the name (key) of the layer you want to freeze. Assuming the name of the layer is feature_layer, you can then simple do:
for name, param in model.named_parameters():
    if not 'feature_layer' in name:
        param.requires_grad = False

So it will freeze every other layers except the last layer feature_layer.
Then your other parameters will follow suit:
optimizer = torch.optim.Adam(params, lr=1e-3)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

Now, to change the criterion, you change it here to one of the default ones, or use a custom one:
loss_fn = torch.nn.CrossEntropyLoss()


Instruction: how does optimizer.step() takes the recent loss of the model?

Input:
I am looking at the example from pytorch of a model:
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py
for epoch in range(2):  # loop over the dataset multiple times
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
        running_loss = 0.0

print('Finished Training')

And I have a very basic question - the optimizer was never inserted or defined into the model (similarly to model.compile in keras). Nor it received the loss or labels of the last batch or epoch.
How does it "knows" to perform optimization step?

Output:
Rather than thinking about how loss and parameters are related, you should consider them as separate events which are not linked. Indeed, there are two distinct elements that have an effect on parameters and their cached gradient.

The autograd mechanism (the process in charge of performing gradient computation) allows you to call backward on a torch.Tensor (your loss) and which will in turn backpropagate through all the nodes tensors that are allowed to compute this final tensor value. Doing so, it will navigate through what's called the computation graph, updating each of the parameters' gradients by changing their grad attribute. This means that at the end of a backward call the network's learned parameters that were used to compute this output will have a grad attribute containing the gradient of the loss with respect to that parameter.
loss.backward()

The optimizer is independent of the backward pass since it doesn't rely on it. You can call backward on your graph once, multiple times, or on different loss terms depending on your use case. The optimizer's task is to take the parameters of the model independently (that is irrespective of the network architecture or its computation graph) and update them using a given optimization routine (for example via Stochastic Gradient Descent, Root Mean Squared Propagation, etc...). It goes through all parameters it was initialized with and updates them using their respective gradient value (which is supposed to be stored in the grad attribute by at least one backpropagation.
optimizer.step()

Important notes:

Keep in mind though that the backward process and the actual update call using the optimizer are linked implicitly only by the fact that the optimizer will use the results computed by the backward preceding call.

In PyTorch parameter gradients are kept in memory so you have to clear them out before performing a new backward call. This is done using the optimizer's zero_grad function. In practice, it clears the grad attribute of the tensors it has registered as parameters.


Instruction: Add padding based on partial sum

Input:
I have four given variables:

group size
total of groups
partial sum
1-D tensor
and I want to add zeros when the sum within a group reached the partial sum. For example:

groupsize = 4
totalgroups = 3
partialsum = 15
d1tensor = torch.tensor([ 3, 12,  5,  5,  5,  4, 11])

The expected result is:
[ 3, 12, 0, 0,  5,  5,  5, 0, 4, 11, 0, 0]
I have no clue how can I achieve that in pure pytorch. In python it would be something like this:
target = [0]*(groupsize*totalgroups)

cursor = 0
current_count = 0
d1tensor = [ 3, 12,  5,  5,  5,  4, 11]

for idx, ele in enumerate(target):
  subgroup_start = (idx//groupsize) *groupsize
  subgroup_end = subgroup_start + groupsize
  if sum(target[subgroup_start:subgroup_end]) < partialsum:
     target[idx] = d1tensor[cursor]
     cursor +=1

Can anyone help me with that? I have already googled it but couldn't find anything.

Output:
Some logic, Numpy and list comprehensions are sufficient here.
I will break it down step by step, you can make it slimmer and prettier afterwards:
import numpy as np

my_val = 15
block_size = 4
total_groups = 3
d1 = [3, 12,  5,  5,  5,  4, 11]
d2 = np.cumsum(d1)
d3 = d2 % my_val == 0 #find where sum of elements is 15 or multiple
split_points= [i+1 for i, x in enumerate(d3) if x] # find index where cumsum == my_val
#### Option 1
split_array = np.split(d1, split_points, axis=0)
padded_arrays = [np.pad(array, (0, block_size - len(array)), mode='constant') for array in split_array] #pad arrays
padded_d1 = np.concatenate(padded_arrays[:total_groups]) #put them together, discard extra group if present
#### Option 2
split_points = [el for el in split_points if el <len(d1)] #make sure we are not splitting on the last element of d1
split_array = np.split(d1, split_points, axis=0)
padded_arrays = [np.pad(array, (0, block_size - len(array)), mode='constant') for array in split_array] #pad arrays
padded_d1 = np.concatenate(padded_arrays)


Instruction: Get 2D output from the embedding layer in pytorch

Input:
I have a X_train with size of (2, 100). I want to use the 250 first of the data, and use the second 250 of this matrix as the input of embedding and convert that to a matrix with size 2*3.
I read a lot about the embedding layer in pytorch, however I did not understand it well. I don't know how to get a 2*3 as the output of the embedding layer. Could you please help me with that? Here is a simple example.
import torch
import torch.nn as nn

X_train = np.random.randint(10, size = (2, 100))

X_train_notmbedding = X_train[:, 0:50] # not embedding  (2,50)
X_train_mbedding    = X_train[:, 50:100] #embedding  (2, 50)
X_train_mbedding = torch.LongTensor([X_train_mbedding])
embedding = nn.Embedding(50, 3)
embeding_output = embedding(X_train_mbedding) # I want to get a embedding output as (2,3)

#X_train_new = torch.cat([X_train_notmbedding, embeding_output], 1) # here I want to build a matrix with size (2, 53)

Output:
From the discussion, it looks like your understanding of Embeddings is not accurate.

Only use 1 Embedding for 1 feature. In your example you are combining dates, ids etc. in 1 Embedding. Even in the medium article, they are using separate embeddings.
Think of Embedding as one-hot encoding on steroids (less memory, data co-relation etc.). If you do not understand one-hot encoding I would start there first.
KWH is already a real value not categorical. Use it as a linear input to the network (after normalization).
ID: I do not know what ID denotes in your data, if it is a unique ID for each datapoint, it is not useful and should be excluded.

If the above does not make sense, I would start with a simple network using LSTM and make it work first before using an advanced architecture.


Instruction: Installing cudatoolkit works with conda install but not with conda create -f

Input:
I have a PyTorch environment file:
name: torch
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.7
  - pytorch::pytorch
  - pytorch::torchvision
  - pytorch::torchaudio
  - pytorch::cudatoolkit
  - numpy
  - scipy
  - scikit-learn
  - matplotlib
  - pillow
  - tqdm
  - joblib
  - visdom
  - jsonpatch
  - pip
  - pip:
    - torchsummary
    - opencv-python==4.1.1.26

Trying to create a conda environment from it with conda create -f torch.yml fails:
(base) prompt@PC:~$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - pytorch::cudatoolkit

The environment is created without issues if I remove cudatoolkit from the list of dependencies.
However, conda install cudatoolkit -c pytorch finds and installs the package without issues. The same happens if I replace cudatoolkit with cudatoolkit=11.3 (the current most recent version listed on the PyTorch website) in both cases.

Output:
I managed to resolve the issue by installing cudatoolkit from the nvidia channel, rather than pytorch. I'm still not sure why cudatoolkit is available from pytorch with one method but not the other, but this solves my issue (although the nvidia version seems to be larger, so it's probably a superset package of pytorch's cudatoolkit). My YAML file now looks like this:
name: ritnet
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.7
  - pytorch::pytorch
  - pytorch::torchvision
  - pytorch::torchaudio
  - nvidia::cudatoolkit=11.3
  - numpy
  - scipy
  - scikit-learn
  - matplotlib
  - pillow
  - tqdm
  - joblib
  - visdom
  - jsonpatch
  - pip
  - pip:
    - torchsummary
    - opencv-python==4.1.1.26


Instruction: Working with large multiple datasets where each dataset contains multiple values - Pytorch

Input:
I'm training a Neural Network and have overall > 15GB of data inside a folder, the folder has multiple pickle files, and each file contains two lists that each holds multiple values.
This looks like the following:
dataset_folder:\

file.pickle
file_2.pickle
...
...
file_n.pickle

Each file_*.pickle contains a variable length list (list x and list y).
How to load all the data to train the model without having memory issue?

Output:
By implementing the custom dataset class provided from Pytorch, we need to implement three methods so pytorch loader can work with your data

__len__
__getitem__
__init__

Let's go through how to implement each one of them seperatly.

__init__
def __init__(self):

 # Original Data has the following format
 """
   dict_object =
   {
     "x":[],
     "y":[]
   }
 """
 DIRECTORY = "data/raw"
 self.dataset_file_name = os.listdir(DIRECTORY)
 self.dataset_file_name_index = 0
 self.dataset_length =0
 self.prefix_sum_idx = list()
 # Loop over each file and calculate the length of overall dataset
 # you might need to check if file_name is file
 for file_name in os.listdir(DIRECTORY):
   with (open(f'{DIRECTORY}/{file_name}', "rb")) as openfile:
     dict_object = pickle.load(openfile)
     curr_page_sum = len(dict_object["x"]) + len(dict_object["y"])
     self.prefix_sum_idx.append(curr_page_sum)
     self.dataset_length += curr_page_sum
 # prefix sum so we have an idea of where each index appeared in which file.
 for i in range (1,len(self.prefix_sum_idx)):
   self.prefix_sum_idx[i] = self.prefix_sum_idx[i] + self.prefix_sum_idx[i-1]

 assert self.prefix_sum_idx[-1] == self.dataset_length
 self.x = []
 self.y = []

As you can see above, the main idea is to use prefix sum to "treat" all the dataset as once, so the logic is whenever you need to get access to a specific index later, you simply look into prefix_sum_idx to see this where this idx appear.

In the image above, let's say we need to access the index 150. Thanks to prefix sum, we are now able to know that 150 exist in the second .pickle file. Still we need a fast mechanism to know where that idx exist in the prefix_sum_idx. This will be explained in the __getitem__

__getitem__
def read_pickle_file(self, idx):
 file_name = self.dataset_file_name[idx]
 dict_object = dict()
 with (open(f'{YOUR_DIRECTORY}/{file_name}', "rb")) as openfile:
     dict_object = pickle.load(openfile)

 self.x = dict_object['x']
 self.y = #some logic here
 ......
 # Some logic here....

def __getitem__(self,idx):

 # Similar to C++ std::upper_bound - O(log n)
 temp = bisect.bisect_right(self.prefix_sum_idx, idx)

 self.read_pickle_file(temp)
 local_idx = idx - self.prefix_sum_idx[temp]

 return self.x[local_idx],self.y[local_idx]

check bisect_right() docs for details on how it works, but simply it returns the rightmost place in the sorted list to insert the given element and keep it sorted. In our approach, we're interested only in the following question, "which file should I access in order to get the appropriate data". More importantly, it does so in O(log n)

__len__
def __len__(self):
 return self.dataset_length

In order to get the length of our dataset, we loop through each file in and accumulate the results as shown in __init__.
The full code sample goes like this:
import pickle
import torch
import torch.nn as nn
import numpy
import os
import bisect
from torch.utils.data import Dataset, DataLoader
from src.data.make_dataset import main
from torch.nn import functional as F

class dataset(Dataset):
  def __init__(self):

    # Original Data has the following format
    """
    dict_object =
    {
        "x":[],
        "y":[]
    }
    """
    DIRECTORY = "data/raw"
    self.dataset_file_name = os.listdir(DIRECTORY)
    self.dataset_file_name_index = 0
    self.dataset_length =0
    self.prefix_sum_idx = list()
    # Loop over each file and calculate the length of overall dataset
    # you might need to check if file_name is file
    for file_name in os.listdir(DIRECTORY):
    with (open(f'{DIRECTORY}/{file_name}', "rb")) as openfile:
        dict_object = pickle.load(openfile)
        curr_page_sum = len(dict_object["x"]) + len(dict_object["y"])
        self.prefix_sum_idx.append(curr_page_sum)
        self.dataset_length += curr_page_sum
    # prefix sum so we have an idea of where each index appeared in which file.
    for i in range (1,len(self.prefix_sum_idx)):
    self.prefix_sum_idx[i] = self.prefix_sum_idx[i] + self.prefix_sum_idx[i-1]

    assert self.prefix_sum_idx[-1] == self.dataset_length
    self.x = []
    self.y = []

def read_pickle_file(self, idx):
 file_name = self.dataset_file_name[idx]
 dict_object = dict()
 with (open(f'{YOUR_DIRECTORY}/{file_name}', "rb")) as openfile:
     dict_object = pickle.load(openfile)

 self.x = dict_object['x']
 self.y = #some logic here
 ......
 # Some logic here....

def __getitem__(self,idx):

 # Similar to C++ std::upper_bound - O(log n)
 temp = bisect.bisect_right(self.prefix_sum_idx, idx)

 self.read_pickle_file(temp)
 local_idx = idx - self.prefix_sum_idx[temp]

 return self.x[local_idx],self.y[local_idx]

def __len__(self):
 return self.dataset_length

large_dataset = dataset()
train_size = int (0.8 * len(large_dataset))
validation_size = len(large_dataset) - train_size

train_dataset, validation_dataset = torch.utils.data.random_split(large_dataset, [train_size, validation_size])
validation_loader = DataLoader(validation_dataset, batch_size=64, num_workers=4, shuffle=False)
train_loader = DataLoader(train_dataset,batch_size=64, num_workers=4,shuffle=False)


Instruction: Getting random output every time on running Next Sentence Prediction code using BERT

Input:
Based on the code provided below, I am trying to run NSP (Next Sentence Prediction) on a custom dataset. The loss after training the model is different every time and the model give different accuracies every time. What am I missing or doing wrong?
pip install transformers[torch]
from transformers import BertTokenizer, BertForNextSentencePrediction
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
with open('clean.txt', 'r') as fp:
    text = fp.read().split('\n')
bag = [item for sentence in text for item in sentence.split('.') if item != '']
bag_size = len(bag)
import random

sentence_a = []
sentence_b = []
label = []

for paragraph in text:
    sentences = [
        sentence for sentence in paragraph.split('.') if sentence != ''
    ]
    num_sentences = len(sentences)
    if num_sentences > 1:
        start = random.randint(0, num_sentences-2)
        # 50/50 whether is IsNextSentence or NotNextSentence
        if random.random() >= 0.5:
            # this is IsNextSentence
            sentence_a.append(sentences[start])
            sentence_b.append(sentences[start+1])
            label.append(0)
        else:
            index = random.randint(0, bag_size-1)
            # this is NotNextSentence
            sentence_a.append(sentences[start])
            sentence_b.append(bag[index])
            label.append(1)

inputs = tokenizer(sentence_a, sentence_b, return_tensors='pt', max_length=512, truncation=True, padding='max_length')
inputs['labels'] = torch.LongTensor([label]).T
class MeditationsDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings
    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    def __len__(self):
        return len(self.encodings.input_ids)

dataset = MeditationsDataset(inputs)
loader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=True)
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
from transformers import AdamW

# activate training mode
model.train()
# initialize optimizer
optim = AdamW(model.parameters(), lr=5e-6)

from tqdm import tqdm  # for our progress bar

epochs = 2

for epoch in range(epochs):
    # setup loop with TQDM and dataloader
    loop = tqdm(loader, leave=True)
    for batch in loop:
        # initialize calculated gradients (from prev step)
        optim.zero_grad()
        # pull all tensor batches required for training
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        token_type_ids = batch['token_type_ids'].to(device)
        labels = batch['labels'].to(device)
        # process
        outputs = model(input_ids, attention_mask=attention_mask,
                        token_type_ids=token_type_ids,
                        labels=labels)
        # extract loss
        loss = outputs.loss
        # calculate loss for every parameter that needs grad update
        loss.backward()
        # update parameters
        optim.step()
        # print relevant info to progress bar
        loop.set_description(f'Epoch {epoch}')
        loop.set_postfix(loss=loss.item())

In the code below I am testing the model on unseen data:
from torch.nn import functional as f
from torch.nn.functional import softmax
prompt = "sentence 1 text"
prompt2 = "sentence 2 text"
output = tokenizer.encode_plus(prompt,prompt2, return_tensors="pt")
result = model(**output)[0]
prob = softmax(result, dim=1)
print(prob)

So, the value of prob and loss is different every single time for the same unseen data which to the best of my knowledge should be same.

Output:
You need to put the model in evaluation mode. if you use i.e. dropout layers while testing the model you should turn it off.
You can do this with
model.eval()

If you don't use this, you will get a different output and loss value because the dropout in your model will close different neurons each time.


Instruction: How to keep track of original images after using transforms in a pytorch model?

Input:
I am working on an AI related issue, where I need to track several human bodyparts on videos. I create a DataLoader with my images and i make several transforms when calling my Dataset class .
Here is a code sample :
transform = transforms.Compose(
        [
            transforms.Resize(img_size),
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
        ]
    )

dataset = NamedClassDataset(annotation_folder_path=path, transform=transform, img_size=img_size, normalized=normalize)
train_set, validation_set = torch.utils.data.random_split(dataset, get_train_test_size(dataset,train_percent))
train_loader = DataLoader(dataset=train_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers,pin_memory=pin_memory)
validation_loader = DataLoader(dataset=validation_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers, pin_memory=pin_memory)

The problem is : after running my model, I display images with the predicted points in order to see their quality. But since images are resized and normalized, I cannot retrieve their original quality and color. I would like to display points on the original images instead of the transformed images and I want to know what is the usual way to do this.
I already have thought of two solutions with their respective disadvantages :

Reverting transformations, but impossible when resize is called since we loose information
Returning an index as a third argument in the __getitem__ method of the NamedClassDataset (along with the image and labels). But pytorch methods expects only two outputs when using __getitem__ which are (image, associated labels).

EDIT : Here is the getitem of my NamedClassDataset class :
def __getitem__(self, index):
        (img_path, coords) = self.annotations.iloc[index].values
        img = Image.open(img_path).convert("RGB")
        w,h = img.size
        # Normalize by img size
        if self.img_size is not None:
            if self.normalized:
                coords = coords/(w,h) # Normalized
            else:
                n_h,n_w = self.img_size
                coords = coords/(w,h)*(n_w,n_h) # Not normalized

        y_coords = torch.flatten(torch.tensor(coords)).float() # Flatten outputs and convert from double to float32

        if self.transform is not None:
            img = self.transform(img)

        return (img, y_coords)

Output:
I managed to do the trick by declaring another dataset with the original images.
# Create the same dataset with untransformed images for visualization purposes
org_dataset = NamedClassDataset(annotation_folder_path="./12_labels/extracted_swimmers", transform=None, img_size=None, normalized=False)
viz_train_set, viz_validation_set = random_split(org_dataset, get_train_test_size(org_dataset,train_percent,_print_size=False), generator=torch.Generator().manual_seed(seed))

And here is what I do in the __getitem__ when transform=None :
        if self.transform is not None:
            tr_img = self.transform(org_img)
            return (tr_img, y_coords)
        return (org_img, y_coords)

I then have access to original images by passing viz sets as parameters. Do note that this is a Dataset and not a Dataloader so you need to take in account your batch size in order to match the predictions.
e.g. :
plot_predictions(viz_set[0+i*batch_size][0], preds[0])

I let the feed open since I strongly believe that a more efficient answer can be provided.


Instruction: Is SGD optimizer in PyTorch actually does Gradient Descent algorithm?

Input:
I'm working on trying to compare the converge rate of SGD and GD algorithms for the neural networks. In PyTorch, we often use SGD optimizer as follows.
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

for epoch in range(epochs):
    running_loss = 0

    for input_batch, labels_batch in train_dataloader:

        input = input_batch

        y_hat = model(input)
        y = labels_batch
        L = loss(y_hat, y)

        optimizer.zero_grad()
        L.backward()

        optimizer.step()

        running_loss += L.item()

My understanding about the optimizer here is that the SGD optimizer actually does the Mini-batch Gradient Descent algorithm because we feed the optimizer one batch of data at one time. So, if we set the batch_size parameter as the size of all data, the code actually does Gradient Descent for the neural network.
Is my understanding correct?

Output:
Your understanding is correct. SGD is just updating weights based on the gradient computed by backpropagation. The flavor of gradient descent that it performs is therefore determined by the data loader.

Gradient descent (aka batch gradient descent): Batch size equal to the size of the entire training dataset.
Stochastic gradient descent: Batch size equal to one and shuffle=True.
Mini-batch gradient descent: Any other batch size and shuffle=True. By far the most common in practical applications.


Instruction: Pytorch Model trained with Lightining has loss stuck at a baseline

Input:
I am building a model for the CIFAR10 dataset.
My model starts with a high loss but only goes to 2.3. Given there are 10 classes and it's logs natural for the loss, it's only giving 10% accuracy (ln(10)=~2.3). I am switching from keras/tensorflow and am lost to what I am doing wrong. Any help/advice/resources would be appreciated.
Model

class Net(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Sequential(nn.Conv2d(3, 6, 3), nn.ReLU())
        self.conv2 = nn.Sequential(nn.Conv2d(6, 12, 3), nn.ReLU())
        self.conv3 = nn.Sequential(nn.Conv2d(12, 24, 3), nn.ReLU())
        self.conv4 = nn.Sequential(nn.Conv2d(24, 128, 3), nn.ReLU())
        self.conv5 = nn.Sequential(nn.Conv2d(128, 256, 3), nn.ReLU())
        self.conv6 = nn.Sequential(nn.Conv2d(256, 256, 3), nn.ReLU())
        self.conv7 = nn.Sequential(nn.Conv2d(256, 512, 3), nn.ReLU())
        self.conv8 = nn.Sequential(nn.Conv2d(512, 512, 3), nn.ReLU())
        self.conv9 = nn.Sequential(nn.Conv2d(512, 512, 3), nn.ReLU(), nn.MaxPool2d(2, 2))
        self.fc1 = nn.Linear(25088, 1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, 512)
        self.fc4 = nn.Linear(512, 84)
        self.last = nn.Linear(84, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = self.conv7(x)
        x = self.conv8(x)
        x = self.conv9(x)
        x = torch.flatten(x, 1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.last(x)
        return x

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        return loss
    def test_step(self,batch,batch_nb):

        x,y = batch
        loss = F.cross_entropy(self(x),y)
        return loss
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

Then I simply do
trainer = pl.Trainer(max_epochs=15, gpus=1)
trainer.fit(net,train_dataloaders=trainloader)

Additionally, when printing out predictions, it always prints "truck".
My dataloading is near identical to the one on the PyTorch tutorial.
transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
batch_size = 128
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=4)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)

Output:
I change some of your code, and after 15 epochs, I get a 0.866 in loss.
For the CIFAR10 dataset, you do not need to create a large network. (This large network needs more time to train. Maybe, for this reason, the loss of your network slowly decreases.)
# !pip install pytorch_lightning

import torch
from torch import nn
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR10
from torchvision import transforms
import pytorch_lightning as pl

class Net(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        loss = F.cross_entropy(self(x),y)
        return loss

# data
transform = transforms.Compose([
    transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
batch_size = 128
trainset = CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=2)

testset = CIFAR10(root='./data', train=False,download=True, transform=transform)
testloader = DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat','deer', 'dog', 'frog', 'horse', 'ship', 'truck')
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)

# model
model = Net()

# training
trainer = pl.Trainer(max_epochs=15, gpus=1)
trainer.fit(model, trainloader, testloader)

Output:
Files already downloaded and verified
Files already downloaded and verified
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name  | Type      | Params
------------------------------------
0 | conv1 | Conv2d    | 456
1 | pool  | MaxPool2d | 0
2 | conv2 | Conv2d    | 2.4 K
3 | fc1   | Linear    | 48.1 K
4 | fc2   | Linear    | 10.2 K
5 | fc3   | Linear    | 850
------------------------------------
62.0 K    Trainable params
0         Non-trainable params
62.0 K    Total params
0.248     Total estimated model params size (MB)
cuda:0
Epoch 14: 100%...470/470 [00:14<00:00, 32.84it/s, loss=0.864, v_num=1]


Instruction: What each parameter returned by cuda.get_device_properties represents

Input: _CudaDeviceProperties(name='NVIDIA GeForce GTX 1050 Ti', major=6, minor=1, total_memory=4095MB, multi_processor_count=6)

Output:
Name of the card: NVIDIA GeForce GTX 1050 Ti
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 4GB
Multiprocessors: 6


Instruction: Why is REINFORCE loss differentiable?

Input:
Here is some sample REINFORCE code found in the PyTorch distributions docs:
probs = policy_network(state)
m = Categorical(probs)
action = m.sample()
next_state, reward = env.step(action)
loss = -m.log_prob(action) * reward
loss.backward()

I don't understand why this loss is differentiable. In particular, how does m.log_prob(action) maintain the computational path of the network output probs? How are m.log_prob(action) and probs are 'connected'?
Edit: I looked at the implementation of log_prob, and it doesn't even seem to reference self.probs anywhere; only self.logits.

Output: As @lejlot noted in the comments, if a Categorical object is constructed with probs rather than logits, then logits is later defined in terms of probs. Hence, when logits is used in log_prob, the gradients from probs are propagated. I missed this connection between logits and probs because it doesn't occur in __init__, but instead, logits is a lazy_property.


Instruction: Save a Bert model with custom forward function and heads on Hugginface

Input:
I have created my own BertClassifier model, starting from a pretrained and then added my own classification heads composed by different layers. After the fine-tuning, I want to save the model using model.save_pretrained() but when I print it upload it from pretrained i don't see my classifier head.
The code is the following. How can I save the all structure on my model and make it full accessible with  AutoModel.from_preatrained('folder_path') ?
. Thanks!
class BertClassifier(PreTrainedModel):
    """Bert Model for Classification Tasks."""
    config_class = AutoConfig
    def __init__(self,config, freeze_bert=True): #tuning only the head
        """
         @param    bert: a BertModel object
         @param    classifier: a torch.nn.Module classifier
         @param    freeze_bert (bool): Set `False` to fine-tune the BERT model
        """
        #super(BertClassifier, self).__init__()
        super().__init__(config)

        # Instantiate BERT model
        # Specify hidden size of BERT, hidden size of our classifier, and number of labels
        self.D_in = 1024 #hidden size of Bert
        self.H = 512
        self.D_out = 2

        # Instantiate the classifier head with some one-layer feed-forward classifier
        self.classifier = nn.Sequential(
            nn.Linear(self.D_in, 512),
            nn.Tanh(),
            nn.Linear(512, self.D_out),
            nn.Tanh()
        )

    def forward(self, input_ids, attention_mask):

         # Feed input to BERT
        outputs = self.bert(input_ids=input_ids,
                             attention_mask=attention_mask)

         # Extract the last hidden state of the token `[CLS]` for classification task
        last_hidden_state_cls = outputs[0][:, 0, :]

         # Feed input to classifier to compute logits
        logits = self.classifier(last_hidden_state_cls)

        return logits

configuration=AutoConfig.from_pretrained('Rostlab/prot_bert_bfd')
model = BertClassifier(config=configuration,freeze_bert=False)

Saving the model after fine-tuning
model.save_pretrained('path')

Loading the fine-tuned model
model = AutoModel.from_pretrained('path')

Printing the model after loading shows I have as the last layer the following and missing my 2 linear layer:
 (output): BertOutput(
          (dense): Linear(in_features=4096, out_features=1024, bias=True)
          (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.0, inplace=False)
          (adapters): ModuleDict()
          (adapter_fusion_layer): ModuleDict()
        )
      )
    )
  )
  (pooler): BertPooler(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (activation): Tanh()
  )
  (prefix_tuning): PrefixTuningPool(
    (prefix_tunings): ModuleDict()
  )
)

Output:
Maybe something is wrong with the config_class attribute inside your BertClassifier class. According to the documentation you need to create an additional config class which inherits form PretrainedConfig and initialises the model_type attribute with the name of your custom model.
The BertClassifier's config_class has to be consistent with your custom config class type.
Afterwards you can register your config and model with the following calls:

AutoConfig.register('CustomModelName', CustomModelConfigClass)
AutoModel.register(CustomModelConfigClass, CustomModelClass)

And load your finetuned model with AutoModel.from_pretrained('YourCustomModelName')
An incomplete example based on your code could look like this:

class BertClassifierConfig(PretrainedConfig):
    model_type="BertClassifier"

class BertClassifier(PreTrainedModel):
    config_class = BertClassifierConfig
    # ...

configuration = BertClassifierConfig()
bert_classifier = BertClassifier(configuration)

# do your finetuning and save your custom model
bert_classifier.save_pretrained("CustomModels/BertClassifier")

# register your config and your model
AutoConfig.register("BertClassifier", BertClassifierConfig)
AutoModel.register(BertClassifierConfig, BertClassifier)

# load your model with AutoModel
bert_classifier_model = AutoModel.from_pretrained("CustomModels/BertClassifier")

Printing the model output should be similiar to this:
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (classifier): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): Tanh()
    (2): Linear(in_features=512, out_features=2, bias=True)
    (3): Tanh()
    (4): Linear(in_features=2, out_features=512, bias=True)
    (5): Tanh()
  )

Hope this helps.
https://huggingface.co/docs/transformers/custom_models#registering-a-model-with-custom-code-to-the-auto-classes


Instruction: How to modify inherited class for additional parameters?

Input:
I am working on a problem statement related to python classes:
I have two classes:
class MCC(object):

    def __init__(self, problem_type, batch_size, dataset):

        self.problem_type = problem_type
        self.batch_size   = batch_size
        self.dataset      = dataset
        self.cls_weights  = weights_calculation()

    def weights_calculation(self):
        class_weights        = (1 - (self.dataset['labels'].value_counts().sort_index()/len(self.dataset))).values
        return class_weights

second class
from transformers import Trainer
class WeightedTrainer(Trainer):

    def compute_loss(self, model, inputs, return_outputs=False):
        outputs        = model(**inputs)
        logits         = outputs.get('logits')
        labels         = inputs.get('labels')
        loss_func      = nn.CrossEntropyLoss(weight = self.class_weights)
        loss           = loss_func(logits, labels)
        return (loss, outputs) if return_outputs else loss

In the second class, I have to pass weight in nn.CrossEntropyLoss as in code loss_func      = nn.CrossEntropyLoss(weight = self.class_weights)
I want to modify the inherited Trainer class to pass a new parameter custom_class_weight in  MCC class.
What I have tried:
class MCC(object):
    def __init__(self, problem_type, batch_size, dataset, model):

        self.problem_type    = problem_type
        self.batch_size      = batch_size
        self.dataset         = dataset
        self.model           = model

        self.cls_weights     = weights_calculation()
        self.WeightedTrainer = WeightedTrainer(self.cls_weights)
        self.trainer         = self.WeightedTrainer(self.model)

    def weights_calculation(self):
        class_weights        = (1 - (self.dataset['labels'].value_counts().sort_index()/len(self.dataset))).values
        return class_weights

class WeightedTrainer(Trainer):

    def __init__(self, custom_class_weight):
        self.custom_class_weight = custom_class_weight

    def compute_loss(self, model, inputs, return_outputs=False):
        outputs        = model(**inputs)
        logits         = outputs.get('logits')
        labels         = inputs.get('labels')
        loss_func      = nn.CrossEntropyLoss(weight = self.custom_class_weight)
        loss           = loss_func(logits, labels)
        return (loss, outputs) if return_outputs else loss

Which is disturbing the inherited's Trainer class properties and thus giving me an error:
TypeError: 'WeightedTrainer' object is not callable

I also tried:
class WeightedTrainer(Trainer):

    def __init__(self, custom_class_weight, **kwargs,):
        self.custom_class_weight = custom_class_weight
        super(WeightedTrainer, self).__init__(**kwargs)

    def compute_loss(self, model, inputs, return_outputs=False):
        outputs        = model(**inputs)
        logits         = outputs.get('logits')
        labels         = inputs.get('labels')
        loss_func      = nn.CrossEntropyLoss(weight = self.custom_class_weight)
        loss           = loss_func(logits, labels)
        return (loss, outputs) if return_outputs else loss

How to create a child class, using the Trainer class as a parent class and modify an inherited class while keeping all default properties of super class?

Output:
Thank you for @JMA valuable comment, Here is the fix.
I was calling self.WeightedTrainer which is not a callable object. Full working code:
class MCC(object):
    def __init__(self, problem_type, batch_size, dataset, model):

        self.problem_type    = problem_type
        self.batch_size      = batch_size
        self.dataset         = dataset
        self.model           = model

        self.cls_weights     = weights_calculation()
        self.custom_trainer  = WeightedTrainer
        self.trainer         = self.WeightedTrainer(self.model, custom_class_weight = self.cls_weights)

    def weights_calculation(self):
        class_weights        = (1 - (self.dataset['labels'].value_counts().sort_index()/len(self.dataset))).values
        return class_weights

class WeightedTrainer(Trainer):

    def __init__(self, custom_class_weight, **kwargs,):
        super().__init__(**kwargs)
        self.custom_class_weight = custom_class_weight

    def compute_loss(self, model, inputs, return_outputs=False):
        outputs        = model(**inputs)
        logits         = outputs.get('logits')
        labels         = inputs.get('labels')
        loss_func      = nn.CrossEntropyLoss(weight = self.custom_class_weight)
        loss           = loss_func(logits, labels)
        return (loss, outputs) if return_outputs else loss


Instruction: The equivalent of torch.nn.Parameter for LibTorch

Input:
I am trying to port a python PyTorch model to LibTorch in C++.
In python the line of code within a subclass of a torch.Module object
self.A = nn.Parameter(A) where A is a torch.tensor object with requires_grad=True.
What would be the equivalent of this for a torch::Tensor in a torch::nn::Module class in C++ ?
The autocomplete in my editor shows the classes ParameterDict, ParameterList,
ParameterDictImpl, ParamaterListImpl, but no Parameter. Do I need to wrap it in a list of size 1 or is there something else I'm missing. I wasn't able to find what I needed from a google search or the documentation, but I wasn't sure precisely what to search to be honest.

Output:
To register a parameter (or tensor which requires gradients) to a module, you could use:
m.register_parameter("A", torch::ones({20, 1, 5, 5}), True);
in libtorch.