Instruction: Lists of PyTorch Lightning sub-models don't get transferred to GPU

Input:
When using PyTorch Lightning on CPU, everything works fine. However when using GPUs, I get a RuntimeError: Expected all tensors to be on the same device.
It seems that the trouble comes from the model using a list of sub-models which don't get passed to the GPU:
class LambdaLayer(LightningModule):
    def __init__(self, fun):
        super(LambdaLayer, self).__init__()
        self.fun = fun

    def forward(self, x):
        return self.fun(x)

class TorchModel(LightningModule):
    def __init__(self):
        super(TorchModel, self).__init__()
        self.cat_layers = [TorchCatEmbedding(cat) for cat in columns_to_embed]
        self.num_layers = [LambdaLayer(lambda x: x[:, idx:idx+1]) for _, idx in numeric_columns]
        self.ffo = TorchFFO(len(self.num_layers) + sum([embed_dim(l) for l in self.cat_layers]), y.shape[1])
        self.softmax = torch.nn.Softmax(dim=1)

model = TorchModel()
trainer = Trainer(gpus=-1)

Before running trainer(model):
>>> model.device
device(type='cpu')

>>> model.ffo.device
device(type='cpu')

>>> model.cat_layers[0].device
device(type='cpu')

After running trainer(model):
>>> model.device
device(type='cuda', index=0) # <---- correct

>>> model.ffo.device
device(type='cuda', index=0) # <---- correct

>>> model.cat_layers[0].device
device(type='cpu') # <---- still showing 'cpu'

Apparently, PyTorch Lightning is not able to transfer the lists of sub-models to the GPU. How to proceed so that the entire model, including list of sub-models (cat_layers and num_layers) is transferred to the GPU?

Output:
Submodules contained in lists are not registered and can't be transformed as is.
You need to use ModuleList instead, i.e.:
...
from torch.nn import ModuleList
...

class TorchModel(LightningModule):
    def __init__(self):
        super(TorchModel, self).__init__()
        self.cat_layers = ModuleList([TorchCatEmbedding(cat) for cat in columns_to_embed])
        self.num_layers = ModuleList([LambdaLayer(lambda x: x[:, idx:idx+1]) for _, idx in numeric_columns])
        self.ffo = TorchFFO(len(self.num_layers) + sum([embed_dim(l) for l in self.cat_layers]), y.shape[1])
        self.softmax = torch.nn.Softmax(dim=1)

edit: I'm not sure what the Lightning equivalent is, or if one such exists, see also PyTorch Lightning - LightningModule for ModuleList / ModuleDict?


Instruction: "ValueError: You have to specify either input_ids or inputs_embeds" when training AutoModelWithLMHead Model (GPT-2)

Input:
I want to fine-tune the AutoModelWithLMHead model from this repository, which is a German GPT-2 model. I have followed the tutorials for pre-processing and fine-tuning. I have prepocessed a bunch of text passages for the fine-tuning, but when beginning training, I receive the following error:
File "GPT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "GPT\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 774, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds

Here is my code for reference:
# Load data
with open("Fine-Tuning Dataset/train.txt", "r", encoding="utf-8") as train_file:
    train_data = train_file.read().split("--")

with open("Fine-Tuning Dataset/test.txt", "r", encoding="utf-8") as test_file:
    test_data = test_file.read().split("--")

# Load pre-trained tokenizer and prepare input
tokenizer = AutoTokenizer.from_pretrained('dbmdz/german-gpt2')

tokenizer.pad_token = tokenizer.eos_token
train_input = tokenizer(train_data, padding="longest")
test_input = tokenizer(test_data, padding="longest")

# Define model

model = AutoModelWithLMHead.from_pretrained("dbmdz/german-gpt2")
training_args = TrainingArguments("test_trainer")

# Evaluation

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = numpy.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_input,
    eval_dataset=test_input,
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate()

Does anyone know the reason for this? Any help is welcome!

Output: I didn't find the concrete answer to this question, but a workaround. For anyone looking for examples on how to fine-tune the GPT models from HuggingFace, you may have a look into this repo. They listed a couple of examples on how to fine-tune different Transformer models, complemented by documented code examples. I used the run_clm.py script and it achieved what I wanted.


Instruction: Emulate fmin in torch 1.7.1

Input:
In the current version there exists torch.fmin Torch Documentation fimin
Unfortunately, my project relies on torch 1.7.1 and I can't upgrade. Is there another way to use min with NaNs in my tensor to emulate fmin. The NaNs are intended and are not a result of poor implementation. So I would like to keep them.

Output:
It's ugly but this works:
>>> a = torch.tensor([2.2, float('nan'), 2.1, float('nan')])
>>> b = torch.tensor([-9.3, 0.1, float('nan'), float('nan')])
>>> c = torch.stack((a,b))
>>> c[c.isnan()] = float('inf')
>>> min_, idx = torch.min(c, dim=0)
>>> min_[min_.isinf()] = float('nan')
>>> min_
tensor([-9.3000,  0.1000,  2.1000,     nan])


Instruction: Custom loss function in pytorch 1.10.1

Input:
I am struggeling with defining a custom loss function for pytorch 1.10.1. My model outputs a float ranging from -1 to +1. The target values are floats of arbitrary range. The loss should be a sum of pruducts if the sign between the model output and target is different.
I have searched the internet for quite some hours, but it seems there have been some changes to pytorch throughout the last versions, so I don't really know which example would best fit to my use case and pytorch 1.10.1.
Here is my approach so far:
class Loss(torch.nn.Module):
    @staticmethod
    def forward(self, output, target) -> Tensor:
        loss = 0.0

        for i in range(len(target)):
            o = output[i,0]
            t = target[i]
            l = o * t
            if l<0:   #if different sign
                loss -= l

        return loss

Question:

Should I subclass torch.nn.Module or torch.autograd.Function?

Do I need to define @staticmethod?

On some examples, I saw ctx instead of self being used and invocations of ctx.save_for_backward etc. Do I need this? What is its purpose?

When subclassing torch.nn.Module, my code complains: 'Tensor' object has no attribute 'children'. What am I missing?

When subclassing torch.autograd.Function, my code complains about not having a backward function defined. How should my backward function look like?

Output:
Custom loss functions can be as simple as a python function. You can simplify this a bit:
def custom_loss(output, target):
    prod = output[:,0]*target
    return -prod[prod<0].sum()


Instruction: Training LSTM over multiple datasets of different timestep number

Input:
I'm new to working with LSTMs and I'm stuggling to understand them even intuitively.
I'm using them for a Regression problem, I'm having about 6000 datasets of ~450 timesteps each and every timestep has 11 features. The target values are 2d ~ [a,b] and they are the same for a single dataset. After training I want to provide the timesteps and predict the 2d y value.
Example:
dataset (1 out of 6000) has ~450 different timesteps of type  x = [1,2,3,4,5,6,7,8,9,10,11] and a target value y = [1,2]

The problems I'm having currently is understanding what exactly LSTM learns in terms of correlation between inputs, what data do I feed exactly and in what order if I'm dealing with multiple datasets? I'm confused of the term batch_size and what happens with the term seq_length if I have a varying sequences... Do I pass the whole 450 timesteps as sequences?
What I do now is merging all the data in a csv file and passing them to the model. I couldn't run it because of memory problems so I reduced it to 5000 timesteps,
below is the LSTM class I use
'''
class LSTM(nn.Module):
def __init__(self, num_classes, input_size, hidden_size, num_layers):
    super(LSTM, self).__init__()

    self.num_classes = num_classes
    self.num_layers = num_layers
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.seq_length = seq_length

    self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                        num_layers=num_layers, batch_first=True)

    self.fc = nn.Linear(hidden_size, num_classes)

def forward(self, x):
    h_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size))

    c_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size))

    # Propagate input through LSTM
    ula, (h_out, _) = self.lstm(x, (h_0, c_0))

    h_out = h_out.view(-1, self.hidden_size)

    out = self.fc(h_out)

    return out, h_out

'''
I don't really need technical answers.. I would really like if somebody could clarify what is going on in this scenario and how I should approach it... I've searched dozens of posts online but it seems I just don't get it or it is not exactly my case.

Output:
I will try to explain this in a way that also explains the vocabulary.
LSTMs are often used for sequential data, for example a time series, where you have data points x_t for multiple time steps t=t0...tN. Here, N would be the sequence length (=seq_length?). Now that means for D-dimensional data, one "dataset" or more precisely, one sequence has the shape N x D.
Let us for now assume that N is equal for all sequences. That means, if you have B sequences, you can stack them into a B x N x D tensor - this would correspond to the actual dataset, which is basically all data that you use. Here, B is your batch axis, basically just meaning the axis where you stack independent sequences. If you choose to train on all data at the same time, you could just pass your complete B x N x D dataset to the model. Then, your batch size would be B. (a Note below)
Now if the sequence length is not equal, there are multiple things you could do. First, you should ask yourself if you want to train on the full sequences. Is it necessary to read the full N steps to get an estimate of the result, or could it be enough to only look at n < N steps? If that is the case, you can sample b (your new batch size, which you can define how you like) sequences of length n, where n < N for all sequences.
If parts of the sequence are not sufficient to estimate the result, it gets more complicated. Then I would suggest to feed the full sequences individually and just train on single sequences. This basically means the batchsize b=1, since you can not stack the sequences as their length differs pairwise. Here, you will feed your model with a b x n x D tensor.
I'm not sure if any of this is "standard procedure", but this is how I would address this.
NOTE: Training on the full dataset is usually not a good practice. Typically, you want to sample b < B random batches from your dataset, which randomizes your training.


Instruction: Is it possible to install different graphic cards and use multi-GPU in pytorch?

Input:
I have a question.
Is it possible to install different graphic cards and use multi-GPU in pytorch?
Is there any other problem?
Ex>
Is the data parallel function of pytorch available in a combination of 3070 (1ea) + 3080 (1ea)?
Thank you in advance for your response.

Output:
I believe it is possible on the technical level, but it would be sub-optimal. The code that divides the load between the different GPUs assume they are the same. Having different GPUs will basically mean that you'll benefit according to the minimal performance of the GPUs you have.
For example, if you have a card with 1GB mem and another with 10GB you will only be able to work with batches suited for the 1GB card and have 9GB un-utilized on the second.


Instruction: mat1 and mat2 shapes cannot be multiplied for GRU

Input:
I am creating a GRU to do some classification for a project, and I'm relatively new to Pytorch and implementing GRUs. I know similar questions like this one have been answered already but I can't seem to bring the same solution over to my own problem. I understand that there is an issue with the shape/order of my fc arrays but after trying to change things I can no longer see the trees for the wood. I would appreciate it if someone could point me in the right direction.
Below I have attached my code and the error. The datasets im using contain 24 features with a label in the 25th column.
# Imports
import pandas as pd
import numpy as np
import torch
import torchvision  # torch package for vision related things
import torch.nn.functional as F  # Parameterless functions, like (some) activation functions
import torchvision.datasets as datasets  # Standard datasets
import torchvision.transforms as transforms  # Transformations we can perform on our dataset for augmentation
from torch import optim  # For optimizers like SGD, Adam, etc.
from torch import nn  # All neural network modules
from torch.utils.data import Dataset, DataLoader  # Gives easier dataset managment by creating mini batches etc.
from tqdm import tqdm  # For a nice progress bar
from sklearn.preprocessing import StandardScaler

# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Hyperparameters
input_size = 24
hidden_size = 128
num_layers = 1
num_classes = 2
sequence_length = 1
learning_rate = 0.005
batch_size = 8
num_epochs = 3

# Recurrent neural network with GRU (many-to-one)
class RNN_GRU(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN_GRU, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size * sequence_length, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        x = x.unsqueeze(0)
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, _ = self.gru(x, h0)
        out = out.reshape(out.shape[0], -1)

        # Decode the hidden state of the last time step
        out = self.fc(out)
        return out

class MyDataset(Dataset):

  def __init__(self,file_name):
    stats_df=pd.read_csv(file_name)

    x=stats_df.iloc[:,0:24].values
    y=stats_df.iloc[:,24].values

    self.x_train=torch.tensor(x,dtype=torch.float32)
    self.y_train=torch.tensor(y,dtype=torch.float32)

  def __len__(self):
    return len(self.y_train)

  def __getitem__(self,idx):
    return self.x_train[idx],self.y_train[idx]

nomDs=MyDataset("nomStats.csv")
atkDs=MyDataset("atkStats.csv")
train_loader=DataLoader(dataset=nomDs,batch_size=batch_size)
test_loader=DataLoader(dataset=atkDs,batch_size=batch_size)

# Initialize network (try out just using simple RNN, or GRU, and then compare with LSTM)
model = RNN_GRU(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# Train Network
for epoch in range(num_epochs):
    for batch_idx, (data, targets) in enumerate(tqdm(train_loader)):
        # Get data to cuda if possible
        data = data.to(device=device).squeeze(1)
        targets = targets.to(device=device)

        # forward
        scores = model(data)
        loss = criterion(scores, targets)

        # backward
        optimizer.zero_grad()
        loss.backward()

        # gradient descent update step/adam step
        optimizer.step()

# Check accuracy on training & test to see how good our model
def check_accuracy(loader, model):
    num_correct = 0
    num_samples = 0

    # Set model to eval
    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device=device).squeeze(1)
            y = y.to(device=device)

            scores = model(x)
            _, predictions = scores.max(1)
            num_correct += (predictions == y).sum()
            num_samples += predictions.size(0)

    # Toggle model back to train
    model.train()
    return num_correct / num_samples

print(f"Accuracy on training set: {check_accuracy(train_loader, model)*100:2f}")
print(f"Accuracy on test set: {check_accuracy(test_loader, model)*100:.2f}")

Traceback (most recent call last):
  File "TESTGRU.py", line 87, in <module>
    scores = model(data)
  File "C:\Users\steph\anaconda3\envs\FYP\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "TESTGRU.py", line 47, in forward
    out = self.fc(out)
  File "C:\Users\steph\anaconda3\envs\FYP\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\steph\anaconda3\envs\FYP\lib\site-packages\torch\nn\modules\linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\steph\anaconda3\envs\FYP\lib\site-packages\torch\nn\functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1024 and 128x2)

Output:
It seems like these lines
        # Forward propagate LSTM
        out, _ = self.gru(x, h0)
        out = out.reshape(out.shape[0], -1)

are the problem.
It appears that you only want to feed the hidden state of the last time step.
This could be read from the output in two ways:

If you want the output of all layers at the last time step, you should use the second return value of out, _ = self.gru(x, h0) not the first.

If you want to use just the last layer's output at the last time    step (which seems to be the case), you should use
out[:, -1, :]. With this change, you may not need the
reshape operation.


Instruction: Pytorch GPU memory keeps increasing with every batch

Input:
I'm training a CNN model on images. Initially, I was training on image patches of size (256, 256) and everything was fine. Then I changed my dataloader to load full HD images (1080, 1920) and I was cropping the images after some processing. In this case, the GPU memory keeps increasing with every batch. Why is this happening?
PS: While tracking losses, I'm doing loss.detach().item() so that loss is not retained in the graph.

Output:
As suggested here, deleting the input, output and loss data helped.
Additionally, I had the data as a dictionary. Just deleting the dictionary isn't sufficient. I had to iterate over the dict elements and delete all of them.


Instruction: Does torch.nn.MultiheadAttention contain normalisation layer and feed forward layer?

Input: Tried to find the source code of multihead attention but could not find any implementation details. I wonder if this module only contains the attention part rather than the whole transformer block (i.e. It does not contain the normalisation layer, residual connection and an additional feedforward neural network)?

Output: According to the source code, the answer is no. MultiheadAttention unsurprisingly implements only the attention function.


Instruction: Solving "CUDA out of memory" when fine-tuning GPT-2 (HuggingFace)

Input:
I get the reoccuring CUDA out of memory error when using the HuggingFace Transformers library to fine-tune a GPT-2 model and can't seem to solve it, despite my 6 GB GPU capacity, which I thought should be enough for fine-tuning on texts. The error reads as follows:
File "GPT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "GPT\lib\site-packages\transformers\modeling_utils.py", line 1763, in forward
    x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
RuntimeError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 6.00 GiB total capacity; 4.28 GiB already allocated; 24.50 MiB free; 4.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I already set batch size to as low as 2 and reduced training examples without success. I also tried to migrate the code to Colab, where the 12GB RAM were quickly consumed.
My examples are rather long, some counting 2.400 characters, but they should be truncated by the model automatically. My (German) examples look like this:
 Er geht in fremde Wohnungen, balgt sich mit Freund und Feind, ist
zudringlich zu unsern Sämereien und Kirschen.  Wenn die Gesellschaft nicht groß
ist, lasse ich sie gelten und streue ihnen sogar Getreide.  Sollten sie hier
aber doch zu viel werden, so hilft die Windbüchse, und sie werden in den
Meierhof hinabgescheucht.  Als einen bösen Feind zeigte sich der Rotschwanz.  Er
flog zu dem Bienenhause und schnappte die Tierchen weg.  Da half nichts, als ihn
ohne Gnade mit der Windbüchse zu töten.

 Ich wollte
Ihnen mein Wort halten, liebe Mama, aber die Versuchung war zu groß.  Da bin ich
eines Abends in den Keller gegangen und hab' aus allen Fässern den Spund
herausgeklopft.  Bis auf den letzten Tropfen ist das Gift ausgeronnen aus den
Fässern.  Der Schade war groß, aber der Teufel war aus dem Haus. «

Andor lachte.  »Mama, das Geschrei hätten Sie hören sollen! Als ob der
Weltuntergang gekommen wäre. Er bedauerte beinahe seine
Schroffheit.  Nun, nachlaufen wird er ihnen nicht, die werden schon selber
kommen.  Aber bewachen wird er seine Kolonie bei Tag und bei Nacht lassen
müssen.  Hol' der Teufel diesen Mercy.  Muß der gerade in Högyész ein Kastell
haben.  Wenn einer von den Schwarzwäldern dahin kommt und ihn verklagt.

Is there a problem with the data formatting maybe?
If anyone has a hint on how to solve this, it would be very welcome.
EDIT: Thank you Timbus Calin for the answer, I described in the comment how adding the block_size flag to the config.json solved the problem. Here is the whole configuration for reference:
{
    "model_name_or_path": "dbmdz/german-gpt2",
    "train_file": "Fine-Tuning Dataset/train.txt",
    "validation_file": "Fine-Tuning Dataset/test.txt",
    "output_dir": "Models",
    "overwrite_output_dir": true,
    "per_device_eval_batch_size": 8,
    "per_device_train_batch_size": 8,
    "block_size": 100,
    "task_type": "text-generation",
    "do_train": true,
    "do_eval": true
}

Output:
If the memory problems still persist, you could opt for
DistillGPT2, as it has a 33% reduction in the parameters of the
network (the forward pass is also twice as fast). Particularly for a small GPU memory like 6GB VRAM, it could
be a solution/alternative to your problem.
At the same time, it depends on how you preprocess the data. Indeed,
the model is capable of "receiving" a maximum length of N tokens
(could be for example 512/768) depending on the models you choose. I
recently trained a named entity recognition model and the model
had a maximum length of 768 tokens. However, when I manually set the
dimension of the padded tokens in my PyTorch DataLoader() to a big
number, I also got OOM memory (even on 3090 24GB VRAM). As I reduced
the dimension of the tokens to a much smaller one (512 instead of
768 for example) the training started to work and I did not get
any issues with the lack of memory.

TLDR: Reducing the number of tokens in the preprocessing phase, regardless of the max capacity of the network, can also help to solve your memories problem.
Note that reducing the number of tokens to process in a sequence is different from the dimension of a token.


Instruction: What is the cleanest way of installing pytorch with CUDA enabled to the latest versions from CLI?

Input:
The way I have installed pytorch with CUDA (on Linux) is by:

Going to the pytorch website and manually filling in the GUI checklist, and copy pasting the resulting command conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Going to the NVIDIA cudatoolkit install website, filling in the GUI, and copy pasting the following code:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-ubuntu2004-11-5-local_11.5.1-495.29.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-5-local_11.5.1-495.29.05-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-5-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

By the way, if I don't install the toolkit from the NVIDIA website then pytorch tells me CUDA is unavailably, probably because the pytorch conda install command doesn't install the drivers.
Is there a way to do all of this in a cleaner way, without manually checking the latest version each time you reinstall, or filling in a GUI?

Output:
TLDR:
You can always try to use
sudo apt install nvidia-cuda-toolkit (to check which version nvcc --version)
conda install pytorch torchvision torchaudio cudatoolkit -c pytorch -c nvidia
(can add -c conda-forge for more robustness of channels)
Warning: Without any specifics like that, you might end up downloading a build that isn't CUDA, so always check before downloading. This is usually not the case if downloading from pytorch and nvidia channels though.
Step by step also explained here.
Consider the following still
The approach you described usually avoids a lot of headaches on a single PC. Another approach is to use NVIDIA's dockers that are pretty much already set up (still have to set up CUDA drivers though), and just expose ports for jupyter notebook or run jobs directly there. This is nice if you don't have to do extra editing.
In general the actual NVIDIA cudatoolkit that you install can be of higher version (to some extent) then the anaconda version of cudatoolkit, meaning that you don't have to be that precise for looking up version (after 11.1 which supports 3090's). Even if you look at the documentation from nvidia, in the end the website which you choose will build up those same commands. As seen on the image below:

Consider Module
On another note, if you have clusters in company or university, they usually have module load XYZ where you can directly load the CUDA support. If you have multiple computers or version of CUDA need installing, might check out this website for more info on modules. This is highly recommended if you don't wanna do reinstalling all the time.
And when you check module avail you would get something like this:


Instruction: PyTorch can't pickle lambda

Input:
I have a model that uses a custom LambdaLayer as follows:
class LambdaLayer(LightningModule):
    def __init__(self, fun):
        super(LambdaLayer, self).__init__()
        self.fun = fun

    def forward(self, x):
        return self.fun(x)

class TorchCatEmbedding(LightningModule):
    def __init__(self, start, end):
        super(TorchCatEmbedding, self).__init__()
        self.lb = LambdaLayer(lambda x: x[:, start:end])
        self.embedding = torch.nn.Embedding(50, 5)

    def forward(self, inputs):
        o = self.lb(inputs).to(torch.int32)
        o = self.embedding(o)
        return o.squeeze()

The model runs perfectly fine on CPU or 1 GPU. However, when running it with PyTorch Lightning over 2+ GPUs, this error happens:
AttributeError: Can't pickle local object 'TorchCatEmbedding.__init__.<locals>.<lambda>'

The purpose of using a lambda function here is that given an inputs tensor, I want to pass only inputs[:, start:end] to the embedding layer.
My questions:

is there an alternative to using a lambda in this case?
if not, what should be done to get the lambda function to work in this context?

Output:
So the problem isn't the lambda function per se, it's that pickle doesn't work with functions that aren't just module-level functions (the way pickle treats functions is just as references to some module-level name). So, unfortunately, if you need to capture the start and end arguments, you won't be able to use a closure, you'd normally just want something like:
def function_maker(start, end):
    def function(x):
        return x[:, start:end]
    return function

But this will get you right back to where you started, as far as the pickling problem is concerned.
So, try something like:
class Slicer:
    def __init__(self, start, end):
        self.start = start
        self.end = end
    def __call__(self, x):
        return x[:, self.start:self.end])

Then you can use:
LambdaLayer(Slicer(start, end))

I'm not familiar with PyTorch, I'm surprised though that it doesn't offer the ability to use a different serialization backend. The pathos/dill project can pickle arbitrary functions, for example, and is often easier to just use that. But I believe the above should solve the problem.


Instruction: RuntimeError: Found dtype Double but expected Float - PyTorch

Input:
I am new to pytorch and I am working on DQN for a timeseries using Reinforcement Learning and I needed to have a complex observation of timeseries and some sensor readings, so I merged two neural networks and I am not sure if that's what is ruining my loss.backward or something else.
I know there is multiple questions with the same title but none worked for me, maybe I am missing something.
First of all, this is my network:
class DQN(nn.Module):
  def __init__(self, list_shape, score_shape, n_actions):
    super(DQN, self).__init__()

    self.FeatureList =  nn.Sequential(
            nn.Conv1d(list_shape[1], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv1d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv1d(64, 64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten()
        )

    self.FeatureScore = nn.Sequential(
            nn.Linear(score_shape[1], 512),
            nn.ReLU(),
            nn.Linear(512, 128)
        )

    t_list_test = torch.zeros(list_shape)
    t_score_test = torch.zeros(score_shape)
    merge_shape = self.FeatureList(t_list_test).shape[1] + self.FeatureScore(t_score_test).shape[1]

    self.FinalNN =  nn.Sequential(
            nn.Linear(merge_shape, 512),
            nn.ReLU(),
            nn.Linear(512, 128),
            nn.ReLU(),
            nn.Linear(128, n_actions),
    )

  def forward(self, list, score):
    listOut = self.FeatureList(list)
    scoreOut = self.FeatureScore(score)
    MergedTensor = torch.cat((listOut,scoreOut),1)
    return self.FinalNN(MergedTensor)

I have a function called calc_loss, and at its end it return the MSE loss as below
  print(state_action_values.dtype)
  print(expected_state_action_values.dtype)
  return nn.MSELoss()(state_action_values, expected_state_action_values)

and the print shows float32 and float64 respectively.
I get the error when I run the loss.backward() as below
LEARNING_RATE = 0.01
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

for i in range(50):
  optimizer.zero_grad()
  loss_v = calc_loss(sample(obs, 500, 200, 64), net, tgt_net)
  print(loss_v.dtype)
  print(loss_v)
  loss_v.backward()
  optimizer.step()

and the print output is as below:
torch.float64
tensor(1887.4831, dtype=torch.float64, grad_fn=)
Update 1:
I tried using a simpler model, yet the same issue, when I tried to cast the inputs to Float, I got an error:
RuntimeError: expected scalar type Double but found Float

What makes the model expects double ?
Update 2:
I tried to add the below line on top after the torch import but same issue of RuntimeError: Found dtype Double but expected Float
>>> torch.set_default_tensor_type(torch.FloatTensor)

But when I used the DoubleTensor I got:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.DoubleTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Output:
The issue wasn't in the input to the network but the criterion of the MSELoss, so it worked fine after casting the criterion to float as below
return nn.MSELoss()(state_action_values.float(), expected_state_action_values.float())

I decided to leave the answer for beginners like me who might be stuck and didn't expect to check the datatype of the loss criterion


Instruction: How can I get the sum of gradients immediately after loss.backward()?

Input:
I am new to Pytorch, and I am trying to do some importance sampling experiments:
During an evaluation epoch, I calculate the loss for each training sample, and obtain the sum of gradients for this training sample. Finally, I will sort the training samples based on gradients they introduced. For example, if sample A shows a very high gradient sum, it must be an important sample to training. Otherwise, it is not a very important sample.
Note that, the gradients calculated here will not be used to update parameters. In other words, they are only used for selecting importance samples.
I know gradients will be ready somewhere after loss.backward(). But what is the easiest way to grab the summed gradients over the entire model? In my current implementation, I am only allowed to modify one small module with only loss availble, so I don’t have “inputs” or “model”. Is it possible to get the gradients from only “loss”?

Output:
Gradients after backward are stored as the grad attribute of tensors that require grad. You can find all tensors involved and sum up their grads. A cleaner way might be to write a backward hook to accumulate gradients to some global variable while backpropagating.
An example is
import torch
import torch.nn as nn

model = nn.Linear(5, 3)
print(model.weight.grad)  # None, since the grads have not been computed yet
print(model.bias.grad)

x = torch.randn(5, 5)
y = model(x)
loss = y.sum()
loss.backward()

print(model.weight.grad)
print(model.bias.grad)

output:
None
None
tensor([[-0.6164,  1.1585, -3.4117, -4.3192, -3.7273],
        [-0.6164,  1.1585, -3.4117, -4.3192, -3.7273],
        [-0.6164,  1.1585, -3.4117, -4.3192, -3.7273]])
tensor([5., 5., 5.])

As you see, you can access the gradients as param.grad. If model is an torch.nn.Module object, you can iterate over its parameters with for param in model.parameters().
Maybe you can also work with backward hooks but I am not that familiar with them to give a code example.


Instruction: Running test calculations in DDP mode with multiple GPUs with PyTorchLightning

Input:
I have a model which I try to use with trainer in DDP mode.
import pytorch_lightning as pl
import torch
import torchvision
from torchmetrics import Accuracy
class Model(pl.LightningModule):

    def __init__(
        self,
        model_name: str,
        num_classes: int,
        model_hparams: Dict["str", Union[str, int]],
        optimizer_name: str,
        optimizer_hparams: Dict["str", Union[str, int]],
    ):

        super().__init__()
        self.save_hyperparameters()
        self.model = torchvision.resnet18(num_classes=num_classes, **model_hparams)

        self.loss_module = CrossEntropyLoss()
        self.example_input_array = torch.zeros((1, 3, 512, 512), dtype=torch.float32)
        # Trying to use in DDP mode
        self.test_accuracy = Accuracy(num_classes=num_classes)

    def forward(self, imgs) -> Tensor:
        return self.model(imgs)

    # <redacted training_*, val_*, etc. as they are not relevant>

    def test_step(self, batch, batch_idx):
        imgs, labels = batch
        preds = self.model(imgs)
        self.test_accuracy.update(preds, labels)
        return labels, preds.argmax(dim=-1)

    def test_epoch_end(self, outputs) -> None:
        num_classes = self.hparams.num_classes
        # Creates table of correct and incorrect predictions
        results = torch.zeros((num_classes, num_classes))
        for output in outputs:
            for label, prediction in zip(*output):
                results[int(label), int(prediction)] += 1
        # Total accuracy. This and `compute` are identical in 1 GPU training
        acc = results.diag().sum() / results.sum()
        self.log("test_acc", self.test_accuracy.compute())
        print(results) # This prints twice

and trainer
trainer = pl.Trainer(
        gpus=torch.cuda.device_count(),
        max_epochs=180,
        callbacks=callbacks,
        strategy="ddp",
        auto_scale_batch_size="binsearch",
    )

However, I get as prints from test
tensor([[0., 0., 0., 0., 0., 5.],
        [0., 7., 0., 0., 0., 0.],
        [0., 3., 0., 0., 0., 2.],
        [0., 3., 0., 0., 0., 0.],
        [0., 3., 0., 0., 0., 2.],
        [0., 1., 0., 0., 0., 4.]])tensor([[0., 0., 0., 0., 0., 6.],
        [0., 2., 0., 0., 0., 0.],
        [0., 4., 0., 0., 0., 2.],
        [0., 2., 0., 0., 0., 1.],
        [0., 3., 0., 0., 0., 2.],
        [0., 5., 0., 0., 0., 3.]])

Also
trainer.fit(model, datamodule=datamodule)
test_results = trainer.test(model, datamodule=datamodule)
print(test_results)
# [{'test_acc': 0.18333333730697632}]
# [{'test_acc': 0.18333333730697632}]

where I would only expect single tensor to be printed. How can I make my calculations over all test predictions rather than by GPU and return the table I create in test_epoch_end from those predictions? I interpreted the documentation as *_epoch_end being executed only on single GPU and am quite lost.

Output:
I think you should use following techniques:

test_epoch_end: In ddp mode, every gpu runs same code in this method. So each gpu computes metric on partial batch not whole batches. You need to synchronize metric and collect to rank==0 gpu to compute
evaluation metric on entire dataset.

torch.distributed.reduce: This method collects and calculate tensors across distributed gpu devices. (docs)

self.trainer.is_global_zero: This flag will be true for rank==0

What is best way to manually compute metric over test set? you should check docs

Using mentioned techniques, you can compute metric over entire dataset and use results tensor after .test. Here is snippet:
import os

import torch
from torch.utils.data import DataLoader

from torchvision import models, transforms
from torchvision.datasets import CIFAR10

from pytorch_lightning import LightningModule, LightningDataModule, Trainer

os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'

class CIFAR(LightningDataModule):
    def __init__(self, img_size=32, batch_size=32):
        super().__init__()
        self.img_size = img_size if isinstance(img_size, tuple) else (img_size, img_size)
        self.batch_size = batch_size

        self.test_transforms = transforms.Compose([
            transforms.Resize(self.img_size),
            transforms.CenterCrop(self.img_size),
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
        ])

    def prepare_data(self) -> None:
        CIFAR10(root='data', train=True, download=True)
        CIFAR10(root='data', train=False, download=True)

    def setup(self, stage=None):
        self.test_ds = CIFAR10(root='data', train=False, download=False, transform=self.test_transforms)

    def test_dataloader(self):
        return DataLoader(self.test_ds, num_workers=4, batch_size=self.batch_size, shuffle=False)

class BasicModule(LightningModule):
    def __init__(self):
        super().__init__()
        self.model = models.resnet18(num_classes=10, pretrained=False)

    def test_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.model(x)
        return y, y_hat.argmax(dim=-1)

    def test_epoch_end(self, outputs):
        results = torch.zeros((10, 10)).to(self.device)
        for output in outputs:
            for label, prediction in zip(*output):
                results[int(label), int(prediction)] += 1
        torch.distributed.reduce(results, 0, torch.distributed.ReduceOp.SUM)
        acc = results.diag().sum() / results.sum()
        if self.trainer.is_global_zero:
            self.log("test_metric", acc, rank_zero_only=True)
            self.trainer.results = results

if __name__ == '__main__':
    data = CIFAR(batch_size=512)
    model = BasicModule()
    trainer = Trainer(max_epochs=2, gpus='0,1', strategy="ddp", precision=16)
    test_results = trainer.test(model, data)
    if trainer.is_global_zero:
        print(test_results)
        print(trainer.results)


Instruction: Bigger batch size improves training by too much

Input:
I am writing a classifier that takes a surname and predicts a language it belongs to. I found that small batch sizes (256 and less) perform poorly compared to big batch sizes (2048 and more). Could someone give me some insight on why this is happening and how to fix it? Thank you.
Training code:
def indices_to_packed(names, input_size):
    names = [F.one_hot(item, input_size).float() for item in names]
    names_packed = pack_sequence(names, enforce_sorted=False)
    return names_packed

def infer(model, data, labels, lengths, device):
    data_packed = indices_to_packed(data, model.rnn.input_size)
    data_packed, labels, lengths = data_packed.to(device), labels.to(device), lengths.to(device)
    preds = model(data_packed, lengths)
    loss = loss_fn(preds, labels)
    return loss, preds

results = {}
epochs = 100
for BATCH_SIZE in [4096, 2048, 256]:
    train_loader = data.DataLoader(train_data, BATCH_SIZE, sampler=train_sampler, collate_fn=partial(my_collate, input_size=input_size, output_size=output_size))
    val_loader = data.DataLoader(val_data, BATCH_SIZE, sampler=val_sampler, collate_fn=partial(my_collate, input_size=input_size, output_size=output_size))
    model = LSTM(input_size, HIDDEN_SIZE, NUM_LAYERS, DROPOUT, output_size)
    optimizer = torch.optim.Adam(model.parameters())
    model.to(device)
    train_losses = []
    val_losses = []
    cur_losses = {}
    duration = 0
    for epoch in range(epochs):
        start = time.time()
        train_loss = 0
        model.train()
        # Using PackedSequence
        for names, langs, lengths in train_loader:
            optimizer.zero_grad()
            loss, _ = infer(model, names, langs, lengths, device)
            loss.backward()
            optimizer.step()
            train_loss += loss

        train_loss /= len(train_data)
        train_losses.append(train_loss.cpu().detach().numpy())

        model.eval()
        val_loss = 0
        with torch.no_grad():
            for names, langs, lengths in val_loader:
                loss, _ = infer(model, names, langs, lengths, device)
                val_loss += loss
        val_loss /= len(val_data)
        val_losses.append(val_loss.cpu().detach().numpy())

        cur_duration = time.time() - start
        duration += cur_duration

        log_line = (f"BATCH_SIZE: {BATCH_SIZE} epoch: {epoch} train loss: "
                    f"{train_loss:.5f} val loss: {val_loss:.5f}")
        print(log_line)
    cur_losses["train_losses"] = train_losses
    cur_losses["val_losses"] = val_losses
    results[BATCH_SIZE] = {"losses" : cur_losses, "duration" : duration, "model": model}

Model:
class LSTM(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, dropout, output_size):
    super().__init__()

    self.rnn = nn.LSTM(input_size, hidden_size, num_layers, dropout=DROPOUT)
    self.linear = nn.Linear(hidden_size, output_size)
    self.softmax = nn.LogSoftmax(dim=1)

def forward(self, x, lengths):
    lstm_out, _ = self.rnn(x)
    # https://discuss.pytorch.org/t/get-each-sequences-last-item-from-packed-sequence/41118/7
    sum_batch_sizes = torch.cat((
        torch.zeros(2, dtype=torch.int64),
        torch.cumsum(lstm_out.batch_sizes, 0)
    ))
    sorted_lengths = lengths[lstm_out.sorted_indices]
    last_seq_idxs = sum_batch_sizes[sorted_lengths] + torch.arange(lengths.size(0))
    last_seq_items = lstm_out.data[last_seq_idxs]
    lstm_last_out = last_seq_items[lstm_out.unsorted_indices]
    linear_out = self.linear(lstm_last_out)
    softmax_out = self.softmax(linear_out)

    return softmax_out

Losses with different batch sizes:

Output:
It looks like there issue is how the loss is calculated.
train_loss += loss line accumulates the loss. When batch size is higher, there will be fewer steps to do. The code normalizes this by dividing by the length of train data, train_loss /= len(train_data), but should probably take into account the batch size: train_loss /= (len(train_data) / BATCH_SIZE).
The same for validation loss, but the effect is different probably because of smaller data size compared to training data.


Instruction: Collecting features from network.foward() in TensorFlow

Input:
So basically I want to achieve the same goal as in this code but in TensorFlow
def get_function(network, loader):
    ''' Collect function (features) from the self.network.module.forward_features() routine '''
    features = []
    for batch_idx, (inputs, targets) in enumerate(loader):
        inputs, targets = inputs.to('cpu'), targets.to('cpu')

        features.append([f.cpu().data.numpy().astype(np.float16) for f in network.forward_features(inputs)])

    return [np.concatenate(list(zip(*features))[i]) for i in range(len(features[0]))]

Are there any clean ways to do this with TensorFlow iterator? Here is the torch code that I want to replicate in TensorFlow. https://pastecode.io/s/b03cpoyv

Output:
To answer your question, I just need to ensure that you understand your original torch code properly. So, here's your workflow
class LeNet(nn.Module):
     def forward:
         # few bunch of layers
         return output

     def forward_features:
         # same as forward function
         return [each layer output]

Now, next, you use the torch_get_function method and retrieve all layers output from the forward_features function that is defined in your model. The torch_get_function gives a total of 4 outputs as a list and you pick only the first feature and concate across the batches in the end.
def torch_get_function(network, loader):
    features = []
    for batch_idx, (inputs, targets) in enumerate(loader):

        print('0', network.forward_features(inputs)[0].shape)
        print('1', network.forward_features(inputs)[1].shape)
        print('2', network.forward_features(inputs)[2].shape)
        print('3', network.forward_features(inputs)[3].shape)
        print()

        features.append([f... for f in network.forward_features(inputs)])
    return [np.concatenate(list(zip(*features))[i]) for i in range(len(features[0]))]

for epoch in epochs:
    dataset = torchvision.datasets.MNIST...
    dataset = torch.utils.data.Subset(dataset, list(range(0, 1000)))
    functloader = torch.utils.data.DataLoader(...)

    # for x , y in functloader:
    #     print('a ', x.shape, y.shape)
    # a  torch.Size([100, 1, 28, 28]) torch.Size([100])

    activs = torch_get_function(net, functloader)
    print(activs[0].shape)
    break

That's why if when I ran your code, I got
# These are the 4 output that returned by forward_features(inputs)
0 torch.Size([100, 10, 12, 12])
1 torch.Size([100, 320])
2 torch.Size([100, 50])
3 torch.Size([100, 10])

...

# In the return statement of forward_features -
# You take only the first index feature and concate across batches.
(1000, 10, 12, 12)

So, the input size of your model is (batch_size, 1, 28, 28) and the final output is like (1000, 10, 12, 12).

Let's do the same in tensorflow, step by step.
import numpy as np
from tqdm import tqdm

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import (Conv2D, Dropout, MaxPooling2D,
                                     Dense, Flatten)

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_test = x_test.astype("float32") / 255.0
x_test = np.reshape(x_test, (-1, 28, 28, 1))
dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
dataset = dataset.shuffle(buffer_size=1024).batch(100)

# it's like torch.utils.data.Subset
dataset = dataset.take(1000)
dataset
<TakeDataset shapes: ((None, 28, 28, 1), (None,)), types: (tf.float32, tf.uint8)>

Let's now build the model. To make it familiar to you, I'm writing in sub-class API.
class LeNet(keras.Model):
    def __init__(self, num_classes, input_size=28):
        super(LeNet, self).__init__()
        self.conv1 = Conv2D(10, (5, 5))
        self.conv2 = Conv2D(20, (5, 5))
        self.conv2_drop = Dropout(rate=0.5)
        self.fc1 = Dense(50)
        self.fc2 = Dense(num_classes)

    def call(self, inputs, training=None):
        x1 = tf.nn.relu(MaxPooling2D(2)(self.conv1(inputs)))
        x2 = tf.nn.relu(MaxPooling2D(2)(self.conv2_drop(self.conv2(x1))))
        x2 = Flatten()(x2)
        x3 = tf.nn.relu(self.fc1(x2))
        x4 = tf.nn.softmax(self.fc2(x3), axis=1)

        # in tf/keras, when we will call model.fit / model.evaluate
        # to train the model only x4 will return
        if training:
            x4
        else: # but when model(input)/model.predict(), we can return many :)
            return [x1, x2, x3, x4]

lenet = LeNet(10)
lenet.build(input_shape=(None, 28, 28, 1))

Get the desired features
features = []

for input, target in tqdm(dataset):
    # lenet(...) will give 4 output as we model
    # but as we're interested on the first index feature...
    features.append(lenet(input, training=False)[0])

print(len(features))
features = np.concatenate(features, axis=0)
features.shape
(10000, 12, 12, 10)

In tensorflow, the channel axis is default set to last, as opposed to the torch. In torch, you received (1000, 10, 12, 12) and in tensorflow, it gives you (10000, 12, 12, 10) but you can change it of course, (how). Here is the working colab.


Instruction: Wrong outputs from torch.sub?

Input:
I’m currently using torch.sub alongside torch.div to obtain the MAPE between my predicted and true labels for my neural network although I’m not getting the answers I’m expecting. According to the example in the documentation, I should be getting a 4x1 tensor, not 4x4.
Could anyone clear this up for me?
print('y_true ', y_true)
y_true tensor([[ 46],
[262],
[ 33],
[ 35]], device=‘cuda:0’, dtype=torch.int16)

print('y_pred ', y_pred)
y_pred tensor([[[308.5075]],
[[375.8983]],
[[389.4587]],
[[406.4957]]], device=‘cuda:0’, grad_fn=)

print('torch.sub ', torch.sub(y_true, y_pred))
torch.sub tensor([[[-262.5075],
[ -46.5075],
[-275.5075],
[-273.5075]],

    [[-329.8983],
     [-113.8983],
     [-342.8983],
     [-340.8983]],

    [[-343.4587],
     [-127.4587],
     [-356.4587],
     [-354.4587]],

    [[-360.4957],
     [-144.4957],
     [-373.4957],
     [-371.4957]]], device='cuda:0', grad_fn=<SubBackward0>)

Output:
That is because y_pred has an extra dimension which means the y_true tensor
probably gets broadcasted to the correct dimension.
If you remove the extra last dimension you get the desired result:
>>> torch.sub(y_true, y_pred[...,0]).shape
torch.Size([4, 1])


Instruction: ValueError in trainer.fit()

Input: I have encountered ValueError: No positive samples in targets, true positive value should be meaningless when I tried to run trainer.fit(model, dataset) for my model, but I've double checked dataset and all of the train/val/test sets had proper number of positive samples. Why would this occur and where should I start to fix this issue? Thanks!

Output:
Before the training loop actually starts, PL trainer will run a sanity check of validation loop for two steps. In that case, these two steps may only have one type of label (negative or positive) and crash your metrics.
Turn it off by setting num_sanity_val_steps=0 in your trainer.
https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#num-sanity-val-steps


Instruction: PyTorch equivalent for Keras sequential model

Input:
How to get the perfect copy of this Keras sequential network in PyTorch?
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

Output:
This is a snippet that works for this case:
    model_torch = nn.Sequential(
         nn.Flatten(),
         nn.Linear(28*28, 128),
         nn.ReLU(),
         nn.Linear(128, 10),
    )


Instruction: Looking for a torch.imshow() 'like' command

Input:
Say that i have a variable image (which is currently located on the gpu), sized [32,1,256,256] where 32 is the batch size, 1 is the amount of channels (gray scale).
Instead of ploting this:
plt.imshow(img[0,0,:,:].cpu().detach(),'gray');plt.show()

I wish i could do
torch.imshow(img,8,'gray') and it will subplot 8 images from my batch
is there any thing like that?

Output: You are looking for torchvision.utils.make_grid: It will convert the [32, 1, 256,256] tensor into a grid of 32 images. You still need to use plt to actually plot the image grid to screen.


Instruction: wandb.plot.line does not work and it just shows a table

Input:
I used this example that was provided by WandB. However, the web interface just shows a table instead of a figure.
data = [[i, random.random() + math.sin(i / 10)] for i in range(100)]
        table = wandb.Table(data=data, columns=["step", "height"])
        wandb.log({'line-plot1': wandb.plot.line(table, "step", "height")})

This is a screenshot from WandB's web interface:

Also, I have the same problem with other kinds of figures and charts that use a table.

Output:
X-Post from the wandb forum

https://community.wandb.ai/t/wandb-plot-confusion-matrix-just-show-a-table/1744
If you click the section called “Custom Charts” above the Table, it’ll show the line plot that you’ve logged.
Logging the Table also is expected behaviour, because this will allow users to interactively explore the logged data in a W&B Table after logging it.


Instruction: How to use the Inception model for transfer learning in PyTorch?

Input:
I have created a PyTorch torchvision model for transfer learning, using the pre-built ResNet50 base model, like this:
        # Create base model from torchvision.models
        model = resnet50(pretrained=True)
        num_features = model.fc.in_features

        # Define the network head and attach it to the model
        model_head = nn.Sequential(
            nn.Linear(num_features, 512),
            nn.ReLU(),
            nn.Dropout(0.25),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes),
        )
        model.fc = model_head

Now I wanted to use the Ineception v3 model instead as base, so I switched from resnet50() above to inception_v3(), the rest stayed as is. However, during training I get the following error:

TypeError: cross_entropy_loss(): argument 'input' (position 1) must be Tensor, not InceptionOutputs

So how can one use the Inception v3 model from torchvision.models as base model for transfer learning?

Output:
From PyTorch documentation about Inceptionv3 architecture:
This network is unique because it has two output layers when training. The primary output is a linear layer at the end of the network. The second output is known as an auxiliary output and is contained in the AuxLogits part of the network.
Have a look at this tutorial: https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#inception-v3 There you can find how to use transfer learning for several models, included ResNet and Inception.


Instruction: Pytorch Index Error ('index out of range in self'): How to Solve?

Input:
I recently encountered a roadblock following a deep learning tutorial on youtube (entire code can be found here). I'm having a problem with part 4.4. The goal is to return a dictionary of article summaries for certain stocks (their tickers are in a list: monitered_tickers).
def summarize(articles):
summaries = []
for article in articles:
    input_ids = tokenizer.encode(article, return_tensors='pt')
    output = model.generate(input_ids, max_length=55, num_beams=5, early_stopping=True)
    summary = tokenizer.decode(output[0], skip_special_tokens=True)
    summaries.append(summary)
return summaries

summaries = {ticker:summarize(articles[ticker]) for ticker in monitered_tickers}
summaries

When I run the code above, I get the following error:
    ---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_10688/3329134555.py in <module>
----> 1 summaries = {ticker:summarize(articles[ticker]) for ticker in monitered_tickers}
      2 summaries

~\AppData\Local\Temp/ipykernel_10688/3329134555.py in <dictcomp>(.0)
----> 1 summaries = {ticker:summarize(articles[ticker]) for ticker in monitered_tickers}
      2 summaries

~\AppData\Local\Temp/ipykernel_10688/3177436574.py in summarize(articles)
      3     for article in articles:
      4         input_ids = tokenizer.encode(article, return_tensors='pt')
----> 5         output = model.generate(input_ids, max_length=40, num_beams=5, early_stopping = True)
      6         summary = tokenizer.decode(output[0], skip_special_tokens=True)
      7         summaries.append(summary)

~\anaconda3\lib\site-packages\torch\autograd\grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---> 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30

~\anaconda3\lib\site-packages\transformers\generation_utils.py in generate(self, inputs, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, max_new_tokens, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, logits_processor, stopping_criteria, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
   1022             # if model is encoder decoder encoder_outputs are created
   1023             # and added to `model_kwargs`
-> 1024             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(
   1025                 inputs_tensor, model_kwargs, model_input_name
   1026             )

~\anaconda3\lib\site-packages\transformers\generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, inputs_tensor, model_kwargs, model_input_name)
    484                 encoder_args = ()
    485
--> 486             model_kwargs["encoder_outputs"]: ModelOutput = encoder(*encoder_args, **encoder_kwargs)
    487
    488         return model_kwargs

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~\anaconda3\lib\site-packages\transformers\models\pegasus\modeling_pegasus.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    754             inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale
    755
--> 756         embed_pos = self.embed_positions(input_shape)
    757
    758         hidden_states = inputs_embeds + embed_pos

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~\anaconda3\lib\site-packages\torch\autograd\grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---> 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30

~\anaconda3\lib\site-packages\transformers\models\pegasus\modeling_pegasus.py in forward(self, input_ids_shape, past_key_values_length)
    138             past_key_values_length, past_key_values_length + seq_len, dtype=torch.long, device=self.weight.device
    139         )
--> 140         return super().forward(positions)
    141
    142

~\anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    156
    157     def forward(self, input: Tensor) -> Tensor:
--> 158         return F.embedding(
    159             input, self.weight, self.padding_idx, self.max_norm,
    160             self.norm_type, self.scale_grad_by_freq, self.sparse)

~\anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2042         # remove once script supports set_grad_enabled
   2043         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2044     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2045
   2046

IndexError: index out of range in self

Wondering how I can fix this. Bit of a newbie so any help on this would be greatly appreciated. Thank you.

Output:
Your article length might exceed the model max input length. Use:
tokenizer.encode(article, return_tensors='pt', max_length=512, truncation=True)


Instruction: Reproducing arithmetic with pytorch's quantized tensors with numpy operations

Input:
I would like to know what exact arithmetic operations I have to do to reproduce results of quantized operations in pytorch.
This is almost duplicate question to:
I want to use Numpy to simulate the inference process of a quantized MobileNet V2 network, but the outcome is different with pytorch realized one
But I would even simplify it with the example of adding two quantized tensors.
For example for addition of two quantized tensors in Resnet architecture I use nn.quantized.FloatFunctional().
self.skip_add = nn.quantized.FloatFunctional()

And during inference I can add two tensors via
out1 = self.skip_add.add(x1, x2)

where x1 and x2 are tensors of torch.Tensor type, quantized with fbgemm backend during post training quantization procedure.
I expected out2_int = x1.int_repr() + x2.int_repr() should be the same as out1.int_repr() (with probably need of clamping in the needed range).
However that is not the case.
Below I dump the example outputs.
So I wonder how can I  get out1 with integer operations?
>print(x1)

      ...,
      [-0.0596, -0.0496, -0.1390,  ..., -0.0596, -0.0695, -0.0099],
      [-0.0893,  0.0000, -0.0695,  ...,  0.0596, -0.0893, -0.0298],
      [-0.1092,  0.0099,  0.0000,  ..., -0.0397, -0.0794, -0.0199]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.009925744496285915,
   zero_point=75)

print(x2)

      ...,
      [ 0.1390, -0.1669, -0.0278,  ..., -0.2225, -0.0556, -0.1112],
      [ 0.0000, -0.1669, -0.0556,  ...,  0.0556,  0.1112, -0.2781],
      [ 0.1390,  0.1669,  0.0278,  ...,  0.2225,  0.4171,  0.0834]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.02780967578291893,
   zero_point=61)

print(x1.int_repr())

      ...,
      [69, 70, 61,  ..., 69, 68, 74],
      [66, 75, 68,  ..., 81, 66, 72],
      [64, 76, 75,  ..., 71, 67, 73]]]], dtype=torch.uint8)

print(x2.int_repr())

      ...,
      [66, 55, 60,  ..., 53, 59, 57],
      [61, 55, 59,  ..., 63, 65, 51],
      [66, 67, 62,  ..., 69, 76, 64]]]], dtype=torch.uint8)

print(out1)

      ...,
      [ 0.0904, -0.2109, -0.1808,  ..., -0.2712, -0.1205, -0.1205],
      [-0.0904, -0.1808, -0.1205,  ...,  0.1205,  0.0301, -0.3013],
      [ 0.0301,  0.1808,  0.0301,  ...,  0.1808,  0.3314,  0.0603]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.03012925386428833,
   zero_point=56)

print(out1.int_repr())

      ...,
      [59, 49, 50,  ..., 47, 52, 52],
      [53, 50, 52,  ..., 60, 57, 46],
      [57, 62, 57,  ..., 62, 67, 58]]]], dtype=torch.uint8)

 print(out2_int)

      [135, 125, 121,  ..., 122, 127, 131],
      [127, 130, 127,  ..., 144, 131, 123],
      [130, 143, 137,  ..., 140, 143, 137]]]], dtype=torch.uint8)

Output:
The answer is twofold:

Integer operations are implemented taking into account that int8 number refer to different domain. Convolution (or matrix-matrix multiplication in general) is implemented with respect to this fact and my answer here I want to use Numpy to simulate the inference process of a quantized MobileNet V2 network, but the outcome is different with pytorch realized one worked for me.
Addition in pytorch is implemented in floats. You need to convert from int to float, make an addition and then convert back to int.

def manual_addition(xq1_int, scale1, zp1, xq2_int, scale2, zp2,
scale_r, zp_r):

  xdq = scale1 * (xq1_int.astype(np.float) - zp1)
  ydq = scale2 * (xq2_int.astype(np.float) - zp2)
  zdq = xdq + ydq
  zq_manual_int = (((zdq / scale_r).round()) + zp_r).round()
  return zq_manual_int #clipping might be needed


Instruction: Tensor matrix multiplication returning vector einsum

Input:
I am confused about the following example of a matrix tensor multiplication that returns a vector. At first glance I thought that it would mean multiplying the first dimension of the tensor dydx by the matrix dLdy but I don't get the expected results as depicted below. So what is the meaning of this einsum ?
import torch
import numpy as np

dLdy = torch.randn(2,2)
dydx = torch.randn(2,2,2)

torch.einsum('jk,jki->i', dLdy, dydx)
tensor([0.3115, 3.7255])

dLdy
tensor([[-0.4845,  0.6838],
    [-1.1723,  1.4914]])

dydx
tensor([[[ 1.5496, -1.2722],
     [ 0.1221,  1.0495]],

    [[-1.4882,  0.0307],
     [-0.5134,  1.6276]]])

(dLdy * dydx[0]).sum()
-0.1985

Output:
For A and B this is contraction (sum) over the first two dimensions jk, so
res(i) = sum_{j,k} A(j,k)B(j,k,i)
for example:
import torch
import numpy as np

dLdy = torch.randn(2,2)
dydx = torch.randn(2,2,2)

print(torch.einsum('jk,jki->i', dLdy, dydx))
print((dLdy * dydx[:,:,0]).sum())
print((dLdy * dydx[:,:,1]).sum())

produces
tensor([4.6025, 1.8987])
tensor(4.6025)
tensor(1.8987)

ie (dLdy * dydx[:,:,0]).sum() is the first element of the resulting vector, etc


Instruction: Why are the parameters of this PyTorch AutoEncoder hardcoded this way?

Input:
Hi I am trying to understand how the following PyTorch AutoEncoder code works.  The code below uses the MNIST dataset which is 28X28.  My question is how the nn.Linear(128,3) parameters where chosen?
I have a dataset which is 512X512 and I would like to modify the code for this AutoEncoder to support.
class LitAutoEncoder(pl.LightningModule):

def __init__(self):
    super().__init__()
    self.encoder = nn.Sequential(nn.Linear(28 * 28, 128), nn.ReLU(), nn.Linear(128, 3))
    self.decoder = nn.Sequential(nn.Linear(3, 128), nn.ReLU(), nn.Linear(128, 28 * 28))

def forward(self, x):
    # in lightning, forward defines the prediction/inference actions
    embedding = self.encoder(x)
    return embedding

def training_step(self, batch, batch_idx):
    # training_step defined the train loop. It is independent of forward
    x, y = batch
    x = x.view(x.size(0), -1)
    z = self.encoder(x)
    x_hat = self.decoder(z)
    loss = F.mse_loss(x_hat, x)
    return loss

def configure_optimizers(self):
    optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
    return optimizer

Output:
I am assuming input image data are in this shape: x.shape == [bs, 1, h, w], where bs is batch size. Then, x is first viewed as [bs, h*w], i.e. [bs, 28*28]. This means all pixels in an image are flattened into a 1D vector.
Then in the encoder:

nn.Linear(28*28, 128) takes flattened input of size [bs, 28*28] and outputs intermediate result of size [bs, 128]
nn.Linear(128, 3): [bs, 128] -> [bs, 3]

Then in the decoder:

nn.Linear(3, 128): [bs, 3] -> [bs, 128]
nn.Linear(128, 28*28): [bs, 128] -> [bs, 28*28]

The final output is then matched against the input.
If you want to use the exact architecture for your 512x512 images, simply change every occurrence of 28*28 in the code to 512*512. However, this is a quite infeasible choice, for these reasons:

For MNIST images, nn.Linear(28*28, 128) contains 28x28x128+128=100480 parameters, while for your images nn.Linear(512*512, 128) contains 512x512x128+128=33554560 parameters. The size is too large, and it may lead to overfitting
The intermediate data [bs, 3] uses only 3 floats to encode a 512x512 image. I don't think you can recover anything with such compression

I'd suggest looking up convolutional architectures for you purpose


Instruction: Saving Pytorch model error: codec can't decode bytes in position 2-3

Input:
I have seen similar questions regarding the loading of a pytorch model, but not the saving of one and the solutions offered on those questions have not solved my problem.
Here is the code I have to save the model:
PATH = "c:\Users\my_name\Desktop\model"
torch.save(model, PATH)

But I am stuck getting the title error. I am saving the model without a checkpoint, before evaluating. Not sure what is going wrong here.

Output: If the path is just my c: drive, for whatever reason the error is fixed. Still unsure about why I cant select a deeper path.


Instruction: how to make small chunks of python dictionary

Input:
hi i have python sub dictionary  like this
import torch
import torch.nn as nn
dic = {
  "A": [0.2822, -0.0958, -0.5165, -0.3812,
        -0.3469,  0.4025, -0.0696, -0.1246,
        -0.1132,  0.4170, -0.0383, -0.4071,
        -0.5407,  0.1519,  0.5630,  0.1276],
  "B": [1.0014, 0.9980, 1.0012, 0.9986,
       1.0001, 0.9999, 1.0016, 1.0014,
       1.0008, 0.9996, 1.0008, 1.0004,
       1.0000, 0.9987, 0.9997, 0.9989]
}

for key 1 i have 16 values, i want to make 2 chunks of size 8 for key A.  how it can be possible?
Like first 8 values stored in separate array and last 8 values stoed ins eparate array or dictionary with their key values.
as shown in this image

Output:
First of all, I'd refer to keys as what they are (i.e. Key 'A' has 16 values, not key 1). It's a good practice to just think of dictionaries as unordered and simply a group of key-value pairs.
Second, using numpy will allow us to split the key we want into two (or more) even groups. If you end up needing to split a list of 30 elements into three lists, this code will still work.
import numpy as np

dic = {
  "A": [0.2822, -0.0958, -0.5165, -0.3812,
        -0.3469,  0.4025, -0.0696, -0.1246,
        -0.1132,  0.4170, -0.0383, -0.4071,
        -0.5407,  0.1519,  0.5630,  0.1276],
  "B": [1.0014, 0.9980, 1.0012, 0.9986,
       1.0001, 0.9999, 1.0016, 1.0014,
       1.0008, 0.9996, 1.0008, 1.0004,
       1.0000, 0.9987, 0.9997, 0.9989]
}

# We give array_split() our list, and how many we want it split into.
a1, a2 = np.array_split(dic['A'], 2) # We get our two lists returned.
dic['A1'] = a1.tolist() # Numpy returns it as an np.array, so let's put it back into a list.
dic['A2'] = a2.tolist()
del(dic['A']) # Remove the now unused key-value.

{'B': [1.0014, 0.998, 1.0012, 0.9986, 1.0001, 0.9999, 1.0016, 1.0014, 1.0008, 0.9996, 1.0008, 1.0004, 1.0, 0.9987, 0.9997, 0.9989],
 'A1': [0.2822, -0.0958, -0.5165, -0.3812, -0.3469, 0.4025, -0.0696, -0.1246],
 'A2': [-0.1132, 0.417, -0.0383, -0.4071, -0.5407, 0.1519, 0.563, 0.1276]}


Instruction: index a list of tensors

Input:
I have a list object named " imgs " of tensors (50 images).
I have an array of indices (indi) of length 29.
how do I index the list of tensors with the array of indices?
when I do the following I get:
imgs[indi]

TypeError: only integer scalar arrays can be converted to a scalar index

Thanks

Output:
Assuming these are normal python lists then you can use a list comprehension
result = [imgs[i] for i in indi]

which will give a list of tensors.
If you further want to make this a single tensor containing the images you can use torch.stack
result = torch.stack([imgs[i] for i in indi], dim=0)


Instruction: Unable to load custom pretrained weight in Pytorch Lightning

Input:
I want to retrain a custom model with my small dataset. I can load the pretrained weight (.pth) and run it in Pytorch. However, I need more functionalities and refactored the code to Pytorch lightning but I can't figure out how to load the pretrained weight into the Pytorch Lightning model.
Please see the details of my code below:
class BDRAR(nn.Module):
    def __init__(self):
        super(BDRAR, self).__init__()
        resnext = ResNeXt101()
        self.layer0 = resnext.layer0
        self.layer1 = resnext.layer1
        self.layer2 = resnext.layer2
        self.layer3 = resnext.layer3
        self.layer4 = resnext.layer4

Pytorch Lightning code:
class liteBDRAR(pl.LightningModule):
    def __init__(self):
        super(liteBDRAR, self).__init__()
        self.model = BDRAR()
        print('Model Created!')

    def forward(self, x):
        return self.model(x)

Pytorch Lightning run:
    path = './ckpt/BDRAR/3000.pth'
    bdrar = liteBDRAR.load_from_checkpoint(path,  strict=False)
    trainer = pl.Trainer(fast_dev_run=True, gpus=1)
    trainer.fit(bdrar)

Error:
keys = model.load_state_dict(checkpoint["state_dict"], strict=strict)
**KeyError: 'state_dict'**

I will appreciate any help.
Thank you.

Output:
It can be that your .pth file is already a state_dict. Try to load pretrained weight in your lightning class.
class liteBDRAR(pl.LightningModule):
    def __init__(self):
        super(liteBDRAR, self).__init__()
        self.model = BDRAR()
        print('Model Created!')

    def load_model(self, path):
        self.model.load_state_dict(torch.load(path, map_location='cuda:0'), strict=False)

path = './ckpt/BDRAR/3000.pth'
model = liteBDRAR()
model.load_model(path)


Instruction: Understanding an example of PyTorch's Einsum function

Input:
I am studying some code and I came across a usage of PyTorch's einsum function that I am not understanding. The docs are here.
The snippet looks like (slightly modified from the original):
import torch
x = torch.rand(64, 64, 25, 25)
y = torch.rand(64, 64, 64, 25)
result = torch.einsum('ncuv,nctv->nctu', x, y)
print(result.shape)
>> torch.Size([64, 64, 64, 25])

So the notation is such that n=64, c=64, u=25, v=25, t=64.
I'm not too sure what's happening. I think that for each 25 dimensional vector in t (64 of them), each one is being multiplied with each of the u=25 vectors of size 25 elementwise and then the results summed, or rather 25 dot products of 25 dimensional vectors?
Any insights appreciated.

Output:
Basically, you can think of it as taking dot products over certain dimensions, and reorganizing the rest.
For simplicity, let's ignore the batching dimensions n and c (since they are consistent before and after ncuv,nctv->nctu), and discuss:
import torch
x = torch.rand(25, 25)
y = torch.rand(64, 25)
result = torch.einsum('uv,tv->tu', x, y)
print(result.shape)
>> torch.Size([64, 25])

Note that v vanishes after einsum, meaning v is the dimension being summed up, while t and u are not. You can interpret it this way: x is a  collection of 25 25-dimensional vectors; y is a collection of 64 25-dimensional vectors. The dot product of the t-th vector in y and the u-th vector in x are computed and put in the t-th row and u-th column of result.
You can also rewrite into a math equation:
result[n,c,t,u] = \sum_{v} x[n,c,u,v] * y[n,c,t,v], for each n, c, t, u

Note two things:

the summation is over the indices that vanish in the summation pattern nctu,ncuv->nctv
indices appearing on the right of the pattern are the indices of the resulting tensor


Instruction: Can the increase in training loss lead to better accuracy?

Input:
I'm working on a competition on Kaggle. First, I trained a Longformer base with the competition dataset and achieved a quite good result on the leaderboard. Due to the CUDA memory limit and time limit, I could only train 2 epochs with a batch size of 1. The loss started at about 2.5 and gradually decreased to 0.6 at the end of my training.
I then continued training 2 more epochs using that saved weights. This time I used a little bit larger learning rate (the one on the Longformer paper) and added the validation data to the training data (meaning I no longer split the dataset 90/10). I did this to try to achieve a better result.
However, this time the loss started at about 0.4 and constantly increased to 1.6 at about half of the first epoch. I stopped because I didn't want to waste computational resources.
Should I have waited more? Could it eventually lead to a better test result? I think the model could have been slightly overfitting at first.

Output:
Your model got fitted to the original training data the first time you trained it. When you added the validation data to the training set the second time around, the distribution of your training data must have changed significantly. Thus, the loss increased in your second training session since your model was unfamiliar with this new distribution.
Should you have waited more? Yes, the loss would have eventually decreased (although not necessarily to a value lower than the original training loss)
Could it have led to a better test result? Probably. It depends on if your validation data contains patterns that are:

Not present in your training data already
Similar to those that your model will encounter in deployment


Instruction: ValueError: Unsupported ONNX opset version: 13

Input:
Goal: successfully run Notebook as is on Jupyter Labs.
Section 2.1 throws a ValueError, I believe because of the version of PyTorch I'm using.

PyTorch 1.7.1
Kernel conda_pytorch_latest_p36

Very similar SO post; the solution was to use the latest PyTorch version... which I am using.

Code:
import onnxruntime

def export_onnx_model(args, model, tokenizer, onnx_model_path):
    with torch.no_grad():
        inputs = {'input_ids':      torch.ones(1,128, dtype=torch.int64),
                    'attention_mask': torch.ones(1,128, dtype=torch.int64),
                    'token_type_ids': torch.ones(1,128, dtype=torch.int64)}
        outputs = model(**inputs)

        symbolic_names = {0: 'batch_size', 1: 'max_seq_len'}
        torch.onnx.export(model,                                            # model being run
                    (inputs['input_ids'],                             # model input (or a tuple for multiple inputs)
                    inputs['attention_mask'],
                    inputs['token_type_ids']),                                         # model input (or a tuple for multiple inputs)
                    onnx_model_path,                                # where to save the model (can be a file or file-like object)
                    opset_version=13,                                 # the ONNX version to export the model to
                    do_constant_folding=True,
                    input_names=['input_ids',                         # the model's input names
                                'input_mask',
                                'segment_ids'],
                    output_names=['output'],                    # the model's output names
                    dynamic_axes={'input_ids': symbolic_names,        # variable length axes
                                'input_mask' : symbolic_names,
                                'segment_ids' : symbolic_names})
        logger.info("ONNX Model exported to {0}".format(onnx_model_path))

export_onnx_model(configs, model, tokenizer, "bert.onnx")

Traceback:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-7aaa4c5455a0> in <module>
     25         logger.info("ONNX Model exported to {0}".format(onnx_model_path))
     26
---> 27 export_onnx_model(configs, model, tokenizer, "bert.onnx")

<ipython-input-7-7aaa4c5455a0> in export_onnx_model(args, model, tokenizer, onnx_model_path)
     22                     dynamic_axes={'input_ids': symbolic_names,        # variable length axes
     23                                 'input_mask' : symbolic_names,
---> 24                                 'segment_ids' : symbolic_names})
     25         logger.info("ONNX Model exported to {0}".format(onnx_model_path))
     26

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/onnx/__init__.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
    228                         do_constant_folding, example_outputs,
    229                         strip_doc_string, dynamic_axes, keep_initializers_as_inputs,
--> 230                         custom_opsets, enable_onnx_checker, use_external_data_format)
    231
    232

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
     89             dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs,
     90             custom_opsets=custom_opsets, enable_onnx_checker=enable_onnx_checker,
---> 91             use_external_data_format=use_external_data_format)
     92
     93

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, _retain_param_name, do_constant_folding, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, enable_onnx_checker, use_external_data_format, onnx_shape_inference, use_new_jit_passes)
    614         # training=TrainingMode.TRAINING or training=TrainingMode.PRESERVE,
    615         # (to preserve whatever the original training mode was.)
--> 616         _set_opset_version(opset_version)
    617         _set_operator_export_type(operator_export_type)
    618         with select_model_mode_for_export(model, training):

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py in _set_opset_version(opset_version)
    506         _export_onnx_opset_version = opset_version
    507         return
--> 508     raise ValueError("Unsupported ONNX opset version: " + str(opset_version))
    509
    510 _operator_export_type = None

ValueError: Unsupported ONNX opset version: 13

Please let me know if there's anything else I can add to post.

Output:
ValueError: Unsupported ONNX opset version N -> install latest PyTorch.
Credit to Tianleiwu on this Git Issue.
As per 1st cell of Notebook:
# Install or upgrade PyTorch 1.8.0 and OnnxRuntime 1.7.0 for CPU-only.

I inserted a new cell right after:
pip install torch==1.10.0  # latest


Instruction: Why are the gradients not equivalent when using loss.backward() v.s torch.auto.grad?

Input:
I ran into this weird behavior when trying to "manually" optimize a network's parameters via SGD. When attempting to update the model's parameters using the following way, it works just fine:
for _ in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m(x), y)
        grad = torch.autograd.grad(loss, m.parameters())
        with torch.no_grad():
            for p, g in zip(m.parameters(), grad):
                p -= 0.1 * g

However, doing the following throws off the model completely:
for _ in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m(x), y)
        loss.backward()
        with torch.no_grad():
            for p in m.parameters():
                p -= 0.1 * p.grad

But to me, both methods should be equivalent. And upon further inspection, when comparing the values of g from grad with the values of p.grad from m.paramters(), it turned out that the gradient values are not the same! I also tried removing with torch.no_grad(): and doing the following, but it didn't work either:
        for p in m.parameters():
            p.data -= 0.1 * p.grad

Can somebody please explain why is this happening? Shouldn't the gradients in both methods have the same values (keeping in mind that both models m are identical)?
REPRODUCIBLE EXAMPLE:
Ensure reproducibility:
device = torch.device('cuda')
torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
np.random.seed(0)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.cuda.empty_cache()

Load the data:
T = transforms.ToTensor()
train_data = datasets.MNIST(root='data', transform=T, download=True)
test_data = datasets.MNIST(root='data', transform=T, train=False, download=True)

BS = 300
epochs = 5
LR = 0.1

train_loader = DataLoader(train_data, batch_size=BS, pin_memory=True)
test_loader = DataLoader(test_data, batch_size=1000, pin_memory=True)

Define the model to be optimized:
class Model(nn.Module):
    def __init__(self, out_dims):
        super().__init__()
        self.conv1 = nn.Conv2d(1, out_dims, 3, stride=3, padding=1)
        self.conv2 = nn.Sequential(nn.Conv2d(out_dims, out_dims * 2, 3), nn.BatchNorm2d(out_dims * 2), nn.ReLU())
        self.conv3 = nn.Sequential(nn.Conv2d(out_dims * 2, out_dims * 4, 4, stride=2, padding=1), nn.BatchNorm2d(out_dims * 4), nn.ReLU(), nn.Flatten())
        self.fc = nn.Linear(out_dims * 4 * 16, 10)

    def forward(self, x):
        return nn.Sequential(*tuple(self.children()))(x)

m1 = Model(5).to(device)
m2 = deepcopy(m1)  # "m2.load_state_dict(m1.state_dict())" doesn't work either

Training and evaluation:
# M1's training:
for _ in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m1(x), y)
        grad = torch.autograd.grad(loss, m1.parameters())
        with torch.no_grad():
            for p, g in zip(m1.parameters(), grad):
                p -= LR * g

# M1's evaluation:
m1.eval()
acc1 = []
with torch.no_grad():
    for x, y in test_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        _, pred = m1(x).max(1)
        acc1.append(metric(pred, y).item())

print(f'Accuracy: {np.mean(acc1) * 100:.4}%')

# M2's training:
for _ in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m2(x), y)
        loss.backward()
        with torch.no_grad():
            for p in m2.parameters():
                p -= LR * p.grad

# M2's evaluation:
m2.eval()
acc2 = []
with torch.no_grad():
    for x, y in test_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        _, pred = m2(x).max(1)
        acc2.append(metric(pred, y).item())

print(f'Accuracy: {np.mean(acc2) * 100:.4}%')

Output: It took me a while to figure out, but the problem was in loss.backward(). Unlike autograd.grad() which computes and returns the gradients, the inplace backward() computes and accumulates the gradients of participating nodes in the computation graph. In other words, the two will have the same effect when used to back-prop once, but every repetition of backward() will add the currently computed gradients to all previous ones (hence the divergence). Resetting the gradients using model.zero_grad() fixes stuff.


Instruction: Torch Tensor with same shape but different storage size

Input:
I'm working on GANs model, the generator creates a tensor with size (3,128,128) which I dumped with the pseudo-code
import torch
image = Generator(noise).clone()
tensor = image[0].detach().cpu()
torch.save(tensor, save_path)

The problem is that tensor costs more storage than the same tensor size, even they have the same shape and dtype
>>> import sys
>>> import torch
>>> tensor = tensor.load(save_path)
>>> rand_t = torch.randn(tensor.shape)
>>> print(tensor.shape, rand_t.shape)
torch.Size([3, 128, 128]) torch.Size([3, 128, 128])
>>> print(tensor.dtype, rand_t.dtype)
torch.float32 torch.float32
>>> print(sys.getsizeof(tensor.storage()))
9830472
>>> print(sys.getsizeof(rand_t.storage()))
196680

I tried to dump those tensors, the output from the generator took 9.2MB and the random tensor took 197.4kB. I did read the pytorch's documents but found nothing. Please help me to figure out what is the difference between them?

Output:
It seems like extracting a sub-tensor directly from the original will bring the whole container with it. The function .clone() can solve it. Example:
>>> import sys
>>> import torch
>>> tensor = torch.randn(10,3,128,128)
>>> sys.getsizeof(tensor.storage())
1966144
>>> sub1 = tensor[0]
>>> sub1.shape
torch.Size([3, 128, 128])
>>> sys.getsizeof(sub1.storage())
1966144
>>> sub2 = tensor[0].clone()
>>> sub2.shape
torch.Size([3, 128, 128])
>>> sys.getsizeof(sub2.storage())
196672

Therefore, in my case, cloning the image individually should solve the problem:
import torch
image = Generator(noise).clone()
tensor = image[0].detach().clone().cpu() # using clone()
torch.save(tensor, save_path)


Instruction: Substitute values in a vector (PyTorch)

Input:
I have a tensor of N unique target labels, randomly selected from [0,R], where N<R (i.e., my target vector can have any length, but only contains N unique labels.). I would like to transform the labels to [0,N]. Is there a function available for this target transform? e.g. input vector: [12, 6, 4, 5, 3, 12, 4] → transformed vector : [4, 3, 1, 2, 0, 4, 1]
My attempt:
I have implemented the following snippet, which works as expected, but might not be the most glorious implementation:
import torch

def my_transform(vec):
    t_ = torch.unique(vec)
    return torch.cat(list(map(lambda x: (t_ == x).nonzero(as_tuple=True)[0], vec)))

t = torch.Tensor([12, 6, 4, 5, 3, 12, 4])
print(my_transform(t))

Output:
You're looking for searchsorted
import torch

t = torch.Tensor([12, 6, 4, 5, 3, 12, 4])
transformed = torch.searchsorted(t.unique(),t)
# tensor([4, 3, 1, 2, 0, 4, 1])


Instruction: Create an instance of AdamParamState

Input:
I need to create an instance of AdamParamState. I looked through the adam.cpp code as an example, and accordingly copied the following code from there. But, with the provided headers, it still does not recognize AdamParamState.
I appreciate any help or comment on this matter.
#include <torch/optim/adam.h>
#include <torch/csrc/autograd/variable.h>
#include <torch/nn/module.h>
#include <torch/serialize/archive.h>
#include <torch/utils.h>

#include <ATen/ATen.h>

void get_state(torch::optim::Optimizer *optimizer){
    for (auto& group : optimizer->param_groups()) {
        for (auto &p : group.params()) {
            if (!p.grad().defined()) {
                continue;
            }

            auto grad = p.grad();
            TORCH_CHECK(!grad.is_sparse(),
                        "Adam does not support sparse gradients"/*, please consider SparseAdam instead*/);
            ska::flat_hash_map<std::string, std::unique_ptr<torch::optim::OptimizerParamState>>& state_ = optimizer->state();
            auto param_state = state_.find(c10::guts::to_string(p.unsafeGetTensorImpl()));
            auto tmp_ = p.dim();
            int tmp_0;
            int tmp_1;
            if (tmp_ > 0)
                tmp_0  = p.size(0);
            if (tmp_ > 1)
                tmp_1  = p.size(1);

            std::cout << tmp_ << tmp_0 << tmp_1 << std::endl;
//                auto& options = static_cast<AdamOptions&>(group.options());
            auto& state = static_cast<AdamParamState&>(*state_[c10::guts::to_string(p.unsafeGetTensorImpl())]);
        }
    }

}

Output:
I found that this works:
auto& state = static_cast<torch::optim::AdamParamState&>(*state_[c10::guts::to_string(p.unsafeGetTensorImpl())]);
very simple and juicy!


Instruction: Torch backward do not return a tensor

Input:
To set up the problem I have as input a matrix X and as output a matrix Y, we obtain Y by a matrix multiplication with W that I pass through an exponential. From what I have understood from torch.backward() with a gradient parameter the formula should be the following

Yet dy_over_dx as a jacobian should be a tensor of sort of size  (I mean not the usual n by n dimensional matrix).
X = torch.tensor( [[2.,1.,-3], [-3,4,2]], requires_grad=True)
W = torch.tensor( [ [3.,2.,1.,-1] , [2,1,3,2 ] , [3,2,1,-2] ], requires_grad=True)

Y = torch.exp(torch.matmul(X, W))
Y.retain_grad()
print(Y)

dL_over_dy = torch.tensor(  [[2,3,-3,9],[-8,1,4,6]])
print(dL_over_dy, dL_over_dy.shape)

Y.backward(dL_over_dy)
print(X.grad)
tensor([[3.6788e-01, 3.6788e-01, 7.3891e+00, 4.0343e+02],
        [1.4841e+02, 7.3891e+00, 5.9874e+04, 1.0966e+03]],
       grad_fn=<ExpBackward>)
tensor([[ 2,  3, -3,  9],
        [-8,  1,  4,  6]]) torch.Size([2, 4])
tensor([[ -3648.6118,   7197.7920,  -7279.4707],
        [229369.6250, 729282.0625, 222789.8281]])

Next if I look at the gradient of Y which I suppose is dy_over_dx I obtain, what do I do not understand here ?
print(Y.grad)
tensor([[ 2.,  3., -3.,  9.],
        [-8.,  1.,  4.,  6.]])

Output:
What you're looking at here is Y.grad, which is dL/dY i.e. none other than dL_over_dy.
To help clarify, let Z = X @ Y (@ is equivalent to matmul), and Y = exp(Z). Then we have with the chain-rule:

Y.grad = dL/dY

Z.grad = dL/dZ = dL/dY . dY/dZ, where dY/dZ = exp(Z) = Y

X.grad = dL/dX = dL/dZ . dZ/dX, where dZ/dX = d(X@W)/dX = W.T

Here is the implementation:
X = torch.tensor([[ 2., 1., -3],
                  [ -3, 4., 2.]], requires_grad=True)

W = torch.tensor([[ 3., 2., 1., -1],
                  [ 2., 1., 3., 2.],
                  [ 3., 2., 1., -2]], requires_grad=True)

Z = torch.matmul(X, W)
Z.retain_grad()
Y = torch.exp(Z)

dL_over_dy = torch.tensor([[ 2., 3., -3, 9.],
                           [ -8, 1., 4., 6.]])

Y.backward(dL_over_dy)

Then we have
>>> dL_over_Z = dL_over_dy*Y
tensor([[ 7.3576e-01,  1.1036e+00, -2.2167e+01,  3.6309e+03],
        [-1.1873e+03,  7.3891e+00,  2.3950e+05,  6.5798e+03]],
       grad_fn=<MulBackward0>)

>>> dL_over_X = dL_over_Z @ W.T
tensor([[ -3648.6118,   7197.7920,  -7279.4707],
        [229369.6250, 729282.0625, 222789.8281]], grad_fn=<MmBackward0>)