Instruction: Libtorch C++: Efficient/correct way for saving/loading Model and Optimizer State Dict for retraining

Input:
I am looking for the correct and most efficient way of saving, loading, and retraining a model in Libtorch (C++) with both the model and optimizer state dict. I believe I have everything correctly set (however this may not be right for saving and loading optimizer state dicts, only the model state dict I am absolutely sure of), my last question is where I set the Optimizer and give it the model parameters.
Saving Model and Optimizer:
// Save model state
torch::serialize::OutputArchive output_model_archive;
myModel.to(torch::kCPU);
myModel.save(output_model_archive);
output_model_archive.save_to(model_state_dict_path);
// Save optim state
torch::serialize::OutputArchive output_optim_archive;
myOptimizer->save(output_optim_archive);
output_optim_archive.save_to(optim_state_dict_path);

Loading model and optim state for retraining.
// Load model state
torch::serialize::InputArchive input_archive;
input_archive.load_from(state_dict);
myModel.load(input_archive);
// Load optim state
torch::serialize::InputArchive input_archive;
input_archive.load_from(state_dict);
myOptimizer->load(input_archive);

When creating the optimizer object, you need to give it the model parameters:
std::shared_ptr<torch::optim::Optimizer> myOptimizer;
myOptimizer.reset(new torch::optim::Adam(myModel.parameters(), torch::optim::AdamOptions(LR)));

Should this be done before the state dicts are loaded, after, or does it matter? For example, I am doing it like:
// Setup model and optimizer object, set model params in optimizer
// Load state dictionaries...
// Train epoch #n...
myOptimizer->step();
// Save state dictionaries

Output:
To answer my own question, the model state dict needs to be loaded and then parameters put into the optimizer object. Then load the state dict into the optimizer object.
My use case was a little more complicated as I was aggregating gradients from multiple nodes where training was happening and doing an optimizer step on a "master" node. I was trying to simplify the problem above for the question, and I assumed I did not need the previous state dict since I was aggregating gradients. That was an incorrect assumption. The flow looks like:
// Load model state dict
// Aggregate gradients
// Load Optimizer state dict / params into optim
// Step


Instruction: Repetitive word predictions in RNN

Input:
Hello dear community,
I am training a Seq2Seq model to generate a question based on a graph. Both train and val loss are converging, but the generated questions (on either train or test set) are nonsense and contain mostly repetition of tokens. I tried various hyper parameters and double checked input and outputs tensors.
Something that I do find odd is that the output out (see below) starts containing some values, which I consider as unusually high. This starts happening around half way through the first epoch:
Out:  tensor([[  0.2016, 103.7198,  90.4739,  ...,   0.9419,   0.4810,  -0.2869]]

My guess for that is vanishing/exploding gradients, which I thought I had handeled by gradient clipping, but now I am not sure about this:
for p in model_params:
        p.register_hook(lambda grad: torch.clamp(
            grad, -clip_value, clip_value))

Below are the training curves (10K samples, batch size=128, lr=0.065, lr_decay=0.99, dropout=0.25)

Encoder (a GNN, learning node embeddings of the input graph, that consists of around 3-4 nodes and edges. A single graph embedding is obtained by pooling the node embeddings and feeding them as the initial hidden state to the Decoder):
class QuestionGraphGNN(torch.nn.Module):
    def __init__(self,
                 in_channels,
                 hidden_channels,
                 out_channels,
                 dropout,
                 aggr='mean'):
        super(QuestionGraphGNN, self).__init__()
        nn1 = torch.nn.Sequential(
            torch.nn.Linear(in_channels, hidden_channels),
            torch.nn.ReLU(),
            torch.nn.Linear(hidden_channels, in_channels * hidden_channels))
        self.conv = NNConv(in_channels, hidden_channels, nn1, aggr=aggr)
        self.lin = nn.Linear(hidden_channels, out_channels)
        self.dropout = dropout

    def forward(self, x, edge_index, edge_attr):
        x = self.conv(x, edge_index, edge_attr)
        x = F.leaky_relu(x)
        x = F.dropout(x, p=self.dropout)
        x = self.lin(x)
        return x

Decoder (The out vector from above is printed in the forward() function):
class DecoderRNN(nn.Module):
    def __init__(self,
                 embedding_size,
                 output_size,
                 dropout):
        super(DecoderRNN, self).__init__()
        self.output_size = output_size
        self.dropout = dropout

        self.embedding = nn.Embedding(output_size, embedding_size)
        self.gru1 = nn.GRU(embedding_size, embedding_size)
        self.gru2 = nn.GRU(embedding_size, embedding_size)
        self.gru3 = nn.GRU(embedding_size, embedding_size)
        self.out = nn.Linear(embedding_size, output_size)
        self.logsoftmax = nn.LogSoftmax(dim=1)

    def forward(self, inp, hidden):
        output = self.embedding(inp).view(1, 1, -1)
        output = F.leaky_relu(output)

        output = F.dropout(output, p=self.dropout)
        output, hidden = self.gru1(output, hidden)

        output = F.dropout(output, p=self.dropout)
        output, hidden = self.gru2(output, hidden)
        output, hidden = self.gru3(output, hidden)

        out = self.out(output[0])
        print("Out: ", out)
        output = self.logsoftmax(out)
        return output, hidden

I am using PyTorchs NLLLoss().
Optimizer is SGD.
I call optimizer.zero_grad() right before the backward and optimizer step and I switch the training/evaluation mode for training, evaluation and testing.
What are your thoughts on this?
Thank you very much!
EDIT
Dimensions of the Encoder:
in_channels=301 (This is the size of the initial node embeddings)
hidden_channels=256
out_channels=301 (This will also be the size of the final graph embedding, after mean pooling the node embeddings)
Dimensions of the Decoder:
embedding_size=301 (the size of the previously pooled graph embedding)
output_size=number of words in my vocabulary. In the training above around 1.2K
I am using top-k sampling and my train loop follows the NMT Tutorial https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model). Similarily, my translation function, that takes the data of a single graph, decodes a question as such:
def translate(self, data):
    # Get node embeddings of the input graph
    h = self.encoder(data.node_embeddings,
                     data.edge_index, data.edge_embeddings)

    # Pool node embeddings into single graph embedding
    graph_embedding = self.get_graph_embeddings(h, data.graph_dict)

    # Pass graph embedding through decoder
    self.encoder.eval()
    self.decoder.eval()
    with torch.no_grad():
        # Initialize first input and hidden state
        decoder_input = decoder_input = torch.tensor(
            [[self.vocab.SOS['idx']]], device=self.device)
        decoder_hidden = graph_embedding.view(1, 1, -1)

        decoder_tokens = []
        for di in range(self.dec_max_length):
            decoder_output, decoder_hidden = self.decoder(
                decoder_input, decoder_hidden)
            topv, topi = decoder_output.data.topk(1)
            if topi.item() == self.vocab.EOS['idx']:
                break
            else:
                word = self.vocab.index2word[topi.item()]
                word = word.upper(
                ) if word == self.vocab.UNK['token'].lower() else word
                decoder_tokens.append(word)
            decoder_input = topi.squeeze().detach()

        return decoder_tokens

Also: At times, the output-vector of the final gru layer (self.gru3(...)) inside the forward() function (5th line from the bottom) outputs a lot of values being (close to) 1 and -1. I suppose these might otherwise be a lot higher/lower without clipping. This might be alright, but seems unusual to me. An example:
tensor([[[-0.9984, -0.9950,  1.0000, -0.9889, -1.0000, -0.9770, -0.0299,
          -0.9996,  0.9996,  1.0000, -0.0176, -0.5815, -0.9998, -0.0265,
          -0.1471,  0.9998, -1.0000, -0.2356,  0.9964,  0.9936, -0.9998,
           0.0652, -0.9999,  0.9999, -1.0000, -0.9998, -0.9999,  0.9998,
          -1.0000, -0.9997,  0.9850,  0.9994, -0.9998, -1.0000, -1.0000,
           0.9977,  0.9015, -0.9982,  1.0000,  0.9980, -1.0000,  0.9859,
           0.6670,  0.9998,  0.3827,  0.9999,  0.9953, -0.9989,  0.1287,
           1.0000,  1.0000, -1.0000,  0.9778,  1.0000,  1.0000, -0.9907, ...

Output:
Your code looks good, and given the training/validation curves you posted, it looks like it's doing alright.
How are you generating text samples? Are you just taking the word the model predicts with the highest probability, appending to the end of your input sequence, and calling forward again? This sampling technique, called greedy sampling, can lead to behavior you described. Maybe another sampling technique could help (see beam search https://medium.com/geekculture/beam-search-decoding-for-text-generation-in-python-9184699f0120)?


Instruction: How to record the Batch Normalization layers under the ONNX format?

Input:
I need to print the structure of a neural network from a Pytorch code. I resorted to use ONNX format: I used the torch.onnx.export function, but what happens is that the Batch Normalization layers are not recorded, since they are included in the convolutional layers (see here and here, for example). After some search, I found this Q&A on StackOverflow, which seems to provide a solution to the above problem, by adding the option training=TrainingMode.TRAINING. Unfortunately, as noted also by others on the web, this option seems to not work, as such option does not seem to be recognized.
---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

<ipython-input-208-8820886c111a> in <module>
     10                   input_names = ['input'],
     11                   output_names = ['output'],
---> 12                   training=TrainingMode.TRAINING)
     13
     14

NameError: name 'TrainingMode' is not defined

Below I provide an example code for showing the problem, I am currently working on Colab.
import torch
import torchvision
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")

class NeuralNetwork(nn.Module):
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.Network = nn.Sequential(
        nn.Conv2d(1,1,3),
        nn.BatchNorm2d(1,track_running_stats=False),
        nn.ReLU(),
    )

  def forward(self,x):
    output = self.Network(x)
    return output

model = NeuralNetwork().to(device)

print(model)

import onnx

net=NeuralNetwork()
torch.onnx.export(net, dummy_input, "test.onnx",
                  verbose=True,
                  export_params=True,
                  opset_version=12,
                  do_constant_folding=True,
                  input_names = ['input'],
                  output_names = ['output'],
                  training=TrainingMode.TRAINING)

Output:
You need to import definition of TrainingMode
with import:
from torch.onnx import TrainingMode

without import:
torch.onnx.TrainingMode.TRAINING

After that you can successfully export onnx model (visualized with netron):


Instruction: GPU Usage While Training LSTM Model Using Darts

Input:
I am currently using the Darts library to do some time series forecasting using an LSTM.
In my arguments I set the model to utilize my GPU and the output from the dashboard during training shows that my GPU is indeed being used.
Looking at the documentation as well confirms that what I am seeing means the GPU is being used for training. https://unit8co.github.io/darts/userguide/gpu_and_tpu_usage.html.
However, looking at my GPU in task manager it says that my GPU usage is at 0%. That does not make any sense to me? Does anyone know what goes on behind the scenes and might be able to explain why the GPU usage would be at 0% during training?

Output: It might be that your GPU is under-used, e.g. because your CPU is not feeding it fast enough. Try playing with the num_loader_workers parameter, increasing the batch size, and reading the recommendations provided here.


Instruction: Simple RNN Error "Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu" How to?

Input:
I'm working on a basic RNN-NLP classifier using PyTorch, and trying to use CUDA for acceleration.(On Google_Colab)
but, I can't solve this error.
The code is written like this.
error message
Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu
RNN class
class RNN(nn.Module):
  def __init__(self, vocab_size, emb_size, hidden_size, output_size):
    super().__init__()
    self.hidden_size = hidden_size
    self.emb = nn.Embedding(vocab_size, emb_size)
    self.rnn = nn.RNN(emb_size, hidden_size, nonlinearity='tanh', batch_first=True)
    self.fc = nn.Linear(hidden_size, output_size)

  def forward(self, x):
    self.batch_size = x.size()[0]
    hidden = self.init_hidden()
    emb = self.emb(x)
    out, hidden = self.rnn(emb, hidden)
    out = self.fc(out[:, -1, :])
    return out

  def init_hidden(self):
    hidden = torch.zeros(1, self.batch_size, self.hidden_size)
    return hidden

device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Setting var
VOCAB_SIZE = len(word_id.keys()) +1
EMB_SIZE = 300
OUTPUT_SIZE = 4
HIDDEN_SIZE = 50

model = RNN(VOCAB_SIZE,EMB_SIZE, HIDDEN_SIZE, OUTPUT_SIZE)

model = model.to(device)

Predict
for i in range(10):
  # datasetet　の、リスト０indexに入力要素
  X, y = dataset_train[i]
  X = X.to(device)
  print(torch.softmax(model(X.unsqueeze(0)), dim=1))

This code works on CPU. but, can't works on "GPU".
Follow this error, I try to make some fix code.
ex) hidden.to(device),,,,
but,I can't solve...
Pleas someone tell me how to solve.
Thank you very much for my question.

Output:
Doesn't doing something like the following work?
device = torch.device("cuda" if torch.cuda.is_available() else "CPU")

class RNN(nn.Module):
  def __init__(self, vocab_size, emb_size, hidden_size, output_size):
    super().__init__()
    self.hidden_size = hidden_size
    self.emb = nn.Embedding(vocab_size, emb_size)
    self.rnn = nn.RNN(emb_size, hidden_size, nonlinearity='tanh', batch_first=True)
    self.fc = nn.Linear(hidden_size, output_size)
    self.to(device)

  def forward(self, x):
    self.batch_size = x.size()[0]
    hidden = self.init_hidden()
    emb = self.emb(x)
    out, hidden = self.rnn(emb, hidden)
    out = self.fc(out[:, -1, :])
    return out

  def init_hidden(self):
    hidden = torch.zeros(1, self.batch_size, self.hidden_size).to(device)
    return hidden


Instruction: NonImplementedError on using torch.onnx.export

Input:
I am trying to convert a pre-saved PyTorch model into a TensorFlow one via ONNX. For now, the following code is to export the model into .onnx format. The neural network has 2 inputs, one hidden layer with 5 neurons and a scalar output.
Here's the code I'm working with:
import torch.nn as nn
from torch.autograd import Variable
import numpy as np

class Model(nn.Module):
    def __init__(self, n_h_layers, n_h_neurons, dim_in, dim_out, in_bound, out_bound):
        super(Model,self).__init__()
        self.n_h_layers=n_h_layers
        self.n_h_neurons=n_h_neurons
        self.dim_in=dim_in
        self.dim_out=dim_out
        self.in_bound=in_bound
        self.out_bound=out_bound
        layer_input = [nn.Linear(dim_in, n_h_neurons, bias=True)]
        layer_output = [nn.ReLU(), nn.Linear(n_h_neurons, dim_out, bias=True), nn.Hardtanh(in_bound, out_bound)]

        # hidden layer
        module_hidden = [[nn.ReLU(), nn.Linear(n_h_neurons, n_h_neurons, bias=True)] for _ in range(n_h_layers - 1)]
        layer_hidden = list(np.array(module_hidden).flatten())

        # nn model
        layers = layer_input + layer_hidden + layer_output
        self.model = nn.Sequential(*layers)

        print(self.model)

trained_nn=torch.load('path')

trained_model=Model(1,5,2,1,-1,1)
trained_model.load_state_dict(trained_nn,strict=False)

dummy_input=Variable(torch.randn(1,2))
torch.onnx.export(trained_model,dummy_input, 'file.onnx', verbose=True)

I have two problems:

Running this snippet raises "NonImplementedError" in _forward_unimplemented in module.py as follows:

File ".../anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 201, in _forward_unimplemented
    raise NotImplementedError

NotImplementedError

I am not aware with Exception handling in python and I do not know what I must change in order to tackle the error.

When I print trained_nn, this is what it gives me:

OrderedDict([('0.weight',
              tensor([[ 0.2035, -0.7679],
                      [ 1.6368, -0.4135],
                      [-0.0908, -0.2335],
                      [ 1.3731, -0.3135],
                      [ 0.6361,  0.2521]])),
             ('0.bias', tensor([-1.6907,  0.7262,  1.4032,  1.2551,  0.8013])),
             ('2.weight',
              tensor([[-0.4603, -0.0719,  0.4082, -1.0235, -0.0538]])),
             ('2.bias', tensor([-1.1568]))])

However, printing trained_model.state_dict() gives me a neural network with a completely different set of weights and biases, although I believe that it should be giving me the exact same model as before as this is what I need to save as onnx file?
OrderedDict([('model.0.weight',
              tensor([[ 0.4817,  0.0928],
                      [-0.4313,  0.1253],
                      [ 0.6681, -0.4029],
                      [ 0.6474,  0.0029],
                      [-0.4663,  0.5029]])),
             ('model.0.bias',
              tensor([-0.2292,  0.6674, -0.3755,  0.0778,  0.0527])),
             ('model.2.weight',
              tensor([[-0.2097, -0.3029,  0.2792,  0.2596,  0.1362]])),
             ('model.2.bias', tensor([-0.1835]))])

Not sure what mistakes I'm making. Any help is appreciated.

Output:
When you are making a subclass of nn.Module you need to implement forward method. In your case you need to add:

class Model(nn.Module):
    def __init__(self, n_h_layers, n_h_neurons, dim_in, dim_out, in_bound, out_bound):
        super(Model, self).__init__()

    ...

    def forward(self, x):
        return self.model(x)

The names of parameters does not match:

model.0.weight != 0.weight
model.0.bias != 0.bias
prefix model is missed.
So when you call load_state_dict() with strict=False the parameters will not be used.
You can rename the parameters to match the model:
trained_nn = torch.load('path')
trained_nn = {f'model.{name}': w for name, w in trained_nn.items()}
trained_model.load_state_dict(trained_nn, strict=True)


Instruction: How to get most of the outputs of the sigmoid function in the range (0, 0.5)?

Input:
I use sigmoid function in my last layer. we know that sigmoid function, limits the outputs of network in range (0,1). I want most of the outputs to be in range (0, 0.5) and very few of them to be in range [0.5, 1). How can I do this in Pytorch to get the desired output?
The following Pytorch code snippet is related to this question:
class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()
    #
    def block(in_feat, out_feat, normalize=True):
        layers = [nn.Linear(in_features=in_feat, out_features=out_feat)]
        if normalize:
            layers.append(nn.BatchNorm1d(out_feat))
        layers.append(nn.LeakyReLU(0.2, inplace=True))
        return layers
    # now we can use this function like below:
    self.model = nn.Sequential(*block(params.input_dim_generator, 500, normalize=False),
                               *block(500, 350),
                               *block(350, 256),
                               nn.Linear(256, 564),
                               nn.Sigmoid())

# forward
    def forward(self, old_vector, z):
        vector_app = torch.cat((old_vector, z), dim=1)
        new_vector = self.model(vector_app)
        new_result = torch.max(new_vector, old_vector).float()
        return new_result

z is a random noise vector in range (0,1) and old_vector is a binary vector(the values are 0 or 1).
the output of this model is:
torch.tensor([0.5167, 0.5281, 0.5804, 0.4372, 1.0000, 1.0000, 1.0000, 0.5501, 1.0000,
        0.6154, 1.0000, 1.0000, 0.4699, 0.5536, 0.5005, 0.4318, 0.5302, 0.4830,
        0.5404, 0.3597, 0.4639, 0.5885, 0.4997, 0.5881, 0.5046, 0.5670, 0.3977,
        0.5186, 0.5859, 0.5398, 0.3954, 0.4839, 0.3310, 0.5208, 0.5420, 0.5056,
        0.5022, 0.6316, 0.6185, 0.5142, 0.5536, 0.4988, 0.5250, 0.4813, 0.5150,
        0.4080, 1.0000, 1.0000, 1.0000, 0.6054, 0.4766, 0.4423, 0.4520, 0.4816,
        0.5159, 0.4582, 1.0000, 0.4550, 0.4956, 1.0000, 0.5934, 1.0000, 0.4809,
        0.5512, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 0.4024, 0.4822, 1.0000,
        0.5310, 1.0000, 0.5127, 1.0000, 0.5441, 0.5063, 1.0000, 0.5511, 0.5544,
        1.0000, 0.4585, 0.5211, 0.5758, 0.4355, 1.0000, 0.5297, 0.4582, 0.4170,
        1.0000, 1.0000, 0.5257, 0.4194, 0.3583, 0.5087, 0.5936, 0.4851, 0.5697,
        0.4261, 0.4736, 0.4551, 1.0000, 0.5667, 0.5650, 1.0000, 0.5069, 0.5901,
        0.4980, 0.5184, 1.0000, 1.0000, 0.5435, 1.0000, 1.0000, 1.0000, 1.0000,
        0.4521, 1.0000, 0.4509, 1.0000, 0.5067, 1.0000, 0.4152, 0.5034, 0.5735,
        0.4040, 1.0000, 0.4492, 1.0000, 0.4405, 1.0000, 1.0000, 0.5667, 0.5639,
        0.4013, 0.4357, 0.4437, 0.4510, 0.4225, 0.5091, 0.5057, 1.0000, 0.5237,
        0.5098, 1.0000, 0.4216, 0.5242, 0.5335, 0.3916, 0.4938, 1.0000, 0.4070,
        0.5210, 1.0000, 1.0000, 0.4050, 0.3960, 0.5750, 0.4906, 0.4991, 1.0000,
        0.3149, 0.2949, 1.0000, 0.4515, 0.3627, 0.4348, 0.3887, 0.5807, 0.5787,
        0.5781, 1.0000, 1.0000, 1.0000, 1.0000, 0.4919, 1.0000, 1.0000, 0.5554,
        0.5515, 1.0000, 0.5472, 0.3342, 0.5705, 0.5076, 0.6348, 0.4436, 0.4683,
        0.4228, 0.6506, 0.4540, 0.5333, 0.4512, 0.6037, 0.5173, 1.0000, 0.4466,
        0.5644, 0.5565, 0.5141, 0.4771, 0.5822, 0.4888, 1.0000, 0.6331, 0.6435,
        1.0000, 0.5012, 1.0000, 0.4864, 1.0000, 0.4994, 0.4326, 0.4347, 0.3606,
        0.5829, 0.5229, 1.0000, 0.5992, 0.5883, 0.4825, 0.6254, 0.4951, 0.4285,
        0.4982, 1.0000, 0.5847, 0.4131, 0.5194, 0.5270, 0.4856, 0.6182, 0.5578,
        1.0000, 0.5460, 0.5023, 0.6279, 0.5727, 0.5997, 0.4903, 0.5633, 0.5070,
        0.5013, 1.0000, 0.4179, 0.5529, 0.6254, 0.5767, 0.3939, 0.5791, 0.4936,
        0.4714, 0.5150, 0.5717, 0.4570, 0.4463, 0.5493, 0.5179, 1.0000, 0.5682,
        0.5451, 0.5266, 0.5571, 1.0000, 1.0000, 0.5506, 0.4710, 0.5951, 1.0000,
        0.5027, 1.0000, 1.0000, 0.4960, 0.6269, 0.4817, 1.0000, 0.4059, 0.4787,
        0.4419, 0.5479, 0.4830, 0.4709, 0.6106, 0.6154, 0.3958, 0.6434, 0.4626,
        0.5954, 0.5083, 0.5121, 1.0000, 0.5139, 1.0000, 0.5428, 1.0000, 0.5278,
        0.5255, 0.5854, 0.4400, 0.4774, 0.4431, 0.4871, 0.3854, 0.6217, 0.5562,
        0.4461, 0.5191, 0.5654, 0.4428, 0.5503, 0.5742, 1.0000, 0.4899, 1.0000,
        0.5229, 0.5428, 0.4285, 0.3038, 0.3029, 0.5145, 0.6747, 0.5685, 0.5268,
        0.4888, 0.6431, 0.5308, 0.6249, 0.4531, 0.5631, 0.4498, 0.4465, 0.5125,
        0.5610, 1.0000, 0.5033, 0.5517, 1.0000, 0.4625, 0.5095, 1.0000, 0.3415,
        0.4749, 1.0000, 0.4567, 1.0000, 0.4417, 0.5623, 1.0000, 0.4780, 0.4218,
        1.0000, 0.5474, 0.6514, 0.5725, 0.4219, 0.5303, 0.3375, 0.5710, 0.5507,
        0.3698, 0.4902, 0.6082, 0.5212, 0.5606, 0.5320, 0.4893, 0.3831, 0.4605,
        0.5409, 0.4605, 0.5774, 0.5709, 0.5020, 0.5771, 0.4032, 0.5832, 0.4454,
        0.4572, 0.4651, 0.4752, 0.5786, 0.4700, 0.3398, 0.4143, 0.4413, 0.4020,
        0.6390, 0.5165, 0.4871, 0.6229, 0.4915, 1.0000, 0.4780, 0.5900, 0.4847,
        0.4583, 0.5889, 0.4291, 0.4095, 0.5258, 1.0000, 0.4875, 1.0000, 0.5174,
        0.4302, 1.0000, 0.5058, 0.5917, 0.5395, 0.3915, 0.4775, 0.4688, 0.4860,
        0.4869, 0.4189, 1.0000, 0.6453, 0.4652, 0.5106, 0.4336, 0.4959, 0.5144,
        1.0000, 1.0000, 0.4382, 0.5917, 1.0000, 0.5123, 0.4299, 0.5447, 1.0000,
        0.5316, 0.4145, 0.5741, 1.0000, 0.4581, 0.5953, 1.0000, 0.4909, 0.3703,
        0.3851, 0.5324, 1.0000, 0.6660, 1.0000, 0.5687, 0.4825, 0.5081, 0.5052,
        0.6288, 0.5371, 0.4286, 1.0000, 0.6535, 0.5556, 0.5390, 0.3320, 1.0000,
        0.6431, 0.5405, 1.0000, 0.3641, 0.4390, 0.6196, 0.4720, 0.5114, 0.4844,
        0.4184, 0.6269, 1.0000, 0.4077, 0.3950, 0.4502, 1.0000, 0.4417, 0.4329,
        0.5803, 0.4967, 0.5248, 0.5182, 0.4417, 0.4066, 0.6219, 0.3435, 1.0000,
        0.4680, 1.0000, 0.5403, 0.4570, 1.0000, 0.5805, 1.0000, 0.5796, 0.5100,
        0.6487, 0.4752, 0.4579, 0.6026, 0.5964, 0.5842, 0.3423, 0.5475, 0.4467,
        0.4494, 0.4782, 0.6054, 0.4499, 0.4691, 0.4700, 0.5006, 0.5895, 0.3947,
        0.5517, 0.4240, 0.5286, 0.4796, 0.5116, 0.5696, 0.4369, 0.4761, 0.5444,
        0.4490, 0.6399, 0.5469, 0.5155, 0.5339, 0.5860, 0.6092, 0.4000, 0.4622,
        0.4235, 0.5554, 0.4088, 0.5798, 0.5034, 0.4752, 0.4337, 0.4786, 0.5766,
        0.4569, 0.5401, 0.4903, 0.4243, 0.3825, 0.6652, 0.4780, 0.5335, 0.4415,
        0.5478, 0.3797, 1.0000, 0.6133, 0.5824, 0.4292, 0.5182, 0.3953, 0.5071,
        0.5131, 0.4735, 1.0000, 0.3457, 0.5933, 0.5329])

Output:
As I noted, reducing the bias of the last layer is one way to decrease the outputs on average. Here's a script that tries a few different values for the bias and plots the effect on the distribution of outputs.
import torch
from torch import nn
import matplotlib.pyplot as plt

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        #
        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_features=in_feat, out_features=out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat))
            layers.append(nn.LeakyReLU(0.2, inplace=True))
            return layers
        # now we can use this function like below:
        self.model = nn.Sequential(*block(input_dim_generator, 500, normalize=False),
                                   *block(500, 350),
                                   *block(350, 256),
                                   nn.Linear(256, 564),
                                   nn.Sigmoid())

    # forward
    def forward(self, old_vector, z):
        vector_app = torch.cat((old_vector, z), dim=1)
        new_vector = self.model(vector_app)
        new_result = torch.max(new_vector, old_vector).float()
        return new_result

z = torch.rand(160 , 70)
input_dim_generator = 634
old_vector = torch.randint(2,(160, 564))

gen = Generator()

bias_shift = 0
bias_delta = -.5

fig, ax = plt.subplots(2,2, figsize = (8,8))

with torch.no_grad():
    for i in range(4):
        ax_cur = ax[i//2][i%2]
        ax_cur.hist(gen(old_vector, z).numpy().ravel(),bins=20)
        ax_cur.set_title(f"{bias_shift} added to bias")
        bias_shift += bias_delta
        gen.model[-2].bias += bias_delta

The resulting histograms:

We can see that reducing the bias impacts the distribution of outputs, but doesn't seem to affect the number of outputs that are equal to exactly 1 (which I suspect come directly from the old vector).


Instruction: GPytorch Runtime Error has different input types

Input:
I am following the simple regression tutorial on gpytorch and get the following error when trying to use 2 dimensional input space during a call to the loss function.
RuntimeError: !(has_different_input_dtypes && !config.promote_inputs_to_common_dtype_ && (has_undefined_outputs || config.enforce_safe_casting_to_output_ || config.cast_common_dtype_to_outputs_)) INTERNAL ASSERT FAILED at "../aten/src/ATen/TensorIterator.cpp":405, please report a bug to PyTorch.

I am not quite sure what it means. Everything but the training data is still: https://github.com/cornellius-gp/gpytorch/blob/master/examples/01_Exact_GPs/Simple_GP_Regression.ipynb

Output:
The issue was my conversion of torch tensor.
I used: torch.from_numpy(array)
Instead I should use: torch.tensor(array)
This is weird, but no issues now.


Instruction: How do a put a different classifier on top of BertForSequenceClassification?

Input:
I have a huggingface model:
model_name = 'bert-base-uncased'
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=1).to(device)

How can I change the default classifier head? Since it's only a single LinearClassifier. I found this issue in the huggingface github which said:

You can also replace self.classifier with your own model.
model = BertForSequenceClassification.from_pretrained("bert-base-multilingual-cased")
model.classifier = new_classifier

where new_classifier is any pytorch model that you want.

However, I can't figure out how the structure of the new_classifier should look like (in particular the inputs and outputs so it can handle batches).

Output:
By looking at the source code of BertForSequenceClassification here, you can see that the classifier is simply a linear layer that project the bert output from hidden_size dimension to num_labels dimension. Suppose you want to change the linear classifier to a two layer MLP with Relu activation, you can do the following:
new_classifier = nn.Sequential(
      nn.Linear(config.hidden_size, config.hidden_size *2),
      nn.ReLU(),
      nn.Linear(config.hidden_size * 2, config.num_labels)
    )
model.classifier = new_classifier

The requirement of the structure of your new classifier is its input dimension and output dimension need to be config.hidden_size dimension and config.num_labels accordingly. The structure of the classifier doesn't rely on the batch size, and module like nn.Linear takes (*, H_dimension) dimension as input so you don't need to specify the batch size when creating the new classifier.


Instruction: How to understand this block of python code?

Input:
Found a piece of python code that can run perfectly,but I couldn't understand how it works.
Would appreciate it if you could explain the lining part for me. I totally don't know what it does
class BoxHead(nn.Module):#pending
    def __init__(self, lengths, num_classes):
        super(BoxHead, self).__init__()
#-------------------------------------------------------
        self.cls_score = nn.Sequential(*tuple([
            module for i in range(len(lengths) - 1)
            for module in (nn.Linear(lengths[i], lengths[i + 1]), nn.ReLU())]
            + [nn.Linear(lengths[-1], num_classes)]))
#-----------------------------------------------------------------------

        self.bbox_pred = nn.Sequential(*tuple([
            module for i in range(len(lengths) - 1)
            for module in (nn.Linear(lengths[i], lengths[i + 1]), nn.ReLU())]
            + [nn.Linear(lengths[-1], 4)]))

Output:
After reading, I assert lengths as a list of numbers.
This is the input of nn.Sequential for cls_score:
# lengths = [num1, num2, num3, ..., numN-1, numN]
# numN-1 just represents its meaning, and is not valid in python syntax
[
    nn.Linear(num1, num2),
    nn.ReLU(),
    nn.Linear(num2, num3),
    nn.ReLU(),
    nn.Linear(num3, num4),
    nn.ReLU(),
    ...,
    nn.Linear(numN-1, numN),
    nn.ReLU(),
    nn.Linear(numN, num_classes)
]

Input of nn.Sequential for bbox_pred is alike, but its last item is nn.Linear(numN, 4)


Instruction: Deep Learning: How does the training process in PyTorch work?

Input:
This picture shows my understanding of the process of running a model and my question about the process of training a model.
Can anyone explain what happens then?
Thanks to @Michael Hearn 's answer, I grabbed information and filled the picture:

Output:
Hopefully someone says a better answer after mine later... So basically you are asking how a neural network functions. So the model is trained on the training data going in it then passes through nodes which run functions on the information and change it as it passes through. Then an output is generated. The output is compared to the test answer. Then through the not magical process of backpropagation the network is corrected. Nodes that were heavily responsible for giving wrong answers on the training set are changed so the next pass through they will be more likely to give the right answer.
Backpropagation with PyTorch


Instruction: Is there a torch function to derive union of two tensors?

Input:
There is a function that can be used to derive union of two tensors in numpy, as below:
import torch
import numpy as np
a = torch.tensor([0, 1, 2])
b = torch.tensor([2, 3, 4])
c = np.union1d(a, b) # c = array([0, 1, 2, 3, 4])
c = torch.from_numpy(c) # c = torch.tensor([0, 1, 2, 3, 4])

However, I am looking for torch function that can be used directly on two tensors. If I use numpy function as above, I must cast the result from numpy to torch, and also must use cpu although the function is applied to tensors.
Is there any union function in torch that can be used directly on two tensors? Or, at least, can it be simply implemented using other torch functions?

Output:
You can use:
torch.cat((a, b)).unique()


Instruction: Transform DVS128Gesture dataset

Input:
I am making a neural network in Python using the DVS128Gesture dataset. I want to transform the default 128x128 trinary frames to 32x32 binary frames, but when I try to use torchvision.transform in the dataset, I am getting this error:
img should be PIL Image. Got <class 'numpy.lib.npyio.NpzFile'>

My code:
import torch
import torchvision
from spikingjelly.datasets.dvs128_gesture import DVS128Gesture

train_data = DVS128Gesture(root_dir, train=True, data_type='event',
                            transform=torchvision.transforms.Compose([
                                torchvision.transforms.Resize(32),
                                torchvision.transforms.Normalize((0.0,), (0.8,)),
                                torchvision.transforms.ToTensor()
                            ]))
test_data = DVS128Gesture(root_dir, train=False, data_type='event',
                           transform=torchvision.transforms.Compose([
                               torchvision.transforms.Resize(32),
                               torchvision.transforms.Normalize((0.0,), (0.8,)),
                               torchvision.transforms.ToTensor()
                           ]))

train_loader = torch.utils.data.DataLoader(train_data, batch_size=bs, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=bs, shuffle=True)

examples = enumerate(test_loader)
batch_idx, (example_data, example_targets) = next(examples)
example_data.shape

I have done the same with the MNIST dataset and everything worked as expected. I think the problem is that I use torchvision.transform in DVS128Gesture, but I am not sure what else I can use.
The same with MNIST:
train_data = torchvision.datasets.MNIST(root_dir, train=True, download=True,
                            transform=torchvision.transforms.Compose([
                                torchvision.transforms.Resize(28),
                                torchvision.transforms.ToTensor(),
                                torchvision.transforms.Normalize((0.0,), (0.8,))
                            ]))
test_data = torchvision.datasets.MNIST(root_dir, train=False, download=True,
                           transform=torchvision.transforms.Compose([
                               torchvision.transforms.Resize(28),
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize((0.0,), (0.8,))
                           ]))

What am I doing wrong?
Stack trace of error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [10], line 2
      1 examples = enumerate(test_loader)
----> 2 batch_idx, (example_data, example_targets) = next(examples)
      3 example_data.shape

File p:\Programs\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py:681, in _BaseDataLoaderIter.__next__(self)
    678 if self._sampler_iter is None:
    679     # TODO(https://github.com/pytorch/pytorch/issues/76750)
    680     self._reset()  # type: ignore[call-arg]
--> 681 data = self._next_data()
    682 self._num_yielded += 1
    683 if self._dataset_kind == _DatasetKind.Iterable and \
    684         self._IterableDataset_len_called is not None and \
    685         self._num_yielded > self._IterableDataset_len_called:

File p:\Programs\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py:721, in _SingleProcessDataLoaderIter._next_data(self)
    719 def _next_data(self):
    720     index = self._next_index()  # may raise StopIteration
--> 721     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    722     if self._pin_memory:
    723         data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

File p:\Programs\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py:49, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---> 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

File p:\Programs\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py:49, in <listcomp>(.0)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---> 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

File p:\Programs\Anaconda3\lib\site-packages\torchvision\datasets\folder.py:232, in DatasetFolder.__getitem__(self, index)
    230 sample = self.loader(path)
    231 if self.transform is not None:
--> 232     sample = self.transform(sample)
    233 if self.target_transform is not None:
    234     target = self.target_transform(target)

File p:\Programs\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py:94, in Compose.__call__(self, img)
     92 def __call__(self, img):
     93     for t in self.transforms:
---> 94         img = t(img)
     95     return img

File p:\Programs\Anaconda3\lib\site-packages\torch\nn\modules\module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File p:\Programs\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py:349, in Resize.forward(self, img)
    341 def forward(self, img):
    342     """
    343     Args:
    344         img (PIL Image or Tensor): Image to be scaled.
   (...)
    347         PIL Image or Tensor: Rescaled image.
    348     """
--> 349     return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)

File p:\Programs\Anaconda3\lib\site-packages\torchvision\transforms\functional.py:430, in resize(img, size, interpolation, max_size, antialias)
    428         warnings.warn("Anti-alias option is always applied for PIL Image input. Argument antialias is ignored.")
    429     pil_interpolation = pil_modes_mapping[interpolation]
--> 430     return F_pil.resize(img, size=size, interpolation=pil_interpolation, max_size=max_size)
    432 return F_t.resize(img, size=size, interpolation=interpolation.value, max_size=max_size, antialias=antialias)

File p:\Programs\Anaconda3\lib\site-packages\torchvision\transforms\functional_pil.py:249, in resize(img, size, interpolation, max_size)
    240 @torch.jit.unused
    241 def resize(
    242     img: Image.Image,
   (...)
    245     max_size: Optional[int] = None,
    246 ) -> Image.Image:
    248     if not _is_pil_image(img):
--> 249         raise TypeError(f"img should be PIL Image. Got {type(img)}")
    250     if not (isinstance(size, int) or (isinstance(size, Sequence) and len(size) in (1, 2))):
    251         raise TypeError(f"Got inappropriate size arg: {size}")

TypeError: img should be PIL Image. Got <class 'numpy.lib.npyio.NpzFile'>

Output:
The error probably comes from the Resize transform (can you provide more details on the stack trace of the error ?).
Resize is a image specific tranform, expecting a PIL image (or a torch Tensor, see the transform documentation), while your dataset DVS128Gesture outputs another object type.


Instruction: PyTorch Lightning: What's the meaning of calling self?

Input:
Consider the following function:
def training_step(self, batch, batch_idx):
    x, y = batch
    y_hat = self(x)
    loss = F.cross_entropy(y_hat, y)
    return loss

What's the meaning of calling self in this context?
How can self act as a callable anyway?
Thanks

Output:
Calling an object with object() is equivalent to calling the __call__ method  of the given class, defined as usual with:
   def __call__(self):

So self() is equivalent to obj.__call__ method of class object and  since it subclasses torch.nn.Module, which will invoke _call_impl, and eventually the forward pass,


Instruction: Best way to implicitly change the value of nn.Parameter() in Pytorch?

Input:
Suppose that I want to optimize a vector v so that its norm is equal to 1. To do that, I defined a network with that vector as follows:
class myNetwork(nn.Module):
    def __init__(self,initial_vector):
        super(myNetwork, self).__init__()
        #Define vector according to an initial column vector
        self.v = nn.Parameter(initial_vector)
    def forward(self,x):
        #Normalize vector so that its norm is equal to 1
        self.v.data = self.v.data / torch.sqrt(self.v.data.transpose(1,0) @ self.v.data)
        #Multiply v times a row vector
        out = x @ self.v
        return out

Is the use of .data the best way to update v? Does it takes into account the normalization during backpropagation?

Output:
You could simply use
    def forward(self,x):
        return x @ self.v / (x**2).sum()

Depending on your loss, or the downstream layers of your network, even skip normalization at all.
    def forward(self,x):
        return x @ self.v

This should work as long as your loss is invariant to scale, the norm should vary only slightly every step, but it is not strictly stable. If you are giving many steps maybe it is worth adding a term thinyvalue * ((myNetwork.v**2).sum()-1)**2 to your loss, to make sure that the norm of $v$ is attracted to 1.


Instruction: PyTorch: Running Inference on multiple GPUs

Input: I have a model that accepts two inputs. I want to run inference on multiple GPUs where one of the inputs is fixed, while the other changes. So, let’s say I use n GPUs, each of them has a copy of the model. First gpu processes the input pair (a_1, b), the second processes (a_2, b) and so on. All the outputs are saved as files, so I don’t need to do a join operation on the outputs. How can I do this with DDP or otherwise?

Output:
I have figured out how to do this using torch.multiprocessing.Queue:
import torch
import torch.multiprocessing as mp
from absl import app, flags
from torchvision.models import AlexNet

FLAGS = flags.FLAGS

flags.DEFINE_integer("num_processes", 2, "Number of subprocesses to use")

def infer(rank, queue):
    """Each subprocess will run this function on a different GPU which is indicated by the parameter `rank`."""
    model = AlexNet()
    device = torch.device(f"cuda:{rank}")
    model.to(device)
    while True:
        a, b = queue.get()
        if a is None:  # check for sentinel value
            break
        x = a + b
        x = x.to(device)
        model(x)
        del a, b  # free memory
        print(f"Inference on process {rank}")

def main(argv):
    queue = mp.Queue()
    processes = []
    for rank in range(FLAGS.num_processes):
        p = mp.Process(target=infer, args=(rank, queue))
        p.start()
        processes.append(p)
    for _ in range(10):
        a_1 = torch.randn(1, 3, 224, 224)
        a_2 = torch.randn(1, 3, 224, 224)
        b = torch.randn(1, 3, 224, 224)
        queue.put((a_1, b))
        queue.put((a_2, b))
    for _ in range(FLAGS.num_processes):
        queue.put((None, None))  # sentinel value to signal subprocesses to exit
    for p in processes:
        p.join()  # wait for all subprocesses to finish

if __name__ == "__main__":
    app.run(main)


Instruction: RuntimeError: NCCL Error 2: unhandled system error

Input:
I upgraded cuda from 9.0 to 10.2 recently, but when I successfully upgraded, my demo as follows will default by "RuntimeError: NCCL Error 2: unhandled system error".
I dont know why and try to look for answer in github or stackoverflow, but I failed. So I hope someone can help me.
import torch
from torchvision import datasets, transforms
import torchvision
from tqdm import tqdm

device_ids = [0, 1] # GPU
BATCH_SIZE = 64

transform = transforms.Compose([transforms.ToTensor()])
data_train = datasets.MNIST(root = "./data/",
                            transform=transform,
                            train=True,
                            download=True)
data_test = datasets.MNIST(root="./data/",
                           transform=transform,
                           train=False)

data_loader_train = torch.utils.data.DataLoader(dataset=data_train,

                                                batch_size=BATCH_SIZE * len(device_ids),
                                                shuffle=True,
                                                num_workers=2)

data_loader_test = torch.utils.data.DataLoader(dataset=data_test,
                                               batch_size=BATCH_SIZE * len(device_ids),
                                               shuffle=True,
                                               num_workers=2)

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = torch.nn.Sequential(
        torch.nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(stride=2, kernel_size=2),
    )
        self.dense = torch.nn.Sequential(
            torch.nn.Linear(14 * 14 * 128, 1024),
            torch.nn.ReLU(),
            torch.nn.Dropout(p=0.5),
            torch.nn.Linear(1024, 10)
    )
    def forward(self, x):
        x = self.conv1(x)
        x = x.view(-1, 14 * 14 * 128)
        x = self.dense(x)
        return x

model = Model()

model = torch.nn.DataParallel(model, device_ids=device_ids)

model = model.cuda(device=device_ids[0])

cost = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())
from time import sleep
n_epochs = 50
for epoch in range(n_epochs):
    running_loss = 0.0
    running_correct = 0
    print("Epoch {}/{}".format(epoch, n_epochs))
    print("-"*10)
    for data in tqdm(data_loader_train):
        X_train, y_train = data

        X_train, y_train = X_train.cuda(device=device_ids[0]), y_train.cuda(device=device_ids[0])
        outputs = model(X_train)
        _,pred = torch.max(outputs.data, 1)
        optimizer.zero_grad()
        loss = cost(outputs, y_train)

        loss.backward()
        optimizer.step()
        running_loss += loss.data.item()
        running_correct += torch.sum(pred == y_train.data)
    testing_correct = 0
    for data in data_loader_test:
        X_test, y_test = data

        X_test, y_test = X_test.cuda(device=device_ids[0]), y_test.cuda(device=device_ids[0])
        outputs = model(X_test)
        _, pred = torch.max(outputs.data, 1)
        testing_correct += torch.sum(pred == y_test.data)
    print("Loss is:{:.4f}, Train Accuracy is:{:.4f}%, Test Accuracy is:{:.4f}".format(torch.true_divide(running_loss, len(data_train)),
                                                                                torch.true_divide(100*running_correct, len(data_train)),
                                                                                torch.true_divide(100*testing_correct, len(data_test))))
torch.save(model.state_dict(), "model_parameter.pkl")

The follows are the Info of the error.
Epoch 0/50
----------
  0%|                                                                                | 0/469 [00:00<?, ?it/s]7aea7ed215cf:50693:50693 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.14<0>
7aea7ed215cf:50693:50693 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

7aea7ed215cf:50693:50693 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
7aea7ed215cf:50693:50693 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.14<0>
7aea7ed215cf:50693:50693 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda10.2
7aea7ed215cf:50693:50809 [1] NCCL INFO Could not enable P2P between dev 1(=3e000) and dev 0(=3d000)
7aea7ed215cf:50693:50809 [1] NCCL INFO Could not enable P2P between dev 0(=3d000) and dev 1(=3e000)
7aea7ed215cf:50693:50809 [1] NCCL INFO Could not enable P2P between dev 1(=3e000) and dev 0(=3d000)
7aea7ed215cf:50693:50809 [1] NCCL INFO Could not enable P2P between dev 0(=3d000) and dev 1(=3e000)
7aea7ed215cf:50693:50808 [0] NCCL INFO Could not enable P2P between dev 1(=3e000) and dev 0(=3d000)
7aea7ed215cf:50693:50808 [0] NCCL INFO Could not enable P2P between dev 0(=3d000) and dev 1(=3e000)
7aea7ed215cf:50693:50808 [0] NCCL INFO Could not enable P2P between dev 1(=3e000) and dev 0(=3d000)
7aea7ed215cf:50693:50808 [0] NCCL INFO Could not enable P2P between dev 0(=3d000) and dev 1(=3e000)
7aea7ed215cf:50693:50809 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
7aea7ed215cf:50693:50808 [0] NCCL INFO Channel 00/02 :    0   1
7aea7ed215cf:50693:50808 [0] NCCL INFO Channel 01/02 :    0   1
7aea7ed215cf:50693:50809 [1] NCCL INFO Setting affinity for GPU 1 to 3ff003ff
7aea7ed215cf:50693:50808 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
7aea7ed215cf:50693:50808 [0] NCCL INFO Setting affinity for GPU 0 to 3ff003ff
7aea7ed215cf:50693:50809 [1] NCCL INFO Could not enable P2P between dev 1(=3e000) and dev 0(=3d000)
7aea7ed215cf:50693:50808 [0] NCCL INFO Could not enable P2P between dev 0(=3d000) and dev 1(=3e000)

7aea7ed215cf:50693:50809 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
7aea7ed215cf:50693:50809 [1] NCCL INFO include/shm.h:41 -> 2

7aea7ed215cf:50693:50809 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-3bd03c4f9664d387-0-0-1 (size 9637888)
7aea7ed215cf:50693:50809 [1] NCCL INFO transport/shm.cc:100 -> 2
7aea7ed215cf:50693:50809 [1] NCCL INFO transport.cc:34 -> 2
7aea7ed215cf:50693:50809 [1] NCCL INFO transport.cc:84 -> 2
7aea7ed215cf:50693:50809 [1] NCCL INFO init.cc:778 -> 2

7aea7ed215cf:50693:50808 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
7aea7ed215cf:50693:50808 [0] NCCL INFO include/shm.h:41 -> 2
7aea7ed215cf:50693:50809 [1] NCCL INFO init.cc:904 -> 2

7aea7ed215cf:50693:50808 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-3bd03c4f9664d387-0-1-0 (size 9637888)
7aea7ed215cf:50693:50808 [0] NCCL INFO transport/shm.cc:100 -> 2
7aea7ed215cf:50693:50808 [0] NCCL INFO transport.cc:34 -> 2
7aea7ed215cf:50693:50808 [0] NCCL INFO transport.cc:84 -> 2
7aea7ed215cf:50693:50808 [0] NCCL INFO init.cc:778 -> 2
7aea7ed215cf:50693:50809 [1] NCCL INFO group.cc:72 -> 2 [Async thread]
7aea7ed215cf:50693:50808 [0] NCCL INFO init.cc:904 -> 2
7aea7ed215cf:50693:50808 [0] NCCL INFO group.cc:72 -> 2 [Async thread]
7aea7ed215cf:50693:50693 [0] NCCL INFO init.cc:973 -> 2
  0%|                                                                                | 0/469 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 73, in <module>
    outputs = model(X_train)
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 172, in replicate
    return replicate(module, device_ids, not torch.is_grad_enabled())
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/replicate.py", line 91, in replicate
    param_copies = _broadcast_coalesced_reshape(params, devices, detach)
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/replicate.py", line 71, in _broadcast_coalesced_reshape
    tensor_copies = Broadcast.apply(devices, *tensors)
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/_functions.py", line 23, in forward
    outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
  File "/root/anaconda3/envs/pytorch171/lib/python3.7/site-packages/torch/nn/parallel/comm.py", line 58, in broadcast_coalesced
    return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: NCCL Error 2: unhandled system error

Output:
This is apparently caused by newer versions of nccl including a data pathway which uses linux shared memory for internode communication (see here). If that system is misconfigured or unavailable, then you might see this issue in any codebase which uses nccl.
Your two choices to fix this are

Correctly set up the linux tmpfs system
Use the NCCL_SHM_DISABLE environment variable to prevent nccl from trying to use this data pathway (see in the documentation here). This will force nccl to fall back to a potentially slower data pathway.


Instruction: How to operate torch.dot() in matrix consist of vectors in pytorch?

Input:
I have tensor like this:
arr1 = np.array([[ 1.6194, -0.6058, -0.8012], [ 1.1483,  1.6538, -0.8062]])
arr2 = np.array([[-0.3180, -1.8249,  0.0499], [-0.4184,  0.6495, -0.4911]])
X = torch.Tensor(arr1)
Y = torch.Tensor(arr2)

I want to do torch.dot on every tensor 1D (2 vectors) inside my 2D tensor
torch.dot(X, Y)

I want to get the result like this tensor([dotResult1, dotResult2]).
But I got the error like this:
RuntimeError: 1D tensors expected, but got 2D and 2D tensors

My main purpose is to do "something" operation on every vector inside my matrix but I don't want to use looping here, does anyone know how to do that?

Output:
Assuming what you are looking for is the tensor : [torch.dot(X[0], Y[0]), torch.dot(X[1], Y[1])]
Then you can do:
(X*Y).sum(axis = 1)

Test:
(X*Y).sum(axis = 1) == torch.tensor([torch.dot(X[0], Y[0]),torch.dot(X[1], Y[1])])

outputs:
tensor([True, True])


Instruction: sum distribution of points on 2d graph

Input:
I have a 10,000 2d points in a pytorch tensor.
The points are between between -1 and 1. i.e. x ∈ {-1, 1}^2
I want to output a heat map to show the distribution of the points.
In order to print the graph I am using matplotlib like so
torch.meshgrid(torch.linspace(-1, 1, 1000), torch.linspace(-1, 1, 1000))
ax.contourf(x_grid, y_grid, grid_values)

However I dont know how to calculate the grid values for this
I would you calculate the grid_values given the tensor of shape (10,000, 2) of points on the graph (note that a point can appear multiple times on the graph and we want to sum the distribtion of these points)

Output:
You are describing a normal 2D histogram, possibly weighted (not sure if your points had a value associated with them as well)?
plt.hist2d(x, y, weights=w, range=[[-1,1],[-1,1]], bins=1000)

will also spit out the associated matrix, with values binned accordingly which you can use to make contours.


Instruction: How to fine-tune gpt-j using Huggingface Trainer

Input:
I'm attempting to fine-tune gpt-j using the huggingface trainer and failing miserably. I followed the example that references bert, but of course, the gpt-j model isn't exactly like the bert model.
The error indicates that the model isn't producing a loss, which is great, except that I have no idea how to make it generate a loss or how to change what the trainer is expecting.
I'm using Transformers 4.22.2. I would like to get this working on a CPU before I try to do anything on Paperspace with a GPU. I did make an initial attempt there using a GPU that received the same error, with slightly different code to use cuda.
I suspect that my approach is entirely wrong. I found a very old example of fine-tuning gpt-j using 8-bit quantization, but even that repository says it is deprecated.
I'm unsure if my mistake is in using the compute_metrics() I found in the bert example or if it is something else. Any advice would be appreciated. Or, maybe it is an issue with the labels I provide the config, but I've tried different permutations.
I understand what a loss function is, but I don't know how it is supposed to be configured in this case.
My Code:
from transformers import Trainer, TrainingArguments, AutoModelForCausalLM
from transformers import GPTJForCausalLM, AutoTokenizer
from datasets import load_dataset
import time
import torch
import os
import numpy as np
import evaluate
import sklearn

start = time.time()

GPTJ_FINE_TUNED_FILE = "./fine_tuned_models/gpt-j-6B"

print("Loading model")
model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", low_cpu_mem_usage=True)
model.config.pad_token_id = model.config.eos_token_id

print("Loading tokenizer")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")
tokenizer.pad_token = tokenizer.eos_token

print("Loading dataset")
current_dataset = load_dataset("wikitext", 'wikitext-103-v1')
current_dataset['train'] = current_dataset['train'].select(range(1200))

def tokenize_function(examples):
    current_tokenizer_result = tokenizer(examples["text"], padding="max_length", truncation=True)
    return current_tokenizer_result

print("Splitting and tokenizing dataset")
tokenized_datasets = current_dataset.map(tokenize_function, batched=True)
small_train_dataset = tokenized_datasets["train"].select(range(100))

print("Preparing training arguments")

training_args = TrainingArguments(output_dir=GPTJ_FINE_TUNED_FILE,
                                  report_to='all',
                                  logging_dir='./logs',
                                  per_device_train_batch_size=1,
                                  label_names=['input_ids', 'attention_mask'],  # 'logits', 'past_key_values'
                                  num_train_epochs=1,
                                  no_cuda=True
                                  )

metric = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset
)

print("Starting training")
trainer.train()
print(f"Finished fine-tuning in {time.time() - start}")

Which leads to the error and stacktrace:
  File "xxx\ft_v3.py", line 66, in <module>
  File "xxx\venv\lib\site-packages\transformers\trainer.py", line 1521, in train
    return inner_training_loop(
  File "xxx\venv\lib\site-packages\transformers\trainer.py", line 1763, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "xxx\venv\lib\site-packages\transformers\trainer.py", line 2499, in training_step
    loss = self.compute_loss(model, inputs)
  File "xxx\venv\lib\site-packages\transformers\trainer.py", line 2544, in compute_loss
    raise ValueError(
ValueError: The model did not return a loss from the inputs, only the following keys: logits,past_key_values. For reference, the inputs it received are input_ids,attention_mask.

Output:
I found what appears to work, though now I'm running low on memory and working through ways of handling it.
The data_collator parameter seems to take care of the exact issue that I was having.
data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
    data_collator=data_collator,
)


Instruction: Installed Pytorch 1.12 in the environment but detects version 1.10.0+cpu

Input:
Recently, I have installed pytorch 1.12.1 in the conda environment. After installation, I checked the version of pytorch using print(torch.__version__), it returns 1.10.0+cpu. I also checked the available packages in the environment. It shows pytorch version 1.12.1 as shown in figure below.
I am unable to understand why it is detecting version 1.10.0+cpu. I even reinstalled Anaconda python in Windows, still it shows same version 1.10.0+cpu even in the base environment.
Can someone please figure it out.

Output: I tried uninstalling the existing pytorch version 1.10.0+cpu using pip/conda but nothing was working. I was finally able to get rid of it by deleting the old Python folder at 'C:\Users\Vikrant\AppData\Roaming\Python'. I created a new environment and installed Pytorch 1.12.1. Now, everything is working fine. Thanks!


Instruction: Unflatten in pytorch

Input:
I need to change the shape of tensor from [2, 48, 196] to [2, 48, 14,14]. I read there a "unflatten" in pytorch. But I couldn't understand how to use  it.
Is there any example?

Output:
Here is example for your question.
import torch

input = torch.randn([2,48,196])
unflatten = torch.nn.Unflatten(2, (14,14))
output = unflatten(input)

If you check output.shape, the shape is [2,48,14,14].
Unflatten function is to expand specific dim to a desired shape. In your case, you want to expand the shape 196 in "dim 2" to new shape of the unflatten dimension "(14,14)".
There are two parameters in Unflatten function.

First parameter is dim. it is specific dimension which you want to
be unflatten. In your case, it is 2.
Second parameter is unflatten_size. It is the new shape of the unflatten dimension of the tensor. So it is (14,14).

Therefore, your Unflatten function should be looked like unflatten = torch.nn.Unflatten(2, (14,14))


Instruction: AttributeError: 'DataLoader' object has no attribute '__getitem__'

Input:
I want to get the first image within Pytorch's dataloader as per this doc here: https://pytorch.org/vision/stable/generated/torchvision.datasets.CIFAR100.html
However I get the error in the title when I try doing this: trainloader.__getitem__(0)
Am I misunderstanding their docs?
Here's my code:
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

trainset = torchvision.datasets.CIFAR100(root= "./data", train = True, transform=preprocess, download=True)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)
trainloader.__getitem__(0)

Output:
Those are the docs for a torch Dataset, which does in fact have a __getitem__ implemented. However, you are calling __getitem__ on trainloader, a DataLoader which does not have that method.
So this should do what you expect:
trainset.__getitem__(0)

When you get something from a Dataset, you are getting a single observation. DataLoaders should not be used to return single items in this manner; instead, they are for fetching a sampled batch of observations. That's why they don't support getting a single obs through __getitem__.
It's also worth reading a bit about how __getitem__ works (for example here); you can use it to subscript an object directly. trainset[0] is equivalent to trainset.__getitem__(0).


Instruction: Unexpected number of fully connected neurons after padding with stride in pytorch

Input:
I'm trying to replicate the procedure of this paper (Re-)Imag(in)ing Price Trends, that trains a 2dCNN based on OHLC charts. They have images of different dimensions (32x15, 64x60 and 96x180) corresponding to 5, 20 and 60 daily bars and thus three dimension-specific architectures. But I end up with a different number of neurons compared to them in the fully connected layer for the 20day horizon (64x60)...
I followed the specification of their architecture that can be summarised as:

Number of blocks: (32x15): 2; (64x60):3 and (96x180):4
Fixed number of filters in 1st block: 64, else number doubles each convolutional block
5x3 (kernel size) convolutional filters (for all image types)
2x1 max-pooling filters (for all image types)
vertical stride of 1, 3, and 3 (only in first layer) for 32x15, 64x60 and 96x180 respectively
vertical dilution rate of 1, 2, and 3 (only in the first layer) for 32x15, 64x60 and 96x180 respectively
padding such that output has SAME dimension as the image itself

I suspect that my issue has something with the workaround for padding="same" in pytorch with asymmetric strides. Since strides can be >1, I went for the Conv2d workaround provided in this answer.
According to their paper (see figure), the CNN for 20 days horizon (64x60) should end up having 46080 neurons in FC layer.
Below is the code for my architecture that gets the following error when resizing.

RuntimeError: shape '[-1, 46080]' is invalid for input of size 30720:

Clearly, the dimensions are incorrect and sensitive to changing the padding calculations. I cannot seem to get this right, not sure how I would else figure out padding for each block in each specific model... Hope someone can help me out. Thanks in advance.
import torch
from torch import nn
import math
from functools import reduce
from operator import __add__
import torch.nn.functional as F

class Conv2dSame(nn.Conv2d):
"""
https://github.com/pytorch/captum/blob/optim-wip/captum/optim/models/_common.py#L144
"""

    def calc_same_pad(self, i: int, k: int, s: int, d: int) -> int:
        pad = max((math.ceil(i / s) - 1) * s + (k - 1) * d + 1 - i, 0)
        return pad

    def forward(self, x: torch.Tensor) -> torch.Tensor:

        ih, iw = x.size()[-2:]
        kh, kw = self.weight.size()[-2:]
        pad_h = self.calc_same_pad(i=ih, k=kh, s=self.stride[0], d=self.dilation[0])
        pad_w = self.calc_same_pad(i=iw, k=kw, s=self.stride[1], d=self.dilation[1])

        if pad_h > 0 or pad_w > 0:
            x = F.pad(
                x, [pad_w // 2, pad_w - pad_w // 2, pad_h // 2, pad_h - pad_h // 2]
            )
        return F.conv2d(
            x,
            self.weight,
            self.bias,
            self.stride,
            self.padding,
            self.dilation,
            self.groups,
        )

class Net20(nn.Module):

    def __init__(self):
        super().__init__()
        self.layer1 = nn.Sequential(
            Conv2dSame(1, 64, kernel_size=(5,3), stride=(3,1), dilation=(2,1)),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(negative_slope=0.01, inplace=True),
            nn.MaxPool2d((2, 1))
        )
        self.layer2 = nn.Sequential(
            Conv2dSame(64, 128, kernel_size=(5,3)),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(negative_slope=0.01, inplace=True),
            nn.MaxPool2d((2, 1))
        )
        self.layer3 = nn.Sequential(
            Conv2dSame(128, 256, kernel_size=(5,3)),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(negative_slope=0.01, inplace=True),
            nn.MaxPool2d((2, 1))
        )
        self.fc1 = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(46080, 1),
        )

    def forward(self, x):
        x = x.reshape(-1,1,64,60)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = x.reshape(-1,46080) # FC neurons according to paper
        x = self.fc1(x)
        return x

Output:
You're right, torch doesn't support 'same' padding and you should implement it yourself for odd size with F.pad.
First I suggest you to inspect MaxPool2D. By default, the output of a 3x3 image by MaxPool2D((2, 2)) is of shape 1x1 (forgetting borders) but you may expect 2x2. Typically if your width starts to be odd you can have a problem. You can try to add ceil_mode=True argument (see this for details).
Now regarding the padding calculation, you should note that there are many possible implementation for padding  because "same" padding is ambiguous in your case. I found an other implementation that also should mimic tensorflow behaviour (see here). It seems not to be exactly what you used so you can try it:
class Conv2dSame(nn.Conv2d):

  def calc_same_pad(self, i: int, k: int, s: int, d: int) -> int:
    # (i + s - 1) // s instead of ceil(i / s)
    pad = max(0, ((i + s - 1) // s - 1) * s + (k - 1) * d + 1 - i)
    return pad

  def forward(self, x: torch.Tensor) -> torch.Tensor:
    # ...
    pad_w, pad_h = self.calc_same_pad(...), self.calc_same_pad(...)
    w_odd, h_odd = pad_w % 2 == 1, pad_h % 2 == 1

    if w_odd or h_odd:
      # Add 1 padding now for odd size
      x = F.pad(x, [0, int(w_odd), 0, int(h_odd)])

    return F.conv2d(
      x,
      kernel_size=self.weight,
      bias=self.bias,
      stride=self.stride,
      padding=(pad_w // 2, pad_h // 2),  # add the rest of the padding here
      dilatation=self.dilation,
      groups=self.groups,
      )

Basically if for instance i = s + 1 = 4 the two methods don't return the same thing. I let you explore that.
Notes:

padding argument in Conv2dSame has no effect (that's logical as the padding is calculated on the fly)
the padding in two steps should be similar to what you did with [pad_w // 2, pad_w - pad_w //2, ...] but I keep it to respect the original code (in case of there is a strange behaviour in Conv2D but I don't think so)

Hope it solves the problem.


Instruction: How can I convert a pytorch tensor of an image of the shape (batch, channels, height, width) to (batch, height, width)?

Input: I want to turn a colored image into a grayscale image by removing the color channels. Is this even possible?

Output:
You can use the Grayscale transform
https://pytorch.org/vision/stable/generated/torchvision.transforms.Grayscale.html
from torchvision.transforms import Grayscale

grayscale_batch = Grayscale()(color_batch)

Which results in a (batch_size, 1, H,W) tensor.
To remove the channel dimension, apply torch.squeeze()


Instruction: How to add a "model as an layer" in pytorch

Input:
Is there any way to use a "pre-trained model as an layer" in a custom net?
Pseudocode:
class Net(nn.Module):

    def __init__(self):

        super(Net, self).__init__()
        self.model_layer = # ?
        self.fc1 = nn.Linear(num_classes_of_model_layer, 320)
        self.fc2 = nn.Linear(320, 160)
        self.fc3 = nn.Linear(160, num_classes)

    def forward(self, x):
        x = # ?
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Output:
Yes you can absolutely use another model a part of your Module, as the other model is also a Module
Do:
self.model_layer = pretrained_model

and make inference as always with x = self.model_layer(x)


Instruction: How to plot a list of torch.tensors? "RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead."

Input:
I am learning Pytorch and was following a tutorial when I came accross this error:
"RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead."
I am adding my losses to a list called final_losses
for i in range(epochs):
    y_pred=model.forward(X_train)
    loss=loss_function(y_pred,y_train)
    final_losses.append(loss)

This is a simple ANN module having 2 fully connected layers and I use Relu function in them.
I am trying to print a graph of epochs vs loss:
plt.plot(range(epochs),final_losses)
plt.show()

When I execute this I am getting the above error.("RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.")
I have printed the these variables for your reference:
Epochs is 150, length of final_losses is 150  and final_losses is [tensor(1.5851, grad_fn=<NllLossBackward0),...
I also tried doing this :
plt.plot(range(epochs),torch.detach(final_losses).numpy())
plt.show()

I am getting the following error:
TypeError: detach(): argument 'input' (position 1) must be Tensor, not list
Please let me know how to solve this.
Thank you!

Output:
You have a list of tensors, rather than a tensor. Modify your initial code to store your losses as numpy arrays (or singular floats by taking the mean):
for i in range(epochs):
    y_pred=model.forward(X_train)
    loss=loss_function(y_pred,y_train)
    final_losses.append(loss.detach().numpy())

If you are on gpu you might also need to convert to cpu in that last line.


Instruction: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0

Input:
I cannot understand why this error keeps popping out.
I start specifying the device variable
if torch.cuda.is_available():
    device = torch.device("cuda")
    n_gpu = torch.cuda.device_count()
    torch.cuda.get_device_name(0)
else:
    device = torch.device("cpu")

The model is the following
class CNN(nn.Module):
def __init__(self, initial_num_channels, num_channels):
    '''
    Args:
    initial_num_channels (int): size of the incoming feature vector
    num_classes (int): size of the output prediction vector
    num_channels (int): constant channel size to use throughout network
    '''
    super(CNN, self).__init__()
    self.convnet = nn.Sequential(

        nn.Conv1d(in_channels=initial_num_channels,
                  out_channels=num_channels, kernel_size=3),

        nn.ELU(),

        nn.Conv1d(in_channels=num_channels, out_channels=num_channels,
                  kernel_size=3, stride=2),

        nn.ELU(),

        nn.Conv1d(in_channels=num_channels, out_channels=num_channels,
                  kernel_size=3, stride=2),

        nn.ELU(),

        nn.Conv1d(in_channels=num_channels, out_channels=num_channels,
                  kernel_size=3),

        nn.ELU() )

def forward(self, x, apply_softmax=False):

    """The forward pass of the classifier

    Args:
        x (torch.Tensor): an input data tensor.
            x.shape should be (batch, dataset._max_seq_length)
        apply_softmax (bool): a flag for the softmax activation
            should be false if used with the Cross Entropy losses
    Returns:
        the resulting tensor. tensor.shape should be (batch, num_classes)
    """

    # input tensor: batch_size x channels x signal_length
    x = self.convnet(x)

    # average through maxpooling
    x = F.avg_pool1d(x, kernel_size=3)
    x = F.dropout(x, p=0.1)

    # go back to 2 dimensions: batch_size x features
    x = x.view(x.size(dim=0), -1)

    # compute the input size of linear combination layer
    num_features = x.size(dim=1)

    # final linear comb layers
    self.fc1 = nn.Linear(num_features, 100)
    self.fc2 = nn.Linear(100, 2)

    # mlp classifier
    x = F.relu(F.dropout(self.fc1(x), p=0.1))
    x = self.fc2(x)

    if apply_softmax:
        x = F.softmax(x, dim=1)

    return x

Here I move it onto the GPU
epochs = 10
model = CNN(initial_num_channels=1, num_channels=256)
# model = MultilayerPerceptron(input_dim=MAX_LEN, hidden_dim=100, output_dim=2)
model = model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Finally I run the training loop
train_loss_set = []
executed = False

for epoch in range(1, epochs + 1):
    train_loop = tqdm(train_dataloader)

    model.train()

    train_steps = 0
    train_loss = 0

    for (idx,(train_input, train_label)) in enumerate(train_loop):

        # add dimension=1 in position 1 to have channels=1
        # ONLY FOR CNN
        train_input = torch.unsqueeze(train_input, 1)

        train_input = train_input.clone().detach().requires_grad_(True).to(device)
        train_label = train_label.clone().detach().to(device)
        train_output = model(train_input)
        loss = criterion(train_output, train_label)
        loss.backward()
        train_loss_set.append(loss.item())
        optimizer.step()
        model.zero_grad()

        train_steps += 1
        train_loss += loss.item()
        train_loss_set.append(loss.item())

    print("Train loss: {}".format(train_loss/train_steps))

The error is triggered during the forward pass of the model, hence I assume, the problem is related to the input. I tried either the classic assignment input = input.to(device) and also the "fancier" one (which you can see above)

Output:
The problem is with the fc1 and fc2 layers which you create in your forward function. These are only created when the model is called on an input, and are not moved to the GPU. To fix this, you can use:
self.fc1 = nn.Linear(num_features, 100, device=x.device)
self.fc2 = nn.Linear(100, 2, device=x.device)

The definition of fc2 could also be moved to the __init__ function to avoid recreating it each time.


Instruction: Unwrapping the module did not yield a `LightningModule`

Input:
I am getting an error running the validate() function of pytorch lighting using the following code.
error:
ValueError: An invalid dataloader was passed to `Trainer.validate(dataloaders=...)`. Either pass the dataloader to the `.validate()` method OR implement `def val_dataloader(self):` in your LightningModule/LightningDataModule.

code:
from torchvision.datasets import MNIST
from torchvision import transforms
from torch.utils.data import DataLoader
...
mnist_val = MNIST(root='data',train=False, download=True, transform=transform)
mnist_val_loader = DataLoader(mnist_val, batch_size=256, num_workers=4)
...
trainer.validate()

I used the data loader into the validate() function but I get the following error:
Unwrapping the module did not yield a `LightningModule`

Output:
I solved with the newer versions of pytorch_lighting putting as input model & data loader to trainer.validate()
trainer.validate(model, mnist_val_loader)


Instruction: convert pytorch model to ONNX

Input:
How to convert a pytorch model to ONNX? I am trying to use this method on Python 3.7:
import torch

model = torch.load("./yolov7x.pt")

#torch.onnx.export(model, "yolo_v7x.onnx")

Even with the commented last line in the 3 lines of code, loading this errors out:
Traceback (most recent call last):
  File "C:\Users\convert_onx.py", line 5, in <module>
    model = torch.load("./yolov7x.pt")
  File "C:\Users\Python37\lib\site-packages\torch\serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\Python37\lib\site-packages\torch\serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'models'

This is the git repo I am working with the Yolo Model 7x:

Ultimate use case is to use this model on Intel's Open VINO toolkit that requires PyTorch models to be converted to ONYX.

Output:
When you are loading the pickled model the source tree must match the one that used when the model was saved. So
ModuleNotFoundError: No module named 'models'

expects this directory to be in your python path: https://github.com/WongKinYiu/yolov7/tree/main/models
To export to ONNX:

Clone the repo https://github.com/WongKinYiu/yolov7

git clone https://github.com/WongKinYiu/yolov7

Set the correct path to it.

import sys
sys.path.insert(0, './yolov7')

or you can set PYTHONPATH environment variable

Also you may need to have specific torch version. I've checked and it seems torch==1.8.0 works fine

Example:
import torch
import sys

sys.path.insert(0, './yolov7')

device = torch.device('cpu')
model = torch.load('yolov7x.pt', map_location=device)['model'].float()
torch.onnx.export(model, torch.zeros((1, 3, 640, 640)), 'yolov7.onnx', opset_version=12)

After that the model was exported to ONNX (visualized with netron):

Usually it is better to save weights as state_dict and keep the source code that can reconstruct the torch.nn.Module so then you can safely use:
model.load_state_dict(torch.load('weights.pt'))


Instruction: Torch - How to calculate average of tensors with the same indexes

Input:
Suppose having two matrices: X(m, n) and index matrix I(m, 1). Every item in index matrix I_k represents the index of the kth element X_k in X.
And suppose the index is in the range of [0, 1, 2, ..., j-1]
I would like to calculate the average of tensors in X with the same index i and return a result matrix R(j, n).
For example,
X = [[1, 1, 1],
     [2, 2, 2],
     [3, 3, 3]]

I = [0, 0, 1]

The result matrix should be:
R = [[torch.mean(([1, 1, 1], [2, 2, 2]))],
     [torch.mean(([3, 3, 3]))]

which equals to:
R = [[1.5, 1.5, 1.5],
     [3, 3, 3]]

My current solution is to traverse through m, stack the tensors with the same index and perform torch.mean.
Is there a way avoiding traversing through m? It seems not elegant and rather time-consuming.

Output:
ret = torch.empty_like(X)
ret.scatter_reduce_(0, I.unsqueeze(-1).expand_as(X), X, "mean", include_self=False)

should do what you want.
Now, note that this is a fairly new method so it may not be particularly performant. If you bump into an issue with this method, you may be better off running scatter_add_ on the tensor X and a tensor of ones and then divide.
If you want to also have a smaller tensor as output, you may want to figure out how many indices and with that infer the size of out.


Instruction: How do I finetune a model while preserving layer names

Input:
When I fine tune a pretrained resnet152 model, I seem to lose all the named layers I’d like access to. I’ve include the simple fine tuned model code, and the print out of named layers of both pretrained and fine tuned.I'd like to maintain the layer names so I can visualize their output in a Class Activation Map.
Code
class ConvNet3(nn.Module):

def init(self):
    super().init()

    model = models.resnet152(pretrained=True)
    model.fc = nn.Linear(2048, 10)
    self.model = model
def forward(self, x):
    return self.model(x) # [batch_size, 10]

import torchvision.models as models
model = ConvNet3().eval()
print([n for n, _ in model.named_children()])

model = models.resnet152(pretrained=True).eval()
print([n for n, _ in model.named_children()])

Output
[‘model’]
[‘conv1’, ‘bn1’, ‘relu’, ‘maxpool’, ‘layer1’, ‘layer2’, ‘layer3’, ‘layer4’, ‘avgpool’, ‘fc’]

Output:
The layers are not lost, you are encapsulating the original Resnet model in your own class. If you use:
print([n for n, _ in model.model.named_children()])

since the Resnet model is stored under the model attribute of the ConvNet3 class.
Unless you need it for another reason, the wrapper class seems unnecessary, a simpler approach would be to do something as follows:
model = models.resnet152(pretrained=True)
model.fc = nn.Linear(2048,10)
model.eval()
print([n for n, _ in model.named_children()])


Instruction: torchaudio.io not properly using ffmpeg

Input:
I am following this tutorial about hardware-accelerated gpu encoding/decoding for PyTorch [https://pytorch.org/audio/main/hw_acceleration_tutorial.html], I am encountering an error with the following code:
import torch
import torchaudio

print(torch.__version__) # 1.14.0.dev20221013+cu116
print(torchaudio.__version__) # 0.13.0.dev20221013+cu116
print(torchaudio._extension._FFMPEG_INITIALIZED) # True

from torchaudio.io import StreamReader
local_src = "vid.mp4"
cuda_conf = {
    "decoder": "h264_cuvid",  # Use CUDA HW decoder
    "hw_accel": "cuda:0",  # Then keep the memory on CUDA:0
}

def decode_vid(src, config):
    frames = []
    s = StreamReader(src)
    s.add_video_stream(5, **config)
    for i, (chunk,) in enumerate(s.stream()):
        frames.append(chunk[0])

if __name__ == "__main__":
    vid = decode_vid(local_src, cuda_conf)

The error message (somewhat truncated) is:

File
"/home/james/PycharmProjects/AlphaPose/Spectronix/Early_Experiments/vid_gpu_decode.py",
line 23, in decode_vid
s.add_video_stream(5, **config)   File "/home/james/anaconda3/envs/alphapose/lib/python3.7/site-packages/torchaudio/io/_stream_reader.py",
line 624, in add_video_stream
hw_accel, RuntimeError: Unsupported codec: "h264_cuvid".

I have an RTX 3090 ti as my GPU, which does support the h264_cuvid decoder, and I have been able to decode a video on the command line running (taken from the tutorial linked above)
sudo ffmpeg -hide_banner -y -vsync 0 -hwaccel cuvid -hwaccel_output_format cuda -c:v h264_cuvid -i "https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4" -c:a copy -c:v h264_nvenc -b:v 5M test.mp4

So it seems torchaudio.io is not properly using ffmpeg. Any insights of how to fix this problem much appreciated. I'm using Ubuntu 22.04.

Output:
RuntimeError: Unsupported codec: "h264_cuvid".
The error happens here, and the StreamReader has not gotten to the point where it executes NVDEC-specific code, so this is generic issue with FFmpeg compatibility.
This suggests that the libavcodec found at runtime is not configured with h264_cuvid.
A possible explanation is that there are multiple installations of FFmpeg in your system and torchaudio is picking up the one without NVDEC support, while when you invoke ffmpeg command, the one with NVDEC support is loaded.
Perhaps you can check your system and see if there are multiple FFmpeg installations and remove the ones without NVDEC support?


Instruction: How to accelerate array_t construction in pybind11

Input:
I used C++ to call python with Pytorch.
C++ generate a vector and send to Python for neural network to inference.
But send the vector is a time consuming process.
A vector contain 500000 float consume 0.5 second turning to array_t.
Is there a faster way to transfer vector to array_t? Any help will be appreciate!
Here is the part of code:
int main(){
        float list[500000];
        std::vector<float> v(list, list+length);
        py::array_t<float> args = py::cast(v); //consume 0.5 second
        py::module_ nd_to_tensor = py::module_::import("inference");
        py::object result = nd_to_tensor.attr("inference")(args);
}

I also tried the second way as below, but it take 1.4 second in Python to make vector into tensor:
PYBIND11_MAKE_OPAQUE(std::vector<float>);
PYBIND11_EMBEDDED_MODULE(vectorbind, m) {
    m.doc() = "C++ type bindings created by py11bind";
    py::bind_vector<std::vector<float>>(m, "Vector");
}

int main(){
    std::vector<float> v(list, list+length);
    py::module_ nd_to_tensor = py::module_::import("inference");
    py::object result = nd_to_tensor.attr("inference")(&v);
}

Here is Python code:
def inference():
    tensor = torch.Tensor(Vector)

Output:
Problem solved
py::array_t<float> args = py::array_t<float>({length}, {4}, &list[0]);

Directly init the array_t will be the best way


Instruction: CUDA 11.6 not compatible with PyTorch 1.12.1

Input:
The PyTorch website says that PyTorch 1.12.1 is compatible with CUDA 11.6, but I get the following error:
NVIDIA GeForce RTX 3060 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

I am using a laptop RTX 3060 and Poetry as my package manager in Python.

>>> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

>>> poetry show
certifi               2022.9.24 Python package for providing Mozilla's CA Bundle.
charset-normalizer    2.1.1     The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
idna                  3.4       Internationalized Domain Names in Applications (IDNA)
numpy                 1.23.4    NumPy is the fundamental package for array computing with Python.
opencv-contrib-python 4.6.0.66  Wrapper package for OpenCV python bindings.
opencv-python         4.6.0.66  Wrapper package for OpenCV python bindings.
pillow                9.2.0     Python Imaging Library (Fork)
requests              2.28.1    Python HTTP for Humans.
torch                 1.12.1    Tensors and Dynamic neural networks in Python with strong GPU acceleration
torchvision           0.13.1    image and video datasets and models for torch deep learning
typing-extensions     4.4.0     Backported and Experimental Type Hints for Python 3.7+
urllib3               1.26.12   HTTP library with thread-safe connection pooling, file post, and more.

What am I missing here? Is this a PyTorch <> CUDA issue or a CUDA <> GPU issue?

Output:
NVIDIA GeForce RTX 3060 Laptop GPU with CUDA capability sm_86 is not
compatible with the current PyTorch installation. The current PyTorch
install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

The build of PyTorch which you have installed doesn't have binary support for your GPU. This is because whoever built the PyTorch you are using chose to build it like that. This isn't a question of CUDA versions or PyTorch versions. It just that many frameworks are built with a limited range of binary architectures in order to keep the size of the packages they distribute small.
NVIDIA provide a method to support forward compatible architectures running older code through JIT recompilation at runtime. Unfortunately the standard PyTorch build system doesn't use it in order to save space in their build distributions, so that cannot help you in this situation.
Your only solution is to either source another build with the appropriate binary support for your GPU included.


Instruction: Nearest Neighbour difference in Numpy/PyTorch

Input:
I need a to write a custom loss function in PyTorch but owing to PyTorch's similarity to NumPy, a Numpy based solution will also work.
I have two tensors(Numpy arrays) p and q of shape (b,...). For each batch element in p I wish to compute the minimum difference w.r.t any of the batch element of q. Sample code is given below:
loss = 0
for outer in range(b):
   tmp_min = 1e+5
   for inner in range(b):
      tmp_loss = torch.abs(p[outer,...] - q[inner,...]) # np.abs(p[outer,...] - q[inner,...])
      if tmp_loss<tmp_min:
          tmp_min = tmp_loss
   loss = loss + tmp_min

Since I will have to compute this loss many many times, is there a way to do it without FOR and IF statements?
Finally, I wish to compute this loss in both directions. So, is there any alternative other than repeating the above code with p and q swapped?

Output:
You can add a singleton dimension to p to leverage broadcasting when doing p-q: (p[:, None, ...] - q[:, ...]).shape == (B, B, ...).
It would work as follows:
def loss_vec(p, q):
    B = p.shape[0]
    assert q.shape[0] == B
    p = p.reshape(B, -1)
    q = q.reshape(B, -1)
    return (p[:, None, :] - q).abs().sum(axis=-1).min(axis=-1).values.sum()

def loss_op(p, q):
    """OP solution as oneliner"""
    return torch.tensor([min([torch.abs(x - y).sum() for y in q]) for x in p]).sum()

B, K, M, N = 11, 3, 5, 7
p = torch.rand(B, K, M, N)
q = torch.rand(B, K, M, N)

value_op_pq = loss_op(p, q)
value_vec_pq = loss_vec(p, q)

assert value_op_pq == value_vec_pq

To compute in both directions, just change the axis=... when computing the min:
def loss_vec_bi(p, q):
    """Returns the loss in both directions"""
    B = p.shape[0]
    assert q.shape[0] == B
    p = p.reshape(B, -1)
    q = q.reshape(B, -1)
    losses = (p[:, None, :] - q).abs().sum(axis=-1)
    return losses.min(axis=-1).values.sum(), losses.min(axis=0).values.sum()

value_op_qp = loss_op(q, p)
value_vec_pq, value_vec_qp = loss_vec_bi(p, q)

assert value_op_pq == value_vec_pq
assert value_op_qp == value_vec_qp


Instruction: How can I make torch.cat faster?

Input:
Below is a simplified version of what I want to do:
import torch
import time

# Create dummy tensors and save them in my_list
my_list = [[]] * 100
for i in range(len(my_list)):
    my_list[i] = torch.randint(0, 1000000000, (100000, 256))
concat_list = torch.tensor([])

# I want to concat two consecutive tensors in my_list
tic = time.time()
for i in range(0, len(my_list), 2):
    concat_list = torch.cat((concat_list, my_list[i]))
    concat_list = torch.cat((concat_list, my_list[i+1]))
    # Do some work at CPU with concat_list
    concat_list = torch.tensor([]) # Empty concat_list
print('time: ', time.time() - tic) # It takes 3.5 seconds in my environment

Is there any way to make above tensor concatenation faster?
I tried to send my_list[i], my_list[i+1], and concat_list to GPU and do the torch.cat function in the device, but I then have to send concat_list back to CPU to do "some work" that I've written above. This takes more time due to frequent GPU-CPU data transfer.
I've also tested converting tensors to lists to do the concatenation with basic Python lists, but this approach was way slower than a simple torch.cat approach.
I've heard that using DataLoader with customized collate_fn can enable concatenation, but I don't know how to implement it.
Is there any faster method possible?

Output:
Your code takes around 11 seconds on my PC. The following takes 4.1 seconds:
# Create dummy tensors and save them in my_list
my_list = [[]] * 100
for i in range(len(my_list)):
    my_list[i] = torch.randint(0, 1000000000, (100000, 256))
tic = time.time()
my_list = torch.stack(my_list)

# I want to concat two consecutive tensors in my_list
for i in range(0, len(my_list), 2):
    concat_list = my_list[i:i+2]
    # Do some work at CPU with concat_list
print('time: ', time.time() - tic) # It takes 3.5 seconds in my environment


Instruction: How can I launch chenrocks/uniter with NVIDIA GeForce RTX 3090 GPU?

Input:
docker run --gpus '"'device=$CUDA_VISIBLE_DEVICES'"' --ipc=host --rm -it \
    --mount src=$(pwd),dst=/src,type=bind \
    --mount src=$OUTPUT,dst=/storage,type=bind \
    --mount src=$PRETRAIN_DIR,dst=/pretrain,type=bind,readonly \
    --mount src=$TXT_DB,dst=/txt,type=bind,readonly \
    --mount src=$IMG_DIR,dst=/img,type=bind,readonly \
    -e NVIDIA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    -w /src chenrocks/uniter

When I run this file, it prints error
NVIDIA Release 19.05 (build 6411784) PyTorch Version 1.1.0a0+828a6a3

...
WARNING: Detected NVIDIA NVIDIA GeForce RTX 3090 GPU, which is not yet supported in this version of the container
ERROR: No supported GPU(s) detected to run this container

it doesn't fit with my NVIDIA GeForce RTX 3090 GPU
so I want to change version to 22.05
but when I run this,
docker run --gpus '"'device=$CUDA_VISIBLE_DEVICES'"' --ipc=host --rm
-it nvcr.io/nvidia/pytorch:22.05-py3 \
    --mount src=$(pwd),dst=/src,type=bind \
    --mount src=$OUTPUT,dst=/storage,type=bind \
    --mount src=$PRETRAIN_DIR,dst=/pretrain,type=bind,readonly \
    --mount src=$TXT_DB,dst=/txt,type=bind,readonly \
    --mount src=$IMG_DIR,dst=/img,type=bind,readonly \
    -e NVIDIA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    -w /src chenrocks/uniter

it prints error
/opt/nvidia/nvidia_entrypoint.sh: line 49: exec: --: invalid option
exec: usage: exec [-cl] [-a name] [command [arguments ...]] [redirection ...]

I'd really appreciate it if you could tell me how to change the version.

Output:
Your second second docker run command specifies 2 images:
docker run ... nvcr.io/nvidia/pytorch:22.05-py3 chenrocks/uniter

You can only pass one.

Also note the general format of the docker run command:
$ docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

UPDATE

But if the docker image's nvidia/pytorch version doesn't fit my GPU, can I not use that docker image? Or is there something I can do?

You could try editing the Dockerfile of the referenced project to build a custom docker image.
Something like this:
git clone https://github.com/ChenRocks/UNITER.git
cd UNITER
# replace the first line of the Dockerfile with:
# FROM nvcr.io/nvidia/pytorch:22.05-py3
docker build .
# ...
# Successfully built <image_id>

Then simply edit your docker run command to use your custom built image:
docker run ... <image_id>

It looks like that at least one other person had a similar issue. Unfortunately the project is not actively maintained so it's difficult to get any kind of support when trying to make it work with the latest hardware.


Instruction: PackagesNotFoundError When Trying to Install intel_extension_for_pytorch

Input:
I am trying to conda install intel_extension_for_pytorch but I keep getting the following error in the command line:
PackagesNotFoundError: The following packages are not available from current channels:

intel_extension_for_pytorch

this is the command that I am using
conda install intel_extension_for_pytorch
edit:
System Info:
Microsoft Windows [Version 10.0.19044.2006]
Processor   11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz, 2995 Mhz, 4 Core(s), 8 Logical Processor(s)

Output:
Currently, the Intel Extension for PyTorch is only supported by Linux OS. Try on a recent Linux version, it should work there.
Check out the docs for more info: https://www.intel.com/content/www/us/en/developer/tools/oneapi/extension-for-pytorch.html


Instruction: How do you specify the bfloat16 mixed precision with the Intel Extension for PyTorch?

Input:
I would like to know how to use mixed precision with PyTorch and Intel Extension for PyTorch.
I have tried to look at the documentation on their GitHub, but I can't find anything that specifies how to go from fp32 to blfoat16.

Output:
The IPEX GitHub might not be the best place to look for API documentation. I would try and use the PyTorch IPEX page, which includes examples of API applications.
This would be an example of how to use fp32
model, optimizer = ipex.optimize(model, optimizer, dtype=torch.float32)
This would be an example of how to use bfloat16
model, optimizer = ipex.optimize(model, optimizer, dtype=torch.bfloat16)