Instruction: My training and validation loss suddenly increased in power of 3

Input:
train function
def train(model, iterator, optimizer, criterion, clip):
    model.train()
    epoch_loss = 0
    for i, batch in enumerate(iterator):
        optimizer.zero_grad()
        output = model(batch.text)
        loss = criterion(output, torch.unsqueeze(batch.labels, 1))
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
        optimizer.step()
        epoch_loss += loss.item()
    return epoch_loss / len(iterator)

main_script
def main(
        train_file,
        test_file,
        config_file,
        checkpoint_path,
        best_model_path
    ):
    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    with open(config_file, 'r') as j:
        config = json.loads(j.read())

    for k,v in config['model'].items():
        v = float(v)
        if v < 1.0:
            config['model'][k] = float(v)
        else:
            config['model'][k] = int(v)

    for k,v in config['training'].items():
        v = float(v)
        if v < 1.0:
            config['training'][k] = float(v)
        else:
            config['training'][k] = int(v)

    train_itr, val_itr, test_itr, vocab_size = data_pipeline(
        train_file,
        test_file,
        config['training']['max_vocab'],
        config['training']['min_freq'],
        config['training']['batch_size'],
        device
    )

    model = CNNNLPModel(
        vocab_size,
        config['model']['emb_dim'],
        config['model']['hid_dim'],
        config['model']['model_layer'],
        config['model']['model_kernel_size'],
        config['model']['model_dropout'],
        device
    )
    optimizer = optim.Adam(model.parameters())
    criterion = nn.CrossEntropyLoss()
    num_epochs = config['training']['n_epoch']
    clip = config['training']['clip']
    is_best = False
    best_valid_loss = float('inf')
    model = model.to(device)
    for epoch in tqdm(range(num_epochs)):

        train_loss = train(model, train_itr, optimizer, criterion, clip)
        valid_loss = evaluate(model, val_itr, criterion)

        if (epoch + 1) % 2 == 0:
            print("training loss {}, validation_loss{}".format(train_loss,valid_loss))

I was training a Convolution Neural Network for binary Text classification. Given a sentence, it detects its a hate speech or not. Training loss and validation loss was fine till 5 epoch after that suddenly the training loss and validation loss shot up suddenly from 0.2 to 10,000.
What could be the reason for such huge increase is loss suddenly?

Output:
Default learning rate of Adam is 0.001, which, depending on task, might be too high.
It looks like instead of converging your neural network became divergent (it left the previous ~0.2 loss minima and fell into different region).
Lowering your learning rate at some point (after 50% or 70% percent of training) would probably fix the issue.
Usually people divide the learning rate by 10 (0.0001 in your case) or by half (0.0005 in your case). Try with dividing by half and see if the issue persist, in general you would want to keep your learning rate as high as possible until divergence occurs as is probably the case here.
This is what schedulers are for (gamma specifies learning rate multiplier, might want to change that to 0.5 first).
One can think of lower learning rate phase as fine-tuning already found solution (placing weights in better region of the loss valley) and might require some patience.


Instruction: Every one of my genetic algorithm models mutates in the same way

Input:
I've been working on a genetic algorithm, where I'm trying to take the best model of my population (based on score) and make the entire population into that model. I have done this successfully, but when I try to mutate each model separately, all the models mutate to the same parameters. I know this is because I use an object and just clone it into a list, but I don't know what to change so that it doesn't work this way. Below I've made a reproducible example to be run in Python 3.9. I know the code isn't particularly small, but this is as small as I can make it. Thanks in advance, any help is appreciated.
import torch
import torch.nn as nn

torch.manual_seed(0) #Reproducibility

population_size = 3 #Defining the size of my population
population = [nn.Linear(1,1) for _ in range(population_size)] #Initializing the population
input = torch.rand(1)# Creating dummy input
def mutate(m): #Function to mutate a model
    if type(m) == nn.Linear:
        m.weight = nn.Parameter(m.weight+torch.randn(m.weight.shape))
        m.bias = nn.Parameter(m.bias+torch.randn(m.bias.shape))

for i in population:
    print (i(input))

population = [x.apply(mutate) for x in population]

print ('\n')

for i in population:
    print (i(input))
#The above works as expected

#I want to fill my entire population with that model.
#I've been filling the population with the best model by doing the following:

best_model = population[0] #Say the first model in the list was the best performing one

population = [best_model for _ in range(population_size)] #This is the line I think needs to change, I just don't know what to change it to.

#This does fill the population with my best model, but when I try to mutate it, every model is mutated to the same parameters

population = [x.apply(mutate) for x in population] #I know this is because I am using best_model while replacing the population, but I don't know how else to copy the model

for i in population:
    print (i(input)) #Just to show that the population all gives the same result

Output:
You can make a deep copy of the model. Make sure to import copy, and then change
population = [best_model for _ in range(population_size)]

to
population = [copy.deepcopy(best_model) for _ in range(population_size)]


Instruction: Deployment with customer handler on Google Cloud Vertex AI

Input:
I'm trying to deploy a TorchServe instance on Google Vertex AI platform but as per their documentation (https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements#response_requirements), it requires the responses to be of the following shape:
{
  "predictions": PREDICTIONS
}

Where PREDICTIONS is an array of JSON values representing the predictions that your container has generated.
Unfortunately, when I try to return such a shape in the postprocess() method of my custom handler, as such:
def postprocess(self, data):
    return {
        "predictions": data
    }

TorchServe returns:
{
  "code": 503,
  "type": "InternalServerException",
  "message": "Invalid model predict output"
}

Please note that data is a list of lists, for example: [[1, 2, 1], [2, 3, 3]]. (Basically, I am generating embeddings from sentences)
Now if I simply return data (and not a Python dictionary), it works with TorchServe but when I deploy the container on Vertex AI, it returns the following error:  ModelNotFoundException. I assumed Vertex AI throws this error since the return shape does not match what's expected (c.f. documentation).
Did anybody successfully manage to deploy a TorchServe instance with custom handler on Vertex AI?

Output: Actually, making sure that the TorchServe processes correctly the input dictionary (instances) solved the issue. It seems like what's on the article did not work for me.


Instruction: Understanding content of pretrained model

Input:
from torchvision import models

model = models.vgg16(pretrained=True).features
for i, module in enumerate(model.modules()):
    print('-' * 60)
    print(type(module))
    print(module)

What is 'module' in model.modules() (modules() Returns an iterator over all modules in the network.)? Why torch.nn.modules.container.Sequential is stored along with single layers?
Output:
------------------------------------------------------------
<class 'torch.nn.modules.container.Sequential'>
Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (3): ReLU(inplace=True)
  (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (6): ReLU(inplace=True)
  (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (8): ReLU(inplace=True)
  (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (11): ReLU(inplace=True)
  (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (13): ReLU(inplace=True)
  (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (15): ReLU(inplace=True)
  (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (18): ReLU(inplace=True)
  (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (20): ReLU(inplace=True)
  (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (22): ReLU(inplace=True)
  (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (25): ReLU(inplace=True)
  (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (27): ReLU(inplace=True)
  (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (29): ReLU(inplace=True)
  (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.pooling.MaxPool2d'>
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.pooling.MaxPool2d'>
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.pooling.MaxPool2d'>
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.pooling.MaxPool2d'>
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.conv.Conv2d'>
Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
------------------------------------------------------------
<class 'torch.nn.modules.activation.ReLU'>
ReLU(inplace=True)
------------------------------------------------------------
<class 'torch.nn.modules.pooling.MaxPool2d'>
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)

Output:
nn.Module.modules will return all modules contained inside the model recursively, that includes parent submodules and children submodules that they may have of their own. You can instead use nn.Module.children to get the direct child submodules:
for child in module.children():
    print(child)

If you want to investigate with recursion you could do:
def print_children(module, i=0):
    if len(list(module.modules())) == 1:
        return print(' '*i, module)
    for child in module.children():
        print_children(child, i+2)


Instruction: How to delete / replace layer in existing model?

Input:
How to delete some layers from pretrained network (for example remove single ReLU activation layer)?
How to replace some layers by type in pretrained network (for example replace MaxPool2d with AvrPool)?

Output:
Assuming you know the structure of your model, you can:
>>> model = torchvision.models(pretrained=True)

Select a submodule and interact with it as you would with any other nn.Module. This will depend on your model's implementation. For example, submodule are often accessible via attributes (e.g. model.features), however this is not always the case, for instance nn.Sequential use indices: model.features[18] to select one of the relu activations. Also do note: not all layers are registered inside the nn.Module, non-parametric functions such as most activation functions can be applied via the functional approach directly in the forward of the module.

For a given nn.Module m you can extract its layer name by using type(m).__name__. A canonical approach is to filter the layers of model.modules and only keep the max pool layers, then replace those with average pool layers:
>>> maxpools = [k for k, m in model.named_modules()
...                if type(m).__name__ == 'MaxPool2d']
['features.4', 'features.9', 'features.16', 'features.23', 'features.30']

We can extract the parent module name for each of those layers:
>>> maxpools = [k.split('.') for k, m in model.named_modules()
...                if type(m).__name__ == 'MaxPool2d']
[['features', '4'],
 ['features', '9'],
 ['features', '16'],
 ['features', '23'],
 ['features', '30']]

Here they all come from the same parent module model.features. Finally, we can fetch the layer reference in order to overwrite their value:
>>> for *parent, k in maxpools:
...     model.get_submodule('.'.join(parent))[int(k)] = nn.AvgPool2d(2,2)

Resulting in:
VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): AvgPool2d(kernel_size=2, stride=2, padding=0)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): AvgPool2d(kernel_size=2, stride=2, padding=0)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): AvgPool2d(kernel_size=2, stride=2, padding=0)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): AvgPool2d(kernel_size=2, stride=2, padding=0)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): AvgPool2d(kernel_size=2, stride=2, padding=0)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)


Instruction: What is default download path for Pytorch models on MacOS?

Input:
What is default download path for Pytorch models (from torchvision) on MacOS?
I can't see anything in home dir ls -la ~/ | grep torch.

Output: You should have the path it displayed when you first download the pre-trained weights for that model, something like this:


Instruction: create Pytorch dataset without downloading

Input:
Quite new here, hope this isn't to much of simple question.
I am trying to get into deep learning, starting with pytorch.
Thing is, all tutorials I see use downloading in order to build a data set for training and testing.  Unfortunately, I am restricted with my internet connection, meaning I can't download directly from the web.
What I can do is download files and transfer them into my computer.
So my question is - in order to use a previously downloaded dataset in pytorch:

Where should I store it?
How do I create a dataset once I have the files on my computer?

If any other information seems important to you I'll be glad to hear, I'm a serious newbie..
Thank you very much!

Output:
Not sure which dataset you are asking for. Since you mentioned the "tutorials", I am guessing that you just want to work with a dataset that comes with some library in PyTorch ecosystem (e.g. torchvision).
Dataset classess in PyTorch ecosystem have a "root" argument in their constrcutors.
mnist = MNIST(root='/some/path', download=True)

You can simply download it on some machine with internet, and transfer the contents of the folder /some/path to your machine at /my/machine/path. And simply point to it and turn off the download
# on your machine without internet
mnist = MNIST(root='/my/machine/path', download=False)

This should work.


Instruction: Why is my Transformer implementation losing to a BiLSTM?

Input:
I am dealing with a sequence tagging problem and I am using a single Transformer Encoder to obtain logits from each element of the sequence. Having experimented both with Transformer and BiLSTM it looks like in my case BiLSTM is working better, so I was wondering if maybe it is because my Transformer implementation has some problem... Below is my implementation of the Transformer Encoder and related functions for creating padding mask and positional embeddings:
def create_mask(src, lengths):
    """Create a mask hiding future tokens
    Parameters:
        src (tensor): the source tensor having shape [batch_size, number_of_steps, features_dimensions]
        length (list): a list of integers representing the length (i.e. number_of_steps) of each sample in the batch."""
    mask = []
    max_len = src.shape[1]
    for index, i in enumerate(src):
        # The mask consists in tensors having false at the step number that doesn't need to be hidden and true otherwise
        mask.append([False if (i+1)>lengths[index] else True for i in range(max_len)])
    return torch.tensor(mask)

class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000, device = 'cpu'):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)
        self.device = device
        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(1, max_len, d_model)
        pe[0, :, 0::2] = torch.sin(position * div_term)
        pe[0, :, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:, :x.size(1), :].to(self.device)
        return self.dropout(x)

class Transformer(nn.Module):
    """Class implementing transformer ecnoder, partially based on
    https://pytorch.org/tutorials/beginner/transformer_tutorial.html"""
    def __init__(self, in_dim, h_dim, n_heads, n_layers, dropout=0.2, drop_out = 0.0, batch_first = True, device = 'cpu', positional_encoding = True):
        super(Transformer, self).__init__()
        self.model_type = 'Transformer'
        self.pos_encoder = PositionalEncoding(in_dim, dropout, device = device)
        encoder_layers = nn.TransformerEncoderLayer(in_dim, n_heads, h_dim, dropout)
        self.transformer_encoder = nn.TransformerEncoder(encoder_layers, n_layers, norm=nn.LayerNorm(in_dim))
        self.in_dim = in_dim
        self.drop_out = drop_out
        self.positional_encoding = positional_encoding

    def forward(self, src, mask = None, line_len=None):
        src = src * math.sqrt(self.in_dim)
        if self.positional_encoding:
            src = self.pos_encoder(src)
        if line_len is not None and mask is None:
            mask = create_mask(src, line_len)
        else:
            mask = None
        output = self.transformer_encoder(src, src_key_padding_mask = mask)
        if self.drop_out:
            output = F.dropout(output, p = self.drop_out)
        return src, output

As it can be seen, the above network outputs the hidden states and then I pass them into an additional linear layer and train with a CrossEntropy loss over two classes and Adam optimizer. I have tried multiple combinations of hyperparameters but the BiLSTM still performs better. Can anyone spot anything off in my Transformer or suggest why I experience such a counterintuitive result?

Output:
This may be surprising, but Transformers don't always beat LSTMs. For example, Language Models with Transformers states:

Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling.

If you run the Transformer tutorial code itself (on which your code is based), you'll also see LSTM do better there. See this thread on stats.SE for more discussion on this topic (disclaimer: both the question and the answer there are mine)


Instruction: Multi-label classification return more than 1 class in each label

Input:
How to train a Multi-label classification model when each label should return more than 1 class?
Example:
Image classification have 2 label: style with 4 classes and layout with 5 classes.
An image in list should return 2 style and 3 layout like [1 0 1 0] [1 1 0 0 1]
My sample net:
class MyModel(nn.Module):
def __init__(self, n__classes=32):
    super().__init__()
    self.base_model = models.resnet50(pretrained=True).to(device)
    last_channel = self.base_model.fc.in_features
    self.base_model.fc = nn.Sequential()

    self.layout = nn.Sequential(
        nn.Dropout(0.2),
        nn.Linear(last_channel, n_classes_layout),
        nn.Sigmoid()
    )
    self.style = nn.Sequential(
        nn.Dropout(0.2),
        nn.Linear(last_channel, n_classes_style),
        nn.Sigmoid()
    )
def forward(self, x):
    base = self.base_model(x)
    return self.layout(base), self.style(base)

def loss_fn(outputs, targets):
    o1, o2 = outputs
    t1, t2 = targets

Output:
I am not sure what you referring label to but it seems you have a multi output model predicting on one hand style, and on the other layout. So I am assuming you are dealing with a multi-task network with two "independent" outputs which are supervised separately with two-loss terms.
Consider one of those two: you can consider this multi-label classification task as outputting n values each one estimating the probability that the corresponding label is present in the input. In your first task, you have four classes, on the second task, you have five.
Unlike a single-label classification task, here you wish to output one activation per label: you can use a sigmoid activation per logit. Then apply the Binary Cross Entropy loss on each of those outputs. In PyTorch, you can use BCELoss, which handles multi-dimensional tensors. In practice you could concatenate the two outputs from the two tasks, do the same with the target labels and call the criterion once.


Instruction: Pytorch: "KeyError: Caught KeyError in DataLoader worker process 0."

Input:
Problem Description:
I am trying to load image data using Pytorch custom dataset. I did a little dive deep and found that my images set consist of 2 types of shape (512,512,3) and (1024,1024) . My assumption is, because of the above reason, it is throwing the below error.
Note: The code is able to read some of the images but, it is throwing the below error message for few of them. This was the reason to do a little EDA on the image data and found that there were 2 different shapes of images in the dataset.
Q1. How to preprocess such image data for training?
Q2. Is there any other reasons why I might be seeing the below error message?
Error message:
KeyError                                  Traceback (most recent call last)
<ipython-input-163-aa3385de8026> in <module>
----> 1 train_features, train_labels = next(iter(train_dataloader))
  2 print(f"Feature batch shape: {train_features.size()}")
  3 print(f"Labels batch shape: {train_labels.size()}")
  4 img = train_features[0].squeeze()
  5 label = train_labels[0]

 ~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils  /data/dataloader.py in __next__(self)
519             if self._sampler_iter is None:
520                 self._reset()
521             data = self._next_data()
522             self._num_yielded += 1
523             if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
1201             else:
1202                 del self._task_info[idx]
1203                 return self._process_data(data)
1204
1205     def _try_put_index(self):

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
1227         self._try_put_index()
1228         if isinstance(data, ExceptionWrapper):
1229             data.reraise()
1230         return data
1231

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
423             # have message field
424             raise self.exc_type(message=msg)
425         raise self.exc_type(msg)
426
427

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pandas  /core/indexes/base.py", line 2898, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1032, in    pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1039, in   pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 16481

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "<ipython-input-161-f38b78d77dcb>", line 19, in __getitem__
img_path =os.path.join(self.img_dir,self.image_ids[idx])
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pandas/core/series.py", line 882, in __getitem__
return self._get_value(key)
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pandas/core/series.py", line 990, in _get_value
loc = self.index.get_loc(label)
File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2900, in get_loc
raise KeyError(key) from err
KeyError: 16481

Code:
from torchvision.io import read_image
import torch
from torchvision import transforms
from sklearn.model_selection import train_test_split
from torch.utils.data import Dataset

class CustomImageDataset(Dataset):
     # init
    def __init__(self,dataset,transforms=None,target_transforms=None):
        #self.train_data = pd.read_csv("Data/train_data.csv")
        self.image_ids = dataset.image_id
        self.image_labels = dataset.label
        self.img_dir = 'Data/images'
        self.transforms = transforms
        self.target_transforms = target_transforms
# len
    def __len__(self):
        return len(self.image_ids)
# getitem
    def __getitem__(self,idx):
        # image path
        img_path =os.path.join(self.img_dir,self.image_ids[idx])
        # image
        image = read_image(img_path)
        label = self.image_labels[idx]
    # transform image
        if self.transforms:
             image = self.transforms(image)
    # transform target
        if self.target_transforms:
             label = self.target_transforms(label)
    return image, label

Code: train_data is the pandas object of the csv file which has the image id, labesl information.
  from sklearn.model_selection import train_test_split
  X_train, X_test = train_test_split(train_data, test_size=0.1, random_state=42)
  train_df = CustomImageDataset(X_train)
  train_dataloader = torch.utils.data.DataLoader(
        train_df,
        batch_size=64,
        num_workers=1,
        shuffle=True,
    )

Output:
found the issue with the code.
Pytorch Custom Dataloader function "getitem" uses idx to retrieve data and my guess is, it know the range of idx from len function, ex: 0, till len(rows in dataset).
In my case, I already had a panda dataset (train_data) with idx as one of the column. When I randomly split it into X_train and X_test, few of the data rows were moved to X_test along with the idx.
Now, when I send X_train to the custom dataloader, it is trying to get row's image_id with an idx and that idx just happens to be in X_test dataset. This lead to error as keyerror: 16481 i.e row with idx=16481 is not present in the X_train dataset. It was moved to X_test during split.
phew...


Instruction: Equivalent of torch EmbeddingBag

Input:
Torch claim that EmbeddingBag with mode="sum" is equivalent to Embedding followed by torch.sum(dim=1), but how can I implement it in detail? Let's say we have
"EE = nn.EmbeddingBag(n, m, mode="sum", sparse=True)", how can we replace the "nn.EmbeddingBag" by "nn.Embeeding" and "torch.sum" equivalently? Many thanks

Output:
Consider the following example where all four implementations yield the same result:

nn.EmbeddingBag:
>>> embedding_sum = nn.EmbeddingBag(10, 3, mode='sum')
>>> embedding_sum(input, torch.zeros(1).long())

nn.functional.embedding_bag:
>>> F.embedding_bag(input, embedding_sum.weight, torch.zeros(1).long(), mode='sum')

nn.Embedding:
>>> embedding = nn.Embedding(10, 3)
>>> embedding.weight = embedding_sum.weight
>>> embedding(input).sum(0)

nn.functional.embedding:
>>> F.embedding(input, embedding_sum.weight).sum(0)


Instruction: Dataloader throwing error TypeError: new(): data must be a sequence (got map)

Input:
I am trying to implement bidirectional LSTM on time series data. The main file calls the dataloader to load the data for the model.
Main.py
import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR

import numpy as np

import time
import utils
import models
from models import rits_i
from models import brits_i
from models import rits
from models import brits

import argparse
import data_loader
import pandas as pd
import ujson as json

from sklearn import metrics

from ipdb import set_trace

parser = argparse.ArgumentParser()
parser.add_argument('--epochs', type = int, default = 1000)
parser.add_argument('--batch_size', type = int, default = 32)
parser.add_argument('--model', type = str)
args = parser.parse_args()

def train(model):
    optimizer = optim.Adam(model.parameters(), lr = 1e-3)

    data_iter = data_loader.get_loader(batch_size = args.batch_size)

    for epoch in range(args.epochs):
        model.train()

        run_loss = 0.0

        for idx, data in enumerate(data_iter):
            data = utils.to_var(data)
            ret = model.run_on_batch(data, optimizer)

            run_loss += ret['loss'].data[0]

            print('\r Progress epoch {}, {:.2f}%, average loss {}'.format(epoch, (idx + 1) * 100.0 / len(data_iter), run_loss / (idx + 1.0)),)

        if epoch % 1 == 0:
            evaluate(model, data_iter)

def evaluate(model, val_iter):
    model.eval()

    labels = []
    preds = []

    evals = []
    imputations = []

    for idx, data in enumerate(val_iter):
        data = utils.to_var(data)
        ret = model.run_on_batch(data, None)

        pred = ret['predictions'].data.cpu().numpy()
        label = ret['labels'].data.cpu().numpy()
        is_train = ret['is_train'].data.cpu().numpy()

        eval_masks = ret['eval_masks'].data.cpu().numpy()
        eval_ = ret['evals'].data.cpu().numpy()
        imputation = ret['imputations'].data.cpu().numpy()

        evals += eval_[np.where(eval_masks == 1)].tolist()
        imputations += imputation[np.where(eval_masks == 1)].tolist()

        # collect test label & prediction
        pred = pred[np.where(is_train == 0)]
        label = label[np.where(is_train == 0)]

        labels += label.tolist()
        preds += pred.tolist()

    labels = np.asarray(labels).astype('int32')
    preds = np.asarray(preds)

    print('AUC {}'.format(metrics.roc_auc_score(labels, preds)))

    evals = np.asarray(evals)
    imputations = np.asarray(imputations)

    print('MAE', np.abs(evals - imputations).mean())
    print('MRE', np.abs(evals - imputations).sum() / np.abs(evals).sum())

def run():
    model = getattr(models, args.model).Model()

    if torch.cuda.is_available():
        model = model.cuda()

    train(model)

if __name__ == '__main__':
    run()

data_loader.py
import os
import time

import ujson as json
import numpy as np
import pandas as pd

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

class MySet(Dataset):
    def __init__(self):
        super(MySet, self).__init__()
        self.content = open('./json/json').readlines()

        indices = np.arange(len(self.content))
        val_indices = np.random.choice(indices, len(self.content) // 5)

        self.val_indices = set(val_indices.tolist())

    def __len__(self):
        return len(self.content)

    def __getitem__(self, idx):
        rec = json.loads(self.content[idx])
        if idx in self.val_indices:
            rec['is_train'] = 0
        else:
            rec['is_train'] = 1
        return rec

def collate_fn(recs):
    forward = map(lambda x: x['forward'], recs)
    backward = map(lambda x: x['backward'], recs)

    def to_tensor_dict(recs):
        values = torch.FloatTensor(map(lambda r: map(lambda x: x['values'], r), recs))
        masks = torch.FloatTensor(map(lambda r: map(lambda x: x['masks'], r), recs))
        deltas = torch.FloatTensor(map(lambda r: map(lambda x: x['deltas'], r), recs))
        forwards = torch.FloatTensor(map(lambda r: map(lambda x: x['forwards'], r), recs))

        evals = torch.FloatTensor(map(lambda r: map(lambda x: x['evals'], r), recs))
        eval_masks = torch.FloatTensor(map(lambda r: map(lambda x: x['eval_masks'], r), recs))

        return {'values': values, 'forwards': forwards, 'masks': masks, 'deltas': deltas, 'evals': evals, 'eval_masks': eval_masks}

    ret_dict = {'forward': to_tensor_dict(forward), 'backward': to_tensor_dict(backward)}

    ret_dict['labels'] = torch.FloatTensor(map(lambda x: x['label'], recs))
    ret_dict['is_train'] = torch.FloatTensor(map(lambda x: x['is_train'], recs))

    return ret_dict

def get_loader(batch_size = 64, shuffle = True):
    data_set = MySet()
    data_iter = DataLoader(dataset = data_set, \
                              batch_size = batch_size, \
                              num_workers = 4, \
                              shuffle = shuffle, \
                              pin_memory = True, \
                              collate_fn = collate_fn
    )

    return data_iter

but I am unable to resolve the error TypeError: new(): data must be a sequence (got map)
Following message is being received in the terminal:
C:\Users\ankit\anaconda3\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2021.2\plugins\python-ce\helpers\pydev\pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 61292 --file C:/Users/ankit/PycharmProjects/BRITS/main.py --epochs 1000 --batch_size 32 --model brits
Connected to pydev debugger (build 212.4746.96)
Traceback (most recent call last):
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 521, in __next__
    data = self._next_data()
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 1229, in _process_data
    data.reraise()
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\_utils.py", line 425, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\utils\data\_utils\worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "C:\Users\ankit\anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "C:\Users\ankit\PycharmProjects\BRITS\data_loader.py", line 48, in collate_fn
    ret_dict = {'forward': to_tensor_dict(forward), 'backward': to_tensor_dict(backward)}
  File "C:\Users\ankit\PycharmProjects\BRITS\data_loader.py", line 38, in to_tensor_dict
    values = torch.FloatTensor(map(lambda r: map(lambda x: x['values'], r), recs))
TypeError: new(): data must be a sequence (got map)

The input data is in the jason format (below is the partial data):
{"forward":[{"evals":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9056552797,0.0,0.0,0.0,-0.6644796629,0.0,0.0,0.0,0.0,0.0,0.2233221525,0.9872348884,1.5032897407,0.0,0.0,0.0,0.0,-0.1197213235,0.0,0.0,-1.2017103673,0.0,0.0,2.16039189,0.0],"deltas":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"forwards":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9056552797,0.0,0.0,0.0,-0.6644796629,0.0,0.0,0.0,0.0,0.0,0.2233221525,0.9872348884,1.5032897407,0.0,0.0,0.0,0.0,-0.1197213235,0.0,0.0,-1.2017103673,0.0,0.0,2.16039189,0.0],"masks":[0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,1,1,1,0,0,0,0,1,0,0,1,0,0,1,0],"values":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9056552797,0.0,0.0,0.0,-0.6644796629,0.0,0.0,0.0,0.0,0.0,0.2233221525,0.9872348884,1.5032897407,0.0,0.0,0.0,0.0,-0.1197213235,0.0,0.0,-1.2017103673,0.0,0.0,2.16039189,0.0],"eval_masks":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"evals":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.5117099666,0.0,0.0,0.0,0.0,0.0,0.2569139277,0.6703649195,0.8156657086,0.0,0.0,0.0,0.0,-0.3034796866,0.0,0.0,0.0,0.0,0.0,-0.5174302535,0.0],"deltas":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,0.0,1.0,1.0,1.0,1.0,1.0,0.0,0.0,0.0,1.0,1.0,1.0,1.0,0.0,1.0,1.0,1.0,1.0,1.0,0.0,1.0],"forwards":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9056552797,0.0,0.0,0.0,-1.5117099666,0.0,0.0,0.0,0.0,0.0,0.2569139277,0.6703649195,0.8156657086,0.0,0.0,0.0,0.0,-0.3034796866,0.0,0.0,-1.2017103673,0.0,0.0,-0.5174302535,0.0],"masks":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,1,1,0,0,0,0,1,0,0,0,0,0,1,0],"values":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.5117099666,0.0,0.0,0.0,0.0,0.0,0.2569139277,0.6703649195,0.8156657086,0.0,0.0,0.0,0.0,-0.3034796866,0.0,0.0,0.0,0.0,0.0,-0.5174302535,0.0],"eval_masks":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"evals":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.3987459261,0.0,0.0,0.0,0.0,0.0,-0.4149215779,-0.0992249514,0.1738832786,0.0,0.0,0.0,0.0,-0.1197213235,0.0,0.0,0.0,0.0,0.0,0.3156699689,0.0],"deltas":[2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,0.0,2.0,2.0,2.0,2.0,2.0,0.0,0.0,0.0,2.0,2.0,2.0,2.0,0.0,2.0,2.0,2.0,2.0,2.0,0.0,2.0]

I researched and got that it might be issue with python 3 but I ran on python 2 but still facing the same issue. Please help me resolve the issue.

Output:
I haven't looked or tried running all of your code, but at a glance, this line is clearly wrong
torch.FloatTensor(map(lambda r: map(lambda x: x['values'], r), recs))

In python3, map() returns a map object, not a list. FloatTensor(..) expects the later, i.e. a list. Just cast all your map objects with list()
torch.FloatTensor(list(map(lambda r: map(lambda x: x['values'], r), recs)))


Instruction: How to select half precision (BFLOAT16 vs FLOAT16) for your trained model?

Input:
how will you decide what precision works best for your inference model? Both BF16 and F16 takes two bytes but they use different number of bits for fraction and exponent.
Range will be different but I am trying to understand why one chose one over other.
Thank you
    |--------+------+----------+----------|
    | Format | Bits | Exponent | Fraction |
    |--------+------+----------+----------|
    | FP32   |   32 |        8 |       23 |
    | FP16   |   16 |        5 |       10 |
    | BF16   |   16 |        8 |        7 |
    |--------+------+----------+----------|

Range
bfloat16: ~1.18e-38 … ~3.40e38 with 3 significant decimal digits.
float16:  ~5.96e−8 (6.10e−5) … 65504 with 4 significant decimal digits precision.

Output:
bfloat16 is generally easier to use, because it works as a drop-in replacement for float32. If your code doesn't create nan/inf numbers or turn a non-0 into a 0 with float32, then it shouldn't do it with bfloat16 either, roughly speaking. So, if your hardware supports it, I'd pick that.
Check out AMP if you choose float16.


Instruction: Functionally is torch.multinomial the same as torch.distributions.categorical.Categorical?

Input: For example, if I provide a probability array of [0.5, 0.5], both functions will sample the index [0,1] with equal probability?

Output:
Yes:

[torch.distributions.categorical.Categorical()] is equivalent to the distribution that torch.multinomial() samples from.

https://pytorch.org/docs/stable/distributions.html#Categorical


Instruction: what is the meaning of 'per-layer learning rate' in Fast R-CNN paper?

Input:
I'm reading a paper about Fast-RCNN model.
In the paper section 2.3 part of 'SGD hyper-parameters', it said that All layers use a per-layer learning rate of 1 for weights and 2 for biases and a global learning rate of 0.001

Is 'per-layer learning rate' same as 'layer-specific learning rate' that give different learning rate by layers? If so, I can't understand how they('per-layer learning rate' and 'global learning rate') can be apply at the same time?

I found the example of 'layer-specific learning rate' in pytorch.
optim.SGD([
                {'params': model.some_layers.parameters()},
                {'params': model.some_layers.parameters(), 'lr': 1}
            ], lr=1e-3, momentum=0.9)

According to paper, Is this the correct approach?

Sorry for may English

Output:
The per-layer terminology in that paper is slightly ambiguous. They aren't referring to the layer-specific learning rates.

All layers use a per-layer learning rate of 1 for weights and 2 for biases and a global learning rate of 0.001.

The concerned statement is w.r.t. Caffe framework in which Fast R-CNN was originally written (github link).
They meant that they're setting the learning rate multiplier of weights and biases to be 1 and 2 respectively.
Check any prototxt file in the repo e.g. CaffeNet/train.prototxt.
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }

Thus, the effective learning rate is going to be base_lr*lr_mult, and here, the base learning rate is 0.001, which is defined in solver.prototxt.


Instruction: How to use SciBERT in the best manner?

Input:
I'm trying to use BERT models to do text classification. As the text is about scientific texts, I intend to use the SicBERT pre-trained model: https://github.com/allenai/scibert
I have faced several limitations which I want to know if there is any solutions for them:

When I want to do tokenization and batching, it only allows me to use max_length of <=512. Is there any way to use more tokens. Doen't this limitation of 512 mean that I am actually not using all the text information during training? Any solution to use all the text?

I have tried to use this pretrained library with other models such as DeBERTa or RoBERTa. But it doesn't let me. I has only worked with BERT. Is there anyway I can do that?

I know this is a general question, but any suggestion that I can improve my fine tuning (from data to hyper parameter, etc)? Currently, I'm getting ~75% accuracy. Thanks

Codes:
tokenizer = BertTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')

encoded_data_train = tokenizer.batch_encode_plus(
    df_train.text.values,
    add_special_tokens=True,
    return_attention_mask=True,
    padding=True,
    max_length=256
)

input_ids_train = encoded_data_train['input_ids']
attention_masks_train = encoded_data_train['attention_mask']
labels_train = torch.tensor(df_train.label.values)

dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)

dataloader_train = DataLoader(dataset_train,
                              sampler=RandomSampler(dataset_train),
                              batch_size=batch_size)

model = BertForSequenceClassification.from_pretrained('allenai/scibert_scivocab_uncased',
                                                      num_labels=len(labels),
                                                      output_attentions=False,
                                                      output_hidden_states=False)

epochs = 1

optimizer = AdamW(model.parameters(), lr=1e-5, eps=1e-8)

scheduler = get_linear_schedule_with_warmup(optimizer,
num_training_steps=len(dataloader_train)*epochs)

Output:
When I want to do tokenization and batching, it only allows me to use max_length of <=512. Is there any way to use more tokens. Doen't this limitation of 512 mean that I am actually not using all the text information during training? Any solution to use all the text?

Yes, you are not using the complete text. And this is one of the limitations of BERT and T5 models, which limit to using 512 and 1024 tokens resp. to the best of my knowledge.
I can suggest you to use Longformer or Bigbird or Reformer models, which can handle sequence lengths up to 16k, 4096, 64k tokens respectively. These are really good for processing longer texts like scientific documents.

I have tried to use this pretrained library with other models such as DeBERTa or RoBERTa. But it doesn't let me. I has only worked with BERT. Is there anyway I can do that?

SciBERT is actually a pre-trained BERT model.
See this issue for more details where they mention the feasibility of converting BERT to ROBERTa:
Since you're working with a BERT model that was pre-trained, you unfortunately won't be able to change the tokenizer now from a WordPiece (BERT) to a Byte-level BPE (RoBERTa).

I know this is a general question, but any suggestion that I can
improve my fine tuning (from data to hyper parameter, etc)? Currently,
I'm getting ~79% accuracy.

I would first try to tune the most important hyperparameter learning_rate. I would then explore different values for hyperparameters of AdamW optimizer and num_warmup_steps hyperparamter of the scheduler.


Instruction: Performance with IPEX

Input:
I tried linear regression & gradient descent in PyTorch. (https://www.analyticsvidhya.com/blog/2021/08/linear-regression-and-gradient-descent-in-pytorch/)
I've imported Ipex with the following command:
import intel_pytorch_extension as ipex

How can I convert the model to ipex? And how can I check that the performance has improved? Any help here?

Output:
After importing IPEX, use the below command while defining a model:
def model(X, IPEX=True):
return X @ w.t() + b

We have verified the code from the link you shared. You could replace model(X) with model(X, IPEX=True)
Performance with IPEX has improved drastically. Without IPEX, we could see some difference between Actual Target values & Model Predictions. Where as, with IPEX values has no difference.


Instruction: Replicate softmax of pytorch

Input:
I am trying to implement the softmax function in pytorch, but cant get the output of my implementation to match the output of pytorch's implementation.
I am trying to do this because I would like to go on to implement a masked softmax that would not include certain indices in the sum for the denominator, and set the output for those masked indexes.
I would like to calculate this for a matrix where each row in the output sums to 1. My current implentation is :
def my_softmax(x):
    exp = x.exp()
    return exp / exp.sum(1, keepdim=True)

However the output isnt the same for my implementation and pytorch's:
>>> t = torch.randn(3, 2)
>>> t
tensor([[-1.1881, -0.1085],
        [ 0.5825,  1.0719],
        [-0.5309, -1.3774]])
>>> my_softmax(t)
tensor([[0.2536, 0.7464],
        [0.3800, 0.6200],
        [0.6998, 0.3002]])
>>> t.softmax(1)
tensor([[0.2536, 0.7464],
        [0.3800, 0.6200],
        [0.6998, 0.3002]])
>>> my_softmax(t) == t.softmax(1)
tensor([[False,  True],
        [False, False],
        [ True,  True]])

Why do these different implementations produce different results?

Output:
This works
import torch

def my_softmax(x):

    means = torch.mean(x, 1, keepdim=True)[0]
    x_exp = torch.exp(x-means)
    x_exp_sum = torch.sum(x_exp, 1, keepdim=True)

    return x_exp/x_exp_sum

t = torch.randn(3, 2)

s1 = my_softmax(t)
s2 = t.softmax(1)

print(torch.allclose(s1, s2))
True

P.S. taken from discussion
https://discuss.pytorch.org/t/how-to-implement-the-exactly-same-softmax-as-f-softmax-by-pytorch/44263


Instruction: StyleGan2 ADA PyTorch Output Image Resolution Assertion Error

Input:
I am trying to produce an image using StyleGan2.
Executing this cell:
!python train.py --gpus=1 --cfg=$config --metrics=None --outdir=./results --data=$dataset_path --snap=$snapshot_count --resume=$resume_from --augpipe=$augs --initstrength=$aug_strength --gamma=$gamma_value --mirror=$mirror_x --mirrory=False --nkimg=$train_count

outputs the following:
Output directory:   ./results/00001-datasets-mirror-11gb-gpu-gamma50-bg-resumecustom
Training data:      /content/datasets.zip
Training duration:  25000 kimg
Number of GPUs:     1
Number of images:   608
Image resolution:   600
Conditional model:  False
Dataset x-flips:    True

Creating output directory...
Launching processes...
Loading training set...

Num images:  1216
Image shape: [3, 600, 600]
Label shape: [0]

then gives me the assertion error down below:
assert img_resolution >= 4 and img_resolution & (img_resolution - 1) == 0
Assertion Error:

I checked the source of the error.
But to be fair I didn't understand it.

Output:
I have found:
The issue is the input dimension of the image must be at least [3, 1024, 1024] mine was [3, 600, 600]


Instruction: slice Pytorch tensors which are saved in a list

Input:
I have the following code segment to generate random samples. The generated samples is a list, where each entry of the list is a tensor. Each tensor has two elements. I would like to extract the first element from all tensors in the list; and extract the second element from all tensors in the list as well. How to perform this kind of tensor slice operation
import torch
import pyro.distributions as dist
num_samples = 250
# note that both covariance matrices are diagonal
mu1 = torch.tensor([0., 5.])
sig1 = torch.tensor([[2., 0.], [0., 3.]])
dist1 = dist.MultivariateNormal(mu1, sig1)
samples1 = [pyro.sample('samples1', dist1) for _ in range(num_samples)]

samples1

Output:
I'd recommend torch.cat with a list comprehension:
col1 = torch.cat([t[0] for t in samples1])
col2 = torch.cat([t[1] for t in samples1])

Docs for torch.cat: https://pytorch.org/docs/stable/generated/torch.cat.html
ALTERNATIVELY
You could turn your list of 1D tensors into a single big 2D tensor using torch.stack, then do a normal slice:
samples1_t = torch.stack(samples1)
col1 = samples1_t[:, 0]  # : means all rows
col2 = samples1_t[:, 1]

Docs for torch.stack: https://pytorch.org/docs/stable/generated/torch.stack.html


Instruction: Get a 128 dim feature vector from a pretrained model without fine tuning in Pytorch

Input:
I need visual features from a set of images in order to have a kind of embedding of a input image. The dimension I need is 128.
I thought of removing the last layer of a pretrained model such as ResNet50, than replace the last fully connected layer with a flatten layer and do a forward pass for each image. The issue is that after the flatten layer ResNet50 gives you a 512 dim feature vector. If I add another fully connected layer with output dim of 128 I need to retrain and fine tuning the network since I'm adding new parameters.
Is there any way to get a 128 dim feature vector from the last layer of 512 dim feature vector without adding learnable parameters?

Output: You could apply a Principal Component Analysis (PCA) on your features retrieved from the pre-trained model to reduce the dimensionality to 128 components.


Instruction: PyTorch attach extra connection when building model

Input:
I have the following Resnet prototype on Pytorch:
Resnet_Classifier(
  (activation): ReLU()
  (model): Sequential(
    (0): Res_Block(
      (mod): Sequential(
        (0): Conv1d(1, 200, kernel_size=(5,), stride=(1,), padding=same)
        (1): ReLU()
        (2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (3): Conv1d(200, 200, kernel_size=(5,), stride=(1,), padding=same)
        (4): ReLU()
        (5): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (6): Conv1d(200, 200, kernel_size=(5,), stride=(1,), padding=same)
        (7): ReLU()
        (8): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (shortcut): Conv1d(1, 200, kernel_size=(1,), stride=(1,), padding=same)
    )
    (1): ReLU()
    (2): Flatten(start_dim=1, end_dim=-1)
    (3): Dropout(p=0.1, inplace=False)
    (4): Linear(in_features=40000, out_features=2, bias=True)
    (5): Softmax(dim=1)
  )
)

Input sample shape is (1, 200).
It seems to be absolutely okay but, when I try to get graph in tensorboard, I get the following structure:

Somehow my Residual block connected with Linear. Does this connection really corresponds my net structure?
Model definition:
class Res_Block(nn.Module):
    def __init__(self, in_ch, out_ch, ks, stride, activation):
        super(Res_Block, self).__init__()
        self.mod = nn.Sequential(
            nn.Conv1d(in_ch, out_ch, ks, stride, padding='same'),
            deepcopy(activation),
            nn.BatchNorm1d(out_ch),

            nn.Conv1d(out_ch, out_ch, ks, stride, padding='same'),
            deepcopy(activation),
            nn.BatchNorm1d(out_ch),

            nn.Conv1d(out_ch, out_ch, ks, stride, padding='same'),
            deepcopy(activation),
            nn.BatchNorm1d(out_ch)
        )
        self.shortcut = nn.Conv1d(in_ch, out_ch, kernel_size=1, stride=1, padding='same')

    def forward(self, X):
        return self.mod(X) + self.shortcut(X)

layers = []
layers.append(Res_Block(1, 200, 5, 1, nn.ReLU()))
layers.append(nn.ReLU())
layers.append(nn.Flatten())
layers.append(nn.Dropout(0.2))
layers.append(nn.Linear(200 * 200, 2))
layers.append(nn.Softmax(dim=1))
R = nn.Sequential(*layers)

Output:
I solved the problem by removing nn.Sequential in Res_Block __init__ and adding self.l1, self.l2 ... instead. (I also removed some layers and added maxpool but only after I solved the problem)
class Res_Block(nn.Module):
    def __init__(self, in_shape, out_ch, ks, stride, activation):
        super(Res_Block, self).__init__()

        self.l1 = nn.Conv1d(in_shape, out_ch, ks, stride, padding='same')
        self.l2 = deepcopy(activation)
        self.l3 = nn.BatchNorm1d(out_ch)
        self.l4 = nn.Conv1d(out_ch, out_ch, ks, stride, padding='same')
        self.l5 = nn.BatchNorm1d(out_ch)

        self.shortcut = nn.Conv1d(in_shape, out_ch, kernel_size=1, stride=1, padding='same')

    def forward(self, X):
        return self.l5(self.l4(self.l3(self.l2(self.l1(X))))) + self.shortcut(X)

The corresponding tensorboard structure is

The only one left question is why did that help me solve the problem.


Instruction: How to get a Histogram of PyTorch tensors in batches?

Input:
Is there a way to get the histograms of torch tensors in batches?
For Example:
x is a tensor of shape (64, 224, 224)
# x will have shape of (64, 256)
x = batch_histogram(x, bins=256, min=0, max=255)

Output:
It is possible to do this with torch.nn.functional.one_hot in a single line of code:
torch.nn.functional.one_hot(data_tensor, num_classes).sum(dim=-2)

The rationale is that one_hot does respect batches and, for each value v in the last dimension of the given tensor, creates a tensor filled with 0s, with the exception of the v-th component, which is 1. We them sum over all such one-hot encodings to obtain how many times v appears in each row of data in the 2-nd last dimension (which was the last dimension in tensor_data).
One possibly serious drawback of this method is memory usage, since each value with be expanded into a tensor of size num_classes (so, the size of tensor_data is multiplied by num_classes). This memory usage is temporary, however, since sum collapses again this extra dimension and the result will typically be smaller than tensor_data. I say "typically" because if num_classes is much larger than the size of the last dimension of tensor_data then the result will be correspondingly larger.
Here's the code with documentation, followed by pytest tests:
def batch_histogram(data_tensor, num_classes=-1):
    """
    Computes histograms of integral values, even if in batches (as opposed to torch.histc and torch.histogram).
    Arguments:
        data_tensor: a D1 x ... x D_n torch.LongTensor
        num_classes (optional): the number of classes present in data.
                                If not provided, tensor.max() + 1 is used (an error is thrown if tensor is empty).
    Returns:
        A D1 x ... x D_{n-1} x num_classes 'result' torch.LongTensor,
        containing histograms of the last dimension D_n of tensor,
        that is, result[d_1,...,d_{n-1}, c] = number of times c appears in tensor[d_1,...,d_{n-1}].
    """
    return torch.nn.functional.one_hot(data_tensor, num_classes).sum(dim=-2)

def test_batch_histogram():
    data = [2, 5, 1, 1]
    expected = [0, 2, 1, 0, 0, 1]
    run_test(data, expected)

    data = [
        [2, 5, 1, 1],
        [3, 0, 3, 1],
    ]
    expected = [
        [0, 2, 1, 0, 0, 1],
        [1, 1, 0, 2, 0, 0],
    ]
    run_test(data, expected)

    data = [
        [[2, 5, 1, 1], [2, 4, 1, 1], ],
        [[3, 0, 3, 1], [2, 3, 1, 1], ],
    ]
    expected = [
        [[0, 2, 1, 0, 0, 1], [0, 2, 1, 0, 1, 0], ],
        [[1, 1, 0, 2, 0, 0], [0, 2, 1, 1, 0, 0], ],
    ]
    run_test(data, expected)

def test_empty_data():
    data = []
    num_classes = 2
    expected = [0, 0]
    run_test(data, expected, num_classes)

    data = [[], []]
    num_classes = 2
    expected = [[0, 0], [0, 0]]
    run_test(data, expected, num_classes)

    data = [[], []]
    run_test(data, expected=None, exception=RuntimeError)  # num_classes not provided for empty data

def run_test(data, expected, num_classes=-1, exception=None):
    data_tensor = torch.tensor(data, dtype=torch.long)

    if exception is None:
        expected_tensor = torch.tensor(expected, dtype=torch.long)
        actual = batch_histogram(data_tensor, num_classes)
        assert torch.equal(actual, expected_tensor)
    else:
        with pytest.raises(exception):
            batch_histogram(data_tensor, num_classes)


Instruction: Is it possible to store some tensors on CPU and other on GPU for training neural network in PyTorch?

Input:
I designed a neural network in PyTorch, which is demanding a lot of GPU memory or else runs with a very small batch size.
The GPU Runtime error is causing due to three lines of code, which stores two new tensors and does some operations.
I don't want to run my code with a small batch size. So, I want to execute those three lines of code (and hence storing those new tensors) on CPU and remaining all other code on GPU as usual.
Is it possible to do?

Output:
It is possible.
You can use the command .to(device=torch.device('cpu') to move the relevant tensors from GPU to CPU, and back to GPU afterwards:
orig_device = a.device  # store the device from which the tensor originated
# move tensors a and b to CPU
a = a.to(device=torch.device('cpu'))
b = b.to(device=torch.device('cpu'))
# do some operation on a and b - it will be executed on CPU
res = torch.bmm(a, b)
# put the result back to GPU
res = res.to(device=orig_device)

A few notes:

Moving tensors between devices, or between GPU and CPU is not an unusual event. The term used to describe it is "model parallel" - you can google it for more details and examples.
Note that .to() operation is not an "in place" operation.
Moving tensor back and forth between GPU and CPU takes time. It might not be worthwhile using "model parallelism" of this type in this case. If you are struggling with GPU space, you might consider gradient accumulation instead


Instruction: Test Intel Extension for Pytorch(IPEX) in multiple-choice from huggingface / transformers

Input:
I am trying out one huggingface sample with SWAG dataset
https://github.com/huggingface/transformers/tree/master/examples/pytorch/multiple-choice
I would like to use Intel Extension for Pytorch in my code to increase the performance.
Here I am using the one without training (run_swag_no_trainer)
In the run_swag_no_trainer.py , I made some changes to use ipex .
#Code before changing is given below:
device = accelerator.device
model.to(device)

#After adding ipex:
import intel_pytorch_extension as ipex
    device = ipex.DEVICE
    model.to(device)

While running the below command, its taking too much time.
export DATASET_NAME=swag

accelerate launch run_swag_no_trainer.py \
  --model_name_or_path bert-base-cased \
  --dataset_name $DATASET_NAME \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$DATASET_NAME/

Is there any other method to test the same on intel ipex?

Output:
First you have to understand, which factors actually increases the running time. Following are these factors:

The large input size.
The data structure; shifted mean, and unnormalized.
The large network depth, and/or width.
Large number of epochs.
The batch size not compatible with physical available memory.
Very small or high learning rate.

For fast running, make sure to work on the above factors, like:

Reduce the input size to the appropriate dimensions that assures no loss in important features.
Always preprocess the input to make it zero mean, and normalized it by dividing it by std. deviation or difference in max, min values.
Keep the network depth and width that is not to high or low. Or always use the standard architecture that are theoretically proven.
Always make sure of the epochs. If you are not able to make any further improvements in your error or accuracy beyond a defined threshold, then there is no need to take more epochs.
The batch size should be decided based on the available memory, and number of CPUs/GPUs. If the batch cannot be loaded fully in memory, then this will lead to slow processing due to lots of paging between memory and the filesystem.
Appropriate learning rate should be determine by trying multiple, and using that which gives the best reduction in error w.r.t. number of epochs.


Instruction: ERROR: while installing Ipex using docker centos image

Input:
Unable to install Ipex using docker centos image
I pulled this docker image: docker pull sysstacks/dlrs-pytorch-centos
Tried to run on my linux machine with this command: docker run -it sysstacks/dlrs-pytorch-centos bash
I was trying to install Ipex using centos docker image (image name: sysstacks/dlrs-pytorch-centos) unfortunately i got this error.
after setting env
enter image description here

[root@0d96884d3a05 /]# python -m pip install torch_ipex==1.9.0 -f https://software.intel.com/ipex-whl-stable
Looking in links: https://software.intel.com/ipex-whl-stable
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 319, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 128, in resolve
    requirements, max_rounds=try_to_avoid_resolution_too_deep
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 473, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _merge_into_criterion
    if not criterion.candidates:
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/resolvelib/structs.py", line 139, in __bool__
    return bool(self._sequence)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in __bool__
    return any(self)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 129, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 30, in _iter_built
    for version, func in infos:
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 272, in iter_index_candidate_infos
    hashes=hashes,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/package_finder.py", line 879, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/package_finder.py", line 824, in find_all_candidates
    page_candidates = list(page_candidates_it)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
    yield from self._candidates_from_page(self._link)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/package_finder.py", line 783, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/collector.py", line 512, in fetch_page
    return _get_html_page(location, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/collector.py", line 422, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index/collector.py", line 137, in _get_html_response
    "Cache-Control": "max-age=0",
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/network/session.py", line 449, in request
    return super().request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/cachecontrol/adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/connection.py", line 506, in _connect_tls_proxy
    ssl_context=ssl_context,
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 432, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 474, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 732, in __init__
    raise ValueError("check_hostname requires server_hostname")

NOTE:
I have tried all these methods shown below but none of them worked
Why requests raise this exception "check_hostname requires server_hostname"?

Output:
I think the issue is with your proxy, can you please try below commands in your docker container:
export HTTP_PROXY=http://x.x.x.x

export https_proxy=http://x.x.x.x

export HTTPS_PROXY=http://x.x.x.x

export http_proxy=http://x.x.x.x

Change x with your proxy ip


Instruction: Missing seed functions to create reproducible RL experiments with Open AI Gym and PyTorch

Input:
I am using a simple DDQN-PER (https://github.com/higgsfield/RL-Adventure/blob/master/4.prioritized%20dqn.ipynb) to run my RL experiments. This could be used as a minimal and verifiable code for my question.
Also, here are the seed functions that I have found to be useful -
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
env.seed(seed)
env.action_space.seed(seed)

Despite this, I cannot reproduce my experiments. I believe I am missing some seed function and will appreciate inputs on the same.

Output:
Check you gym Version. There has been a major recent upgrade with v0.20 .
Here is also a blog post, describing that the new environments should be deterministic, quoting the essence:

Determinism
Up until now, the ALE hasn’t been completely deterministic. There
exist games that implement their own PRNG seeded by various methods
which depend on the console’s state. We now properly control for this
source of stochasticity and can now say with confidence that the
emulator is 100% deterministic.


Instruction: converting tensorflow transpose to pytorch

Input:
What is the equivalent function of
x = tf.transpose(y, perm=[2, 0, 1])

in pytorch?
For instance, If I have
import torch
y=torch.randn(10, 480, 640)

How can I convert y to x in pytorch?

Output:
You can use torch.Tensor.permute:
>>> x = y.permute(2, 0, 1)


Instruction: Save a model weights when a program receives TIME LIMIT while learning on a SLURM cluster

Input:
I use a deep learning models written in pytorch_lightning (pytorch) and train them on slurm clusters. I submit job like this:
sbatch --gpus=1 -t 100 python train.py

When requested GPU time ends, slurm kills my program and shows such message:
Epoch 0: : 339it [01:10,  4.84it/s, loss=-34]  slurmstepd: error: *** JOB 375083 ON cn-007 CANCELLED AT 2021-10-04T22:20:54 DUE TO TIME LIMIT ***

How can I configure a Trainer to save model when available time end?
I know about automatic saving after each epoch, but I have only one long epoch that lasts >10 hours, so this case is not working for me.

Output:
You can use Slurm's signalling mechanism to pass a signal to your application when it's within a certain number of seconds of the timelimit (see man sbatch). In your submission script use --signal=USR1@30 to send USR1 30 seconds before the timelimit is reached. Your submit script would contain these lines:
#SBATCH -t 100
#SBATCH --signal=USR1@30
srun python train.py

Then, in your code, you can handle that signal like this:
import signal

def handler(signum, frame):
    print('Signal handler got signal ', signum)
    # e.g. exit(0), or call your pytorch save routines

# enable the handler
signal.signal(signal.SIGUSR1, handler)

# your code here

You need to call your Python application via srun in order for Slurm to be able to propagate the signal to the Python process. (You can probably use --signal on the command line to sbatch, I tend to prefer writing self-contained submit scripts :))
Edit: This link has a nice summary of the issues involved with signal propagation and Slurm.


Instruction: Sideffects when changind some elements of nn.Parameter

Input:
I want to have some nn.Parameter that depend on data, e.g. different values for different people. I realized while training one person, the results of the other people also change. I created a small demo below.
I tried to make everything deterministic and as expected, changing the size of nn.Parameter does not change the results. After training for 100 epochs, I always get [1.0605, 1.0715, 1.0622, 0.9702] for all_calibration_params[0]. But all other parameters of all_calibration_params also change, and they all change by the same value, with all_calibration_params = nn.Parameter(torch.ones(10, 4)) I get [0.8455, 0.8455, 0.8455, 0.8455] for all 9 other values. I would expect these values to stay [1, 1, 1, 1]. Can anyone please explain to me, why the other values change even though I do not use them during training?
Thank you.
import torch
from pytorch_lightning import LightningModule, Trainer, seed_everything
from torch import nn
from torch.utils.data import Dataset, DataLoader
import torch.nn.functional as F

class DemoDataset(Dataset):
    """
    Generate random data, only for person 0.
    Data is always random but output is always 1.
    """

    def __init__(self):
        pass

    def __len__(self) -> int:
        return 1_000

    def __getitem__(self, idx):
        return {
            'person': torch.zeros(1),
            'data': torch.rand(1),
            'output': torch.ones(1),
        }

class DemoModel(LightningModule):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.all_calibration_params = nn.Parameter(torch.ones(3, 4))  # all values are 1

        self.fc_1 = nn.Linear(1, 4)
        self.fc_2 = nn.Linear(4 + 4, 1)

    def forward(self, person: torch.Tensor, data: torch.Tensor):
        x = self.fc_1(data)
        calibration_params = self.all_calibration_params[person].squeeze(1)
        x = torch.cat((x, calibration_params), dim=1)
        x = self.fc_2(x)

        return x

class Model(DemoModel):
    def __init__(self, learning_rate: float, weight_decay: float, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.learning_rate = learning_rate
        self.weight_decay = weight_decay

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.learning_rate, weight_decay=self.weight_decay)

    def training_step(self, train_batch: dict, batch_idx: int):
        data, person, output = train_batch['data'], train_batch['person'].long(), train_batch['output']

        pred = self(person, data)

        loss = F.mse_loss(pred, output)
        return loss

if __name__ == '__main__':
    seed_everything(42)

    model = Model(learning_rate=1e-4, weight_decay=1e-5)

    dataset = DemoDataset()
    dataloader = DataLoader(dataset, batch_size=64)

    print(model.all_calibration_params)

    trainer = Trainer(
        max_epochs=100,
        deterministic=True
    )
    trainer.fit(model, dataloader)

    print(model.all_calibration_params)

    # model.all_calibration_params[1] should be [1, 1, 1, 1]
    print(model.all_calibration_params[1], torch.ones(1, 4))

Output:
tensor([[1.0605, 1.0715, 1.0622, 0.9702],
        [0.8455, 0.8455, 0.8455, 0.8455],
        [0.8455, 0.8455, 0.8455, 0.8455]], requires_grad=True)

EDIT:
I added a small test if the other elements in nn.Parameter change the output, but the do not, so I do not understand why they are affected while training.
batch = next(iter(dataloader))
data, person = batch['data'], batch['person'].long()
output_org = model(person, data)
print(output_org[0])

model.freeze()
model.all_calibration_params[1:] = torch.ones(2, 4)
model.unfreeze()

output_mod = model(person, data)
print(output_mod[0])
print(torch.equal(output_org, output_mod))

Output:
tensor([0.6334], grad_fn=<SelectBackward>)
tensor([0.6334], grad_fn=<SelectBackward>)
True

Output:
Your dataset always returns a 0 index for the 'person', i.e. self.all_calibration_params[person] will always select the first rows.
The reason the other two rows are getting updated is because of the weight decay that was set on the optimizer. This will update all parameters regardless of whether they were used or not during the model inference.


Instruction: Getting multiple variables in TensorFlow and other frameworks (overheads)

Input:
If I have the following graph and want to get the values of tensors T1 and T2 in TF without eager execution, how would I do this? I only know of eval() or session.run() (running that twice could be an option) or tf.print(), but printing is not desired (for performance reasons).

Especially, how is this functionality implemented in TensorFlow? Does this impose a big overhead towards just getting T2? I would be happy to be pointed to relevant resources as well.
I'm generally looking for discussions on this -- if people want to add comparisons to how other frameworks do this (Caffe, Torch, CNTK, Theano, Chainer, DyNet, etc.), that's great! In the end, I am trying to understand how these frameworks could be expanded by operators that return operator-specfic metrics that a user can use to monitor training.
Thanks!

Output:
you can pass multiple parameters to session.run, and it will run the network once and return each of those parameters.
For example (from the docs):
   a = tf.constant([10, 20])
   b = tf.constant([1.0, 2.0])
   u, v = session.run([a, b])


Instruction: Pytorch: '<=' not supported between instances of 'float' and 'function'

Input:
I am trying to calculate Intersection over Union (IOU) score. Here's my code implementation, which works fine.
def IoU(predict: torch.Tensor, target: torch.Tensor):

    i = (predict & target).float().sum()
    u = (predict | target).float().sum()
    x = i/u
    IOU = x.item()

    return IoU

But when I run my unit test:
def test_IoU1():
    pred = torch.tensor([[1, 0], [1, 0]])
    target = torch.tensor([[1, 0], [1, 1]])

    iou = IoU(pred,target)

    assert 0.66 <= iou
    assert iou <= 2/3

I get:
 TypeError: '<=' not supported between instances of 'float' and 'function'

How do I fix this without changing anything on the unit test? Thank you

Output:
In this function
def IoU(predict: torch.Tensor, target: torch.Tensor):

    i = (predict & target).float().sum()
    u = (predict | target).float().sum()
    x = i/u
    IOU = x.item()

    return IoU

You are returning IoU which is the name of function, I suppose you need to return IOU. So the correct way would be -
def IoU(predict: torch.Tensor, target: torch.Tensor):

    i = (predict & target).float().sum()
    u = (predict | target).float().sum()
    x = i/u
    IOU = x.item()

    return IOU


Instruction: Pytorch different outputs between with transpose

Input:
Let I have a tensor dimension of (B, N^2, C)
and I reshape it into (B, C, N, N).
I think that I have two choices below
A = torch.rand(5, 100, 20) # Original Tensor

# First Method
B = torch.transpose(2, 1)
B = B.view(5, 20, 10, 10)

# Second Method
C = A.view(5, 20, 10, 10)

Both methods work but the outputs are slightly different and I cannot catch the difference between them.
Thanks

Output:
The difference between B and C is that you have used torch.transpose which means you have swapped two axes, this means you have changed the layout of the memory. The view at the end is just a nice interface for you to access your data but it has no effect on the underlying data of your tensor. What it comes down to is a contiguous memory data buffer.
If you take a smaller example, something we can grasp more easily:
>>> A = torch.rand(1, 4, 3)
tensor([[[0.2656, 0.5920, 0.3774],
         [0.8447, 0.5984, 0.0614],
         [0.5160, 0.8048, 0.6260],
         [0.1644, 0.3144, 0.1040]]])

Here swapping axis=1 and axis=2 comes down to a batched transpose (in mathematical terms):
>>> B = A.transpose(2, 1)
tensor([[[0.4543, 0.7447, 0.7814, 0.3444],
         [0.9766, 0.2732, 0.4766, 0.0387],
         [0.0123, 0.7260, 0.8939, 0.8581]]])

In terms of memory layout A has the following memory arangement:
>>> A.flatten()
tensor([0.4543, 0.9766, 0.0123, 0.7447, 0.2732, 0.7260, 0.7814, 0.4766, 0.8939,
        0.3444, 0.0387, 0.8581])

While B has a different layout. By layout I mean memory arrangement, I am not referring to its shape which is irrelevant:
>>> B.flatten()
tensor([0.4543, 0.7447, 0.7814, 0.3444, 0.9766, 0.2732, 0.4766, 0.0387, 0.0123,
        0.7260, 0.8939, 0.8581])

As I said reshaping i.e. building a view on top of a tensor doesn't change its memory layout, it's an abstraction level to better manipulate tensors.
So in the end, yes you end up with two different results: C shares the same data as A, while B is a copy and has a different memory layout.


Instruction: Why torch.autograd.grad() returns None with torch.cat?

Input:
I know the torch.autograd.grad() returns None if the gradient is stopped somehow, however, I am wondering what is wrong with the following snippet?
x = torch.rand(6, requires_grad=True)

y = x.pow(2).sum()
z = torch.cat([x])

grad1 = torch.autograd.grad(y, x, allow_unused=True)
grad2 = torch.autograd.grad(y, z, allow_unused=True)

print(f'grad1 = {grad1}, grad = {grad2}')

The output is grad1 = (tensor([0.3705, 0.7468, 0.6102, 1.8640, 0.3518, 0.5397]),), grad = (None,).
I am expecting the grad2 is the same to grad1, because z is essentially the x. May I know why please?

Update: After reading the post and the help from @Ivan, I conclude the reason is x is a leaf node of y but z is not any more. x is the leaf node of both y and z in the computation graph, but there is no direct path from z to y, so the torch.autograd.grad returns None.
Note: The returned value None does not necessarily guarantee the values are 0.

Output:
Tensor z was not used to compute the value of y, as such it is not connected to its computation graph, and you won't get a gradient on z, since it's not connected to y.
On the other hand, the following will work:
>>> y = x.pow(2).sum()
>>> torch.autograd.grad(y, x, allow_unused=True)
(tensor([0.3134, 1.6802, 0.1989, 0.8495, 1.9203, 1.0905]),)

>>> z = torch.cat([x])
>>> y = z.pow(2).sum()
>>> torch.autograd.grad(y, z, allow_unused=True)
(tensor([0.3134, 1.6802, 0.1989, 0.8495, 1.9203, 1.0905]),)


Instruction: Get grads of parameters w.r.t a loss term in pytorch

Input:
I my Pytorch training i use a composite loss function defined as :
.
In order to update the weights alpha and beta, i need to compute three values :  which are the the means of the gradients of the loss terms w.r.t to all the weights in the network.
Is there an efficient way to write it in pytorch ?
My training code look like :
for epoc in range(1, nb_epochs+1):
  #init
  optimizer_fo.zero_grad()
  #get the current loss
  loss_total = mynet_fo.loss(tensor_xy_dirichlet,g_boundaries_d,tensor_xy_inside,tensor_f_inter,tensor_xy_neuman,g_boundaries_n)
  #compute gradients
  loss_total.backward(retain_graph=True)
  #optimize
  optimizer_fo.step()

Where my .loss() function directly return the sum of the terms. I've thinking of make a second forward pass and call backward on each Loss term independently but it would be very expensive.

Output:
1- Using torch.autograd.grad
You can get the different terms of your gradient only by back-propagating multiple times on your network. In order to avoid having to perform multiple inferences on your input, you can use the torch.autograd.grad utility function instead of doing a conventional backward pass backward. This means you won't pollute the gradients coming from the different terms.
Here is a minimal example that shows the basic idea:
>>> x = torch.rand(1, 10, requires_grad=True)
>>> lossA = x.pow(2).sum()
>>> lossB = x.mean()

Then perform one backward pass on each term out of place. You have to retain the graph on all calls but the last:
>>> gradA = torch.autograd.grad(lossA, x, retain_graph=True)
(tensor([[1.5810, 0.6684, 0.1467, 0.6618, 0.5067, 0.2368, 0.0971, 0.4533, 0.3511,
          1.9858]]),)

>>> gradB = torch.autograd.grad(lossB, x)
(tensor([[0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000,
          0.1000]]),)

This method has some limitations since you are receiving your parameters' gradient as a tuple which is not that convenient.

2- Caching the results of backward
An alternative solution consists in caching the gradient after each successive backward call:
>>> lossA = x.pow(2).sum()
>>> lossB = x.mean()

>>> lossA.backward(retain_graph=True)

Store the gradient and clear the .grad attributes (don't forget to do so otherwise the gradient of lossA will pollute gradB. You will have to adapt this to the general case when handling multiple tensor parameters:
>>> x.gradA = x.grad
>>> x.grad = None

Backward pass on the next loss term:
>>> lossB.backward()
>>> x.gradB = x.grad

Then you can interact with each gradient term locally (i.e. on each parameter separately):
>>> x.gradA, x.gradB
(tensor([[1.5810, 0.6684, 0.1467, 0.6618, 0.5067, 0.2368, 0.0971, 0.4533, 0.3511,
          1.9858]]),
 tensor([[0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000, 0.1000,
          0.1000]]))

The latter method seems more practical.

This essentially comes down to torch.autograd.grad vs torch.autograd.backward, i.e. out-of-place vs in-place... and will ultimately depends on your needs. You can read more about these two functions here.


Instruction: Output of nn.Linear is different for the same input

Input:
In torch==1.7.1+cu101, I have two tensors
import torch
a = torch.rand(1,5,10)
b = torch.rand(100,1,10)

and a feed-forward network
import torch.nn as nn
l = nn.Linear(10,10)

I force one row of them to be equal:
a[0,0] = b[80][0].clone()

Then I feed both tensors to l:
r1 = l(a)
r2 = l(b)

Apparently, since a[0,0] is equal to b[80,0], r1[0,0] must be equal to r2[80,0].
But it turns out to be like:
(r1[0,0] == r2[80,0]).all()
>>> False

I've fixed the randomness by:
seed = 42

random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = True

Does anyone know why (r1[0,0] == r2[80,0]).all() is False?

Output:
If you were to print r1[0, 0] and r2[80, 0], you'd see that they are extremely similar. Even identical within the number of printed digits.
However, if you print r1[0, 0] - r2[80, 0] you'll see that the resulting entries are not perfectly 0.0 (though they are close to it), meaning that r1[0, 0] and r2[80, 0] are close but not perfectly identical.
Now, if we were to do take those individual vectors out first, and then pass them through the linear layer like this:
r1_small = l(a[0, 0])
r2_small = l(b[80, 0])
print((r1_small == r2_small).all())  # tensor(True)

we get that they are perfectly identical, even despite being floats.
So, that means that some difference is introduced by the identical vectors being smaller parts of bigger tensors when they are passed through the linear layer.
It's also very much worth noting that the same difference does not arise when the first n-1 dimensions are all powers of 2:
a2 = torch.randn(8, 8, 10)
b2 = torch.randn(4, 16, 10)
a2[0, 0] = b2[1, 0].clone()
r1_2 = l(a2)
r2_2 = l(b2)
print((r1_2[0, 0] == r2_2[1, 0]).all())  # tensor(True)

So, while I don't know the details, I suspect it has to do with byte alignment.
In general, testing for perfect equality between float values that should be mathematically equal will not always give the expected results. So how do we handle these differences? You can use torch.isclose or torch.allclose to check for imperfect equality.


Instruction: pytorch optim.SGD with momentum how to check "velocity"?

Input:
I have a noob question: from the SGD doc they provided the equation of SGD with momentum, which indicates that apart from current gradient weight.grad, we also need to save the velocity from the previous step (something like weight.prev_v?). I know nn.Parameter object has .data and .grad attributes, but does it also saves a .prev_v? Do you know how pytorch works? Thanks!
Edit: Basically I'd like to know where does pytorch save velocity from the previous step?

Output:
Those are stored inside the state attribute of the optimizer. In the case of torch.optim.SGD the momentum values are stored a dictionary under the 'momentum_buffer' key, as you can see in the source code.
Here is a minimal example:
>>> m = nn.Linear(10,10)
>>> optim = torch.optim.SGD(m.parameters(), lr=1.e-3, momentum=.9)
>>> m(torch.rand(1, 10)).mean().backward()
>>> optim.step()

>>> optim.state
defaultdict(dict, {0: {}, Parameter containing: ...})

>>> list(optim.state.values())[0]
{'momentum_buffer': tensor([...])}


Instruction: Is there a function for picking the max value in a pytorch tensor which follows a requirenment

Input:
I am programming an othello bot in python using reinforcment learning and pytorch. In the program, I scan the board for legal moves. The AI should choose the move with the highest probability of beeing good, and that is legal according to the previose calculation. Here I need a function that works something like this:
a = torch.tensor([1,2,3,4,5])
b = torch.tensor([True, True, False, True, False], dtype=bool)
print(torch.somefunction(a,b))

The output should be the id of the max value in a, in this case 3. Does this function exist? And if not, are there any other whay of doing this?

Output:
Assuming there is at least one non-negative value in your tensor, you multiply it by the mask itself to remove excluded values in the sorting:
>>> torch.argmax(a*b)
tensor(3)

If that's not the case, you can still get away with it, using torch.where by replacing the excluded values with some value that will get ignored by the argmax (e.g. a.min()):
>>> torch.where(b, a, a.min()).argmax()
tensor(3)


Instruction: Tuple of Tensors

Input:
I recently asked one part of this question. I am building a chatbot, and there is a function that makes the problems. The function is given below:
def variable_from_sentence(sentence):
  vec, length = indexes_from_sentence(sentence)
  inputs = [vec]
  lengths_inputs = [length]
  if hp.cuda:
    batch_inputs = Variable(torch.stack(torch.Tensor(inputs),1).cuda())
  else:
    batch_inputs = Variable(torch.stack(torch.Tensor(inputs),1))
  return batch_inputs, lengths_inputs

But when I try to run the chatbot code , it gives me this error:

stack(): argument 'tensors' (position 1) must be tuple of Tensors, not tensor

For that reason, I fixed the function like this:
def variable_from_sentence(sentence):
  vec, length = indexes_from_sentence(sentence)
  inputs = [vec]
  lengths_inputs = [length]
  if hp.cuda:
    batch_inputs = torch.stack(inputs, 1).cuda()
  else:
    batch_inputs = torch.stack(inputs, 1)
  return batch_inputs, lengths_inputs

But it still gives me error, and the error is like this:

TypeError: expected Tensor as element 0 in argument 0, but got list

What should I do now in this situation?

Output:
Since the vec and length are both integers, you can use torch.tensor directly:
def variable_from_sentence(sentence):
    vec, length = indexes_from_sentence(sentence)
    inputs = [vec]
    lengths_inputs = [length]
    if hp.cuda:
        batch_inputs = torch.tensor(inputs, device='cuda')
    else:
        batch_inputs = torch.tensor(inputs)
    return batch_inputs, lengths_inputs


Instruction: pyro: how to specify conditional distribution

Input:
I am trying to use pyro for specifying a Bayesian network. I have a child node D which is continuous and it has three discrete nodes are parents, each with 10 possible states:
So, I first define my discrete nodes as:
import torch
import pyro
import pyro.distributions as dist

def model(data):
    A = pyro.sample("A", dist.Dirichlet(torch.ones(10)))
    B = pyro.sample("B", dist.Dirichlet(torch.ones(10)))
    C = pyro.sample("C", dist.Dirichlet(torch.ones(10)))

Now, as I understand I need to define P(D|A, B, C). I want to model this as a normal distribution but not sure how to do this conditioning. My plan is to put priors on this distribution parameters and then use MCMC or HMC to estimate the posterior distribution and learn the model parameters.
However, not sure how to proceed with the model definition.

Output:
Good thing about pyro is that the model definition is very pythonic. The underlying PyTorch machinery can track dependencies for you.
All you need is use the samples A, B and C and compute the parameters of the conditional p(D|A,B,C)
def cond_mean(a, b, c):
    return # use a,b,c to compute mean

def cond_scale(a, b, c):
    return # use a,b,c to compute scale

def model(data):
    A = pyro.sample("A", dist.Dirichlet(torch.ones(10)))
    B = pyro.sample("B", dist.Dirichlet(torch.ones(10)))
    C = pyro.sample("C", dist.Dirichlet(torch.ones(10)))

    D = pyro.sample("D", dist.Normal(loc=cond_mean(A, B, C), scale=cond_scale(A, B, C)
    ...


Instruction: When I was trying to run IPEX on DevCloud it is showing "Illegal instruction"

Input:
I was trying image classification using IPEX on DevCloud  but it is showing "Illegal instruction" for me.

qsub -I -l nodes=1:gpu:ppn=2 -d .
python3 inf.py

Output:
qsub -I -l nodes=1:gpu:ppn=2 -d .

This command is used to assign one full GPU node.
IPEX don't support gpu as mentioned here:
https://github.com/intel/intel-extension-for-pytorch/issues/149#issuecomment-812348623
This could be the reason for that error.
Try using below commands:
qsub -I
python3 inf.py