Instruction: RuntimeError: Found dtype Long but expected Float: When using criterion

Input:
I realized this question has been asked many times but I cannot find any solution which fixes my issue. I found this code online and tried to run it to understand how it actually works. I found the error is RuntimeError: Found dtype Long but expected Float. It happens at errD_real = criterion(output, label). I tried to add .float() to change the type, but it does not work. I use Pytorch 1.7.0, Python 3.8.3, Anaconda 1.9.12, Cudnn 7.6.0. The error and code are below.
runfile('C:/Summer2020/computer_vision/final_project/copy_of_chest_x_ray_shenanigans_(dcgan).py', wdir='C:/Summer2020/computer_vision/final_project')
Generator(
  (main): Sequential(
    (0): ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace=True)
    (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace=True)
    (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): ReLU(inplace=True)
    (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (13): Tanh()
  )
)
Discriminator(
  (main): Sequential(
    (0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (1): LeakyReLU(negative_slope=0.2, inplace=True)
    (2): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (4): LeakyReLU(negative_slope=0.2, inplace=True)
    (5): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (6): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): LeakyReLU(negative_slope=0.2, inplace=True)
    (8): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (9): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (10): LeakyReLU(negative_slope=0.2, inplace=True)
    (11): Conv2d(512, 1, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (12): Sigmoid()
  )
)
Starting Training Loop...
Type of output is  <class 'torch.Tensor'>
Type of label is  <class 'torch.Tensor'>
Traceback (most recent call last):

  File "C:\Summer2020\computer_vision\final_project\copy_of_chest_x_ray_shenanigans_(dcgan).py", line 337, in <module>
    main()

  File "C:\Summer2020\computer_vision\final_project\copy_of_chest_x_ray_shenanigans_(dcgan).py", line 243, in main
    errD_real = criterion(output, label)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\modules\loss.py", line 530, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\functional.py", line 2525, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(

RuntimeError: Found dtype Long but expected Float

from __future__ import print_function
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

def main():
    # Set random seed for reproducibility
    manualSeed = 999
    #manualSeed = random.randint(1, 10000) # use if you want new results
    random.seed(manualSeed)
    torch.manual_seed(manualSeed)

    # Load dataset

    dataroot = 'C:\Spring 2020/Machine Learning and Computer Vision/Project Database/chest xray/chest_xray/chest_xray/train/'

    # Number of workers for dataloader
    workers = 4

    # Batch size during training
    batch_size = 128

    # Spatial size of training images. All images will be resized to this
    #   size using a transformer.
    image_size = 64

    # Number of channels in the training images. For color images this is 3
    nc = 3

    # Size of z latent vector (i.e. size of generator input)
    nz = 100

    # Size of feature maps in generator
    ngf = 64

    # Size of feature maps in discriminator
    ndf = 64

    # Number of training epochs
    num_epochs = 5

    # Learning rate for optimizers
    lr = 0.0002

    # Beta1 hyperparam for Adam optimizers
    beta1 = 0.5

    # Number of GPUs available. Use 0 for CPU mode.
    ngpu = 1

    # Create the dataset
    dataset = dset.ImageFolder(root='C:\Spring 2020/Machine Learning and Computer Vision/Project Database/chest xray/chest_xray/chest_xray/train/',
                               transform=transforms.Compose([
                                   transforms.Resize(image_size),
                                  #  transforms.Grayscale(1),
                                   transforms.CenterCrop(image_size),
                                   transforms.ToTensor(),
                                   transforms.Normalize((0.5), (0.5)),
                               ]))
    # Create the dataloader
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                             shuffle=True, num_workers=workers)

    # Decide which device we want to run on
    device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

    # Plot some training images
    real_batch = next(iter(dataloader))
    plt.figure(figsize=(8,8))
    plt.axis("off")
    plt.title("Training Images")
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))

    # Custom weights initialization called on netG and netD
    def weights_init(m):
        classname = m.__class__.__name__
        if classname.find('Conv') != -1:
            nn.init.normal_(m.weight.data, 0.0, 0.02)
        elif classname.find('BatchNorm') != -1:
            nn.init.normal_(m.weight.data, 1.0, 0.02)
            nn.init.constant_(m.bias.data, 0)

    """Generator Code"""

    class Generator(nn.Module):
        def __init__(self, ngpu):
            super(Generator, self).__init__()
            self.ngpu = ngpu
            self.main = nn.Sequential(
                # input is Z, going into a convolution
                nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
                nn.BatchNorm2d(ngf * 8),
                nn.ReLU(True),
                # state size. (ngf*8) x 4 x 4
                nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf * 4),
                nn.ReLU(True),
                # state size. (ngf*4) x 8 x 8
                nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf * 2),
                nn.ReLU(True),
                # state size. (ngf*2) x 16 x 16
                nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf),
                nn.ReLU(True),
                # state size. (ngf) x 32 x 32
                nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
                nn.Tanh()
                # state size. (nc) x 64 x 64
            )

        def forward(self, input):
            return self.main(input)

    # Create the generator
    netG = Generator(ngpu).to(device)

    # Handle multi-gpu if desired
    if (device.type == 'cuda') and (ngpu > 1):
        netG = nn.DataParallel(netG, list(range(ngpu)))

    # Apply the weights_init function to randomly initialize all weights
    #  to mean=0, stdev=0.2.
    netG.apply(weights_init)

    # Print the model
    print(netG)

    """Discriminator Code

    """

    class Discriminator(nn.Module):
        def __init__(self, ngpu):
            super(Discriminator, self).__init__()
            self.ngpu = ngpu
            self.main = nn.Sequential(
                # input is (nc) x 64 x 64
                nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf) x 32 x 32
                nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 2),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*2) x 16 x 16
                nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 4),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*4) x 8 x 8
                nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 8),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*8) x 4 x 4
                nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
                nn.Sigmoid()
            )

        def forward(self, input):
            return self.main(input)

    # Create the Discriminator
    netD = Discriminator(ngpu).to(device)

    # Handle multi-gpu if desired
    if (device.type == 'cuda') and (ngpu > 1):
        netD = nn.DataParallel(netD, list(range(ngpu)))

    # Apply the weights_init function to randomly initialize all weights
    #  to mean=0, stdev=0.2.
    netD.apply(weights_init)

    # Print the model
    print(netD)

    """Train the GAN!

    """

    # Initialize BCELoss function
    criterion = nn.BCELoss()

    # Create batch of latent vectors that we will use to visualize
    #  the progression of the generator
    fixed_noise = torch.randn(64, nz, 1, 1, device=device)

    # Establish convention for real and fake labels during training
    real_label = 1
    fake_label = 0

    # Setup Adam optimizers for both G and D
    optimizerD = optim.Adam(netD.parameters(), lr=0.0002, betas=(beta1, 0.999))
    optimizerG = optim.Adam(netG.parameters(), lr=0.001, betas=(beta1, 0.999))

    # Commented out IPython magic to ensure Python compatibility.
    # Training Loop

    # Lists to keep track of progress
    img_list = []
    G_losses = []
    D_losses = []
    iters = 0
    num_epochs = 35

    print("Starting Training Loop...")
    # For each epoch
    for epoch in range(num_epochs):
        # For each batch in the dataloader
        for i, data in enumerate(dataloader, 0):

            ############################
            # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
            ###########################
            ## Train with all-real batch
            netD.zero_grad()
            # Format batch
            real_cpu = data[0].to(device)
            b_size = real_cpu.size(0)
            label = torch.full((b_size,), real_label, device=device) # 64 = batchSize
            # print("label " + str(label.size()))
            # Forward pass real batch through D
            output = netD(real_cpu).view(-1) # 256??
            print('Type of output is ', type(output))
            print('Type of label is ', type(label))
            # print("output " + str(output.size()))
            # Calculate loss on all-real batch
            errD_real = criterion(output, label)
            # Calculate gradients for D in backward pass
            errD_real.backward()
            D_x = output.mean().item()

            ## Train with all-fake batch
            # Generate batch of latent vectors
            noise = torch.randn(b_size, nz, 1, 1, device=device)
            # Generate fake image batch with G
            fake = netG(noise)
            # print("fake " + str(fake.size()))
            label.fill_(fake_label)
            # Classify all fake batch with D
            output = netD(fake.detach()).view(-1)
            # Calculate D's loss on the all-fake batch
            errD_fake = criterion(output, label)
            # Calculate the gradients for this batch
            errD_fake.backward()
            D_G_z1 = output.mean().item()
            # Add the gradients from the all-real and all-fake batches
            errD = errD_real + errD_fake
            # Update D
            optimizerD.step()

            ############################
            # (2) Update G network: maximize log(D(G(z)))
            ###########################
            netG.zero_grad()
            label.fill_(real_label)  # fake labels are real for generator cost
            # Since we just updated D, perform another forward pass of all-fake batch through D
            output = netD(fake).view(-1)
            # Calculate G's loss based on this output
            errG = criterion(output, label)
            # Calculate gradients for G
            errG.backward()
            D_G_z2 = output.mean().item()
            # Update G
            optimizerG.step()

            # Output training stats
            if i % 50 == 0:
                print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                       % (epoch, num_epochs, i, len(dataloader),
                         errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

            # Save Losses for plotting later
            G_losses.append(errG.item())
            D_losses.append(errD.item())

            # Check how the generator is doing by saving G's output on fixed_noise
            if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
                with torch.no_grad():
                    fake = netG(fixed_noise).detach().cpu()
                img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

            iters += 1

    plt.figure(figsize=(10,5))
    plt.title("Generator and Discriminator Loss During Training")
    plt.plot(G_losses,label="G")
    plt.plot(D_losses,label="D")
    plt.xlabel("iterations")
    plt.ylabel("Loss")
    plt.legend()
    plt.show()

    fig = plt.figure(figsize=(8,8))
    plt.axis("off")
    ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
    ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)

    HTML(ani.to_jshtml())

    # Grab a batch of real images from the dataloader
    real_batch = next(iter(dataloader))

    # Plot the real images
    plt.figure(figsize=(80,80))
    plt.subplot(1,2,1)
    plt.axis("off")
    plt.title("Real Images")
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

    # Plot the fake images from the last epoch
    plt.subplot(1,2,2)
    plt.axis("off")
    plt.title("Fake Images")
    plt.imshow(np.transpose(img_list[-1],(1,2,0)))
    plt.show()

if __name__ == '__main__':
    main()

Output:
I think I just found the solution is to change
from
label = torch.full((b_size,), real_label, device=device)

to
label = torch.full((b_size,), real_label, device=device, dtype=torch.float)


Instruction: Decaying the learning rate from the 100th epoch

Input:
Knowing that
learning_rate = 0.0004
optimizer = torch.optim.Adam(
    model.parameters(),
    lr=learning_rate, betas=(0.5, 0.999)
)

is there a way of decaying the learning rate from the 100th epoch?
Is this a good practice:
decayRate = 0.96
my_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=my_optimizer, gamma=decayRate)

Output:
from torch.optim.lr_scheduler import MultiStepLR

# reduce the learning rate by 0.1 after epoch 100
scheduler = MultiStepLR(optimizer, milestones=[100,], gamma=0.1)

Please refer: MultiStepLR for more information.


Instruction: How can i add a Bi-LSTM layer on top of bert model?

Input:
I'm using pytorch and I'm using the base pretrained bert to classify sentences for hate speech.
I want to implement a Bi-LSTM layer that takes as an input all outputs of the latest
transformer encoder from the bert model as a new model (class that implements nn.Module), and i got confused with the nn.LSTM parameters.
I tokenized the data using
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=int(data['class'].nunique()),output_attentions=False,output_hidden_states=False)

My data-set has 2 columns: class(label), sentence.
Can someone help me with this?
Thank you in advance.
Edit:
Also, after processing the input in the bi-lstm, the network sends the  final hidden state to a fully connected network that performs classication using the softmax activation function. how can I do that ?

Output:
You can do it as follows:
from transformers import BertModel
class CustomBERTModel(nn.Module):
    def __init__(self):
          super(CustomBERTModel, self).__init__()
          self.bert = BertModel.from_pretrained("bert-base-uncased")
          ### New layers:
          self.lstm = nn.LSTM(768, 256, batch_first=True,bidirectional=True)
          self.linear = nn.Linear(256*2, <number_of_classes>)

    def forward(self, ids, mask):
          sequence_output, pooled_output = self.bert(
               ids,
               attention_mask=mask)

          # sequence_output has the following shape: (batch_size, sequence_length, 768)
          lstm_output, (h,c) = self.lstm(sequence_output) ## extract the 1st token's embeddings
          hidden = torch.cat((lstm_output[:,-1, :256],lstm_output[:,0, 256:]),dim=-1)
          linear_output = self.linear(hidden.view(-1,256*2)) ### assuming that you are only using the output of the last LSTM cell to perform classification

          return linear_output

tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
model = CustomBERTModel()


Instruction: How can i get all outputs of the last transformer encoder in bert pretrained model and not just the cls token output?

Input:
I'm using pytorch and this is the model from huggingface transformers link:
from transformers import BertTokenizerFast, BertForSequenceClassification
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased",
                                                     num_labels=int(data['class'].nunique()),
                                                     output_attentions=False,
                                                     output_hidden_states=False)

and in the forward function I'm building, I'm calling x1, x2 = self.bert(sent_id, attention_mask=mask)
Now, as far as I know, x2 is the cls output(which is the output of the first transformer encoder) but yet again, I don't think I understand the output of the model.
but I want the output of all the 12 last transformer encoders.
How can I do that in pytorch ?

Output:
Ideally, if you want to look into the outputs of all the layer, you should use BertModel and not BertForSequenceClassification. Because, BertForSequenceClassification is inherited from BertModel and adds a linear layer on top of the BERT model.
from transformers import BertModel
my_bert_model = BertModel.from_pretrained("bert-base-uncased")

### Add your code to map the model to device, data to device, and obtain input_ids and mask

sequence_output, pooled_output = my_bert_model(ids, attention_mask=mask)

# sequence_output has the following shape: (batch_size, sequence_length, 768), which contains output for all tokens in the last layer of the BERT model.

sequence_output contains output for all tokens in the last layer of the BERT model.
In order to obtain the outputs of all the transformer encoder layers, you can use the following:
my_bert_model = BertModel.from_pretrained("bert-base-uncased")
sequence_output, pooled_output, all_layer_output = model(ids, attention_mask=mask, output_hidden_states=True)

all_layer_output is a output tuple containing the outputs embeddings layer + outputs of all the layer. Each element in the tuple will have a shape (batch_size, sequence_length, 768)
Hence, to get the sequence of outputs at layer-5, you can use all_layer_output[5]. As, all_layer_output[0] contains outputs of the embeddings.


Instruction: What does this code in PyTorch do? How can I express it with tensorflow

Input:
I found a code that would solve my problem that looks like this:
(self.conv_diag(input_tensor.diagonal(dim1=2, dim2=3))).diag_embed(dim1=2, dim2=3)

While self.conv_diag is a layer I have defined before.
As far as I understood it extracts the diagonal of a subtensor in the second and third dimension puts it into the layer and constructs a new tensor filled with zeros and replaces its second and third dimension with the new values calculated by my layer.
What I have found to extract the diagonal is
tf.math.reduce_diag(input_tensor)

but I cannot choose the axis and I have not yet found an equivalent function to replace torch.diag_embed()
How can I express it in Tensorflow?

Output:
This is maybe what you are looking for:
tf.linalg.diag(
    diagonal, name='diag', k=0, num_rows=-1, num_cols=-1, padding_value=0,
    align='RIGHT_LEFT'
)


Instruction: Load csv and Image dataset in pytorch

Input:
I am doing image classification with PyTorch. I have a separate Images folder and train and test csv file with images ids and labels . I don’t have any an idea about how to combine those images and ID and converting into tensors.

train.csv : contains all ID of Image like 4325.jpg, 2345.jpg,…so on and contains Labels like cat,dog.
Image_data : contains all the images of with ID name.

Output:
You can create custom dataset class by inherting pytorch's torch.utils.data.Dataset.
The assumption for the following custom dataset class is

csv file format is

filename
label

4325.jpg
cat

2345.jpg
dog

All images are inside images folder.

class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, csv_path, images_folder, transform = None):
        self.df = pd.read_csv(csv_path)
        self.images_folder = images_folder
        self.transform = transform
        self.class2index = {"cat":0, "dog":1}

    def __len__(self):
        return len(self.df)
    def __getitem__(self, index):
        filename = self.df[index, "FILENAME"]
        label = self.class2index[self.df[index, "LABEL"]]
        image = PIL.Image.open(os.path.join(self.images_folder, filename))
        if self.transform is not None:
            image = self.transform(image)
        return image, label

Now you can use this class to load the training and test dataset using both csv file and image folder.

train_dataset = CustomDataset("path - to - train.csv", "path - to - images - folder"  )
test_dataset = CustomDataset("path - to - test.csv", "path - to - images - folder"  )

image, label = train_dataset[0]


Instruction: How do I specify and install the latest version of PyTorch via Conda in AWS Sagemaker?

Input:
I'm attempting to use a recent version of PyTorch (1.7.0) in a Conda environment on Sagemaker by specifying the package version in an environment.yml file. However, I'm getting a ResolvePackageNotFound error. Note that I'm just working in a Jupyter notebook with a kernel corresponding to this Conda environment. I'm not using a deep learning image.
Steps to reproduce:
Save the code below as a .yml file, navigate to where this .yml file is stored, and run conda env create environment.yml.
name: test_env
dependencies:
  - numpy
  - pandas
  - pytorch>=1.7.0
  - torchvision
  - scipy
  - ipykernel
  - torchvision

I've tried this on instances of various types (ml.p2.xlarge, ml.p3.2xlarge, and ml.p3.8xlarge) and have gotten the same error each time. I tried it with conda version 4.8.3 and 4.9.2 as well. If I specify pytorch>=1.5.0, I'm able to create the environment successfully.
Does anyone have any idea why I can't create an environment successfully with more recent versions of PyTorch? Based on this documentation, I wonder if Sagemaker has preinstalled certain versions of PyTorch, and something is going awry when I attempt to use a more recent version.

Output:
Sagemaker instances do not always have support for latest packages.
Check this link for the list of supported images in Sagemaker instances.


Instruction: Load trained model on another machine - fastai, torch, huggingface

Input:
I am using fastai with pytorch to fine tune XLMRoberta from huggingface.
I've trained the model and everything is fine on the machine where I trained it.
But when I try to load the model on another machine I get OSError - Not Found - No such file or directory pointing to .cache/torch/transformers/. The issue is the path of a vocab_file.
I've used fastai's Learner.export to export the model in .pkl file, but I don't believe that issue is related to fastai since I found the same issue appearing in flairNLP.
It appears that the path to the cache folder, where the vocab_file is stored during the training, is embedded in the .pkl file:

The error comes from transformer's XLMRobertaTokenizer __setstate__:
def __setstate__(self, d):
    self.__dict__ = d
    self.sp_model = spm.SentencePieceProcessor()
    self.sp_model.Load(self.vocab_file)

which tries to load the vocab_file using the path from the file.
I've tried patching this method using:
pretrained_model_name = "xlm-roberta-base"
vocab_file = XLMRobertaTokenizer.from_pretrained(pretrained_model_name).vocab_file

def _setstate(self, d):
    self.__dict__ = d
    self.sp_model = spm.SentencePieceProcessor()
    self.sp_model.Load(vocab_file)

XLMRobertaTokenizer.__setstate__ = MethodType(_setstate, XLMRobertaTokenizer(vocab_file))

And that successfully loaded the model but caused other problems like missing model attributes and other unwanted issues.
Can someone please explain why is the path embedded inside the file, is there a way to configure it without reexporting the model or if it has to be reexported how to configure it dynamically using fastai, torch and huggingface.

Output:
I faced the same error. I had fine tuned XLMRoberta on downstream classification task with fastai version = 1.0.61. I'm loading the model inside docker.
I'm not sure about why the path is embedded, but I found a workaround. Posting for future readers who might be looking for workaround as retraining is usually not possible.

I created /home/.cache/torch/transformer/ inside the docker image.

RUN mkdir -p /home/<username>/.cache/torch/transformers

Copied the files (which were not found in docker) from my local /home/.cache/torch/transformer/ to docker image /home/.cache/torch/transformer/

 COPY filename:/home/<username>/.cache/torch/transformers/filename


Instruction: I got this error when installing PyTorch in Pycharm. I installed torchvision and torch successfully but the problem is in PyTorch

Input:
I try to install PyTorch in cmd to import it in pycharm project. it gives me numerous errors after Running setup.py install for PyTorch ... error.
ERROR: Command errored out with exit status 1:
     command: 'c:\users\sarah\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarah\AppData\Local\Temp\pip-record-k6l1ba85\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\sarah\AppData\Roaming\Python\Python38\Include\PyTorch'
         cwd: C:\Users\sarah\AppData\Local\Temp\pip-install-uejad0d4\pytorch_87cb825d32c24f6ca6f350ea09c367a4\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\sarah\AppData\Local\Temp\pip-install-uejad0d4\pytorch_87cb825d32c24f6ca6f350ea09c367a4\setup.py", line 11, in <module>
        raise Exception(message)
    Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\sarah\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarah\\AppData\\Local\\Temp\\pip-install-uejad0d4\\pytorch_87cb825d32c24f6ca6f350ea09c367a4\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarah\AppData\Local\Temp\pip-record-k6l1ba85\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\sarah\AppData\Roaming\Python\Python38\Include\PyTorch' Check the logs for full command output.

Output:
The best way to install PyTorch is using pip or conda, with the commands provided on their website: https://pytorch.org/
This way you can choose which OS you are using, which version of CUDA (or no CUDA), and whether you are using conda or pip.
Please notice that as of today (12/14/20), I tried installing PyTorch in a new environment and it failed because of a new Numpy version. I solved this by installing Numpy first:
pip install numpy==1.18

And then PyTorch as usual from their website. Notice that their pip install command specifies torch as the package name and not pytorch, which you seem to have used.


Instruction: How to solve the failure of getting a python file from cpp extension of pytorch using setuptools?

Input:
I wanted to try a github project named deformable kernels, and followed the steps described in the README.md file:
conda env create -f environment.yml

cd deformable_kernels/ops/deform_kernel;
pip install -e .;

The structure of deformable_kernel/ops/deform_kernel is showed here:
.
csrc
    filter_sample_depthwise_cuda.cpp
    filter_sample_depthwise_cuda.h
    filter_sample_depthwise_cuda_kernel.cu
    nd_linear_sample_cuda.cpp
    nd_linear_sample_cuda.h
    nd_linear_sample_cuda_kernel.cu
functions
    filter_sample_depthwise.py
    __init__.py
    nd_linear_sample.py
__init__.py
modules
    filter_sample_depthwise.py
    __init__.py
setup.py

And the content of file setup.py is showed here:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='filter_sample_depthwise',
    ext_modules=[
        CUDAExtension(
            'filter_sample_depthwise_cuda',
            [
                'csrc/filter_sample_depthwise_cuda.cpp',
                'csrc/filter_sample_depthwise_cuda_kernel.cu',
            ]
        ),
    ],
    cmdclass={'build_ext': BuildExtension}
)

setup(
    name="nd_linear_sample",
    ext_modules=[
        CUDAExtension(
            "nd_linear_sample_cuda",
            [
                "csrc/nd_linear_sample_cuda.cpp",
                "csrc/nd_linear_sample_cuda_kernel.cu",
            ],
        )
    ],
    cmdclass={"build_ext": BuildExtension},
)

When I install this directory using command pip install -e ., it failed and the result is:
Obtaining file:///home/xxx/Downloads/deformable_kernels/deformable_kernels/ops/deform_kernel
ERROR: More than one .egg-info directory found in /tmp/pip-pip-egg-info-pta6z__q

So I tried to separate the 2 setup()s in different setup.py files. It worked but I didn't get a python file. Instead a .so file was generated.
Does anyone know how to solve a problem like this?

Output: Check your pip version. I've had the same error (when installing other things in dev mode with pip) and downgrading to pip version 20.0.2 worked. Unsure why, but I've seen other folks on the internet solve the problem similarly.


Instruction: Efficient numpy broadcasting not found

Input:
It may be an easy problem but I could not find any practical solution. My code has following code segment involving 3 nested for loops. The target is to create a specialized intensity matrix for my algorithms for both prediction and ground_truth image matrix as follows:
    for i in range (batch):
      for j in range (img_width):
        for k in range (img_height):
            tensor=prediction[i][j][:]-prediction[i][k][:]
            extracted_intensity_pred[i][j][k]=torch.norm(tensor, 2)
            tensor=ground truth[i][j][:]-ground_truth[i][k][:]
            extracted_intensity_ground_truth[i][j][k]=torch.norm(tensor, 2)

This nested for loop structure is slowing the execution intensively. Is there any broadcasting implementation(in numpy or pytorch tensor based) that may be used?

Output:
first lets clean up some notation; [:] does nothing
But first what's the dimensions, mostly 3d?
for i in range (batch):
      for j in range (img_width):
        for k in range (img_height):
            tensor = prediction[i,j,:] - prediction[i,k,:]
            # looks like a prediction[:,:,None]-prediction[:,None,:]; making 4d?
            extracted_intensity_pred[i,j,k] = torch.norm(tensor, 2)
            # what can torch.norm work with?

so maybe it's just
 tensor = prediction[:,:,None] - prediction(:,None,:]
 extracted_intensity_pred = torch.norm(tensor, ?)


Instruction: CNN forward function , AutoTuning the number of layers

Input:
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features, trial):
        # we optimize the number of layers, hidden units and dropout ratio in each layer.
        n_layers = self.trial.suggest_int("n_layers", 1, 5)
        p = self.trial.suggest_uniform("dropout_1{}".format(i), 0, 1.0)
        layers = []
        for i in range(n_layers):
            self.out_features = self.trial.suggest_int("n_units_1{}".format(i), 16, 160,step =2)
            kernel_size = trial.suggest_int('kernel_size', 2, 7)
            layers.append(nn.Conv1d(1, self.out_features,kernel_size,1))
            layers.append(nn.RReLU())
            layers.append(nn.BatchNorm1d(self.out_features)
            layers.append(nn.Dropout(p))

            self.in_features = self.out_features

        layers.append(nn.Conv1d(self.in_features, 16,kernel_size,1))
        layers.append(nn.RReLU())
        return nn.Sequential(*layers)

As you can see above, I did some Optuna tuning of parameters , including tuning on the number of layers.
def forward(self,x):
    # shape x for conv 1d op
    x = x.view(-1, 1, self.in_features)
    x = self.conv1(x)
    x = F.rrelu(x)
    x = F.max_pool1d(x, 64, 64)
    x = self.conv2(x)
    x = F.rrelu(x)
    x = F.max_pool1d(x, 64, 64)
    x = x.view(-1, self.n_conv)

    x = self.dp(x)
    x = self.fc3(x)
    x = F.log_softmax(x, dim=1)
    return x

I now need to do the same for the forward function above, I did the pseudo code below but it won't run, kindly advise how. The main issue is to incorporate the for loop function for the forward function.
def forward(self,x):
    # shape x for conv 1d op
    x = x.view(-1, 1, self.in_features)
    for i in range(n_layers):
        layers.append(self.conv1(x))
        layers.append(F.rrelu(x))
        layers.append(F.max_pool1d(x, 64, 64))
    x = x.view(-1, self.n_conv)
    x = self.dp(x)
    x = self.fc3(x)
    #x = F.sigmoid(x)
    x = F.log_softmax(x, dim=1)
    return x

Output:
there are a bunch of errors that make it hard to understand what you intended to do :

Why would you build a nn.Sequential model in the __init__and not use it ?
What is this return instruction in __init__ ??
The successive convolution layers you create do not have matching channel sizes (in_channels is always 1). The out_feature of one iteration should be the in_features of the next iteration
Your pseudocode for the forward function appends tensors in a layers list (which you did not declare btw) and then does not use this list.
At the beginning of the forward, you reshape your input with x = x.view(-1, 1, self.in_features), but at that point in_features does not match at all the numer of input channels for the first convolution layer.

long story short : correct all the above errors, and then something like :
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features, trial):
        # do your stuff here ...
        self._model = nn.Sequential(*layers)
    def forward(self, x):
        return self._model(x)

should work


Instruction: how to calculate mahalanobis distance in pytorch?

Input: What is the most efficient way to calculate the mahalanobis distance: in pytorch?

Output:
Based on SciPy's implementation of the mahalanobis distance, you would do this in PyTorch. Assuming u and v are 1D and cov is the 2D covariance matrix.
def mahalanobis(u, v, cov):
    delta = u - v
    m = torch.dot(delta, torch.matmul(torch.inverse(cov), delta))
    return torch.sqrt(m)

Note: scipy.spatial.distance.mahalanobis takes in the inverse of the covariance matrix.


Instruction: Loss is “nan” when fine-tuning HuggingFace NLI model (both RoBERTa/BART)

Input:
I'm using HuggingFace's Transformer's library and I’m trying to fine-tune a pre-trained NLI model (ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli) on a dataset of around 276.000 hypothesis-premise pairs. I’m following the instructions from the docs here and here. I have the impression that the fine-tuning works (it does the training and saves the checkpoints), but trainer.train() and trainer.evaluate() return "nan" for the loss.
What I've tried:

I tried using both ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli and facebook/bart-large-mnli to make sure that it's not linked to specific model, but I get the issue for both models
I tried following the advice in this related github issue, but adding num_labels=3 to the config file does not solve the issue. (I think my issue is different because the models are already fine-tuned on NLI in my case)
I tried many different ways of changing my input data because I suspected that there could be an issue with my input data, but I also couldn't solve it that way.
The probable source of the issue: I inspected the prediction output from the model during training and the weird thing is the prediction value always seems to be "0" (entailment) in 100% of cases (see printed output in the code below). This is clearly an error. I think the source for this is that the logits the model seems to return during training are torch.tensor([[np.nan, np.nan, np.nan]]) and when you apply .argmax(-1) to this, you get torch.tensor(0). The big mystery for me is why the logits would become "nan", because the model does not do that when I use the same input data only outside of the trainer.
=> Does anyone know where this issues comes from? See my code below.

Thanks a lot in advance for any suggestion!
Here is my code:
### load model & tokenize
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

max_length = 256
hg_model_hub_name = "ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli"
# also tried: hg_model_hub_name = "facebook/bart-large-mnli"
tokenizer = AutoTokenizer.from_pretrained(hg_model_hub_name)
model = AutoModelForSequenceClassification.from_pretrained(hg_model_hub_name)
model.config

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")
if device == "cuda":
  model = model.half()
model.to(device)
model.train();

#... some data preprocessing

encodings_train = tokenizer(premise_train, hypothesis_train, return_tensors="pt", max_length=max_length,
                            return_token_type_ids=True, truncation=False, padding=True)
encodings_val = tokenizer(premise_val, hypothesis_val, return_tensors="pt", max_length=max_length,
                          return_token_type_ids=True, truncation=False, padding=True)
encodings_test = tokenizer(premise_test, hypothesis_test, return_tensors="pt", max_length=max_length,
                           return_token_type_ids=True, truncation=False, padding=True)

### create pytorch dataset object
class XDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels
    def __getitem__(self, idx):
        item = {key: torch.as_tensor(val[idx]) for key, val in self.encodings.items()}
        #item = {key: torch.as_tensor(val[idx]).to(device) for key, val in self.encodings.items()}
        item['labels'] = torch.as_tensor(self.labels[idx])
        #item['labels'] = self.labels[idx]
        return item
    def __len__(self):
        return len(self.labels)

dataset_train = XDataset(encodings_train, label_train)
dataset_val = XDataset(encodings_val, label_val)
dataset_test = XDataset(encodings_test, label_test)

# compute metrics with trainer
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
def compute_metrics(pred):
    labels = pred.label_ids
    print(labels)
    preds = pred.predictions.argmax(-1)
    print(preds)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary', pos_label=0)
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

## training
from transformers import Trainer, TrainingArguments

# https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments
training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=1,              # total number of training epochs
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=100,
)

trainer = Trainer(
    model=model,                         # the instantiated  Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=dataset_train,         # training dataset
    eval_dataset=dataset_val             # evaluation dataset
)

trainer.train()
# output: TrainOutput(global_step=181, training_loss=nan)
trainer.evaluate()
# output:
[2 2 2 0 0 2 2 2 0 2 0 0 2 2 2 2 0 2 0 2 2 2 2 0 2 0 2 0 0 2 0 0 2 0 0 0 2
 0 2 0 0 0 0 0 2 0 0 2 2 2 0 2 2 2 2 2 0 0 0 0 2 0 0 0 2 2 0 0 0 2 0 0 0 2
 2 0 2 0 0 2 2 2 0 2 2 0 0 0 0 0 0 0 2 0 0 0 0 2 0 2 2 0 2 0 0 2 2 2 2 2 2
 2 0 0 0 0 2 0 0 2 0 0 0 0 2 2 2 0 0 0 0 0 2 0 0 2 0 2 0 2 0 2 0 0 2 2 0 0
 2 2 2 2 2 2 0 0 2 2 2 2 0 2 0 0 2 2 2 0 0 2 0 2 0 2 0 0 0 0 0 0 2 0 0 2 2
 0 2 2 2 0 2 2 0 2 2 2 2 2 2 0 0 2 0 0 2 2 0 0 0 2 0 2 2 2 0 0 0 0 0 0 0 0
 2 0 2 2 2 0 2 0 0 2 0 2 2 0 0 0 0 2 2 2 0 0 0 2 2 2 2 0 2 0 2 2 2]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

{'epoch': 1.0,
 'eval_accuracy': 0.5137254901960784,
 'eval_f1': 0.6787564766839378,
 'eval_loss': nan,
 'eval_precision': 0.5137254901960784,
 'eval_recall': 1.0}

Edit:  I've also opened a github issue with a but more detailed description of the issue here: https://github.com/huggingface/transformers/issues/9160

Output: I received a good answer from the HuggingFace team on github. The issue was the model.half(), which has the advantage of increasing speed and reducing memory usage, but it also changes the model in a way that it produces the error. removing the model.half() solved the issue for me. For details, see https://github.com/huggingface/transformers/issues/9160


Instruction: What memory does Transformer Decoder Only use?

Input: I've been reading a lot about transformers and self attention and have seen both BERT and GPT-2 are a newer version that only use an encoder transformer (BERT) and decoder transformer (GPT-2). I've been trying to build a decoder only model for myself for next sequence prediction but am confused by one thing. I'm using PyTorch and have looked at thereSeq2Seq tutorial and then looked into the Transformer Decoder Block which is made up of Transformer Decoder Layers. My confusion comes from the memory these need to be passed as well. In the documentation they say memory is the last layer of the encoder block which makes sense for a Seq2Seq model but I'm wanting to make a decoder only model. So my question is what do you pass a decoder only model like GPT-2 for memory if you do not have an encoder?

Output: After further investigation I believe I can now answer this myself. A decoder only transformer doesn't actually use any memory as there is no encoder-decoder self attention in it like there is in a encoder-decoder transformer. A decoder only transformer looks a lot like an encoder transformer only instead it uses a masked self attention layer over a self attention layer. In order to do this you can pass a square subsequent mask (upper triangle) so that the model cannot look forward to achieve a decoder only model like found in GPT-2/GPT-3.


Instruction: Adam optimizer with warmup on PyTorch

Input:
In the paper Attention is all you need, under section 5.3, the authors suggested to increase the learning rate linearly and then decrease proportionally to the inverse square root of steps.

How do we implement this in PyTorch with Adam optimizer? Preferably without additional packages.

Output:
PyTorch provides learning-rate-schedulers for implementing various methods of adjusting the learning rate during the training process.
Some simple LR-schedulers are are already implemented and can be found here: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
In your special case you can - just like the other LR-schedulers do - subclass _LRScheduler for implementing a variable schedule based on the number of epochs. For a bare-bones method you only need to implement __init__() and get_lr() methods.
Just note that many of these schedulers expect you to call .step() once per epoch. But you can also update it more frequently or even pass a custom argument just like in the cosine-annealing LR-scheduler: https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CosineAnnealingLR


Instruction: How to store wrong predictions during evaluation on the CNN

Input:
During evaluation, I want to store unique ids that are wrongly predicted to do some more processing.
It is a multiclass prediction problem
Here is the code during the evaluation:
 outputs = model(imgs)
 loss = criterion(outputs, targets)  # Prediction error

 val_loss += loss.item()
 predicted = torch.argmax(outputs, dim=1)
 t_predicted +=predicted.cpu().tolist()

 total += targets.size(0)
 good_answers = (predicted == targets)
 correct += good_answers.sum().item()

Knowing that ids is a list of the ids of the images,  When I try to get the ids that are wrong:
wrong_ids += ids[~(good_answers.to('cpu'))]

I get this error:
add(): argument 'other' (position 1) must be Tensor

Output:
The question contained a tensorflow tag, so I was preparing an answer. After completing my write up, I've found that this tag is removed. However, I believe my answer can give insight into this general question of whether they're using tf or pytorch.

Data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# train set / data
x_train = x_train.astype('float32') / 255

# validation set / data
x_test = x_test.astype('float32') / 255

# train set / target
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
# validation set / target
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)

print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

Train
import tensorflow as tf

# declare input shape
input = tf.keras.Input(shape=(32,32,3))
# Block 1
x = tf.keras.layers.Conv2D(32, 3, strides=2, activation="relu")(input)
x = tf.keras.layers.MaxPooling2D(3)(x)
# Now that we apply global max pooling.
gap = tf.keras.layers.GlobalMaxPooling2D()(x)
# Finally, we add a classification layer.
output = tf.keras.layers.Dense(10, activation='softmax')(gap)
# bind all
func_model = tf.keras.Model(input, output)

print('\nFunctional API')
func_model.compile(
          metrics=['accuracy'],
          loss      = 'categorical_crossentropy',
          optimizer = tf.keras.optimizers.Adam()
          )

func_model.fit(x_train, y_train)

Error Prediction
# Predict the values from the validation dataset
y_pred = func_model.predict(x_test)
# Convert predictions classes to one hot vectors
y_pred_classes = np.argmax(y_pred, axis = 1)
y_test = np.argmax(y_test, axis=1)

# Errors are difference between predicted labels and true labels
errors = (y_pred_classes - y_test != 0)

y_pred_classes_errors = y_pred_classes[errors]
y_pred_errors = y_pred[errors]
y_true_errors = y_test[errors]
x_test_errors = x_test[errors]

# Probabilities of the wrong predicted numbers
y_pred_errors_prob = np.max(y_pred_errors, axis = 1)

# Predicted probabilities of the true values in the error set
true_prob_errors = np.diagonal(np.take(y_pred_errors, y_true_errors, axis=1))

# Difference between the probability of the predicted label and the true label
delta_pred_true_errors = y_pred_errors_prob - true_prob_errors

# Sorted list of the delta prob errors
sorted_dela_errors = np.argsort(delta_pred_true_errors)

# Top 6 errors
most_important_errors = sorted_dela_errors[-6:]

Display
import matplotlib.pyplot as plt

def display_errors(errors_index,img_errors,pred_errors, obs_errors):
    """ This function shows 6 images with their predicted and real labels"""
    n = 0
    nrows = 2
    ncols = 3
    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)
    for row in range(nrows):
        for col in range(ncols):
            error = errors_index[n]
            ax[row,col].imshow((img_errors[error]).reshape((32,32,3)))
            ax[row,col].set_title("Predicted label :{}\nTrue label :{}".format(pred_errors[error],obs_errors[error]))
            n += 1

# Show the top 6 errors
display_errors(most_important_errors, x_test_errors, y_pred_classes_errors, y_true_errors)


Instruction: How to set hydra's parameter HYDRA_FULL_ERROR?

Input:
When I use hydra in a python pytorch project, the operation result prompt “Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.”
But i don't konw how to set it.

Output:
You set an environment variable in the shell.
For a specific run:
$ HYDRA_FULL_ERROR=1 python foo.py

Or for all runs in this shell session:
$ export HYDRA_FULL_ERROR=1
$ python foo.py

However, you shouldn't normally need to set it. This is more of a debugging backdoor in case of issues with Hydra itself.
If you hit a case where you can only understand your issue after setting HYDRA_FULL_ERROR, please file an issue.


Instruction: Torch gather middle dimension

Input:
Let a be a (n, d, l) tensor. Let indices be a (n, 1) tensor, containing indices. I want to gather from a in the middle dimension tensors from indices given by indices. The resulting tensor would therefore be of shape (n, l).
n = 3
d = 2
l = 3

a = tensor([[[ 0,  1,  2],
             [ 3,  4,  5]],

            [[ 6,  7,  8],
             [ 9, 10, 11]],

            [[12, 13, 14],
             [15, 16, 17]]])

indices = tensor([[0],
                  [1],
                  [0]])

# Shape of result is (n, l)
result = tensor([[ 0,  1,  2],  # a[0, 0, :] since indices[0] == 0

                 [ 9, 10, 11],  # a[1, 1, :] since indices[1] == 1

                 [12, 13, 14]]) # a[2, 0, :] since indices[2] == 0

This is indeed similar to a.gather(1, indices), but gather won't work since indices does not have the same shape as a. How can I use gather in this setting? Or what should I use?

Output:
You can create the indices manually. The indices tensor has to be flattened if it has the shape of your example data.
a[torch.arange(len(a)),indices.view(-1)]
# equal to a[[0,1,2],[0,1,0]]

Out:
tensor([[ 0,  1,  2],
        [ 9, 10, 11],
        [12, 13, 14]])


Instruction: Why is my dynamic pytorch model definition not complete?

Input:
I tried to create a custom pytorch model class in a way that would allow variable number of hidden layers. Everything seems to "work" in the code, however even if I set 10 hidden layers, none of them show up in the print out of the model definition. I am wondering why this is happening? I can see obviously it has something to do with the way I use a loop to apply the hidden layers.
Code:
class FFModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(FFModel, self).__init__()
        self.input_layer = nn.Linear(input_dim, hidden_dim).to(device)
        self.hidden_layers = [ nn.Linear(hidden_dim, hidden_dim).to(device) for _ in range(layer_dim) ]
        self.activation = nn.LeakyReLU().to(device)
        self.output_layer = nn.Linear(hidden_dim, output_dim).to(device)

    def forward(self, x):
        y_hat = self.activation( self.input_layer(x) )
        for hidden_layer in self.hidden_layers:
            y_hat = self.activation( hidden_layer(y_hat) )
        out = self.output_layer(y_hat)
        return out

model = FFModel(2, 5, 10, 1)
model

Output:
FFModel(
  (input_layer): Linear(in_features=2, out_features=5, bias=True)
  (activation): LeakyReLU(negative_slope=0.01)
  (output_layer): Linear(in_features=5, out_features=1, bias=True)
)

Indeed, if I inspect deeper and parse through the parameters, even though this model should have 10 layers, the parameters are not correct for a model of 10 hidden layers. Clearly this model doesn't have 10 layers as I thought it would.
Why are the hidden layers all gone?
for param in model.parameters():
    print (param)

out:
Parameter containing:
tensor([[-0.1064, -0.0171],
        [-0.5892, -0.5391],
        [-0.6420, -0.5128],
        [ 0.0128,  0.2155],
        [ 0.3406,  0.1825]], device='cuda:0', requires_grad=True)
Parameter containing:
tensor([-0.5432, -0.0650,  0.6447,  0.4615,  0.4772], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([[ 0.0615,  0.0901,  0.3399, -0.3341,  0.2029]], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([0.1742], device='cuda:0', requires_grad=True)

Output:
PyTorch cannot see your list object. You need to use nn.ModuleList:
self.hidden_layers = nn.ModuleList([nn.Linear(hidden_dim, hidden_dim).to(device) for _ in range(layer_dim)])


Instruction: How to convert a yolo darknet format into .csv file

Input:
I have a few annotations that is originally in YOLO format. I need to convert it into yolo csv format in order to train with my transformers model.
Sample .csv file I need:
Sample annotation file in CSV format
The csv attributes include: image_id, width, height and coordinates of the image's bounding box.
Any help would be appreciated!

Output:
first of all i should say there is no straight way to convert those format into csv. you should read files and parse their data.
Step 1: import libraries
we need to read txt files ( yolo labels ) from a directory and save them into csv. so we need these libraries :
import os
import glob
import pandas as pd
import numpy as np

Step 2: get list of your yolo labels
open directory where you have your yolo labels with OS library and read txt files with glob:
os.chdir(r'D:\karami\Labeled\train1\labels')
myFiles = glob.glob('*.txt')

my labels are in my labels folder so i set that directory to that.
Step 3: read lines and split data
you have your labels list in myFiles variable
you should iterate on it and read the first line of each file and split its data.
in the image you shared we have absolute coordinate of bounding boxes. according to yolo darknet document we have :

object-class x_center y_center width height
x_center,y_center,width,height - float values relative to width and
height of image, it can be equal from (0.0 to 1.0] for example:  =
<absolute_x> / <image_width> or  = <absolute_height> /
<image_height>

you can see full description in here
so we need to multiply our coordinates into width and height of images.
yolo file doesnt contain image size, so you should define them before
after that use your splitted string. then save your data frame. code :
width=1024
height=1024
image_id=0
final_df=[]
for item in myFiles:
    row=[]
    bbox_temp=[]
    with open(item, 'rt') as fd:
        first_line = fd.readline()
        splited = first_line.split();

        row.append(image_id)
        row.append(width)
        row.append(height)
        try:
            bbox_temp.append(float(splited[1])*width)
            bbox_temp.append(float(splited[2])*height)
            bbox_temp.append(float(splited[3])*width)
            bbox_temp.append(float(splited[4])*height)
            row.append(bbox_temp)
            final_df.append(row)
        except:
            print("file is not in YOLO format!")
df = pd.DataFrame(final_df,columns=['image_id', 'width', 'height','bbox'])
df.to_csv("saved.csv",index=False)

here is my csv file :


Instruction: "MisconfigurationError: No TPU devices were found" even when TPU is connected in PyTorch Lightning

Input:
Have been frustrated over the past few hours over a problem, though It's likely its a problem I started myself hah.
I'm trying to connect to the TPU in Colab. I'm pretty sure I've gotten all the import stuff down. My code is here. I'm not completely set on everything, so the entire document isn't functional, but you should be able to see my attempts at TPU connection.
I'm running Pytorch in version 1.5.0 and torchvision in 0.6.0 because I was finding I couldn't install XLA with anything later than 1.5.0. I'm running XLA in version 20200325.
This is the image that seems so confusing: It states that we have a connection with xla: 1 yet when trying to flag it in the trainer I get an error saying no TPUs can be found.

If anyone could help me, that would be amazing.
Thanks,
A

Output:
I suffered the same problem, and these steps solved the problem.

Follow the PyTorch-Lightning docs: TPU SUPPORT
Add another notebook cell:

%%capture
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py > /dev/null
!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev > /dev/null
!pip install pytorch-lightning > /dev/null


Instruction: PyTorch LinearLayer+BatchNorm1d with a 3D input

Input: I would like to apply a BatchNorm1d after a Linear. My input is a 3D multivariate time series of shape [batch_size, n_variables, timesteps]. The Linear performs the linear transformation on the third dimension so that the new shape is [batch_size, n_variables, LinearLayer_out_features]. My problem occurs with the BatchNorm1d, I would like to apply it on the third dimension but, for a 3D input, BatchNorm1d operation is done over the second dimension (even for a 3D tensor). Do you have any suggestion on how to do that?

Output:
Why not transpose the input to BatchNorm1d and then transpose it back?
m=Linear(.....)
m=torch.transpose(BatchNorm1D(torch.transpose(m,1,2)),1,2)

This doesn't create a copy of your tensor.
https://pytorch.org/docs/stable/generated/torch.transpose.html


Instruction: Running two different independent PyTorch programs on a single GPU

Input:
I have a single NVIDIA GPU which has a memory of 16GB. I have to run two different (and independent; meaning, two different problems: one is a vision type task, another is NLP task) Python programs. The codes are written using PyTorch and both the codes can use GPU.
I have tested that program 1 takes roughly 5GB of GPU memory, and the rest is free. If I run the two programs, will it hamper the model performance or will it cause any process conflicts?
Linked question; but it does not necessarily mean PyTorch codes

Output:
I do not know the details of how this works, but I can tell from experience that both programs will run well (as long as they do not need more than 16GB of RAM when combined), and execution times should stay roughly the same.
However, computer vision usually requires a lot of IO (mostly reading images), if the other task needs to read files too, this part may become slower than when running both programs individually.


Instruction: Tweak positional encodings shape (DETR model) to support batchsize > 1

Input:
Referencing from this notebook, and would like to scale this to support batch size > 1, as it state on in the comments Only batch size 1 supported.. I'm having trouble tweaking the pos statement inside the forward(). How to go about doing this? Any tips will be very helpful too.
This is the original code taken from the notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb#scrollTo=h91rsIPl7tVl):
import torch
from torch import nn
from torchvision.models import resnet50
import torchvision.transforms as T
torch.set_grad_enabled(False);

class DETRdemo(nn.Module):
    """
    Demo DETR implementation.

    Demo implementation of DETR in minimal number of lines, with the
    following differences wrt DETR in the paper:
    * learned positional encoding (instead of sine)
    * positional encoding is passed at input (instead of attention)
    * fc bbox predictor (instead of MLP)
    The model achieves ~40 AP on COCO val5k and runs at ~28 FPS on Tesla V100.
    Only batch size 1 supported.
    """
    def __init__(self, num_classes, hidden_dim=256, nheads=8,
                 num_encoder_layers=6, num_decoder_layers=6):
        super().__init__()

        # create ResNet-50 backbone
        self.backbone = resnet50()
        del self.backbone.fc

        # create conversion layer
        self.conv = nn.Conv2d(2048, hidden_dim, 1)

        # create a default PyTorch transformer
        self.transformer = nn.Transformer(
            hidden_dim, nheads, num_encoder_layers, num_decoder_layers)

        # prediction heads, one extra class for predicting non-empty slots
        # note that in baseline DETR linear_bbox layer is 3-layer MLP
        self.linear_class = nn.Linear(hidden_dim, num_classes + 1)
        self.linear_bbox = nn.Linear(hidden_dim, 4)

        # output positional encodings (object queries)
        self.query_pos = nn.Parameter(torch.rand(100, hidden_dim))

        # spatial positional encodings
        # note that in baseline DETR we use sine positional encodings
        self.row_embed = nn.Parameter(torch.rand(50, hidden_dim // 2))
        self.col_embed = nn.Parameter(torch.rand(50, hidden_dim // 2))

    def forward(self, inputs):
        # propagate inputs through ResNet-50 up to avg-pool layer
        x = self.backbone.conv1(inputs)
        x = self.backbone.bn1(x)
        x = self.backbone.relu(x)
        x = self.backbone.maxpool(x)

        x = self.backbone.layer1(x)
        x = self.backbone.layer2(x)
        x = self.backbone.layer3(x)
        x = self.backbone.layer4(x)

        # convert from 2048 to 256 feature planes for the transformer
        h = self.conv(x)

        # construct positional encodings
        H, W = h.shape[-2:]
        pos = torch.cat([ ## <--- trouble scaling `pos` to support batch size > 1
            self.col_embed[:W].unsqueeze(0).repeat(H, 1, 1),
            self.row_embed[:H].unsqueeze(1).repeat(1, W, 1),
        ], dim=-1).flatten(0, 1).unsqueeze(1)

        # propagate through the transformer
        h = self.transformer(pos + 0.1 * h.flatten(2).permute(2, 0, 1),
                             self.query_pos.unsqueeze(1)).transpose(0, 1)

        # finally project transformer outputs to class labels and bounding boxes
        return {'pred_logits': self.linear_class(h),
                'pred_boxes': self.linear_bbox(h).sigmoid()}

For development:
model = DETRdemo(num_classes=10)
x = torch.ones((1,3,128,128)) # <-- this is ok
y = model(x)

model = DETRdemo(num_classes=10)
x = torch.ones((2,3,128,128)) # <-- this is NOT ok
y = model(x)

Output:
Have a look at their code on github: https://github.com/facebookresearch/detr
The code there allows for arbitrary batch sizes, see e.g. the Evaluation section in their Readme file.


Instruction: AssertionError: No inf checks were recorded for this optimizer in Pytorch's AutomaticMixedPrecision

Input:
I'm using AutomaticMixedPrecision feature of PyTorch to train a network with smaller footprint and precision.
At a certain point some embeddings from the network have NaNs in their tensors, so I'd like to replace those with 0s in order to perform online hard negative samples mining.
However, after replacing the NaNs in the tensor like this:
tensor[torch.isnan(tensor)] = 0

I get the following error while doing the next scaler ste (scaler.step(optimizer):
    assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.

What's the correct way to zero out NaNs while getting rid of this error?

Output:
could you show us your full code. Generally it is advisable to just skip the step (batch) if it has NaNs.
Also take a look at torch.nan_to_num.


Instruction: Replacing a max pooling layer with an average pooling layer on a VGG model

Input:
I'm following this article and I try to implement this function:
def replace_max_pooling(model):
 '''
 The function replaces max pooling layers with average pooling layers with
 the following properties: kernel_size=2, stride=2, padding=0.
 '''
 for layer in model.layers:
    if layer is max pooling:
        replace

But I get an error on the iteration that says:

ModuleAttributeError: 'VGG' object has no attribute 'layers'...

How can I do this correctly?

Output:
The VGG model provided by Torchvision contains three components: the features sub-module, avgpool (the adaptive average pool), and the classifier. You need to be looking into the head of the network, where the convolution and pool layers are located: features.
You can loop over the layers of a nn.Module with named_children(). However there are other ways of going about this. You can use isinstance to determine if the layer is of a particular type.
In this particular model, layers are named by there index. So in order to locate the appropriate layers in the nn.Module and overwrite them, we can convert the names to  ints.
for i, layer in m.features.named_children():
    if isinstance(layer, torch.nn.MaxPool2d):
        m.features[int(i)] = nn.AvgPool2d(kernel_size=2, stride=2, padding=0)

Having setup beforehand:
import torch
import torch.nn as nn
m = models.vgg16()


Instruction: What is hp_metric in TensorBoard and how to get rid of it?

Input:
I am new to Tensorboard.
I am using fairly simple code running an experiment, and this is the output:

I don't remember asking for a hp_metric graph, yet here it is.
What is it and how do I get rid of it?

Full code to reproduce, using Pytorch Lightning (not that I think anyone should have to reproduce this to answer):
Please notice the ONLY line dereferencing TensorBoard is
self.logger.experiment.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)

import torch
from torch import nn
import torch.nn.functional as F
from typing import List, Optional
from pytorch_lightning.core.lightning import LightningModule
from Testing.Research.toy_datasets.ClustersDataset import ClustersDataset
from torch.utils.data import DataLoader
from Testing.Research.config.ConfigProvider import ConfigProvider
from pytorch_lightning import Trainer, seed_everything
from torch import optim
import os
from pytorch_lightning.loggers import TensorBoardLogger

class VAEFC(LightningModule):
    # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73
    # for possible upgrades, see https://arxiv.org/pdf/1602.02282.pdf
    # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational-auto-encoder
    def __init__(self, encoder_layer_sizes: List, decoder_layer_sizes: List, config):
        super(VAEFC, self).__init__()
        self._config = config
        self.logger: Optional[TensorBoardLogger] = None

        assert len(encoder_layer_sizes) >= 3, "must have at least 3 layers (2 hidden)"
        # encoder layers
        self._encoder_layers = nn.ModuleList()
        for i in range(1, len(encoder_layer_sizes) - 1):
            enc_layer = nn.Linear(encoder_layer_sizes[i - 1], encoder_layer_sizes[i])
            self._encoder_layers.append(enc_layer)

        # predict mean and covariance vectors
        self._mean_layer = nn.Linear(encoder_layer_sizes[
                                         len(encoder_layer_sizes) - 2],
                                     encoder_layer_sizes[len(encoder_layer_sizes) - 1])
        self._logvar_layer = nn.Linear(encoder_layer_sizes[
                                           len(encoder_layer_sizes) - 2],
                                       encoder_layer_sizes[len(encoder_layer_sizes) - 1])

        # decoder layers
        self._decoder_layers = nn.ModuleList()
        for i in range(1, len(decoder_layer_sizes)):
            dec_layer = nn.Linear(decoder_layer_sizes[i - 1], decoder_layer_sizes[i])
            self._decoder_layers.append(dec_layer)

        self._recon_function = nn.MSELoss(reduction='mean')

    def _encode(self, x):
        for i in range(len(self._encoder_layers)):
            layer = self._encoder_layers[i]
            x = F.relu(layer(x))

        mean_output = self._mean_layer(x)
        logvar_output = self._logvar_layer(x)
        return mean_output, logvar_output

    def _reparametrize(self, mu, logvar):
        if not self.training:
            return mu
        std = logvar.mul(0.5).exp_()
        if std.is_cuda:
            eps = torch.cuda.FloatTensor(std.size()).normal_()
        else:
            eps = torch.FloatTensor(std.size()).normal_()
        reparameterized = eps.mul(std).add_(mu)
        return reparameterized

    def _decode(self, z):
        for i in range(len(self._decoder_layers) - 1):
            layer = self._decoder_layers[i]
            z = F.relu((layer(z)))

        decoded = self._decoder_layers[len(self._decoder_layers) - 1](z)
        # decoded = F.sigmoid(self._decoder_layers[len(self._decoder_layers)-1](z))
        return decoded

    def _loss_function(self, recon_x, x, mu, logvar, reconstruction_function):
        """
        recon_x: generating images
        x: origin images
        mu: latent mean
        logvar: latent log variance
        """
        binary_cross_entropy = reconstruction_function(recon_x, x)  # mse loss TODO see if mse or cross entropy
        # loss = 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
        kld_element = mu.pow(2).add_(logvar.exp()).mul_(-1).add_(1).add_(logvar)
        kld = torch.sum(kld_element).mul_(-0.5)
        # KL divergence Kullback–Leibler divergence, regularization term for VAE
        # It is a measure of how different two probability distributions are different from each other.
        # We are trying to force the distributions closer while keeping the reconstruction loss low.
        # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73

        # read on weighting the regularization term here:
        # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational
        # -auto-encoder
        return binary_cross_entropy + kld * self._config.regularization_factor

    def training_step(self, batch, batch_index):
        orig_batch, noisy_batch, _ = batch
        noisy_batch = noisy_batch.view(noisy_batch.size(0), -1)

        recon_batch, mu, logvar = self.forward(noisy_batch)

        loss = self._loss_function(
            recon_batch,
            orig_batch, mu, logvar,
            reconstruction_function=self._recon_function
        )
        # self.logger.experiment.add_scalars("losses", {"train_loss": loss})
        self.logger.experiment.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)
        # self.logger.experiment.add_scalar("train_loss", loss, self.current_epoch)
        self.logger.experiment.flush()
        return loss

    def train_dataloader(self):
        default_dataset, train_dataset, test_dataset = ClustersDataset.clusters_dataset_by_config()
        train_dataloader = DataLoader(train_dataset, batch_size=self._config.batch_size, shuffle=True)
        return train_dataloader

    def test_dataloader(self):
        default_dataset, train_dataset, test_dataset = ClustersDataset.clusters_dataset_by_config()
        test_dataloader = DataLoader(test_dataset, batch_size=self._config.batch_size, shuffle=True)
        return test_dataloader

    def configure_optimizers(self):
        optimizer = optim.Adam(model.parameters(), lr=self._config.learning_rate)
        return optimizer

    def forward(self, x):
        mu, logvar = self._encode(x)
        z = self._reparametrize(mu, logvar)
        decoded = self._decode(z)
        return decoded, mu, logvar

if __name__ == "__main__":
    config = ConfigProvider.get_config()
    seed_everything(config.random_seed)
    latent_dim = config.latent_dim
    enc_layer_sizes = config.enc_layer_sizes + [latent_dim]
    dec_layer_sizes = [latent_dim] + config.dec_layer_sizes
    model = VAEFC(config=config, encoder_layer_sizes=enc_layer_sizes, decoder_layer_sizes=dec_layer_sizes)

    logger = TensorBoardLogger(save_dir='tb_logs', name='VAEFC')
    logger.hparams = config  # TODO only put here relevant stuff
    # trainer = Trainer(gpus=1)
    trainer = Trainer(deterministic=config.is_deterministic,
                      #auto_lr_find=config.auto_lr_find,
                      #log_gpu_memory='all',
                      # min_epochs=99999,
                      max_epochs=config.num_epochs,
                      default_root_dir=os.getcwd(),
                      logger=logger
                      )
    # trainer.tune(model)
    trainer.fit(model)
    print("done training vae with lightning")

ClustersDataset.py
from torch.utils.data import Dataset
import matplotlib.pyplot as plt
import torch
import numpy as np
from Testing.Research.config.ConfigProvider import ConfigProvider

class ClustersDataset(Dataset):
    __default_dataset = None
    __default_dataset_train = None
    __default_dataset_test = None

    def __init__(self, cluster_size: int, noise_factor: float = 0, transform=None, n_clusters=2, centers_radius=4.0):
        super(ClustersDataset, self).__init__()
        self._cluster_size = cluster_size
        self._noise_factor = noise_factor
        self._n_clusters = n_clusters
        self._centers_radius = centers_radius
        # self._transform = transform
        self._size = self._cluster_size * self._n_clusters

        self._create_data_clusters()
        self._combine_clusters_to_array()
        self._normalize_data()
        self._add_noise()

        # self._plot()
        pass

    @staticmethod
    def clusters_dataset_by_config():
        if ClustersDataset.__default_dataset is not None:
            return \
                ClustersDataset.__default_dataset, \
                ClustersDataset.__default_dataset_train, \
                ClustersDataset.__default_dataset_test
        config = ConfigProvider.get_config()
        default_dataset = ClustersDataset(
            cluster_size=config.cluster_size,
            noise_factor=config.noise_factor,
            transform=None,
            n_clusters=config.n_clusters,
            centers_radius=config.centers_radius
        )

        train_size = int(config.train_size * len(default_dataset))
        test_size = len(default_dataset) - train_size
        train_dataset, test_dataset = torch.utils.data.random_split(default_dataset, [train_size, test_size])

        ClustersDataset.__default_dataset = default_dataset
        ClustersDataset.__default_dataset_train = train_dataset
        ClustersDataset.__default_dataset_test = test_dataset

        return default_dataset, train_dataset, test_dataset

    def _create_data_clusters(self):
        self._clusters = [torch.zeros((self._cluster_size, 2)) for _ in range(self._n_clusters)]
        centers_radius = self._centers_radius
        for i, c in enumerate(self._clusters):
            r, x, y = 3.0, centers_radius * np.cos(i * np.pi * 2 / self._n_clusters), centers_radius * np.sin(
                i * np.pi * 2 / self._n_clusters)
            cluster_length = 1.1
            cluster_start = i * 2 * np.pi / self._n_clusters
            cluster_end = cluster_length * (i + 1) * 2 * np.pi / self._n_clusters
            cluster_inds = torch.linspace(start=cluster_start, end=cluster_end, steps=self._cluster_size,
                                          dtype=torch.float)
            c[:, 0] = r * torch.sin(cluster_inds) + y
            c[:, 1] = r * torch.cos(cluster_inds) + x

    def _plot(self):
        plt.figure()
        plt.scatter(self._noisy_values[:, 0], self._noisy_values[:, 1], s=1, color='b', label="noisy_values")
        plt.scatter(self._values[:, 0], self._values[:, 1], s=1, color='r', label="values")
        plt.legend(loc="upper left")
        plt.show()

    def _combine_clusters_to_array(self):
        size = self._size
        self._values = torch.zeros(size, 2)
        self._labels = torch.zeros(size, dtype=torch.long)
        for i, c in enumerate(self._clusters):
            self._values[i * self._cluster_size: (i + 1) * self._cluster_size, :] = self._clusters[i]
            self._labels[i * self._cluster_size: (i + 1) * self._cluster_size] = i

    def _add_noise(self):
        size = self._size

        mean = torch.zeros(size, 2)
        std = torch.ones(size, 2)
        noise = torch.normal(mean, std)
        self._noisy_values = torch.zeros(size, 2)
        self._noisy_values[:] = self._values
        self._noisy_values = self._noisy_values + noise * self._noise_factor

    def _normalize_data(self):
        values_min, values_max = torch.min(self._values), torch.max(self._values)
        self._values = (self._values - values_min) / (values_max - values_min)
        self._values = self._values * 2 - 1

    def __len__(self):
        return self._size  # number of samples in the dataset

    def __getitem__(self, index):
        item = self._values[index, :]
        noisy_item = self._noisy_values[index, :]
        # if self._transform is not None:
        #     noisy_item = self._transform(item)
        return item, noisy_item, self._labels[index]

    @property
    def values(self):
        return self._values

    @property
    def noisy_values(self):
        return self._noisy_values

Config values (ConfigProvider just returns those as an object)
num_epochs: 15
batch_size: 128
learning_rate: 0.0001
auto_lr_find: False

noise_factor: 0.1
regularization_factor: 0.0

cluster_size: 5000
n_clusters: 5
centers_radius: 4.0
train_size: 0.8

latent_dim: 8

enc_layer_sizes: [2, 200, 200, 200]
dec_layer_sizes: [200, 200, 200, 2]

retrain_vae: False
random_seed: 11
is_deterministic: True

Output:
It's the default setting of tensorboard in pytorch lightning. You can set default_hp_metric to false to get rid of this metric.
TensorBoardLogger(save_dir='tb_logs', name='VAEFC', default_hp_metric=False)

The hp_metric helps you track the model performance across different hyperparameters. You can check it at hparams in your tensorboard.


Instruction: How to draw a scatter plot in Tensorboard Pytorch?

Input:
Assuming I want a generic scatter plot drawn in TensorBoard that draws the 1st batch[:, 0], batch[:, 1] of every epoch.
How can that be done in TensorBoard?
An old similar question (2017 january) has a workaround, but I hope we now (2020 december) have the technology for a real solution.
Not enough is my attempt:
if self._current_epoch == 0:
    self.logger.experiment.add_scalars("epoch", {"batch": batch[:, 1]}, batch[:, 0])

Gives me the wonderful error

assert(scalar.squeeze().ndim == 0), 'scalar should be 0D'

Output:
If I understand your question right, you could use add_images, add_figure to add image or figure to tensorboard(docs).
Sample code:
from torch.utils.tensorboard import SummaryWriter
import numpy as np
import matplotlib.pyplot as plt

# create summary writer
writer = SummaryWriter('lightning_logs')

# write dummy image to tensorboard
img_batch = np.zeros((16, 3, 100, 100))
writer.add_images('my_image_batch', img_batch, 0)

# write dummy figure to tensorboard
plt.imshow(np.transpose(img_batch[0], [1, 2, 0]))
plt.title('example title')
writer.add_figure('my_figure_batch', plt.gcf(), 0)
writer.close()


Instruction: How to cache big data in memory (efficiently) in complex variables across executions of Python scripts?

Input:
I am trying to call (from Java Spring beans) Python Pytorch scripts that contains trained neural networks for the use: my Pytorch neural networks are neural functions that accepts state, encodes it and returns the action, decodes all this is according to learned/trained policy.
So - each time when I am trying to invoke Python script I should construc torch.nn, load weights and biases from some external store (DB, file) and then execute this nn to get single answer, pretty expensive operation.
How can I keep torch.nn instance (with loaded weights and biases) in memory and make it available immediately for each execution of Python script?
Memcached is not the solution, because it can keep string or binary values only and it is quite expensive to serialized and deserialize torch.nn instance. One suggestion was Tensorflow Serving, I am currently researching it, so - I don't yet know whether this is the answer.
It is quite possible that Tensorflow has some caching technologies which I can use for the Pytroch as well?

Output:
If I understand your use case correctly, what you need is a model server that keeps the model loaded and ideally also handles any exceptions from incorrect data.
One rather straightforward way to transform your inference script into a tensorflow-serving-like callable service is the python library flask. Another way seems to be a new tool called torchserve.


Instruction: tensor type attributes in bert model returned as string

Input:
I am new to nlp and i want to build a bert model for sentiment Analysis so i am following this tuto
https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/
but i am getting the error bellow
bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)

last_hidden_state, pooled_output = bert_model(
   input_ids=encoding['input_ids'],
   attention_mask=encoding['attention_mask']
)
last_hidden_state.shape
pooled_output.shape

When i want to execute last_hidden_state.shape I get an error:
'str' object has no attribute 'shape'
why does it return last_hidden_state and pooled_output as str and not tensors.
Thank you.

Output:
it seems there was A couple of changes were introduced when the switch from version 3 to version 4 was done in hugging face and can be solved like below
bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME, return_dict=False)


Instruction: What are the numbers in torch.transforms.normalize and how to select them?

Input:
I am following some tutorials and I keep seeing different numbers that seem quite arbitrary to me in the transforms section
namely,
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

or
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

or
transform = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

or others.
I wonder where these numbers arise, and how to know to select the correct ones?
I am about to use MNIST for sanity, but very soon to use my own unique dataset and will probably need my own normaliztion.

Output:
Normalize in pytorch context subtracts from each instance (MNIST image in your case) the mean (the first number) and divides by the standard deviation (second number). This takes place for each channel separately, meaning in mnist you only need 2 numbers because images are grayscale, but on let's say cifar10 which has colored images you would use something along the lines of your last sform (3 numbers for mean and 3 for std).
So basically each input image in MNIST gets transformed from [0,255] to [0,1] because you transform an image to Tensor (source: https://pytorch.org/docs/stable/torchvision/transforms.html --
Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] if the PIL Image belongs to one of the modes (L, LA, P, I, F, RGB, YCbCr, RGBA, CMYK, 1) or if the numpy.ndarray has dtype = np.uint8)
After that you want your input image to have values in a range like [0,1] or [-1,1] to help your model converge to the right direction (Many reasons why scaling takes place, e.g. NNs prefer inputs around that range to avoid gradient saturation). Now as you probably noticed passing 0.5 and 0.5 in Normalize would yield vales in range:
Min of input image = 0 -> 0-0.5 = -0.5 -> gets divided by 0.5 std -> -1
Max of input image = 255 -> toTensor -> 1 -> (1 - 0.5) / 0.5 -> 1
so it transforms your data in a range [-1, 1]


Instruction: Convert image to tensor with range [0,255] instead of [0,1]?

Input:
I'm trying to use transforms.compose to convert my images into normalized images with a range of [0,255] instead of normalizing it as [0,1] for training my model. How do I make my code do this. Currently it normalizes the images from [0,1]. How would i just multiply this up to 255 to make it 0-255 or is it not that simple?
def build_model(self):
        """ DataLoader """
        train_transform = transforms.Compose([
            transforms.RandomHorizontalFlip(),
            transforms.Resize((self.img_size + 30, self.img_size+30)),
            transforms.RandomCrop(self.img_size),
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
        ])
        test_transform = transforms.Compose([
            transforms.Resize((self.img_size, self.img_size)),
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
        ])

Output:
Ideally you would normalize values between [0, 1] then standardize by calculating the mean and std of your whole training set and apply it to all datasets (training, validation and test set).
The following is essentially a x in [x_min, x_max] -> x' in [0, 1] mapping:
x_min, x_max = x.min(), x.max()
x = (x - x_min) / (x_max-x_min)

Then standardize, for instance with the z-score, which makes mean(x')=0 and std(x')=1:
mean, std = x.mean(), x.std()
x = (x - mean) / std

Back to your question, torchvision.transforms.Normalize is described as:
output[channel] = (input[channel] - mean[channel]) / std[channel]

If you divide your std argument by 255 then you will end multiply by 255 the output.
Here's an exemple with shape (b=1, c=3, h=1, w=3):
> x = torch.tensor([[[[0.2, 0.3, 0.6]], [[0.1, 0.4, 0.2]], [[0.1, 0.8, 0.6]]]])

> mean, std = x.mean(), x.std()
tensor(0.3667), tensor(0.2500)

> t = T.Normalize(mean=mean, std=std/255)

> t(x).mean(), t(x).std()
tensor(~0), tensor(255.)

However, if you're just looking to multiply your data by 255 inside the torchvision.transforms.Compose pipeline you can just do:
T.Compose([
    # other transforms
    T.ToTensor(),
    T.Normalize(mean=(0,)*3, std=(255,)*3)
])

Or with just a lambda:
T.Compose([
    # other transforms
    T.ToTensor(),
    lambda x: x*255
])

Having imported torchvision.transforms as T.


Instruction: How do I create a DataLoaders using rows of a DataFrame?

Input:
I am trying to create a model that will predict the next row of values. There are 7 columns, but I am only using the first 6. I figure that if I pass in the datetimes in column 7 to the model, that will guarantee overfitting. Here is a screenshot of the DataFrame:

I am using an arbitrary number of rows, 100 in this case, to make this prediction. All I need to know is some way to create a DataLoader where the y value is the row that I want to predict, and the x value is the 100 preceding rows.
If there is a way to do this with a DataBlock, that would be preferred. I have thought about using .loc and .iloc, but I do not know how I would use those to create a DataLoader.

Output:
Create your custom dataset like this:
class TimeSeriesDataset:
    def __init__(self, df, input_features: list,
                 output_features: list, lookback=99, lookahead=1):
        self.df = df
        self.lookback = lookback
        self.lookahead = lookahead

    def __len__(self):
        return len(self.df) - self.lookback

    def __getitem__(self, idx):
        idx += self.lookback
        lookback = self.df.iloc[idx-self.lookback:idx]
        lookahead = self.df.iloc[idx]
        lookback = lookback[self.input_features].values
        lookahead = lookahead[self.output_features].values
        X = T.tensor(lookback)
        y = T.tensor(lookahead)
        return X, y

Then make your dataloader like this.
dataset = TimeSeriesDataset(df, input_features, ouput_features)
dataloader = DataLoader(dataset, batch_size=batch_size)


Instruction: Calculate alpha values with torch mean?

Input:
I'm trying to calculate the alpha values as explained here.
I have as argument a tensor with shape (1, 512, 14, 14). To calculate alpha values I need to calculate the average of all dimensions except the channel dimension, so the output will have the shape (1, k, 1, 1) which is essentialy (k,).
How can I do this in PyTorch?
Thanks!

Output:
You could permute the first and second axis to keep the channel dimension on dim=0, then flatten all other dimensions, and lastly, take the mean on that new axis:
x.permute(1, 0, 2, 3).flatten(start_dim=1).mean(dim=1)

Here are the shapes, step by step:
>>> x.permute(1, 0, 2, 3).shape
(512, 1, 14, 14)

>>> x.permute(1, 0, 2, 3).flatten(start_dim=1).shape
(512, 1, 196)

>>> x.permute(1, 0, 2, 3).flatten(start_dim=1).mean(dim=1).shape
(512,)


Instruction: Pytorch tensor dimension multiplication

Input:
I'm trying to implement the grad-camm algorithm:
https://arxiv.org/pdf/1610.02391.pdf
My arguments are:
activations: Tensor with shape torch.Size([1, 512, 14, 14])
alpha values : Tensor with shape torch.Size([512])
I want to multiply each activation (in dimension index 1 (sized 512)) in each corresponding alpha value: for example if the i'th index out of the 512 in the activation is 4 and the i'th alpha value is 5, then my new i'th activation would be 20.
The shape of the output should be torch.Size([1, 512, 14, 14])

Output:
Assuming the desired output is of shape (1, 512, 14, 14).
You can achieve this with torch.einsum:
torch.einsum('nchw,c->nchw', x, y)

Or with a simple dot product, but you will first need to add a couple of additional dimensions on y:
x*y[None, :, None, None]

Here's an example with x.shape = (1, 4, 2, 2) and y = (4,):
>>> x = torch.arange(16).reshape(1, 4, 2, 2)
tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]],

         [[ 8,  9],
          [10, 11]],

         [[12, 13],
          [14, 15]]]])

>>> y = torch.arange(1, 5)
tensor([1, 2, 3, 4])

>>> x*y[None, :, None, None]
tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 8, 10],
          [12, 14]],

         [[24, 27],
          [30, 33]],

         [[48, 52],
          [56, 60]]]])


Instruction: ImportError: cannot import name 'AdultDataset' from 'dataset'

Input:
Please I am working on AdultDataset for a classification task
I found out: from dataset import AdultDataset
is giving the error below:
ImportError: cannot import name 'AdultDataset' from 'dataset'

Import Relevant Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split

import torch
import torch.nn as nn

import torch.optim as optim
from torch.autograd import Variable
from torch.utils.data import DataLoader

from dataset import AdultDataset

So when I am trying to create a 3 layer Feed Forward Neural network
using pytorch that takes as input the a dataset entries and classifies
if the individuals gains more or less than 50K (i.e., the fnlwgt label)
starting with the code below I get an error
train_dataset = AdultDataset(X_train, y_train)
test_dataset = AdultDataset(X_test, y_test)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-49-4ab31bdb6258> in <module>
      1 # Using norm_D01
      2
----> 3 train_dataset = AdultDataset(X_train, y_train)
      4 test_dataset = AdultDataset(X_test, y_test)
      5

NameError: name 'AdultDataset' is not defined

Output:
I believe your problem can be fixed like this:
from aif360.datasets import AdultDataset

Maybe you are using an old guide?


Instruction: Pytorch torch.load ModuleNotFoundError: No module named 'utils'

Input:
I'm trying to load a pretrained model with torch.load.
I get the following error:
ModuleNotFoundError: No module named 'utils'

I've checked that the path I am using is correct by opening it from the command line. What could be causing this?
Here's my code:
import torch
import sys

PATH = './gan.pth'

model = torch.load(PATH)
model.eval()

EDIT:
Entire error stack:
Traceback (most recent call last):
  File "load.py", line 6, in <module>
    model = torch.load(PATH)
  File "C:\Users\user\anaconda3\envs\pytorch-flask\lib\site-packages\torch\serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\user\anaconda3\envs\pytorch-flask\lib\site-packages\torch\serialization.py", line 774, in _legacy_load
    result = unpickler.load()
ModuleNotFoundError: No module named 'utils'

Output:
EDIT this answer doesn't provide the answer for the question but addresses another issue in the given code
the .pth file just stores the parameters of a model, not the model itself. When you want to load a model you will need the .pt/-h file and the python code of your model class. Then you can load it like this:

# your model
class YourModel(nn.Modules):
    def __init__(self):
        super(YourModel, self).__init__()
        . . .

    def forward(self, x):
        . . .

# the pytorch save-file in which you stored your trained model
model_file = "<your path>"

model = Model()
model = model.load_state_dict(torch.load(model_file))

model.eval()